ganesha: store exports and export counter in RADOS

Allow the ganesha driver to store ganesha exports
and export counter as Ceph RADOS objects. This enables
highly available(HA) Ganesha servers in manila deployments
to store their config in a HA storage.

Implements: blueprint ganesha-ha-rados
Change-Id: Ia51156055fa10d0661e662c9c998829864f1a204

doc/source/admin/cephfs_driver.rst

@@ -285,6 +285,42 @@ The following options are set in the driver backend section above:
   recommended to set this option even if the ganesha server is co-located
   with the :term:`manila-share` service.
 
+
+With NFS-Ganesha (v2.5.4 or later), Ceph (v12.2.2 or later), the driver (Queens
+or later) can store NFS-Ganesha exports and export counter in Ceph RADOS
+objects. This is useful for highly available NFS-Ganesha deployments to store
+its configuration efficiently in an already available distributed storage
+system. Set additional options in the NFS driver section to enable the driver
+to do this.
+
+.. code-block:: ini
+
+    [cephfsnfs1]
+    ganesha_rados_store_enable = True
+    ganesha_rados_store_pool_name = cephfs_data
+    driver_handles_share_servers = False
+    share_backend_name = CEPHFSNFS1
+    share_driver = manila.share.drivers.cephfs.driver.CephFSDriver
+    cephfs_protocol_helper_type = NFS
+    cephfs_conf_path = /etc/ceph/ceph.conf
+    cephfs_auth_id = manila
+    cephfs_cluster_name = ceph
+    cephfs_enable_snapshots = False
+    cephfs_ganesha_server_is_remote= False
+    cephfs_ganesha_server_ip = 172.24.4.3
+
+
+The following ganesha library (See manila's ganesha library documentation for
+more details) related options are set in the driver backend section above:
+
+* ``ganesha_rados_store_enable`` to True for persisting Ganesha exports and
+  export counter in Ceph RADOS objects.
+
+* ``ganesha_rados_store_pool_name`` to the Ceph RADOS pool that stores Ganesha
+  exports and export counter objects. If you want to use one of the backend
+  CephFS's RADOS pools, then using CephFS's data pool is preferred over using
+  its metadata pool.
+
 Edit ``enabled_share_backends`` to point to the driver's backend section
 using the section name, ``cephfnfs1``.
 

doc/source/contributor/ganesha.rst

@@ -30,21 +30,37 @@ Supported operations
 
 - Deny NFS Share access
 
+Supported manila drivers
+------------------------
+
+- CephFS driver uses ``ganesha.GaneshaNASHelper2`` library class
+
+- GlusterFS driver uses ``ganesha.GaneshaNASHelper`` library class
+
 Requirements
 ------------
 
 - Preferred:
 
   `NFS-Ganesha <https://github.com/nfs-ganesha/nfs-ganesha/wiki>`_ v2.4 or
-  later, which allows dynamic update of access rules. And use manila's
+  later, which allows dynamic update of access rules. Use with manila's
   ``ganesha.GaneshaNASHelper2`` class as described later in
-  :ref:`ganesha_using_library`.
+  :ref:`using_ganesha_library`.
+
+  (or)
+
+  `NFS-Ganesha <https://github.com/nfs-ganesha/nfs-ganesha/wiki>`_ v2.5.4 or
+  later that allows dynamic update of access rules, and can make use of highly
+  available Ceph RADOS (distributed object storage) as its shared storage for
+  NFS client recovery data, and exports. Use with Ceph v12.2.2 or later, and
+  ``ganesha.GaneshaNASHelper2`` library class in manila Queens release or
+  later.
 
 - For use with limitations documented in :ref:`ganesha_known_issues`:
 
   `NFS-Ganesha <https://github.com/nfs-ganesha/nfs-ganesha/wiki>`_ v2.1 to
-  v2.3. And use manila's ``ganesha.GaneshaNASHelper`` class as described later
+  v2.3. Use with manila's ``ganesha.GaneshaNASHelper`` class as described later
-  in :ref:`ganesha_using_library`.
+  in :ref:`using_ganesha_library`.
 
 NFS-Ganesha configuration
 -------------------------
@@ -76,6 +92,53 @@ The above paths can be customized through manila configuration as follows:
 
     ``%include <ganesha_export_dir>/INDEX.conf``
 
+In versions 2.5.4 or later, Ganesha can store NFS client recovery data in
+Ceph RADOS, and also read exports stored in Ceph RADOS. These features are
+useful to make Ganesha server that has access to a Ceph (luminous or later)
+storage backend, highly available. The Ganesha library class
+`GaneshaNASHelper2` (in manila Queens or later) allows you to store Ganesha
+exports directly in a shared storage, RADOS objects, by setting the following
+manila config options in the driver section:
+
+- `ganesha_rados_store_enable` = 'True' to persist Ganesha exports and export
+  counter in Ceph RADOS objects
+- `ganesha_rados_store_pool_name` = name of the Ceph RADOS pool to store
+  Ganesha exports and export counter objects
+- `ganesha_rados_export_index` = name of the Ceph RADOS object used to store
+  a list of export RADOS object URLs (defaults to 'ganesha-export-index')
+
+Check out the `cephfs_driver` documentation for an example driver section
+that uses these options.
+
+To allow Ganesha to read from RADOS objects add the below code block in
+ganesha's configuration file, substituting values per your setup.
+
+.. code-block:: console
+
+    # To read exports from RADOS objects
+    RADOS_URLS {
+        ceph_conf = "/etc/ceph/ceph.conf";
+        userid = "admin";
+    }
+    # Replace with actual pool name, and export index object
+    %url rados://<ganesha_rados_store_pool_name>/<ganesha_rados_export_index>
+    # To store client recovery data in the same RADOS pool
+    NFSv4 {
+        RecoveryBackend = "rados_kv";
+    }
+    RADOS_KV {
+        ceph_conf = "/etc/ceph/ceph.conf";
+        userid = "admin";
+        # Replace with actual pool name
+        pool = <ganesha_rados_store_pool_name>;
+    }
+
+For a fresh setup, make sure to create the Ganesha export index object as an
+empty object before starting the Ganesha server.
+
+.. code-block:: console
+
+    echo | sudo rados -p ${GANESHA_RADOS_STORE_POOL_NAME} put ganesha-export-index -
 
 Further Ganesha related manila configuration
 --------------------------------------------
@@ -87,13 +150,22 @@ itself).
 These are:
 
 - `ganesha_service_name` = name of the system service representing Ganesha,
-    defaults to ganesha.nfsd
+  defaults to ganesha.nfsd
 - `ganesha_db_path` = location of on-disk database storing permanent Ganesha
-    state
+  state, e.g. a export ID counter to generate export IDs for shares
+
+  (or)
+
+  When `ganesha_rados_store_enabled` is set to True, the ganesha export
+  counter is stored in a Ceph RADOS object instead of in a SQLite database
+  local to the manila driver. The counter can be optionally configured with,
+  `ganesha_rados_export_counter` = name of the Ceph RADOS object used as the
+  Ganesha export counter (defaults to 'ganesha-export-counter')
+
 - `ganesha_export_template_dir` = directory from where Ganesha loads
     export customizations (cf. "Customizing Ganesha exports").
 
-.. _ganesha_using_library:
+.. _using_ganesha_library:
 
 Using Ganesha Library in drivers
 --------------------------------

manila/exception.py

@@ -954,3 +954,8 @@ class LockCreationFailed(ManilaException):
 
 class LockingFailed(ManilaException):
     message = _('Lock acquisition failed.')
+
+
+# Ganesha library
+class GaneshaException(ManilaException):
+    message = _("Unknown NFS-Ganesha library exception.")

manila/share/driver.py

@@ -171,6 +171,21 @@ ganesha_opts = [
                default='/etc/manila/ganesha-export-templ.d',
                help='Path to directory containing Ganesha export '
                     'block templates. (Ganesha module only.)'),
+    cfg.BoolOpt('ganesha_rados_store_enable',
+                default=False,
+                help='Persist Ganesha exports and export counter '
+                     'in Ceph RADOS objects, highly available storage.'),
+    cfg.StrOpt('ganesha_rados_store_pool_name',
+               help='Name of the Ceph RADOS pool to store Ganesha exports '
+                    'and export counter.'),
+    cfg.StrOpt('ganesha_rados_export_counter',
+               default='ganesha-export-counter',
+               help='Name of the Ceph RADOS object used as the Ganesha '
+                    'export counter.'),
+    cfg.StrOpt('ganesha_rados_export_index',
+               default='ganesha-export-index',
+               help='Name of the Ceph RADOS object used to store a list '
+                    'of the export RADOS object URLS.'),
 ]
 
 CONF = cfg.CONF

manila/share/drivers/cephfs/driver.py

@@ -122,7 +122,7 @@ class CephFSDriver(driver.ExecuteMixin, driver.GaneshaMixin,
         self.protocol_helper = protocol_helper_class(
             self._execute,
             self.configuration,
-            volume_client=self.volume_client)
+            ceph_vol_client=self.volume_client)
 
         self.protocol_helper.init_helper()
 
@@ -321,7 +321,7 @@ class NativeProtocolHelper(ganesha.NASHelperBase):
                                constants.ACCESS_LEVEL_RO)
 
     def __init__(self, execute, config, **kwargs):
-        self.volume_client = kwargs.pop('volume_client')
+        self.volume_client = kwargs.pop('ceph_vol_client')
         super(NativeProtocolHelper, self).__init__(execute, config,
                                                    **kwargs)
 
@@ -456,11 +456,12 @@ class NFSProtocolHelper(ganesha.GaneshaNASHelper2):
             LOG.info("NFS-Ganesha server's location defaulted to driver's "
                      "hostname: %s", self.ganesha_host)
 
-        self.volume_client = kwargs.pop('volume_client')
-
         super(NFSProtocolHelper, self).__init__(execute, config_object,
                                                 **kwargs)
 
+        if not hasattr(self, 'ceph_vol_client'):
+            self.ceph_vol_client = kwargs.pop('ceph_vol_client')
+
     def get_export_locations(self, share, cephfs_volume):
         export_location = "{server_address}:{path}".format(
             server_address=self.ganesha_host,
@@ -485,7 +486,7 @@ class NFSProtocolHelper(ganesha.GaneshaNASHelper2):
     def _fsal_hook(self, base, share, access):
         """Callback to create FSAL subblock."""
         ceph_auth_id = ''.join(['ganesha-', share['id']])
-        auth_result = self.volume_client.authorize(
+        auth_result = self.ceph_vol_client.authorize(
             cephfs_share_path(share), ceph_auth_id, readonly=False,
             tenant_id=share['project_id'])
         # Restrict Ganesha server's access to only the CephFS subtree or path,
@@ -501,15 +502,15 @@ class NFSProtocolHelper(ganesha.GaneshaNASHelper2):
     def _cleanup_fsal_hook(self, base, share, access):
         """Callback for FSAL specific cleanup after removing an export."""
         ceph_auth_id = ''.join(['ganesha-', share['id']])
-        self.volume_client.deauthorize(cephfs_share_path(share),
+        self.ceph_vol_client.deauthorize(cephfs_share_path(share),
-                                       ceph_auth_id)
+                                         ceph_auth_id)
 
     def _get_export_path(self, share):
         """Callback to provide export path."""
         volume_path = cephfs_share_path(share)
-        return self.volume_client._get_path(volume_path)
+        return self.ceph_vol_client._get_path(volume_path)
 
     def _get_export_pseudo_path(self, share):
         """Callback to provide pseudo path."""
         volume_path = cephfs_share_path(share)
-        return self.volume_client._get_path(volume_path)
+        return self.ceph_vol_client._get_path(volume_path)

manila/share/drivers/ganesha/__init__.py

@@ -24,6 +24,7 @@ import six
 
 from manila.common import constants
 from manila import exception
+from manila.i18n import _
 from manila.share.drivers.ganesha import manager as ganesha_manager
 from manila.share.drivers.ganesha import utils as ganesha_utils
 
@@ -167,6 +168,45 @@ class GaneshaNASHelper(NASHelperBase):
 class GaneshaNASHelper2(GaneshaNASHelper):
     """Perform share access changes using Ganesha version >= 2.4."""
 
+    def __init__(self, execute, config, tag='<no name>', **kwargs):
+        super(GaneshaNASHelper2, self).__init__(execute, config, **kwargs)
+        if self.configuration.ganesha_rados_store_enable:
+            self.ceph_vol_client = kwargs.pop('ceph_vol_client')
+
+    def init_helper(self):
+        """Initializes protocol-specific NAS drivers."""
+        kwargs = {
+            'ganesha_config_path': self.configuration.ganesha_config_path,
+            'ganesha_export_dir': self.configuration.ganesha_export_dir,
+            'ganesha_service_name': self.configuration.ganesha_service_name
+        }
+        if self.configuration.ganesha_rados_store_enable:
+            kwargs['ganesha_rados_store_enable'] = (
+                self.configuration.ganesha_rados_store_enable)
+            if not self.configuration.ganesha_rados_store_pool_name:
+                raise exception.GaneshaException(
+                    _('"ganesha_rados_store_pool_name" config option is not '
+                      'set in the driver section.'))
+            kwargs['ganesha_rados_store_pool_name'] = (
+                self.configuration.ganesha_rados_store_pool_name)
+            kwargs['ganesha_rados_export_index'] = (
+                self.configuration.ganesha_rados_export_index)
+            kwargs['ganesha_rados_export_counter'] = (
+                self.configuration.ganesha_rados_export_counter)
+            kwargs['ceph_vol_client'] = (
+                self.ceph_vol_client)
+        else:
+            kwargs['ganesha_db_path'] = self.configuration.ganesha_db_path
+        self.ganesha = ganesha_manager.GaneshaManager(
+            self._execute, self.tag, **kwargs)
+        system_export_template = self._load_conf_dir(
+            self.configuration.ganesha_export_template_dir,
+            must_exist=False)
+        if system_export_template:
+            self.export_template = system_export_template
+        else:
+            self.export_template = self._default_config_hook()
+
     def _get_export_path(self, share):
         """Subclass this to return export path."""
         raise NotImplementedError()
@@ -186,8 +226,8 @@ class GaneshaNASHelper2(GaneshaNASHelper):
         confdict = {}
         existing_access_rules = []
 
-        if self.ganesha._check_export_file_exists(share['name']):
+        if self.ganesha.check_export_exists(share['name']):
-            confdict = self.ganesha._read_export_file(share['name'])
+            confdict = self.ganesha._read_export(share['name'])
             existing_access_rules = confdict["EXPORT"]["CLIENT"]
             if not isinstance(existing_access_rules, list):
                 existing_access_rules = [existing_access_rules]

manila/share/drivers/ganesha/manager.py

@@ -20,6 +20,7 @@ import sys
 
 from oslo_log import log
 from oslo_serialization import jsonutils
+from oslo_utils import importutils
 import six
 
 from manila import exception
@@ -204,6 +205,19 @@ def mkconf(confdict):
     return s.getvalue()
 
 
+rados = None
+
+
+def setup_rados():
+    global rados
+    if not rados:
+        try:
+            rados = importutils.import_module('rados')
+        except ImportError:
+            raise exception.ShareBackendException(
+                _("python-rados is not installed"))
+
+
 class GaneshaManager(object):
     """Ganesha instrumentation class."""
 
@@ -227,30 +241,54 @@ class GaneshaManager(object):
                     stdout=e.stdout, stderr=e.stderr, exit_code=e.exit_code,
                     cmd=e.cmd)
         self.execute = _execute
+        self.ganesha_service = kwargs['ganesha_service_name']
         self.ganesha_export_dir = kwargs['ganesha_export_dir']
         self.execute('mkdir', '-p', self.ganesha_export_dir)
-        self.ganesha_db_path = kwargs['ganesha_db_path']
+
-        self.execute('mkdir', '-p', os.path.dirname(self.ganesha_db_path))
+        self.ganesha_rados_store_enable = kwargs.get(
-        self.ganesha_service = kwargs['ganesha_service_name']
+            'ganesha_rados_store_enable')
-        # Here we are to make sure that an SQLite database of the
+        if self.ganesha_rados_store_enable:
-        # required scheme exists at self.ganesha_db_path.
+            setup_rados()
-        # The following command gets us there -- provided the file
+            self.ganesha_rados_store_pool_name = (
-        # does not yet exist (otherwise it just fails). However,
+                kwargs['ganesha_rados_store_pool_name'])
-        # we don't care about this condition, we just execute the
+            self.ganesha_rados_export_counter = (
-        # command unconditionally (ignoring failure). Instead we
+                kwargs['ganesha_rados_export_counter'])
-        # directly query the db right after, to check its validity.
+            self.ganesha_rados_export_index = (
-        self.execute("sqlite3", self.ganesha_db_path,
+                kwargs['ganesha_rados_export_index'])
-                     'create table ganesha(key varchar(20) primary key, '
+            self.ceph_vol_client = (
-                     'value int); insert into ganesha values("exportid", '
+                kwargs['ceph_vol_client'])
-                     '100);', run_as_root=False, check_exit_code=False)
+            try:
-        self.get_export_id(bump=False)
+                self._get_rados_object(self.ganesha_rados_export_counter)
+            except rados.ObjectNotFound:
+                self._put_rados_object(self.ganesha_rados_export_counter,
+                                       six.text_type(1000))
+        else:
+            self.ganesha_db_path = kwargs['ganesha_db_path']
+            self.execute('mkdir', '-p', os.path.dirname(self.ganesha_db_path))
+            # Here we are to make sure that an SQLite database of the
+            # required scheme exists at self.ganesha_db_path.
+            # The following command gets us there -- provided the file
+            # does not yet exist (otherwise it just fails). However,
+            # we don't care about this condition, we just execute the
+            # command unconditionally (ignoring failure). Instead we
+            # directly query the db right after, to check its validity.
+            self.execute(
+                "sqlite3", self.ganesha_db_path,
+                'create table ganesha(key varchar(20) primary key, '
+                'value int); insert into ganesha values("exportid", '
+                '100);', run_as_root=False, check_exit_code=False)
+            self.get_export_id(bump=False)
 
     def _getpath(self, name):
         """Get the path of config file for name."""
         return os.path.join(self.ganesha_export_dir, name + ".conf")
 
-    def _write_file(self, path, data):
+    @staticmethod
-        """Write data to path atomically."""
+    def _get_export_rados_object_name(name):
+        return 'ganesha-export-' + name
+
+    def _write_tmp_conf_file(self, path, data):
+        """Write data to tmp conf file."""
         dirpath, fname = (getattr(os.path, q + "name")(path) for q in
                           ("dir", "base"))
         tmpf = self.execute('mktemp', '-p', dirpath, "-t",
@@ -259,17 +297,18 @@ class GaneshaManager(object):
             'sh', '-c',
             'echo %s > %s' % (pipes.quote(data), pipes.quote(tmpf)),
             message='writing ' + tmpf)
+        return tmpf
+
+    def _write_conf_file(self, name, data):
+        """Write data to config file for name atomically."""
+        path = self._getpath(name)
+        tmpf = self._write_tmp_conf_file(path, data)
         try:
             self.execute('mv', tmpf, path)
         except exception.ProcessExecutionError:
             LOG.error('mv temp file ({0}) to {1} failed.'.format(tmpf, path))
             self.execute('rm', tmpf)
             raise
-
-    def _write_conf_file(self, name, data):
-        """Write data to config file for name atomically."""
-        path = self._getpath(name)
-        self._write_file(path, data)
         return path
 
     def _mkindex(self):
@@ -285,15 +324,32 @@ class GaneshaManager(object):
             self._write_conf_file("INDEX", index)
         _mkindex()
 
+    def _read_export_rados_object(self, name):
+        return parseconf(self._get_rados_object(
+            self._get_export_rados_object_name(name)))
+
     def _read_export_file(self, name):
-        """Return the dict of the export identified by name."""
         return parseconf(self.execute("cat", self._getpath(name),
                                       message='reading export ' + name)[0])
 
-    def _check_export_file_exists(self, name):
+    def _read_export(self, name):
-        """Check whether export exists."""
+        """Return the dict of the export identified by name."""
+        if self.ganesha_rados_store_enable:
+            return self._read_export_rados_object(name)
+        else:
+            return self._read_export_file(name)
+
+    def _check_export_rados_object_exists(self, name):
         try:
-            self.execute('test', '-f', self._getpath(name), makelog=False,
+            self._get_rados_object(
+                self._get_export_rados_object_name(name))
+            return True
+        except rados.ObjectNotFound:
+            return False
+
+    def _check_file_exists(self, path):
+        try:
+            self.execute('test', '-f', path, makelog=False,
                          run_as_root=False)
             return True
         except exception.GaneshaCommandFailure as e:
@@ -302,8 +358,25 @@ class GaneshaManager(object):
             else:
                 raise
 
-    def _write_export_file(self, name, confdict):
+    def _check_export_file_exists(self, name):
-        """Write confdict to the export file of name."""
+        return self._check_file_exists(self._getpath(name))
+
+    def check_export_exists(self, name):
+        """Check whether export exists."""
+        if self.ganesha_rados_store_enable:
+            return self._check_export_rados_object_exists(name)
+        else:
+            return self._check_export_file_exists(name)
+
+    def _write_export_rados_object(self, name, data):
+        """Write confdict to the export RADOS object of name."""
+        self._put_rados_object(self._get_export_rados_object_name(name),
+                               data)
+        # temp export config file required for DBus calls
+        return self._write_tmp_conf_file(self._getpath(name), data)
+
+    def _write_export(self, name, confdict):
+        """Write confdict to the export file or RADOS object of name."""
         for k, v in ganesha_utils.walk(confdict):
             # values in the export block template that need to be
             # filled in by Manila are pre-fixed by '@'
@@ -311,11 +384,21 @@ class GaneshaManager(object):
                 msg = _("Incomplete export block: value %(val)s of attribute "
                         "%(key)s is a stub.") % {'key': k, 'val': v}
                 raise exception.InvalidParameterValue(err=msg)
-        return self._write_conf_file(name, mkconf(confdict))
+        if self.ganesha_rados_store_enable:
+            return self._write_export_rados_object(name, mkconf(confdict))
+        else:
+            return self._write_conf_file(name, mkconf(confdict))
+
+    def _rm_file(self, path):
+        self.execute("rm", "-f", path)
 
     def _rm_export_file(self, name):
         """Remove export file of name."""
-        self.execute("rm", self._getpath(name))
+        self._rm_file(self._getpath(name))
+
+    def _rm_export_rados_object(self, name):
+        """Remove export object of name."""
+        self._delete_rados_object(self._get_export_rados_object_name(name))
 
     def _dbus_send_ganesha(self, method, *args, **kwargs):
         """Send a message to Ganesha via dbus."""
@@ -329,70 +412,158 @@ class GaneshaManager(object):
         """Remove an export from Ganesha runtime with given export id."""
         self._dbus_send_ganesha("RemoveExport", "uint16:%d" % xid)
 
+    def _add_rados_object_url_to_index(self, name):
+        """Add an export RADOS object's URL to the RADOS URL index."""
+
+        # TODO(rraja): Ensure that the export index object's update is atomic,
+        # e.g., retry object update until the object version between the 'get'
+        # and 'put' operations remains the same.
+        index_data = self._get_rados_object(self.ganesha_rados_export_index)
+
+        want_url = "%url rados://{0}/{1}".format(
+            self.ganesha_rados_store_pool_name,
+            self._get_export_rados_object_name(name))
+
+        if index_data:
+            self._put_rados_object(
+                self.ganesha_rados_export_index,
+                '\n'.join([index_data, want_url])
+            )
+        else:
+            self._put_rados_object(self.ganesha_rados_export_index, want_url)
+
+    def _remove_rados_object_url_from_index(self, name):
+        """Remove an export RADOS object's URL from the RADOS URL index."""
+
+        # TODO(rraja): Ensure that the export index object's update is atomic,
+        # e.g., retry object update until the object version between the 'get'
+        # and 'put' operations remains the same.
+        index_data = self._get_rados_object(self.ganesha_rados_export_index)
+        if not index_data:
+            return
+
+        unwanted_url = "%url rados://{0}/{1}".format(
+            self.ganesha_rados_store_pool_name,
+            self._get_export_rados_object_name(name))
+
+        rados_urls = index_data.split('\n')
+        new_rados_urls = [url for url in rados_urls if url != unwanted_url]
+
+        self._put_rados_object(self.ganesha_rados_export_index,
+                               '\n'.join(new_rados_urls))
+
     def add_export(self, name, confdict):
         """Add an export to Ganesha specified by confdict."""
         xid = confdict["EXPORT"]["Export_Id"]
         undos = []
         _mkindex_called = False
         try:
-            path = self._write_export_file(name, confdict)
+            path = self._write_export(name, confdict)
-            undos.append(lambda: self._rm_export_file(name))
+            if self.ganesha_rados_store_enable:
+                undos.append(lambda: self._rm_export_rados_object(name))
+                undos.append(lambda: self._rm_file(path))
+            else:
+                undos.append(lambda: self._rm_export_file(name))
 
             self._dbus_send_ganesha("AddExport", "string:" + path,
                                     "string:EXPORT(Export_Id=%d)" % xid)
             undos.append(lambda: self._remove_export_dbus(xid))
 
-            _mkindex_called = True
+            if self.ganesha_rados_store_enable:
-            self._mkindex()
+                # Clean up temp export file used for the DBus call
+                self._rm_file(path)
+                self._add_rados_object_url_to_index(name)
+            else:
+                _mkindex_called = True
+                self._mkindex()
         except Exception:
             for u in undos:
                 u()
-            if not _mkindex_called:
+            if not self.ganesha_rados_store_enable and not _mkindex_called:
                 self._mkindex()
             raise
 
     def update_export(self, name, confdict):
         """Update an export to Ganesha specified by confdict."""
         xid = confdict["EXPORT"]["Export_Id"]
-        old_confdict = self._read_export_file(name)
+        old_confdict = self._read_export(name)
 
-        path = self._write_export_file(name, confdict)
+        path = self._write_export(name, confdict)
         try:
             self._dbus_send_ganesha("UpdateExport", "string:" + path,
                                     "string:EXPORT(Export_Id=%d)" % xid)
         except Exception:
-            # Revert the export file update.
+            # Revert the export update.
-            self._write_export_file(name, old_confdict)
+            self._write_export(name, old_confdict)
             raise
+        finally:
+            if self.ganesha_rados_store_enable:
+                # Clean up temp export file used for the DBus update call
+                self._rm_file(path)
 
     def remove_export(self, name):
         """Remove an export from Ganesha."""
         try:
-            confdict = self._read_export_file(name)
+            confdict = self._read_export(name)
             self._remove_export_dbus(confdict["EXPORT"]["Export_Id"])
         finally:
-            self._rm_export_file(name)
+            if self.ganesha_rados_store_enable:
-            self._mkindex()
+                self._delete_rados_object(
+                    self._get_export_rados_object_name(name))
+                self._remove_rados_object_url_from_index(name)
+            else:
+                self._rm_export_file(name)
+                self._mkindex()
+
+    def _get_rados_object(self, obj_name):
+        """Get data stored in Ceph RADOS object as a text string."""
+        return self.ceph_vol_client.get_object(
+            self.ganesha_rados_store_pool_name, obj_name).decode()
+
+    def _put_rados_object(self, obj_name, data):
+        """Put data as a byte string in a Ceph RADOS object."""
+        return self.ceph_vol_client.put_object(
+            self.ganesha_rados_store_pool_name,
+            obj_name,
+            data.encode())
+
+    def _delete_rados_object(self, obj_name):
+        return self.ceph_vol_client.delete_object(
+            self.ganesha_rados_store_pool_name,
+            obj_name)
 
     def get_export_id(self, bump=True):
         """Get a new export id."""
         # XXX overflowing the export id (16 bit unsigned integer)
         # is not handled
-        if bump:
+        if self.ganesha_rados_store_enable:
-            bumpcode = 'update ganesha set value = value + 1;'
+            # TODO(rraja): Ensure that the export counter object's update is
+            # atomic, e.g., retry object update until the object version
+            # between the 'get' and 'put' operations remains the same.
+            export_id = int(
+                self._get_rados_object(self.ganesha_rados_export_counter))
+            if not bump:
+                return export_id
+            export_id += 1
+            self._put_rados_object(self.ganesha_rados_export_counter,
+                                   str(export_id))
+            return export_id
         else:
-            bumpcode = ''
+            if bump:
-        out = self.execute(
+                bumpcode = 'update ganesha set value = value + 1;'
-            "sqlite3", self.ganesha_db_path,
+            else:
-            bumpcode + 'select * from ganesha where key = "exportid";',
+                bumpcode = ''
-            run_as_root=False)[0]
+            out = self.execute(
-        match = re.search('\Aexportid\|(\d+)$', out)
+                "sqlite3", self.ganesha_db_path,
-        if not match:
+                bumpcode + 'select * from ganesha where key = "exportid";',
-            LOG.error("Invalid export database on "
+                run_as_root=False)[0]
-                      "Ganesha node %(tag)s: %(db)s.",
+            match = re.search('\Aexportid\|(\d+)$', out)
-                      {'tag': self.tag, 'db': self.ganesha_db_path})
+            if not match:
-            raise exception.InvalidSqliteDB()
+                LOG.error("Invalid export database on "
-        return int(match.groups()[0])
+                          "Ganesha node %(tag)s: %(db)s.",
+                          {'tag': self.tag, 'db': self.ganesha_db_path})
+                raise exception.InvalidSqliteDB()
+            return int(match.groups()[0])
 
     def restart_service(self):
         """Restart the Ganesha service."""

manila/tests/share/drivers/cephfs/test_driver.py

@@ -121,11 +121,11 @@ class CephFSDriverTestCase(test.TestCase):
         if protocol_helper == 'cephfs':
             driver.NativeProtocolHelper.assert_called_once_with(
                 self._execute, self._driver.configuration,
-                volume_client=self._driver._volume_client)
+                ceph_vol_client=self._driver._volume_client)
         else:
             driver.NFSProtocolHelper.assert_called_once_with(
                 self._execute, self._driver.configuration,
-                volume_client=self._driver._volume_client)
+                ceph_vol_client=self._driver._volume_client)
 
         self._driver.protocol_helper.init_helper.assert_called_once_with()
 
@@ -366,7 +366,7 @@ class NativeProtocolHelperTestCase(test.TestCase):
         self._native_protocol_helper = driver.NativeProtocolHelper(
             None,
             self.fake_conf,
-            volume_client=MockVolumeClientModule.CephFSVolumeClient()
+            ceph_vol_client=MockVolumeClientModule.CephFSVolumeClient()
         )
 
     def test_get_export_locations(self):
@@ -546,7 +546,7 @@ class NFSProtocolHelperTestCase(test.TestCase):
         self._nfs_helper = driver.NFSProtocolHelper(
             self._execute,
             self.fake_conf,
-            volume_client=self._volume_client)
+            ceph_vol_client=self._volume_client)
 
     @ddt.data(False, True)
     def test_init_executor_type(self, ganesha_server_is_remote):
@@ -563,7 +563,7 @@ class NFSProtocolHelperTestCase(test.TestCase):
         driver.NFSProtocolHelper(
             self._execute,
             fake_conf,
-            volume_client=MockVolumeClientModule.CephFSVolumeClient()
+            ceph_vol_client=MockVolumeClientModule.CephFSVolumeClient()
         )
 
         if ganesha_server_is_remote:
@@ -590,7 +590,7 @@ class NFSProtocolHelperTestCase(test.TestCase):
         driver.NFSProtocolHelper(
             self._execute,
             fake_conf,
-            volume_client=MockVolumeClientModule.CephFSVolumeClient()
+            ceph_vol_client=MockVolumeClientModule.CephFSVolumeClient()
         )
 
         driver.ganesha_utils.RootExecutor.assert_has_calls(

manila/tests/share/drivers/ganesha/test_manager.py

@@ -13,6 +13,7 @@
 #    License for the specific language governing permissions and limitations
 #    under the License.
 
+import copy
 import re
 
 import ddt
@@ -29,6 +30,7 @@ from manila import utils
 test_export_id = 101
 test_name = 'fakefile'
 test_path = '/fakedir0/export.d/fakefile.conf'
+test_tmp_path = '/fakedir0/export.d/fakefile.conf.RANDOM'
 test_ganesha_cnf = """EXPORT {
     Export_Id = 101;
     CLIENT {
@@ -65,6 +67,35 @@ manager_fake_kwargs = {
 }
 
 
+class MockRadosClientModule(object):
+    """Mocked up version of Ceph's RADOS client interface."""
+
+    class ObjectNotFound(Exception):
+        pass
+
+
+@ddt.ddt
+class MiscTests(test.TestCase):
+
+    @ddt.data({'import_exc': None},
+              {'import_exc': ImportError})
+    @ddt.unpack
+    def test_setup_rados(self, import_exc):
+        manager.rados = None
+        with mock.patch.object(
+                manager.importutils,
+                'import_module',
+                side_effect=import_exc) as mock_import_module:
+            if import_exc:
+                self.assertRaises(
+                    exception.ShareBackendException, manager.setup_rados)
+            else:
+                manager.setup_rados()
+                self.assertEqual(mock_import_module.return_value,
+                                 manager.rados)
+            mock_import_module.assert_called_once_with('rados')
+
+
 class GaneshaConfigTests(test.TestCase):
     """Tests Ganesha config file format convertor functions."""
 
@@ -152,23 +183,40 @@ class GaneshaManagerTestCase(test.TestCase):
     """Tests GaneshaManager."""
 
     def instantiate_ganesha_manager(self, *args, **kwargs):
-        with mock.patch.object(
+        ganesha_rados_store_enable = kwargs.get('ganesha_rados_store_enable',
-                manager.GaneshaManager,
+                                                False)
-                'get_export_id',
+        if ganesha_rados_store_enable:
-                return_value=100) as self.mock_get_export_id:
             with mock.patch.object(
                     manager.GaneshaManager,
-                    'reset_exports') as self.mock_reset_exports:
+                    '_get_rados_object') as self.mock_get_rados_object:
-                with mock.patch.object(
+                return manager.GaneshaManager(*args, **kwargs)
-                        manager.GaneshaManager,
+        else:
-                        'restart_service') as self.mock_restart_service:
+            with mock.patch.object(
-                    return manager.GaneshaManager(*args, **kwargs)
+                    manager.GaneshaManager,
+                    'get_export_id',
+                    return_value=100) as self.mock_get_export_id:
+                return manager.GaneshaManager(*args, **kwargs)
 
     def setUp(self):
         super(GaneshaManagerTestCase, self).setUp()
         self._execute = mock.Mock(return_value=('', ''))
         self._manager = self.instantiate_ganesha_manager(
             self._execute, 'faketag', **manager_fake_kwargs)
+        self._ceph_vol_client = mock.Mock()
+        self._setup_rados = mock.Mock()
+        self._execute2 = mock.Mock(return_value=('', ''))
+        self.mock_object(manager, 'rados', MockRadosClientModule)
+        self.mock_object(manager, 'setup_rados', self._setup_rados)
+        fake_kwargs = copy.copy(manager_fake_kwargs)
+        fake_kwargs.update(
+            ganesha_rados_store_enable=True,
+            ganesha_rados_store_pool_name='fakepool',
+            ganesha_rados_export_counter='fakecounter',
+            ganesha_rados_export_index='fakeindex',
+            ceph_vol_client=self._ceph_vol_client
+        )
+        self._manager_with_rados_store = self.instantiate_ganesha_manager(
+            self._execute2, 'faketag', **fake_kwargs)
         self.mock_object(utils, 'synchronized',
                          mock.Mock(return_value=lambda f: f))
 
@@ -227,6 +275,49 @@ class GaneshaManagerTestCase(test.TestCase):
             *fake_args, message='fakemsg', makelog=False)
         self.assertFalse(manager.LOG.error.called)
 
+    @ddt.data(False, True)
+    def test_init_with_rados_store_and_export_counter_exists(
+            self, counter_exists):
+        fake_execute = mock.Mock(return_value=('', ''))
+        fake_kwargs = copy.copy(manager_fake_kwargs)
+        fake_kwargs.update(
+            ganesha_rados_store_enable=True,
+            ganesha_rados_store_pool_name='fakepool',
+            ganesha_rados_export_counter='fakecounter',
+            ganesha_rados_export_index='fakeindex',
+            ceph_vol_client=self._ceph_vol_client
+        )
+        if counter_exists:
+            self.mock_object(
+                manager.GaneshaManager, '_get_rados_object', mock.Mock())
+        else:
+            self.mock_object(
+                manager.GaneshaManager, '_get_rados_object',
+                mock.Mock(side_effect=MockRadosClientModule.ObjectNotFound))
+        self.mock_object(manager.GaneshaManager, '_put_rados_object')
+
+        test_mgr = manager.GaneshaManager(
+            fake_execute, 'faketag', **fake_kwargs)
+
+        self.assertEqual('/fakedir0/fakeconfig', test_mgr.ganesha_config_path)
+        self.assertEqual('faketag', test_mgr.tag)
+        self.assertEqual('/fakedir0/export.d', test_mgr.ganesha_export_dir)
+        self.assertEqual('ganesha.fakeservice', test_mgr.ganesha_service)
+        fake_execute.assert_called_once_with(
+            'mkdir', '-p', '/fakedir0/export.d')
+        self.assertTrue(test_mgr.ganesha_rados_store_enable)
+        self.assertEqual('fakepool', test_mgr.ganesha_rados_store_pool_name)
+        self.assertEqual('fakecounter', test_mgr.ganesha_rados_export_counter)
+        self.assertEqual('fakeindex', test_mgr.ganesha_rados_export_index)
+        self.assertEqual(self._ceph_vol_client, test_mgr.ceph_vol_client)
+        self._setup_rados.assert_called_with()
+        test_mgr._get_rados_object.assert_called_once_with('fakecounter')
+        if counter_exists:
+            self.assertFalse(test_mgr._put_rados_object.called)
+        else:
+            test_mgr._put_rados_object.assert_called_once_with(
+                'fakecounter', six.text_type(1000))
+
     def test_ganesha_export_dir(self):
         self.assertEqual(
             '/fakedir0/export.d', self._manager.ganesha_export_dir)
@@ -236,84 +327,78 @@ class GaneshaManagerTestCase(test.TestCase):
             '/fakedir0/export.d/fakefile.conf',
             self._manager._getpath('fakefile'))
 
-    def test_write_file(self):
+    def test_get_export_rados_object_name(self):
-        test_data = 'fakedata'
+        self.assertEqual(
+            'ganesha-export-fakeobj',
+            self._manager._get_export_rados_object_name('fakeobj'))
+
+    def test_write_tmp_conf_file(self):
         self.mock_object(manager.pipes, 'quote',
                          mock.Mock(side_effect=['fakedata',
-                                                'fakefile.conf.RANDOM']))
+                                                test_tmp_path]))
         test_args = [
             ('mktemp', '-p', '/fakedir0/export.d', '-t',
              'fakefile.conf.XXXXXX'),
-            ('sh', '-c', 'echo fakedata > fakefile.conf.RANDOM'),
+            ('sh', '-c', 'echo fakedata > %s' % test_tmp_path)]
-            ('mv', 'fakefile.conf.RANDOM', test_path)]
         test_kwargs = {
-            'message': 'writing fakefile.conf.RANDOM'
+            'message': 'writing %s' % test_tmp_path
         }
 
         def return_tmpfile(*args, **kwargs):
             if args == test_args[0]:
-                return ('fakefile.conf.RANDOM\n', '')
+                return (test_tmp_path + '\n', '')
-
         self.mock_object(self._manager, 'execute',
                          mock.Mock(side_effect=return_tmpfile))
-        self._manager._write_file(test_path, test_data)
+
+        ret = self._manager._write_tmp_conf_file(test_path, 'fakedata')
+
         self._manager.execute.assert_has_calls([
             mock.call(*test_args[0]),
-            mock.call(*test_args[1], **test_kwargs),
+            mock.call(*test_args[1], **test_kwargs)])
-            mock.call(*test_args[2])])
         manager.pipes.quote.assert_has_calls([
             mock.call('fakedata'),
-            mock.call('fakefile.conf.RANDOM')])
+            mock.call(test_tmp_path)])
+        self.assertEqual(test_tmp_path, ret)
 
-    def test_write_file_with_mv_error(self):
+    @ddt.data(True, False)
+    def test_write_conf_file_with_mv_error(self, mv_error):
         test_data = 'fakedata'
-        self.mock_object(manager.pipes, 'quote',
-                         mock.Mock(side_effect=['fakedata',
-                                                'fakefile.conf.RANDOM']))
         test_args = [
-            ('mktemp', '-p', '/fakedir0/export.d', '-t',
+            ('mv', test_tmp_path, test_path),
-             'fakefile.conf.XXXXXX'),
+            ('rm', test_tmp_path)]
-            ('sh', '-c', 'echo fakedata > fakefile.conf.RANDOM'),
+        self.mock_object(self._manager, '_getpath',
-            ('mv', 'fakefile.conf.RANDOM', test_path),
+                         mock.Mock(return_value=test_path))
-            ('rm', 'fakefile.conf.RANDOM')]
+        self.mock_object(self._manager, '_write_tmp_conf_file',
-        test_kwargs = {
+                         mock.Mock(return_value=test_tmp_path))
-            'message': 'writing fakefile.conf.RANDOM'
-        }
 
         def mock_return(*args, **kwargs):
             if args == test_args[0]:
-                return ('fakefile.conf.RANDOM\n', '')
+                if mv_error:
-            if args == test_args[2]:
+                    raise exception.ProcessExecutionError()
-                raise exception.ProcessExecutionError()
+                else:
+                    return ('', '')
 
         self.mock_object(self._manager, 'execute',
                          mock.Mock(side_effect=mock_return))
-        self.assertRaises(
-            exception.ProcessExecutionError,
-            self._manager._write_file,
-            test_path,
-            test_data
-        )
-        self._manager.execute.assert_has_calls([
-            mock.call(*test_args[0]),
-            mock.call(*test_args[1], **test_kwargs),
-            mock.call(*test_args[2])],
-            mock.call(*test_args[3])
-        )
-        manager.pipes.quote.assert_has_calls([
-            mock.call('fakedata'),
-            mock.call('fakefile.conf.RANDOM')])
 
-    def test_write_conf_file(self):
+        if mv_error:
-        test_data = 'fakedata'
+            self.assertRaises(
-        self.mock_object(self._manager, '_getpath',
+                exception.ProcessExecutionError,
-                         mock.Mock(return_value=test_path))
+                self._manager._write_conf_file, test_name, test_data)
-        self.mock_object(self._manager, '_write_file')
+        else:
-        ret = self._manager._write_conf_file(test_name, test_data)
+            ret = self._manager._write_conf_file(test_name, test_data)
-        self.assertEqual(test_path, ret)
+
         self._manager._getpath.assert_called_once_with(test_name)
-        self._manager._write_file.assert_called_once_with(
+        self._manager._write_tmp_conf_file.assert_called_once_with(
             test_path, test_data)
+        if mv_error:
+            self._manager.execute.assert_has_calls([
+                mock.call(*test_args[0]),
+                mock.call(*test_args[1])])
+        else:
+            self._manager.execute.assert_has_calls([
+                mock.call(*test_args[0])])
+            self.assertEqual(test_path, ret)
 
     def test_mkindex(self):
         test_ls_output = 'INDEX.conf\nfakefile.conf\nfakefile.txt'
@@ -328,6 +413,25 @@ class GaneshaManagerTestCase(test.TestCase):
             'INDEX', test_index)
         self.assertIsNone(ret)
 
+    def test_read_export_rados_object(self):
+        self.mock_object(self._manager_with_rados_store,
+                         '_get_export_rados_object_name',
+                         mock.Mock(return_value='fakeobj'))
+        self.mock_object(self._manager_with_rados_store, '_get_rados_object',
+                         mock.Mock(return_value=test_ganesha_cnf))
+        self.mock_object(manager, 'parseconf',
+                         mock.Mock(return_value=test_dict_unicode))
+
+        ret = self._manager_with_rados_store._read_export_rados_object(
+            test_name)
+
+        (self._manager_with_rados_store._get_export_rados_object_name.
+         assert_called_once_with(test_name))
+        (self._manager_with_rados_store._get_rados_object.
+         assert_called_once_with('fakeobj'))
+        manager.parseconf.assert_called_once_with(test_ganesha_cnf)
+        self.assertEqual(test_dict_unicode, ret)
+
     def test_read_export_file(self):
         test_args = ('cat', test_path)
         test_kwargs = {'message': 'reading export fakefile'}
@@ -344,23 +448,62 @@ class GaneshaManagerTestCase(test.TestCase):
         manager.parseconf.assert_called_once_with(test_ganesha_cnf)
         self.assertEqual(test_dict_unicode, ret)
 
-    def test_check_export_file_exists(self):
+    @ddt.data(False, True)
-        self.mock_object(self._manager, '_getpath',
+    def test_read_export_with_rados_store(self, rados_store_enable):
-                         mock.Mock(return_value=test_path))
+        self._manager.ganesha_rados_store_enable = rados_store_enable
+        self.mock_object(self._manager, '_read_export_file',
+                         mock.Mock(return_value=test_dict_unicode))
+        self.mock_object(self._manager, '_read_export_rados_object',
+                         mock.Mock(return_value=test_dict_unicode))
+
+        ret = self._manager._read_export(test_name)
+
+        if rados_store_enable:
+            self._manager._read_export_rados_object.assert_called_once_with(
+                test_name)
+            self.assertFalse(self._manager._read_export_file.called)
+        else:
+            self._manager._read_export_file.assert_called_once_with(test_name)
+            self.assertFalse(self._manager._read_export_rados_object.called)
+        self.assertEqual(test_dict_unicode, ret)
+
+    @ddt.data(True, False)
+    def test_check_export_rados_object_exists(self, exists):
+        self.mock_object(
+            self._manager_with_rados_store,
+            '_get_export_rados_object_name', mock.Mock(return_value='fakeobj'))
+        if exists:
+            self.mock_object(
+                self._manager_with_rados_store, '_get_rados_object')
+        else:
+            self.mock_object(
+                self._manager_with_rados_store, '_get_rados_object',
+                mock.Mock(side_effect=MockRadosClientModule.ObjectNotFound))
+
+        ret = self._manager_with_rados_store._check_export_rados_object_exists(
+            test_name)
+
+        (self._manager_with_rados_store._get_export_rados_object_name.
+         assert_called_once_with(test_name))
+        (self._manager_with_rados_store._get_rados_object.
+         assert_called_once_with('fakeobj'))
+        if exists:
+            self.assertTrue(ret)
+        else:
+            self.assertFalse(ret)
+
+    def test_check_file_exists(self):
         self.mock_object(self._manager, 'execute',
                          mock.Mock(return_value=(test_ganesha_cnf,)))
 
-        ret = self._manager._check_export_file_exists(test_name)
+        ret = self._manager._check_file_exists(test_path)
 
-        self._manager._getpath.assert_called_once_with(test_name)
         self._manager.execute.assert_called_once_with(
             'test', '-f', test_path, makelog=False, run_as_root=False)
         self.assertTrue(ret)
 
     @ddt.data(1, 4)
-    def test_check_export_file_exists_error(self, exit_code):
+    def test_check_file_exists_error(self, exit_code):
-        self.mock_object(self._manager, '_getpath',
-                         mock.Mock(return_value=test_path))
         self.mock_object(
             self._manager, 'execute',
             mock.Mock(side_effect=exception.GaneshaCommandFailure(
@@ -368,30 +511,93 @@ class GaneshaManagerTestCase(test.TestCase):
         )
 
         if exit_code == 1:
-            ret = self._manager._check_export_file_exists(test_name)
+            ret = self._manager._check_file_exists(test_path)
             self.assertFalse(ret)
         else:
             self.assertRaises(exception.GaneshaCommandFailure,
-                              self._manager._check_export_file_exists,
+                              self._manager._check_file_exists,
-                              test_name)
+                              test_path)
 
-        self._manager._getpath.assert_called_once_with(test_name)
         self._manager.execute.assert_called_once_with(
             'test', '-f', test_path, makelog=False, run_as_root=False)
 
-    def test_write_export_file(self):
+    def test_check_export_file_exists(self):
+        self.mock_object(self._manager, '_getpath',
+                         mock.Mock(return_value=test_path))
+        self.mock_object(self._manager, '_check_file_exists',
+                         mock.Mock(return_value=True))
+
+        ret = self._manager._check_export_file_exists(test_name)
+
+        self._manager._getpath.assert_called_once_with(test_name)
+        self._manager._check_file_exists.assert_called_once_with(test_path)
+        self.assertTrue(ret)
+
+    @ddt.data(False, True)
+    def test_check_export_exists_with_rados_store(self, rados_store_enable):
+        self._manager.ganesha_rados_store_enable = rados_store_enable
+        self.mock_object(self._manager, '_check_export_file_exists',
+                         mock.Mock(return_value=True))
+        self.mock_object(self._manager, '_check_export_rados_object_exists',
+                         mock.Mock(return_value=True))
+
+        ret = self._manager.check_export_exists(test_name)
+
+        if rados_store_enable:
+            (self._manager._check_export_rados_object_exists.
+             assert_called_once_with(test_name))
+            self.assertFalse(self._manager._check_export_file_exists.called)
+        else:
+            self._manager._check_export_file_exists.assert_called_once_with(
+                test_name)
+            self.assertFalse(
+                self._manager._check_export_rados_object_exists.called)
+        self.assertTrue(ret)
+
+    def test_write_export_rados_object(self):
+        self.mock_object(self._manager, '_get_export_rados_object_name',
+                         mock.Mock(return_value='fakeobj'))
+        self.mock_object(self._manager, '_put_rados_object')
+        self.mock_object(self._manager, '_getpath',
+                         mock.Mock(return_value=test_path))
+        self.mock_object(self._manager, '_write_tmp_conf_file',
+                         mock.Mock(return_value=test_tmp_path))
+
+        ret = self._manager._write_export_rados_object(test_name, 'fakedata')
+
+        self._manager._get_export_rados_object_name.assert_called_once_with(
+            test_name)
+        self._manager._put_rados_object.assert_called_once_with(
+            'fakeobj', 'fakedata')
+        self._manager._getpath.assert_called_once_with(test_name)
+        self._manager._write_tmp_conf_file.assert_called_once_with(
+            test_path, 'fakedata')
+        self.assertEqual(test_tmp_path, ret)
+
+    @ddt.data(True, False)
+    def test_write_export_with_rados_store(self, rados_store_enable):
+        self._manager.ganesha_rados_store_enable = rados_store_enable
         self.mock_object(manager, 'mkconf',
                          mock.Mock(return_value=test_ganesha_cnf))
         self.mock_object(self._manager, '_write_conf_file',
                          mock.Mock(return_value=test_path))
-        ret = self._manager._write_export_file(test_name, test_dict_str)
+        self.mock_object(self._manager, '_write_export_rados_object',
+                         mock.Mock(return_value=test_path))
+
+        ret = self._manager._write_export(test_name, test_dict_str)
+
         manager.mkconf.assert_called_once_with(test_dict_str)
-        self._manager._write_conf_file.assert_called_once_with(
+        if rados_store_enable:
-            test_name, test_ganesha_cnf)
+            self._manager._write_export_rados_object.assert_called_once_with(
+                test_name, test_ganesha_cnf)
+            self.assertFalse(self._manager._write_conf_file.called)
+        else:
+            self._manager._write_conf_file.assert_called_once_with(
+                test_name, test_ganesha_cnf)
+            self.assertFalse(self._manager._write_export_rados_object.called)
         self.assertEqual(test_path, ret)
 
-    def test_write_export_file_error_incomplete_export_block(self):
+    def test_write_export_error_incomplete_export_block(self):
-
         test_errordict = {
             u'EXPORT': {
                 u'Export_Id': '@config',
@@ -402,20 +608,47 @@ class GaneshaManagerTestCase(test.TestCase):
                          mock.Mock(return_value=test_ganesha_cnf))
         self.mock_object(self._manager, '_write_conf_file',
                          mock.Mock(return_value=test_path))
+
         self.assertRaises(exception.InvalidParameterValue,
-                          self._manager._write_export_file,
+                          self._manager._write_export,
                           test_name, test_errordict)
+
         self.assertFalse(manager.mkconf.called)
         self.assertFalse(self._manager._write_conf_file.called)
 
-    def test_rm_export_file(self):
+    def test_rm_file(self):
         self.mock_object(self._manager, 'execute',
                          mock.Mock(return_value=('', '')))
+        ret = self._manager._rm_export_file(test_name)
+
+        self._manager.execute.assert_called_once_with('rm', '-f', test_path)
+        self.assertIsNone(ret)
+
+    def test_rm_export_file(self):
         self.mock_object(self._manager, '_getpath',
                          mock.Mock(return_value=test_path))
+        self.mock_object(self._manager, '_rm_file')
+
         ret = self._manager._rm_export_file(test_name)
+
         self._manager._getpath.assert_called_once_with(test_name)
-        self._manager.execute.assert_called_once_with('rm', test_path)
+        self._manager._rm_file.assert_called_once_with(test_path)
+        self.assertIsNone(ret)
+
+    def test_rm_export_rados_object(self):
+        self.mock_object(self._manager_with_rados_store,
+                         '_get_export_rados_object_name',
+                         mock.Mock(return_value='fakeobj'))
+        self.mock_object(self._manager_with_rados_store,
+                         '_delete_rados_object')
+
+        ret = self._manager_with_rados_store._rm_export_rados_object(
+            test_name)
+
+        (self._manager_with_rados_store._get_export_rados_object_name.
+         assert_called_once_with(test_name))
+        (self._manager_with_rados_store._delete_rados_object.
+         assert_called_once_with('fakeobj'))
         self.assertIsNone(ret)
 
     def test_dbus_send_ganesha(self):
@@ -440,22 +673,99 @@ class GaneshaManagerTestCase(test.TestCase):
             'RemoveExport', 'uint16:101')
         self.assertIsNone(ret)
 
-    def test_add_export(self):
+    @ddt.data('',
-        self.mock_object(self._manager, '_write_export_file',
+              '%url rados://fakepool/fakeobj2')
+    def test_add_rados_object_url_to_index_with_index_data(
+            self, index_data):
+        self.mock_object(
+            self._manager_with_rados_store, '_get_rados_object',
+            mock.Mock(return_value=index_data))
+        self.mock_object(
+            self._manager_with_rados_store, '_get_export_rados_object_name',
+            mock.Mock(return_value='fakeobj1'))
+        self.mock_object(
+            self._manager_with_rados_store, '_put_rados_object')
+
+        ret = (self._manager_with_rados_store.
+               _add_rados_object_url_to_index('fakename'))
+
+        (self._manager_with_rados_store._get_rados_object.
+         assert_called_once_with('fakeindex'))
+        (self._manager_with_rados_store._get_export_rados_object_name.
+         assert_called_once_with('fakename'))
+        if index_data:
+            urls = ('%url rados://fakepool/fakeobj2\n'
+                    '%url rados://fakepool/fakeobj1')
+        else:
+            urls = '%url rados://fakepool/fakeobj1'
+        (self._manager_with_rados_store._put_rados_object.
+         assert_called_once_with('fakeindex', urls))
+        self.assertIsNone(ret)
+
+    @ddt.data('',
+              '%url rados://fakepool/fakeobj1\n'
+              '%url rados://fakepool/fakeobj2')
+    def test_remove_rados_object_url_from_index_with_index_data(
+            self, index_data):
+        self.mock_object(
+            self._manager_with_rados_store, '_get_rados_object',
+            mock.Mock(return_value=index_data))
+        self.mock_object(
+            self._manager_with_rados_store, '_get_export_rados_object_name',
+            mock.Mock(return_value='fakeobj1'))
+        self.mock_object(
+            self._manager_with_rados_store, '_put_rados_object')
+
+        ret = (self._manager_with_rados_store.
+               _remove_rados_object_url_from_index('fakename'))
+
+        if index_data:
+            (self._manager_with_rados_store._get_rados_object.
+             assert_called_once_with('fakeindex'))
+            (self._manager_with_rados_store._get_export_rados_object_name.
+             assert_called_once_with('fakename'))
+            urls = '%url rados://fakepool/fakeobj2'
+            (self._manager_with_rados_store._put_rados_object.
+             assert_called_once_with('fakeindex', urls))
+        else:
+            (self._manager_with_rados_store._get_rados_object.
+             assert_called_once_with('fakeindex'))
+            self.assertFalse(self._manager_with_rados_store.
+                             _get_export_rados_object_name.called)
+            self.assertFalse(self._manager_with_rados_store.
+                             _put_rados_object.called)
+        self.assertIsNone(ret)
+
+    @ddt.data(False, True)
+    def test_add_export_with_rados_store(self, rados_store_enable):
+        self._manager.ganesha_rados_store_enable = rados_store_enable
+        self.mock_object(self._manager, '_write_export',
                          mock.Mock(return_value=test_path))
         self.mock_object(self._manager, '_dbus_send_ganesha')
+        self.mock_object(self._manager, '_rm_file')
+        self.mock_object(self._manager, '_add_rados_object_url_to_index')
         self.mock_object(self._manager, '_mkindex')
+
         ret = self._manager.add_export(test_name, test_dict_str)
-        self._manager._write_export_file.assert_called_once_with(
+
+        self._manager._write_export.assert_called_once_with(
             test_name, test_dict_str)
         self._manager._dbus_send_ganesha.assert_called_once_with(
             'AddExport', 'string:' + test_path,
             'string:EXPORT(Export_Id=101)')
-        self._manager._mkindex.assert_called_once_with()
+        if rados_store_enable:
+            self._manager._rm_file.assert_called_once_with(test_path)
+            self._manager._add_rados_object_url_to_index(test_name)
+            self.assertFalse(self._manager._mkindex.called)
+        else:
+            self._manager._mkindex.assert_called_once_with()
+            self.assertFalse(self._manager._rm_file.called)
+            self.assertFalse(
+                self._manager._add_rados_object_url_to_index.called)
         self.assertIsNone(ret)
 
     def test_add_export_error_during_mkindex(self):
-        self.mock_object(self._manager, '_write_export_file',
+        self.mock_object(self._manager, '_write_export',
                          mock.Mock(return_value=test_path))
         self.mock_object(self._manager, '_dbus_send_ganesha')
         self.mock_object(
@@ -463,9 +773,11 @@ class GaneshaManagerTestCase(test.TestCase):
             mock.Mock(side_effect=exception.GaneshaCommandFailure))
         self.mock_object(self._manager, '_rm_export_file')
         self.mock_object(self._manager, '_remove_export_dbus')
+
         self.assertRaises(exception.GaneshaCommandFailure,
                           self._manager.add_export, test_name, test_dict_str)
-        self._manager._write_export_file.assert_called_once_with(
+
+        self._manager._write_export.assert_called_once_with(
             test_name, test_dict_str)
         self._manager._dbus_send_ganesha.assert_called_once_with(
             'AddExport', 'string:' + test_path,
@@ -475,135 +787,269 @@ class GaneshaManagerTestCase(test.TestCase):
         self._manager._remove_export_dbus.assert_called_once_with(
             test_export_id)
 
-    def test_add_export_error_during_write_export_file(self):
+    @ddt.data(True, False)
+    def test_add_export_error_during_write_export_with_rados_store(
+            self, rados_store_enable):
+        self._manager.ganesha_rados_store_enable = rados_store_enable
         self.mock_object(
-            self._manager, '_write_export_file',
+            self._manager, '_write_export',
             mock.Mock(side_effect=exception.GaneshaCommandFailure))
-        self.mock_object(self._manager, '_dbus_send_ganesha')
         self.mock_object(self._manager, '_mkindex')
-        self.mock_object(self._manager, '_rm_export_file')
+
-        self.mock_object(self._manager, '_remove_export_dbus')
         self.assertRaises(exception.GaneshaCommandFailure,
                           self._manager.add_export, test_name, test_dict_str)
-        self._manager._write_export_file.assert_called_once_with(
-            test_name, test_dict_str)
-        self.assertFalse(self._manager._dbus_send_ganesha.called)
-        self._manager._mkindex.assert_called_once_with()
-        self.assertFalse(self._manager._rm_export_file.called)
-        self.assertFalse(self._manager._remove_export_dbus.called)
 
-    def test_add_export_error_during_dbus_send_ganesha(self):
+        self._manager._write_export.assert_called_once_with(
-        self.mock_object(self._manager, '_write_export_file',
+            test_name, test_dict_str)
+        if rados_store_enable:
+            self.assertFalse(self._manager._mkindex.called)
+        else:
+            self._manager._mkindex.assert_called_once_with()
+
+    @ddt.data(True, False)
+    def test_add_export_error_during_dbus_send_ganesha_with_rados_store(
+            self, rados_store_enable):
+        self._manager.ganesha_rados_store_enable = rados_store_enable
+        self.mock_object(self._manager, '_write_export',
                          mock.Mock(return_value=test_path))
         self.mock_object(
             self._manager, '_dbus_send_ganesha',
             mock.Mock(side_effect=exception.GaneshaCommandFailure))
         self.mock_object(self._manager, '_mkindex')
         self.mock_object(self._manager, '_rm_export_file')
+        self.mock_object(self._manager, '_rm_export_rados_object')
+        self.mock_object(self._manager, '_rm_file')
         self.mock_object(self._manager, '_remove_export_dbus')
+
         self.assertRaises(exception.GaneshaCommandFailure,
                           self._manager.add_export, test_name, test_dict_str)
-        self._manager._write_export_file.assert_called_once_with(
+
+        self._manager._write_export.assert_called_once_with(
             test_name, test_dict_str)
         self._manager._dbus_send_ganesha.assert_called_once_with(
             'AddExport', 'string:' + test_path,
             'string:EXPORT(Export_Id=101)')
-        self._manager._rm_export_file.assert_called_once_with(test_name)
+        if rados_store_enable:
-        self._manager._mkindex.assert_called_once_with()
+            self._manager._rm_export_rados_object.assert_called_once_with(
+                test_name)
+            self._manager._rm_file.assert_called_once_with(test_path)
+            self.assertFalse(self._manager._rm_export_file.called)
+            self.assertFalse(self._manager._mkindex.called)
+        else:
+            self._manager._rm_export_file.assert_called_once_with(test_name)
+            self._manager._mkindex.assert_called_once_with()
+            self.assertFalse(self._manager._rm_export_rados_object.called)
+            self.assertFalse(self._manager._rm_file.called)
         self.assertFalse(self._manager._remove_export_dbus.called)
 
-    def test_update_export(self):
+    @ddt.data(True, False)
+    def test_update_export_with_rados_store(self, rados_store_enable):
+        self._manager.ganesha_rados_store_enable = rados_store_enable
         confdict = {
             'EXPORT': {
                 'Export_Id': 101,
                 'CLIENT': {'Clients': 'ip1', 'Access_Level': 'ro'},
             }
         }
-        self.mock_object(self._manager, '_read_export_file',
+        self.mock_object(self._manager, '_read_export',
                          mock.Mock(return_value=test_dict_unicode))
-        self.mock_object(self._manager, '_write_export_file',
+        self.mock_object(self._manager, '_write_export',
                          mock.Mock(return_value=test_path))
         self.mock_object(self._manager, '_dbus_send_ganesha')
+        self.mock_object(self._manager, '_rm_file')
 
         self._manager.update_export(test_name, confdict)
 
-        self._manager._read_export_file.assert_called_once_with(test_name)
+        self._manager._read_export.assert_called_once_with(test_name)
-        self._manager._write_export_file.assert_called_once_with(test_name,
+        self._manager._write_export.assert_called_once_with(test_name,
-                                                                 confdict)
+                                                            confdict)
         self._manager._dbus_send_ganesha.assert_called_once_with(
             'UpdateExport', 'string:' + test_path,
             'string:EXPORT(Export_Id=101)')
+        if rados_store_enable:
+            self._manager._rm_file.assert_called_once_with(test_path)
+        else:
+            self.assertFalse(self._manager._rm_file.called)
 
-    def test_update_export_error(self):
+    @ddt.data(True, False)
+    def test_update_export_error_with_rados_store(self, rados_store_enable):
+        self._manager.ganesha_rados_store_enable = rados_store_enable
         confdict = {
             'EXPORT': {
                 'Export_Id': 101,
                 'CLIENT': {'Clients': 'ip1', 'Access_Level': 'ro'},
             }
         }
-        self.mock_object(self._manager, '_read_export_file',
+        self.mock_object(self._manager, '_read_export',
                          mock.Mock(return_value=test_dict_unicode))
-        self.mock_object(self._manager, '_write_export_file',
+        self.mock_object(self._manager, '_write_export',
                          mock.Mock(return_value=test_path))
         self.mock_object(
             self._manager, '_dbus_send_ganesha',
             mock.Mock(side_effect=exception.GaneshaCommandFailure))
+        self.mock_object(self._manager, '_rm_file')
 
         self.assertRaises(exception.GaneshaCommandFailure,
                           self._manager.update_export, test_name, confdict)
 
-        self._manager._read_export_file.assert_called_once_with(test_name)
+        self._manager._read_export.assert_called_once_with(test_name)
-        self._manager._write_export_file.assert_has_calls([
+        self._manager._write_export.assert_has_calls([
             mock.call(test_name, confdict),
             mock.call(test_name, test_dict_unicode)])
         self._manager._dbus_send_ganesha.assert_called_once_with(
             'UpdateExport', 'string:' + test_path,
             'string:EXPORT(Export_Id=101)')
+        if rados_store_enable:
+            self._manager._rm_file.assert_called_once_with(test_path)
+        else:
+            self.assertFalse(self._manager._rm_file.called)
 
-    def test_remove_export(self):
+    @ddt.data(True, False)
-        self.mock_object(self._manager, '_read_export_file',
+    def test_remove_export_with_rados_store(self, rados_store_enable):
+        self._manager.ganesha_rados_store_enable = rados_store_enable
+        self.mock_object(self._manager, '_read_export',
                          mock.Mock(return_value=test_dict_unicode))
-        methods = ('_remove_export_dbus', '_rm_export_file', '_mkindex')
+        self.mock_object(self._manager, '_get_export_rados_object_name',
+                         mock.Mock(return_value='fakeobj'))
+        methods = ('_remove_export_dbus', '_rm_export_file', '_mkindex',
+                   '_remove_rados_object_url_from_index',
+                   '_delete_rados_object')
         for method in methods:
             self.mock_object(self._manager, method)
+
         ret = self._manager.remove_export(test_name)
-        self._manager._read_export_file.assert_called_once_with(test_name)
+
+        self._manager._read_export.assert_called_once_with(test_name)
         self._manager._remove_export_dbus.assert_called_once_with(
             test_dict_unicode['EXPORT']['Export_Id'])
-        self._manager._rm_export_file.assert_called_once_with(test_name)
+        if rados_store_enable:
-        self._manager._mkindex.assert_called_once_with()
+            (self._manager._get_export_rados_object_name.
+             assert_called_once_with(test_name))
+            self._manager._delete_rados_object.assert_called_once_with(
+                'fakeobj')
+            (self._manager._remove_rados_object_url_from_index.
+             assert_called_once_with(test_name))
+            self.assertFalse(self._manager._rm_export_file.called)
+            self.assertFalse(self._manager._mkindex.called)
+        else:
+            self._manager._rm_export_file.assert_called_once_with(test_name)
+            self._manager._mkindex.assert_called_once_with()
+            self.assertFalse(
+                self._manager._get_export_rados_object_name.called)
+            self.assertFalse(self._manager._delete_rados_object.called)
+            self.assertFalse(
+                self._manager._remove_rados_object_url_from_index.called)
         self.assertIsNone(ret)
 
-    def test_remove_export_error_during_read_export_file(self):
+    @ddt.data(True, False)
+    def test_remove_export_error_during_read_export_with_rados_store(
+            self, rados_store_enable):
+        self._manager.ganesha_rados_store_enable = rados_store_enable
         self.mock_object(
-            self._manager, '_read_export_file',
+            self._manager, '_read_export',
             mock.Mock(side_effect=exception.GaneshaCommandFailure))
-        methods = ('_remove_export_dbus', '_rm_export_file', '_mkindex')
+        self.mock_object(self._manager, '_get_export_rados_object_name',
+                         mock.Mock(return_value='fakeobj'))
+        methods = ('_remove_export_dbus', '_rm_export_file', '_mkindex',
+                   '_remove_rados_object_url_from_index',
+                   '_delete_rados_object')
         for method in methods:
             self.mock_object(self._manager, method)
+
         self.assertRaises(exception.GaneshaCommandFailure,
                           self._manager.remove_export, test_name)
-        self._manager._read_export_file.assert_called_once_with(test_name)
-        self.assertFalse(self._manager._remove_export_dbus.called)
-        self._manager._rm_export_file.assert_called_once_with(test_name)
-        self._manager._mkindex.assert_called_once_with()
 
-    def test_remove_export_error_during_remove_export_dbus(self):
+        self._manager._read_export.assert_called_once_with(test_name)
-        self.mock_object(self._manager, '_read_export_file',
+        self.assertFalse(self._manager._remove_export_dbus.called)
+        if rados_store_enable:
+            (self._manager._get_export_rados_object_name.
+             assert_called_once_with(test_name))
+            self._manager._delete_rados_object.assert_called_once_with(
+                'fakeobj')
+            (self._manager._remove_rados_object_url_from_index.
+             assert_called_once_with(test_name))
+            self.assertFalse(self._manager._rm_export_file.called)
+            self.assertFalse(self._manager._mkindex.called)
+        else:
+            self._manager._rm_export_file.assert_called_once_with(test_name)
+            self._manager._mkindex.assert_called_once_with()
+            self.assertFalse(
+                self._manager._get_export_rados_object_name.called)
+            self.assertFalse(self._manager._delete_rados_object.called)
+            self.assertFalse(
+                self._manager._remove_rados_object_url_from_index.called)
+
+    @ddt.data(True, False)
+    def test_remove_export_error_during_remove_export_dbus_with_rados_store(
+            self, rados_store_enable):
+        self._manager.ganesha_rados_store_enable = rados_store_enable
+        self.mock_object(self._manager, '_read_export',
                          mock.Mock(return_value=test_dict_unicode))
+        self.mock_object(self._manager, '_get_export_rados_object_name',
+                         mock.Mock(return_value='fakeobj'))
         self.mock_object(
             self._manager, '_remove_export_dbus',
             mock.Mock(side_effect=exception.GaneshaCommandFailure))
-        methods = ('_rm_export_file', '_mkindex')
+        methods = ('_rm_export_file', '_mkindex',
+                   '_remove_rados_object_url_from_index',
+                   '_delete_rados_object')
         for method in methods:
             self.mock_object(self._manager, method)
+
         self.assertRaises(exception.GaneshaCommandFailure,
                           self._manager.remove_export, test_name)
-        self._manager._read_export_file.assert_called_once_with(test_name)
+
+        self._manager._read_export.assert_called_once_with(test_name)
         self._manager._remove_export_dbus.assert_called_once_with(
             test_dict_unicode['EXPORT']['Export_Id'])
-        self._manager._rm_export_file.assert_called_once_with(test_name)
+        if rados_store_enable:
-        self._manager._mkindex.assert_called_once_with()
+            (self._manager._get_export_rados_object_name.
+             assert_called_once_with(test_name))
+            self._manager._delete_rados_object.assert_called_once_with(
+                'fakeobj')
+            (self._manager._remove_rados_object_url_from_index.
+             assert_called_once_with(test_name))
+            self.assertFalse(self._manager._rm_export_file.called)
+            self.assertFalse(self._manager._mkindex.called)
+        else:
+            self._manager._rm_export_file.assert_called_once_with(test_name)
+            self._manager._mkindex.assert_called_once_with()
+            self.assertFalse(
+                self._manager._get_export_rados_object_name.called)
+            self.assertFalse(self._manager._delete_rados_object.called)
+            self.assertFalse(
+                self._manager._remove_rados_object_url_from_index.called)
+
+    def test_get_rados_object(self):
+        self.mock_object(self._ceph_vol_client, 'get_object',
+                         mock.Mock(return_value=b'fakedata'))
+
+        ret = self._manager_with_rados_store._get_rados_object('fakeobj')
+
+        self._ceph_vol_client.get_object.assert_called_once_with(
+            'fakepool', 'fakeobj')
+        self.assertEqual(b'fakedata'.decode(), ret)
+
+    def test_put_rados_object(self):
+        self.mock_object(self._ceph_vol_client, 'put_object',
+                         mock.Mock(return_value=None))
+
+        ret = self._manager_with_rados_store._put_rados_object(
+            'fakeobj', 'fakedata')
+
+        self._ceph_vol_client.put_object.assert_called_once_with(
+            'fakepool', 'fakeobj', 'fakedata'.encode())
+        self.assertIsNone(ret)
+
+    def test_delete_rados_object(self):
+        self.mock_object(self._ceph_vol_client, 'delete_object',
+                         mock.Mock(return_value=None))
+
+        ret = self._manager_with_rados_store._delete_rados_object('fakeobj')
+
+        self._ceph_vol_client.delete_object.assert_called_once_with(
+            'fakepool', 'fakeobj')
+        self.assertIsNone(ret)
 
     def test_get_export_id(self):
         self.mock_object(self._manager, 'execute',
@@ -640,6 +1086,27 @@ class GaneshaManagerTestCase(test.TestCase):
             'select * from ganesha where key = "exportid";',
             run_as_root=False)
 
+    @ddt.data(True, False)
+    def test_get_export_id_with_rados_store_and_bump(self, bump):
+        self.mock_object(self._manager_with_rados_store,
+                         '_get_rados_object', mock.Mock(return_value='1000'))
+        self.mock_object(self._manager_with_rados_store, '_put_rados_object')
+
+        ret = self._manager_with_rados_store.get_export_id(bump=bump)
+
+        if bump:
+            (self._manager_with_rados_store._get_rados_object.
+             assert_called_once_with('fakecounter'))
+            (self._manager_with_rados_store._put_rados_object.
+             assert_called_once_with('fakecounter', '1001'))
+            self.assertEqual(1001, ret)
+        else:
+            (self._manager_with_rados_store._get_rados_object.
+             assert_called_once_with('fakecounter'))
+            self.assertFalse(
+                self._manager_with_rados_store._put_rados_object.called)
+            self.assertEqual(1000, ret)
+
     def test_restart_service(self):
         self.mock_object(self._manager, 'execute')
         ret = self._manager.restart_service()

manila/tests/share/drivers/test_ganesha.py

@@ -322,12 +322,19 @@ class GaneshaNASHelper2TestCase(test.TestCase):
         CONF.set_default('ganesha_export_template_dir',
                          '/fakedir2/faketempl.d')
         CONF.set_default('ganesha_service_name', 'ganesha.fakeservice')
+        CONF.set_default('ganesha_rados_store_enable', True)
+        CONF.set_default('ganesha_rados_store_pool_name', 'ceph_pool')
+        CONF.set_default('ganesha_rados_export_index', 'fake_index')
+        CONF.set_default('ganesha_rados_export_counter', 'fake_counter')
+
         self._context = context.get_admin_context()
         self._execute = mock.Mock(return_value=('', ''))
+        self.ceph_vol_client = mock.Mock()
         self.fake_conf = config.Configuration(None)
         self.fake_conf_dir_path = '/fakedir0/exports.d'
         self._helper = ganesha.GaneshaNASHelper2(
-            self._execute, self.fake_conf, tag='faketag')
+            self._execute, self.fake_conf, tag='faketag',
+            ceph_vol_client=self.ceph_vol_client)
         self._helper.ganesha = mock.Mock()
         self._helper.export_template = {}
         self.share = fake_share.fake_share()
@@ -335,9 +342,100 @@ class GaneshaNASHelper2TestCase(test.TestCase):
         self.rule2 = fake_share.fake_access(access_level='rw',
                                             access_to='10.0.0.2')
 
+    @ddt.data(False, True)
+    def test_init_helper_with_rados_store(self, rados_store_enable):
+        CONF.set_default('ganesha_rados_store_enable', rados_store_enable)
+        mock_template = mock.Mock()
+        mock_ganesha_manager = mock.Mock()
+        self.mock_object(ganesha.ganesha_manager, 'GaneshaManager',
+                         mock.Mock(return_value=mock_ganesha_manager))
+        self.mock_object(self._helper, '_load_conf_dir',
+                         mock.Mock(return_value={}))
+        self.mock_object(self._helper, '_default_config_hook',
+                         mock.Mock(return_value=mock_template))
+
+        ret = self._helper.init_helper()
+
+        if rados_store_enable:
+            kwargs = {
+                'ganesha_config_path': '/fakedir0/fakeconfig',
+                'ganesha_export_dir': '/fakedir0/export.d',
+                'ganesha_service_name': 'ganesha.fakeservice',
+                'ganesha_rados_store_enable': True,
+                'ganesha_rados_store_pool_name': 'ceph_pool',
+                'ganesha_rados_export_index': 'fake_index',
+                'ganesha_rados_export_counter': 'fake_counter',
+                'ceph_vol_client': self.ceph_vol_client
+            }
+        else:
+            kwargs = {
+                'ganesha_config_path': '/fakedir0/fakeconfig',
+                'ganesha_export_dir': '/fakedir0/export.d',
+                'ganesha_service_name': 'ganesha.fakeservice',
+                'ganesha_db_path': '/fakedir1/fake.db'
+            }
+        ganesha.ganesha_manager.GaneshaManager.assert_called_once_with(
+            self._execute, '<no name>', **kwargs)
+        self._helper._load_conf_dir.assert_called_once_with(
+            '/fakedir2/faketempl.d', must_exist=False)
+        self.assertEqual(mock_ganesha_manager, self._helper.ganesha)
+        self._helper._default_config_hook.assert_called_once_with()
+        self.assertEqual(mock_template, self._helper.export_template)
+        self.assertIsNone(ret)
+
+    @ddt.data(False, True)
+    def test_init_helper_conf_dir_empty(self, conf_dir_empty):
+        mock_template = mock.Mock()
+        mock_ganesha_manager = mock.Mock()
+        self.mock_object(ganesha.ganesha_manager, 'GaneshaManager',
+                         mock.Mock(return_value=mock_ganesha_manager))
+        if conf_dir_empty:
+            self.mock_object(self._helper, '_load_conf_dir',
+                             mock.Mock(return_value={}))
+        else:
+            self.mock_object(self._helper, '_load_conf_dir',
+                             mock.Mock(return_value=mock_template))
+        self.mock_object(self._helper, '_default_config_hook',
+                         mock.Mock(return_value=mock_template))
+
+        ret = self._helper.init_helper()
+
+        ganesha.ganesha_manager.GaneshaManager.assert_called_once_with(
+            self._execute, '<no name>',
+            ganesha_config_path='/fakedir0/fakeconfig',
+            ganesha_export_dir='/fakedir0/export.d',
+            ganesha_service_name='ganesha.fakeservice',
+            ganesha_rados_store_enable=True,
+            ganesha_rados_store_pool_name='ceph_pool',
+            ganesha_rados_export_index='fake_index',
+            ganesha_rados_export_counter='fake_counter',
+            ceph_vol_client=self.ceph_vol_client)
+        self._helper._load_conf_dir.assert_called_once_with(
+            '/fakedir2/faketempl.d', must_exist=False)
+        self.assertEqual(mock_ganesha_manager, self._helper.ganesha)
+        if conf_dir_empty:
+            self._helper._default_config_hook.assert_called_once_with()
+        else:
+            self.assertFalse(self._helper._default_config_hook.called)
+        self.assertEqual(mock_template, self._helper.export_template)
+        self.assertIsNone(ret)
+
+    def test_init_helper_with_rados_store_pool_name_not_set(self):
+        self.mock_object(ganesha.ganesha_manager, 'GaneshaManager')
+        self.mock_object(self._helper, '_load_conf_dir')
+        self.mock_object(self._helper, '_default_config_hook')
+        self._helper.configuration.ganesha_rados_store_pool_name = None
+
+        self.assertRaises(
+            exception.GaneshaException, self._helper.init_helper)
+
+        self.assertFalse(ganesha.ganesha_manager.GaneshaManager.called)
+        self.assertFalse(self._helper._load_conf_dir.called)
+        self.assertFalse(self._helper._default_config_hook.called)
+
     def test_update_access_add_export(self):
         mock_gh = self._helper.ganesha
-        self.mock_object(mock_gh, '_check_export_file_exists',
+        self.mock_object(mock_gh, 'check_export_exists',
                          mock.Mock(return_value=False))
         self.mock_object(mock_gh, 'get_export_id',
                          mock.Mock(return_value=100))
@@ -364,7 +462,7 @@ class GaneshaNASHelper2TestCase(test.TestCase):
             self._context, self.share, access_rules=[self.rule1],
             add_rules=[], delete_rules=[])
 
-        mock_gh._check_export_file_exists.assert_called_once_with('fakename')
+        mock_gh.check_export_exists.assert_called_once_with('fakename')
         mock_gh.get_export_id.assert_called_once_with()
         self._helper._get_export_path.assert_called_once_with(self.share)
         (self._helper._get_export_pseudo_path.assert_called_once_with(
@@ -380,10 +478,10 @@ class GaneshaNASHelper2TestCase(test.TestCase):
               [{'Access_Type': 'ro', 'Clients': '10.0.0.1'}])
     def test_update_access_update_export(self, client):
         mock_gh = self._helper.ganesha
-        self.mock_object(mock_gh, '_check_export_file_exists',
+        self.mock_object(mock_gh, 'check_export_exists',
                          mock.Mock(return_value=True))
         self.mock_object(
-            mock_gh, '_read_export_file',
+            mock_gh, '_read_export',
             mock.Mock(return_value={'EXPORT': {'CLIENT': client}})
         )
         result_confdict = {
@@ -398,7 +496,7 @@ class GaneshaNASHelper2TestCase(test.TestCase):
             self._context, self.share, access_rules=[self.rule1, self.rule2],
             add_rules=[self.rule2], delete_rules=[])
 
-        mock_gh._check_export_file_exists.assert_called_once_with('fakename')
+        mock_gh.check_export_exists.assert_called_once_with('fakename')
         mock_gh.update_export.assert_called_once_with('fakename',
                                                       result_confdict)
         self.assertFalse(mock_gh.add_export.called)
@@ -406,12 +504,12 @@ class GaneshaNASHelper2TestCase(test.TestCase):
 
     def test_update_access_remove_export(self):
         mock_gh = self._helper.ganesha
-        self.mock_object(mock_gh, '_check_export_file_exists',
+        self.mock_object(mock_gh, 'check_export_exists',
                          mock.Mock(return_value=True))
         self.mock_object(self._helper, '_cleanup_fsal_hook')
         client = {'Access_Type': 'ro', 'Clients': '10.0.0.1'}
         self.mock_object(
-            mock_gh, '_read_export_file',
+            mock_gh, '_read_export',
             mock.Mock(return_value={'EXPORT': {'CLIENT': client}})
         )
 
@@ -419,7 +517,7 @@ class GaneshaNASHelper2TestCase(test.TestCase):
             self._context, self.share, access_rules=[],
             add_rules=[], delete_rules=[self.rule1])
 
-        mock_gh._check_export_file_exists.assert_called_once_with('fakename')
+        mock_gh.check_export_exists.assert_called_once_with('fakename')
         mock_gh.remove_export.assert_called_once_with('fakename')
         self._helper._cleanup_fsal_hook.assert_called_once_with(
             None, self.share, None)
@@ -428,7 +526,7 @@ class GaneshaNASHelper2TestCase(test.TestCase):
 
     def test_update_access_export_file_already_removed(self):
         mock_gh = self._helper.ganesha
-        self.mock_object(mock_gh, '_check_export_file_exists',
+        self.mock_object(mock_gh, 'check_export_exists',
                          mock.Mock(return_value=False))
         self.mock_object(ganesha.LOG, 'warning')
         self.mock_object(self._helper, '_cleanup_fsal_hook')
@@ -437,7 +535,7 @@ class GaneshaNASHelper2TestCase(test.TestCase):
             self._context, self.share, access_rules=[],
             add_rules=[], delete_rules=[self.rule1])
 
-        mock_gh._check_export_file_exists.assert_called_once_with('fakename')
+        mock_gh.check_export_exists.assert_called_once_with('fakename')
         ganesha.LOG.warning.assert_called_once_with(mock.ANY, mock.ANY)
         self.assertFalse(mock_gh.add_export.called)
         self.assertFalse(mock_gh.update_export.called)

releasenotes/notes/ganesha-store-exports-and-export-counter-in-ceph-rados-052b925f8ea460f4.yaml

@@ -0,0 +1,4 @@
+---
+features:
+  - Added ganesha driver feature to store NFS-Ganesha's exports and
+    export counter directly in a HA storage, Ceph's RADOS objects.