Avoid the need for users to manually edit PasteDeploy config in order to switch pipelines.

Define multiple pipelines in glace-api.conf to reflect the
various supported deployment flavors (minimal, with caching,
with cache management, with keystone-based auth etc.).

Add an optional paste_deploy.flavor config variable to allow the
user select the appropriate pipeline without having to edit the
paste config (i.e. uncommenting lines as before). For example
in glance-api.conf, a setting of:

  [paste_deploy]
  flavor = keystone+caching

identifies the following pipeline in glace-api-paste.ini:

  [pipeline:glance-api-keystone+caching]
  pipeline = versionnegotiation authtoken auth-context cache apiv1app

the advantage being that the user need not be concerned with
the precise sequence of filters required to realize the QoS
they desire.

Modify the functional tests that patch configuration (i.e. the
keystone and caching tests) to use the new deployment_flavor
mechanism.

Extend the TestConfigOpts to support option groups.

Change-Id: Ide843ada11bce115b7dc650440397853c6409b03

doc/source/authentication.rst

@@ -185,18 +185,18 @@ The ``context_class`` variable is needed to specify the
 Registry-specific request context, which contains the extra access
 checks used by the Registry.
 
-Again, to enable using Keystone authentication, the application
+Again, to enable using Keystone authentication, the appropriate
-pipeline must be modified.  By default, it looks like:
+application pipeline must be selected.  By default, it looks like:
 
-  [pipeline:glance-registry]
+  [pipeline:glance-registry-keystone]
-  pipeline = context registryapp
-
-This must be changed by replacing ``context`` with ``authtoken`` and
-``auth-context``::
-
-  [pipeline:glance-registry]
   pipeline = authtoken auth-context registryapp
 
+To enable the above application pipeline, in your main ``glance-registry.conf``
+configuration file, select the appropriate deployment flavor like so::
+
+  [paste_deploy]
+  flavor = keystone
+
 Sharing Images With Others
 --------------------------
 

doc/source/configuring.rst

@@ -495,13 +495,15 @@ image files is transparent and happens using a piece of middleware that can
 optionally be placed in the server application pipeline.
 
 This pipeline is configured in the PasteDeploy configuration file,
-<component>-paste.ini.
+<component>-paste.ini. You should not generally have to edit this file
+directly, as it ships with ready-made pipelines for all common deployment
+flavors.
 
 Enabling the Image Cache Middleware
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-To enable the image cache middleware, you would insert the cache middleware
+To enable the image cache middleware, the cache middleware must occur in
-into your application pipeline **after** the appropriate context middleware.
+the application pipeline **after** the appropriate context middleware.
 
 The cache middleware should be in your ``glance-api-paste.ini`` in a section
 titled ``[filter:cache]``. It should look like this::
@@ -510,19 +512,18 @@ titled ``[filter:cache]``. It should look like this::
   paste.filter_factory = glance.common.wsgi:filter_factory
   glance.filter_factory = glance.api.middleware.cache:CacheFilter
 
+A ready-made application pipeline including this filter is defined in
+the ``glance-api-paste.ini`` file, looking like so::
 
-For example, suppose your application pipeline in the ``glance-api-paste.ini``
+  [pipeline:glance-api-caching]
-file looked like so::
-
-  [pipeline:glance-api]
-  pipeline = versionnegotiation context apiv1app
-
-In the above application pipeline, you would add the cache middleware after the
-context middleware, like so::
-
-  [pipeline:glance-api]
   pipeline = versionnegotiation context cache apiv1app
 
+To enable the above application pipeline, in your main ``glance-api.conf``
+configuration file, select the appropriate deployment flavor like so::
+
+  [paste_deploy]
+  flavor = caching
+
 And that would give you a transparent image cache on the API server.
 
 Configuration Options Affecting the Image Cache

etc/glance-api-paste.ini

@@ -1,17 +1,46 @@
+# Default minimal pipeline
 [pipeline:glance-api]
 pipeline = versionnegotiation context apiv1app
-# NOTE: use the following pipeline for keystone
-# pipeline = versionnegotiation authtoken auth-context apiv1app
 
-# To enable transparent caching of image files replace pipeline with below:
+# Use the following pipeline for keystone auth
-# pipeline = versionnegotiation context cache apiv1app
+# i.e. in glance-api.conf:
-# NOTE: use the following pipeline for keystone auth (with caching)
+#   [paste_deploy]
-# pipeline = versionnegotiation authtoken auth-context cache apiv1app
+#   flavor = keystone
+#
+[pipeline:glance-api-keystone]
+pipeline = versionnegotiation authtoken auth-context apiv1app
 
-# To enable Image Cache Management API replace pipeline with below:
+# Use the following pipeline to enable transparent caching of image files
-# pipeline = versionnegotiation context cachemanage apiv1app
+# i.e. in glance-api.conf:
-# NOTE: use the following pipeline for keystone auth (with caching)
+#   [paste_deploy]
-# pipeline = versionnegotiation authtoken auth-context cachemanage apiv1app
+#   flavor = caching
+#
+[pipeline:glance-api-caching]
+pipeline = versionnegotiation context cache apiv1app
+
+# Use the following pipeline for keystone auth with caching
+# i.e. in glance-api.conf:
+#   [paste_deploy]
+#   flavor = keystone+caching
+#
+[pipeline:glance-api-keystone+caching]
+pipeline = versionnegotiation authtoken auth-context cache apiv1app
+
+# Use the following pipeline to enable the Image Cache Management API
+# i.e. in glance-api.conf:
+#   [paste_deploy]
+#   flavor = cachemanagement
+#
+[pipeline:glance-api-cachemanagement]
+pipeline = versionnegotiation context cache cachemanage apiv1app
+
+# Use the following pipeline for keystone auth with cache management
+# i.e. in glance-api.conf:
+#   [paste_deploy]
+#   flavor = keystone+cachemanagement
+#
+[pipeline:glance-api-keystone+cachemanagement]
+pipeline = versionnegotiation authtoken auth-context cache cachemanage apiv1app
 
 [app:apiv1app]
 paste.app_factory = glance.common.wsgi:app_factory

etc/glance-registry-paste.ini

@@ -1,7 +1,14 @@
+# Default minimal pipeline
 [pipeline:glance-registry]
 pipeline = context registryapp
-# NOTE: use the following pipeline for keystone
+
-# pipeline = authtoken auth-context registryapp
+# Use the following pipeline for keystone auth
+# i.e. in glance-registry.conf:
+#   [paste_deploy]
+#   flavor = keystone
+#
+[pipeline:glance-registry-keystone]
+pipeline = authtoken auth-context registryapp
 
 [app:registryapp]
 paste.app_factory = glance.common.wsgi:app_factory

glance/common/config.py

@@ -31,6 +31,12 @@ from glance.common import cfg
 from glance.common import wsgi
 
 
+paste_deploy_group = cfg.OptGroup('paste_deploy')
+paste_deploy_opts = [
+    cfg.StrOpt('flavor'),
+    ]
+
+
 class GlanceConfigOpts(cfg.CommonConfigOpts):
 
     def __init__(self, default_config_files=None, **kwargs):
@@ -96,6 +102,28 @@ def setup_logging(conf):
     root_logger.addHandler(handler)
 
 
+def _register_paste_deploy_opts(conf):
+    """
+    Idempotent registration of paste_deploy option group
+
+    :param conf: a cfg.ConfigOpts object
+    """
+    conf.register_group(paste_deploy_group)
+    conf.register_opts(paste_deploy_opts, group=paste_deploy_group)
+
+
+def _get_deployment_flavor(conf):
+    """
+    Retrieve the paste_deploy.flavor config item, formatted appropriately
+    for appending to the application name.
+
+    :param conf: a cfg.ConfigOpts object
+    """
+    _register_paste_deploy_opts(conf)
+    flavor = conf.paste_deploy.flavor
+    return '' if not flavor else ('-' + flavor)
+
+
 def load_paste_app(conf, app_name=None):
     """
     Builds and returns a WSGI app from a paste config file.
@@ -112,6 +140,10 @@ def load_paste_app(conf, app_name=None):
     if app_name is None:
         app_name = conf.prog
 
+    # append the deployment flavor to the application name,
+    # in order to identify the appropriate paste pipeline
+    app_name += _get_deployment_flavor(conf)
+
     # Assume paste config is in a paste.ini file corresponding
     # to the last config file
     conf_file = os.path.abspath(conf.config_file[-1].replace(".conf",

glance/tests/functional/__init__.py

@@ -84,6 +84,7 @@ class Server(object):
         self.paste_conf_base = None
         self.server_control = './bin/glance-control'
         self.exec_env = None
+        self.deployment_flavor = ''
 
     def write_conf(self, **kwargs):
         """
@@ -188,7 +189,6 @@ class ApiServer(Server):
         self.rbd_store_chunk_size = 4
         self.delayed_delete = delayed_delete
         self.owner_is_tenant = True
-        self.cache_pipeline = ""  # Set to cache for cache middleware
         self.image_cache_dir = os.path.join(self.test_dir,
                                             'cache')
         self.image_cache_driver = 'sqlite'
@@ -225,9 +225,17 @@ scrub_time = 5
 scrubber_datadir = %(scrubber_datadir)s
 image_cache_dir = %(image_cache_dir)s
 image_cache_driver = %(image_cache_driver)s
+[paste_deploy]
+flavor = %(deployment_flavor)s
 """
         self.paste_conf_base = """[pipeline:glance-api]
-pipeline = versionnegotiation context %(cache_pipeline)s apiv1app
+pipeline = versionnegotiation context apiv1app
+
+[pipeline:glance-api-caching]
+pipeline = versionnegotiation context cache apiv1app
+
+[pipeline:glance-api-cachemanagement]
+pipeline = versionnegotiation context cache cache_manage apiv1app
 
 [app:apiv1app]
 paste.app_factory = glance.common.wsgi:app_factory
@@ -281,6 +289,8 @@ sql_idle_timeout = 3600
 api_limit_max = 1000
 limit_param_default = 25
 owner_is_tenant = %(owner_is_tenant)s
+[paste_deploy]
+flavor = %(deployment_flavor)s
 """
         self.paste_conf_base = """[pipeline:glance-registry]
 pipeline = context registryapp

glance/tests/functional/keystone_utils.py

@@ -130,6 +130,10 @@ class AdminServer(KeystoneServer):
                                           auth_port, admin_port)
 
 
+def patch_copy(base, src, offset, old, new):
+    base.insert(src + offset, base[src].replace(old, new))
+
+
 def conf_patch(server, **subs):
     # First, pull the configuration file
     paste_base = server.paste_conf_base.split('\n')
@@ -137,13 +141,15 @@ def conf_patch(server, **subs):
     # Need to find the pipeline
     for idx, text in enumerate(paste_base):
         if text.startswith('[pipeline:glance-'):
-            # OK, the line to modify is the next one...
+            # OK, the lines to repeat in modified form
-            modidx = idx + 1
+            # are this and the next one...
+            modidx = idx
             break
 
-    # Now we need to replace the default context field...
+    # Now we need to add a new pipeline, replacing the default context field...
-    paste_base[modidx] = paste_base[modidx].replace('context',
+    server.deployment_flavor = 'tokenauth+keystoneshim'
-                                                    'tokenauth keystone_shim')
+    patch_copy(paste_base, modidx, 2, ']', '-tokenauth+keystoneshim]')
+    patch_copy(paste_base, modidx + 1, 2, 'context', 'tokenauth keystone_shim')
 
     # Put the conf back together and append the keystone pieces
     server.paste_conf_base = '\n'.join(paste_base) + """

glance/tests/functional/test_bin_glance_cache_manage.py

@@ -36,11 +36,12 @@ class TestBinGlanceCacheManage(functional.FunctionalTest):
     """Functional tests for the bin/glance CLI tool"""
 
     def setUp(self):
-        self.cache_pipeline = "cache cache_manage"
         self.image_cache_driver = "sqlite"
 
         super(TestBinGlanceCacheManage, self).setUp()
 
+        self.api_server.deployment_flavor = "cachemanagement"
+
         # NOTE(sirp): This is needed in case we are running the tests under an
         # environment in which OS_AUTH_STRATEGY=keystone. The test server we
         # spin up won't have keystone support, so we need to switch to the
@@ -85,6 +86,7 @@ class TestBinGlanceCacheManage(functional.FunctionalTest):
         Test that cache index command works
         """
         self.cleanup()
+        self.api_server.deployment_flavor = ''
         self.start_servers()  # Not passing in cache_manage in pipeline...
 
         api_port = self.api_port

glance/tests/functional/test_cache_middleware.py

@@ -436,11 +436,12 @@ class TestImageCacheXattr(functional.FunctionalTest,
 
         self.inited = True
         self.disabled = False
-        self.cache_pipeline = "cache"
         self.image_cache_driver = "xattr"
 
         super(TestImageCacheXattr, self).setUp()
 
+        self.api_server.deployment_flavor = "caching"
+
         if not xattr_writes_supported(self.test_dir):
             self.inited = True
             self.disabled = True
@@ -480,11 +481,12 @@ class TestImageCacheManageXattr(functional.FunctionalTest,
 
         self.inited = True
         self.disabled = False
-        self.cache_pipeline = "cache cache_manage"
         self.image_cache_driver = "xattr"
 
         super(TestImageCacheManageXattr, self).setUp()
 
+        self.api_server.deployment_flavor = "cachemanagement"
+
         if not xattr_writes_supported(self.test_dir):
             self.inited = True
             self.disabled = True
@@ -524,10 +526,11 @@ class TestImageCacheSqlite(functional.FunctionalTest,
 
         self.inited = True
         self.disabled = False
-        self.cache_pipeline = "cache"
 
         super(TestImageCacheSqlite, self).setUp()
 
+        self.api_server.deployment_flavor = "caching"
+
     def tearDown(self):
         if os.path.exists(self.api_server.image_cache_dir):
             shutil.rmtree(self.api_server.image_cache_dir)
@@ -561,11 +564,12 @@ class TestImageCacheManageSqlite(functional.FunctionalTest,
 
         self.inited = True
         self.disabled = False
-        self.cache_pipeline = "cache cache_manage"
         self.image_cache_driver = "sqlite"
 
         super(TestImageCacheManageSqlite, self).setUp()
 
+        self.api_server.deployment_flavor = "cachemanagement"
+
     def tearDown(self):
         if os.path.exists(self.api_server.image_cache_dir):
             shutil.rmtree(self.api_server.image_cache_dir)

glance/tests/unit/test_config.py

@@ -16,6 +16,7 @@
 #    under the License.
 
 import os.path
+import shutil
 import unittest
 
 import stubout
@@ -23,8 +24,9 @@ import stubout
 from glance.api.middleware import version_negotiation
 from glance.api.v1 import images
 from glance.api.v1 import members
-from glance.common import config
+from glance.common import config, context, utils
 from glance.image_cache import pruner
+from glance.tests import utils as test_utils
 
 
 class TestPasteApp(unittest.TestCase):
@@ -35,19 +37,39 @@ class TestPasteApp(unittest.TestCase):
     def tearDown(self):
         self.stubs.UnsetAll()
 
-    def test_load_paste_app(self):
+    def _do_test_load_paste_app(self,
-        conf = config.GlanceConfigOpts()
+                                expected_app_type,
-        conf(['--config-file',
+                                paste_group={},
-              os.path.join(os.getcwd(), 'etc/glance-api.conf')])
+                                paste_append=None):
 
-        self.stubs.Set(config, 'setup_logging', lambda *a: None)
+        conf = test_utils.TestConfigOpts(groups=paste_group)
-        self.stubs.Set(images, 'create_resource', lambda *a: None)
+
-        self.stubs.Set(members, 'create_resource', lambda *a: None)
+        def _appendto(orig, copy, str):
+            shutil.copy(orig, copy)
+            with open(copy, 'ab') as f:
+                f.write(str or '')
+                f.flush()
+
+        paste_from = os.path.join(os.getcwd(), 'etc/glance-api-paste.ini')
+        paste_to = os.path.join(conf.temp_file.replace('.conf',
+                                                       '-paste.ini'))
+        _appendto(paste_from, paste_to, paste_append)
 
         app = config.load_paste_app(conf, 'glance-api')
 
-        self.assertEquals(version_negotiation.VersionNegotiationFilter,
+        self.assertEquals(expected_app_type, type(app))
-                          type(app))
+
+    def test_load_paste_app(self):
+        type = version_negotiation.VersionNegotiationFilter
+        self._do_test_load_paste_app(type)
+
+    def test_load_paste_app_with_paste_flavor(self):
+        paste_group = {'paste_deploy': {'flavor': 'incomplete'}}
+        pipeline = '[pipeline:glance-api-incomplete]\n' + \
+                   'pipeline = context apiv1app'
+
+        type = context.ContextMiddleware
+        self._do_test_load_paste_app(type, paste_group, paste_append=pipeline)
 
     def test_load_paste_app_with_conf_name(self):
         def fake_join(*args):

glance/tests/utils.py

@@ -30,35 +30,66 @@ from glance.common import config
 
 
 class TestConfigOpts(config.GlanceConfigOpts):
+    """
+    Support easily controllable config for unit tests, avoiding the
+    need to manipulate config files directly.
 
-    def __init__(self, test_values):
+    Configuration values are provided as a dictionary of key-value pairs,
+    in the simplest case feeding into the DEFAULT group only.
+
+    Non-default groups may also populated via nested dictionaries, e.g.
+
+      {'snafu': {'foo': 'bar', 'bells': 'whistles'}}
+
+    equates to config of form:
+
+      [snafu]
+      foo = bar
+      bells = whistles
+
+    The config so provided is dumped to a temporary file, with its path
+    exposed via the temp_file property.
+
+    :param test_values: dictionary of key-value pairs for the
+                        DEFAULT group
+    :param groups:      nested dictionary of key-value pairs for
+                        non-default groups
+    """
+
+    def __init__(self, test_values={}, groups={}):
         super(TestConfigOpts, self).__init__()
         self._test_values = test_values
+        self._test_groups = groups
+
+        self.temp_file = os.path.join(tempfile.mkdtemp(), 'testcfg.conf')
+
         self()
 
     def __call__(self):
-        config_file = self._write_tmp_config_file()
+        self._write_tmp_config_file()
         try:
             super(TestConfigOpts, self).\
-                __call__(['--config-file', config_file])
+                __call__(['--config-file', self.temp_file])
         finally:
-            os.remove(config_file)
+            os.remove(self.temp_file)
 
     def _write_tmp_config_file(self):
         contents = '[DEFAULT]\n'
         for key, value in self._test_values.items():
             contents += '%s = %s\n' % (key, value)
 
-        (fd, path) = tempfile.mkstemp(prefix='testcfg')
+        for group, settings in self._test_groups.items():
-        try:
+            contents += '[%s]\n' % group
-            os.write(fd, contents)
+            for key, value in settings.items():
-        except Exception, e:
+                contents += '%s = %s\n' % (key, value)
-            os.close(fd)
-            os.remove(path)
-            raise e
 
-        os.close(fd)
+        try:
-        return path
+            with open(self.temp_file, 'wb') as f:
+                f.write(contents)
+                f.flush()
+        except Exception, e:
+            os.remove(self.temp_file)
+            raise e
 
 
 class skip_test(object):