In-depth documentation on paste.deploy configuration files.

get_logger now uses python's log routing to separate multiple log_names and log_facilities in the same process.

doc/source/deployment_guide.rst

@@ -134,9 +134,80 @@ can be found in the :doc:`Ring Overview <overview_ring>`.
 General Server Configuration
 ----------------------------
 
-Swift uses paste.deploy to manage server configurations. Default configuration
+Swift uses paste.deploy (http://pythonpaste.org/deploy/) to manage server
-options are set in the `[DEFAULT]` section, and any options specified there
+configurations. Default configuration options are set in the `[DEFAULT]`
-can be overridden in any of the other sections.
+section, and any options specified there can be overridden in any of the other
+sections BUT ONLY BY USING THE SYNTAX ``set option_name = value``. This is the
+unfortunate way paste.deploy works and I'll try to explain it in full.
+
+First, here's an example paste.deploy configuration file::
+
+    [DEFAULT]
+    name1 = globalvalue
+    name2 = globalvalue
+    name3 = globalvalue
+    set name4 = globalvalue
+
+    [pipeline:main]
+    pipeline = myapp
+
+    [app:myapp]
+    use = egg:mypkg#myapp
+    name2 = localvalue
+    set name3 = localvalue
+    set name5 = localvalue
+    name6 = localvalue
+
+The resulting configuration that myapp receives is::
+
+    global {'__file__': '/etc/mypkg/wsgi.conf', 'here': '/etc/mypkg',
+            'name1': 'globalvalue',
+            'name2': 'globalvalue',
+            'name3': 'localvalue',
+            'name4': 'globalvalue',
+            'name5': 'localvalue',
+            'set name4': 'globalvalue'}
+    local {'name6': 'localvalue'}
+
+So, `name1` got the global value which is fine since it's only in the `DEFAULT`
+section anyway.
+
+`name2` got the global value from `DEFAULT` even though it's seemingly
+overridden in the `app:myapp` subsection. This is just the unfortunate way
+paste.deploy works (at least at the time of this writing.)
+
+`name3` got the local value from the `app:myapp` subsection because it using
+the special paste.deploy syntax of ``set option_name = value``. So, if you want
+a default value for most app/filters but want to overridde it in one
+subsection, this is how you do it.
+
+`name4` got the global value from `DEFAULT` since it's only in that section
+anyway. But, since we used the ``set`` syntax in the `DEFAULT` section even
+though we shouldn't, notice we also got a ``set name4`` variable. Weird, but
+probably not harmful.
+
+`name5` got the local value from the `app:myapp` subsection since it's only
+there anyway, but notice that it is in the global configuration and not the
+local configuration. This is because we used the ``set`` syntax to set the
+value. Again, weird, but not harmful since Swift just treats the two sets of
+configuration values as one set anyway.
+
+`name6` got the local value from `app:myapp` subsection since it's only there,
+and since we didn't use the ``set`` syntax, it's only in the local
+configuration and not the global one. Though, as indicated above, there is no
+special distinction with Swift.
+
+That's quite an explanation for something that should be so much simpler, but
+it might be important to know how paste.deploy interprets configuration files.
+The main rule to remember when working with Swift configuration files is:
+
+.. note::
+
+    Use the ``set option_name = value`` syntax in subsections if the option is
+    also set in the ``[DEFAULT]`` section. Don't get in the habit of always
+    using the ``set`` syntax or you'll probably mess up your non-paste.deploy
+    configuration files.
+
 
 ---------------------------
 Object Server Configuration
@@ -170,10 +241,10 @@ Option              Default        Description
 use                                paste.deploy entry point for the object
                                    server.  For most cases, this should be
                                    `egg:swift#object`.
-log_name            object-server  Label used when logging
+set log_name        object-server  Label used when logging
-log_facility        LOG_LOCAL0     Syslog log facility
+set log_facility    LOG_LOCAL0     Syslog log facility
-log_level           INFO           Logging level
+set log_level       INFO           Logging level
-log_requests        True           Whether or not to log each request
+set log_requests    True           Whether or not to log each request
 user                swift          User to run as
 node_timeout        3              Request timeout to external services
 conn_timeout        0.5            Connection timeout to external services
@@ -271,9 +342,9 @@ Option              Default           Description
 use                                   paste.deploy entry point for the 
                                       container server.  For most cases, this 
                                       should be `egg:swift#container`.
-log_name            container-server  Label used when logging
+set log_name        container-server  Label used when logging
-log_facility        LOG_LOCAL0        Syslog log facility
+set log_facility    LOG_LOCAL0        Syslog log facility
-log_level           INFO              Logging level
+set log_level       INFO              Logging level
 node_timeout        3                 Request timeout to external services
 conn_timeout        0.5               Connection timeout to external services
 ==================  ================  ========================================
@@ -358,9 +429,9 @@ Option              Default         Description
 use                                 Entry point for paste.deploy for the account
                                     server.  For most cases, this should be
                                     `egg:swift#account`.
-log_name            account-server  Label used when logging
+set log_name        account-server  Label used when logging
-log_facility        LOG_LOCAL0      Syslog log facility
+set log_facility    LOG_LOCAL0      Syslog log facility
-log_level           INFO            Logging level
+set log_level       INFO            Logging level
 ==================  ==============  ==========================================
 
 [account-replicator]
@@ -439,10 +510,10 @@ use                                            Entry point for paste.deploy for
                                                the proxy server.  For most
                                                cases, this should be
                                                `egg:swift#proxy`.
-log_name                      proxy-server     Label used when logging
+set log_name                  proxy-server     Label used when logging
-log_facility                  LOG_LOCAL0       Syslog log facility
+set log_facility              LOG_LOCAL0       Syslog log facility
-log_level                     INFO             Log level
+set log_level                 INFO             Log level
-log_headers                   True             If True, log headers in each
+set log_headers               True             If True, log headers in each
                                                request
 recheck_account_existence     60               Cache timeout in seconds to
                                                send memcached for account
@@ -500,10 +571,10 @@ use                                                    Entry point for
                                                        auth. To use the swauth
                                                        set to:
                                                        `egg:swift#swauth`
-log_name               auth-server                     Label used when logging
+set log_name           auth-server                     Label used when logging
-log_facility           LOG_LOCAL0                      Syslog log facility
+set log_facility       LOG_LOCAL0                      Syslog log facility
-log_level              INFO                            Log level
+set log_level          INFO                            Log level
-log_headers            True                            If True, log headers in
+set log_headers        True                            If True, log headers in
                                                        each request
 reseller_prefix        AUTH                            The naming scope for the
                                                        auth service. Swift

etc/account-server.conf-sample

@@ -7,18 +7,27 @@
 # swift_dir = /etc/swift
 # devices = /srv/node
 # mount_check = true
+# You can specify default log routing here if you want:
+# log_name = swift
+# log_facility = LOG_LOCAL0
+# log_level = INFO
 
 [pipeline:main]
 pipeline = account-server
 
 [app:account-server]
 use = egg:swift#account
-# log_name = account-server
+# You can override the default log routing for this app here:
-# log_facility = LOG_LOCAL0
+# set log_name = account-server
-# log_level = INFO
+# set log_facility = LOG_LOCAL0
+# set log_level = INFO
+# set log_requests = True
 
 [account-replicator]
+# You can override the default log routing for this app here (don't use set!):
 # log_name = account-replicator
+# log_facility = LOG_LOCAL0
+# log_level = INFO
 # vm_test_mode = no
 # log_facility = LOG_LOCAL0
 # log_level = INFO
@@ -36,7 +45,10 @@ use = egg:swift#account
 # reclaim_age = 86400
 
 [account-stats]
+# You can override the default log routing for this app here (don't use set!):
 # log_name = account-stats
+# log_facility = LOG_LOCAL0
+# log_level = INFO
 # cf_account = AUTH_7abbc116-8a07-4b63-819d-02715d3e0f31
 # container_name = account_stats
 # proxy_server_conf = /etc/swift/proxy-server.conf
@@ -44,14 +56,20 @@ use = egg:swift#account
 # log_level = INFO
 
 [account-auditor]
+# You can override the default log routing for this app here (don't use set!):
 # log_name = account-auditor
+# log_facility = LOG_LOCAL0
+# log_level = INFO
 # Will audit, at most, 1 account per device per interval
 # interval = 1800
 # log_facility = LOG_LOCAL0
 # log_level = INFO
 
 [account-reaper]
+# You can override the default log routing for this app here (don't use set!):
 # log_name = account-reaper
+# log_facility = LOG_LOCAL0
+# log_level = INFO
 # concurrency = 25
 # interval = 3600
 # node_timeout = 10

etc/auth-server.conf-sample

@@ -7,6 +7,10 @@
 # swift_dir = /etc/swift
 # cert_file = Default is no cert; format is path like /etc/swift/auth.crt
 # key_file = Default is no key; format is path like /etc/swift/auth.key
+# You can specify default log routing here if you want:
+# log_name = swift
+# log_facility = LOG_LOCAL0
+# log_level = INFO
 
 [pipeline:main]
 pipeline = auth-server
@@ -15,11 +19,12 @@ pipeline = auth-server
 use = egg:swift#auth
 # Highly recommended to change this.
 super_admin_key = devauth
-# log_name = auth-server
+# You can override the default log routing for this app here:
-# log_facility = LOG_LOCAL0
+# set log_name = proxy-server
-# log_level = INFO
+# set log_facility = LOG_LOCAL0
+# set log_level = INFO
+# set log_headers = False
 # reseller_prefix = AUTH
 # default_cluster_url = http://127.0.0.1:8080/v1
 # token_life = 86400
-# log_headers = False
 # node_timeout = 10

etc/container-server.conf-sample

@@ -7,20 +7,29 @@
 # swift_dir = /etc/swift
 # devices = /srv/node
 # mount_check = true
+# You can specify default log routing here if you want:
+# log_name = swift
+# log_facility = LOG_LOCAL0
+# log_level = INFO
 
 [pipeline:main]
 pipeline = container-server
 
 [app:container-server]
 use = egg:swift#container
-# log_name = container-server
+# You can override the default log routing for this app here:
-# log_facility = LOG_LOCAL0
+# set log_name = container-server
-# log_level = INFO
+# set log_facility = LOG_LOCAL0
+# set log_level = INFO
+# set log_requests = True
 # node_timeout = 3
 # conn_timeout = 0.5
 
 [container-replicator]
+# You can override the default log routing for this app here (don't use set!):
 # log_name = container-replicator
+# log_facility = LOG_LOCAL0
+# log_level = INFO
 # vm_test_mode = no
 # per_diff = 1000
 # concurrency = 8
@@ -31,7 +40,10 @@ use = egg:swift#container
 # reclaim_age = 604800
 
 [container-updater]
+# You can override the default log routing for this app here (don't use set!):
 # log_name = container-updater
+# log_facility = LOG_LOCAL0
+# log_level = INFO
 # interval = 300
 # concurrency = 4
 # node_timeout = 3
@@ -40,6 +52,9 @@ use = egg:swift#container
 # slowdown = 0.01
 
 [container-auditor]
+# You can override the default log routing for this app here (don't use set!):
 # log_name = container-auditor
+# log_facility = LOG_LOCAL0
+# log_level = INFO
 # Will audit, at most, 1 container per device per interval
 # interval = 1800

etc/object-server.conf-sample

@@ -7,16 +7,21 @@
 # swift_dir = /etc/swift
 # devices = /srv/node
 # mount_check = true
+# You can specify default log routing here if you want:
+# log_name = swift
+# log_facility = LOG_LOCAL0
+# log_level = INFO
 
 [pipeline:main]
 pipeline = object-server
 
 [app:object-server]
 use = egg:swift#object
-# log_name = object-server
+# You can override the default log routing for this app here:
-# log_facility = LOG_LOCAL0
+# set log_name = object-server
-# log_level = INFO
+# set log_facility = LOG_LOCAL0
-# log_requests = True
+# set log_level = INFO
+# set log_requests = True
 # node_timeout = 3
 # conn_timeout = 0.5
 # network_chunk_size = 65536
@@ -27,7 +32,10 @@ use = egg:swift#object
 # mb_per_sync = 512
 
 [object-replicator]
+# You can override the default log routing for this app here (don't use set!):
 # log_name = object-replicator
+# log_facility = LOG_LOCAL0
+# log_level = INFO
 # vm_test_mode = no
 # daemonize = on
 # run_pause = 30
@@ -45,7 +53,10 @@ use = egg:swift#object
 # reclaim_age = 604800
 
 [object-updater]
+# You can override the default log routing for this app here (don't use set!):
 # log_name = object-updater
+# log_facility = LOG_LOCAL0
+# log_level = INFO
 # interval = 300
 # concurrency = 1
 # node_timeout = 10
@@ -54,7 +65,10 @@ use = egg:swift#object
 # slowdown = 0.01
 
 [object-auditor]
+# You can override the default log routing for this app here (don't use set!):
 # log_name = object-auditor
+# log_facility = LOG_LOCAL0
+# log_level = INFO
 # files_per_second = 20
 # bytes_per_second = 10000000
 # log_time = 3600

etc/proxy-server.conf-sample

@@ -7,6 +7,10 @@
 # user = swift
 # cert_file = /etc/swift/proxy.crt
 # key_file = /etc/swift/proxy.key
+# You can specify default log routing here if you want:
+# log_name = swift
+# log_facility = LOG_LOCAL0
+# log_level = INFO
 
 [pipeline:main]
 # For DevAuth:
@@ -16,10 +20,11 @@ pipeline = catch_errors healthcheck cache ratelimit auth proxy-server
 
 [app:proxy-server]
 use = egg:swift#proxy
-# log_name = proxy-server
+# You can override the default log routing for this app here:
-# log_facility = LOG_LOCAL0
+# set log_name = proxy-server
-# log_level = INFO
+# set log_facility = LOG_LOCAL0
-# log_headers = False
+# set log_level = INFO
+# set log_headers = False
 # recheck_account_existence = 60
 # recheck_container_existence = 60
 # object_chunk_size = 8192
@@ -39,6 +44,11 @@ use = egg:swift#proxy
 # Only needed for DevAuth
 [filter:auth]
 use = egg:swift#auth
+# You can override the default log routing for this filter here:
+# set log_name = auth-server
+# set log_facility = LOG_LOCAL0
+# set log_level = INFO
+# set log_headers = False
 # The reseller prefix will verify a token begins with this prefix before even
 # attempting to validate it with the external authentication server. Also, with
 # authorization, only Swift storage accounts with this prefix will be
@@ -54,10 +64,11 @@ use = egg:swift#auth
 # Only needed for Swauth
 [filter:swauth]
 use = egg:swift#swauth
-# log_name = auth-server
+# You can override the default log routing for this filter here:
-# log_facility = LOG_LOCAL0
+# set log_name = auth-server
-# log_level = INFO
+# set log_facility = LOG_LOCAL0
-# log_headers = False
+# set log_level = INFO
+# set log_headers = False
 # The reseller prefix will verify a token begins with this prefix before even
 # attempting to validate it. Also, with authorization, only Swift storage
 # accounts with this prefix will be authorized by this middleware. Useful if
@@ -82,15 +93,30 @@ super_admin_key = swauthkey
 
 [filter:healthcheck]
 use = egg:swift#healthcheck
+# You can override the default log routing for this filter here:
+# set log_name = auth-server
+# set log_facility = LOG_LOCAL0
+# set log_level = INFO
+# set log_headers = False
 
 [filter:cache]
 use = egg:swift#memcache
+# You can override the default log routing for this filter here:
+# set log_name = auth-server
+# set log_facility = LOG_LOCAL0
+# set log_level = INFO
+# set log_headers = False
 # Default for memcache_servers is below, but you can specify multiple servers
 # with the format: 10.1.2.3:11211,10.1.2.4:11211
 # memcache_servers = 127.0.0.1:11211
 
 [filter:ratelimit]
 use = egg:swift#ratelimit
+# You can override the default log routing for this filter here:
+# set log_name = auth-server
+# set log_facility = LOG_LOCAL0
+# set log_level = INFO
+# set log_headers = False
 # clock_accuracy should represent how accurate the proxy servers' system clocks
 # are with each other. 1000 means that all the proxies' clock are accurate to
 # each other within 1 millisecond.  No ratelimit should be higher than the
@@ -118,14 +144,29 @@ use = egg:swift#ratelimit
 
 [filter:domain_remap]
 use = egg:swift#domain_remap
+# You can override the default log routing for this filter here:
+# set log_name = auth-server
+# set log_facility = LOG_LOCAL0
+# set log_level = INFO
+# set log_headers = False
 # storage_domain = example.com
 # path_root = v1
 
 [filter:catch_errors]
 use = egg:swift#catch_errors
+# You can override the default log routing for this filter here:
+# set log_name = auth-server
+# set log_facility = LOG_LOCAL0
+# set log_level = INFO
+# set log_headers = False
 
 [filter:cname_lookup]
 # Note: this middleware requires python-dnspython
 use = egg:swift#cname_lookup
+# You can override the default log routing for this filter here:
+# set log_name = auth-server
+# set log_facility = LOG_LOCAL0
+# set log_level = INFO
+# set log_headers = False
 # storage_domain = example.com
 # lookup_depth = 1

swift/common/utils.py

@@ -382,7 +382,7 @@ class NamedFormatter(logging.Formatter):
         return msg
 
 
-def get_logger(conf, name=None, log_to_console=False):
+def get_logger(conf, name=None, log_to_console=False, log_route=None):
     """
     Get the current system logger using config settings.
 
@@ -396,33 +396,41 @@ def get_logger(conf, name=None, log_to_console=False):
     :param name: Name of the logger
     :param log_to_console: Add handler which writes to console on stderr
     """
-    root_logger = logging.getLogger()
+    if not conf:
-    if hasattr(get_logger, 'handler') and get_logger.handler:
+        conf = {}
-        root_logger.removeHandler(get_logger.handler)
+    if not hasattr(get_logger, 'root_logger_configured'):
-        get_logger.handler.close()
+        get_logger.root_logger_configured = True
-        get_logger.handler = None
+        get_logger(conf, name, log_to_console, log_route='root')
+    if name is None:
+        name = conf.get('log_name', 'swift')
+    if not log_route:
+        log_route = name
+    if log_route == 'root':
+        logger = logging.getLogger()
+    else:
+        logger = logging.getLogger(log_route)
+    if not hasattr(get_logger, 'handlers'):
+        get_logger.handlers = {}
+    facility = getattr(SysLogHandler, conf.get('log_facility', 'LOG_LOCAL0'),
+                       SysLogHandler.LOG_LOCAL0)
+    if facility in get_logger.handlers:
+        logger.removeHandler(get_logger.handlers[facility])
+        get_logger.handlers[facility].close()
+        del get_logger.handlers[facility]
     if log_to_console:
         # check if a previous call to get_logger already added a console logger
         if hasattr(get_logger, 'console') and get_logger.console:
-            root_logger.removeHandler(get_logger.console)
+            logger.removeHandler(get_logger.console)
         get_logger.console = logging.StreamHandler(sys.__stderr__)
-        root_logger.addHandler(get_logger.console)
+        logger.addHandler(get_logger.console)
-    if conf is None:
+    get_logger.handlers[facility] = \
-        root_logger.setLevel(logging.INFO)
+        SysLogHandler(address='/dev/log', facility=facility)
-        adapted_logger = LogAdapter(root_logger)
+    logger.addHandler(get_logger.handlers[facility])
-        return adapted_logger
+    logger.setLevel(
-    if name is None:
-        name = conf.get('log_name', 'swift')
-    get_logger.handler = SysLogHandler(address='/dev/log',
-        facility=getattr(SysLogHandler,
-                         conf.get('log_facility', 'LOG_LOCAL0'),
-                         SysLogHandler.LOG_LOCAL0))
-    root_logger.addHandler(get_logger.handler)
-    root_logger.setLevel(
         getattr(logging, conf.get('log_level', 'INFO').upper(), logging.INFO))
-    adapted_logger = LogAdapter(root_logger)
+    adapted_logger = LogAdapter(logger)
     formatter = NamedFormatter(name, adapted_logger)
-    get_logger.handler.setFormatter(formatter)
+    get_logger.handlers[facility].setFormatter(formatter)
     if hasattr(get_logger, 'console'):
         get_logger.console.setFormatter(formatter)
     return adapted_logger