From 778cb9dedc96f774242e38b301f35c89e002c0a0 Mon Sep 17 00:00:00 2001 From: gholt Date: Sun, 23 Jan 2011 10:42:31 -0800 Subject: [PATCH 1/5] In-depth documentation on paste.deploy configuration files --- doc/source/deployment_guide.rst | 77 +++++++++++++++++++++++++++++++-- 1 file changed, 74 insertions(+), 3 deletions(-) diff --git a/doc/source/deployment_guide.rst b/doc/source/deployment_guide.rst index 40854b0a1f..c0047847e5 100644 --- a/doc/source/deployment_guide.rst +++ b/doc/source/deployment_guide.rst @@ -134,9 +134,80 @@ can be found in the :doc:`Ring Overview `. General Server Configuration ---------------------------- -Swift uses paste.deploy to manage server configurations. Default configuration -options are set in the `[DEFAULT]` section, and any options specified there -can be overridden in any of the other sections. +Swift uses paste.deploy (http://pythonpaste.org/deploy/) to manage server +configurations. Default configuration options are set in the `[DEFAULT]` +section, and any options specified there can be overridden in any of the other +sections BUT ONLY BY USING THE SYNTAX ``set option_name = value``. This is the +unfortunate way paste.deploy works and I'll try to explain it in full. + +First, here's an example paste.deploy configuration file:: + + [DEFAULT] + name1 = globalvalue + name2 = globalvalue + name3 = globalvalue + set name4 = globalvalue + + [pipeline:main] + pipeline = myapp + + [app:myapp] + use = egg:mypkg#myapp + name2 = localvalue + set name3 = localvalue + set name5 = localvalue + name6 = localvalue + +The resulting configuration that myapp receives is:: + + global {'__file__': '/etc/mypkg/wsgi.conf', 'here': '/etc/mypkg', + 'name1': 'globalvalue', + 'name2': 'globalvalue', + 'name3': 'localvalue', + 'name4': 'globalvalue', + 'name5': 'localvalue', + 'set name4': 'globalvalue'} + local {'name6': 'localvalue'} + +So, `name1` got the global value which is fine since it's only in the `DEFAULT` +section anyway. + +`name2` got the global value from `DEFAULT` even though it's seemingly +overridden in the `app:myapp` subsection. This is just the unfortunate way +paste.deploy works (at least at the time of this writing.) + +`name3` got the local value from the `app:myapp` subsection because it using +the special paste.deploy syntax of ``set option_name = value``. So, if you want +a default value for most app/filters but want to overridde it in one +subsection, this is how you do it. + +`name4` got the global value from `DEFAULT` since it's only in that section +anyway. But, since we used the ``set`` syntax in the `DEFAULT` section even +though we shouldn't, notice we also got a ``set name4`` variable. Weird, but +probably not harmful. + +`name5` got the local value from the `app:myapp` subsection since it's only +there anyway, but notice that it is in the global configuration and not the +local configuration. This is because we used the ``set`` syntax to set the +value. Again, weird, but not harmful since Swift just treats the two sets of +configuration values as one set anyway. + +`name6` got the local value from `app:myapp` subsection since it's only there, +and since we didn't use the ``set`` syntax, it's only in the local +configuration and not the global one. Though, as indicated above, there is no +special distinction with Swift. + +That's quite an explanation for something that should be so much simpler, but +it might be important to know how paste.deploy interprets configuration files. +The main rule to remember when working with Swift configuration files is: + +.. note:: + + Always use the ``set option_name = value`` syntax in subsections if the + option is also set in the ``[DEFAULT]`` section. Don't get in the habit of + always using the ``set`` syntax or you'll probably mess up your + non-paste.deploy configuration files. + --------------------------- Object Server Configuration From e274741f25d7da02199398d287a58961bba2190d Mon Sep 17 00:00:00 2001 From: gholt Date: Sun, 23 Jan 2011 10:50:55 -0800 Subject: [PATCH 2/5] Remove the misleading "Always" --- doc/source/deployment_guide.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/source/deployment_guide.rst b/doc/source/deployment_guide.rst index c0047847e5..900cb8577b 100644 --- a/doc/source/deployment_guide.rst +++ b/doc/source/deployment_guide.rst @@ -203,10 +203,10 @@ The main rule to remember when working with Swift configuration files is: .. note:: - Always use the ``set option_name = value`` syntax in subsections if the - option is also set in the ``[DEFAULT]`` section. Don't get in the habit of - always using the ``set`` syntax or you'll probably mess up your - non-paste.deploy configuration files. + Use the ``set option_name = value`` syntax in subsections if the option is + also set in the ``[DEFAULT]`` section. Don't get in the habit of always + using the ``set`` syntax or you'll probably mess up your non-paste.deploy + configuration files. --------------------------- From cc638e7ed5243496e1cde80cdf9dfa5af139678e Mon Sep 17 00:00:00 2001 From: gholt Date: Sun, 23 Jan 2011 12:32:10 -0800 Subject: [PATCH 3/5] get_logger now uses python's log routing to separate multiple log_names in the same process --- swift/common/utils.py | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/swift/common/utils.py b/swift/common/utils.py index 299980493a..661e23eeb3 100644 --- a/swift/common/utils.py +++ b/swift/common/utils.py @@ -382,7 +382,7 @@ class NamedFormatter(logging.Formatter): return msg -def get_logger(conf, name=None, log_to_console=False): +def get_logger(conf, name=None, log_to_console=False, log_route=None): """ Get the current system logger using config settings. @@ -396,31 +396,39 @@ def get_logger(conf, name=None, log_to_console=False): :param name: Name of the logger :param log_to_console: Add handler which writes to console on stderr """ - root_logger = logging.getLogger() + if not hasattr(get_logger, 'root_logger_configured'): + get_logger.root_logger_configured = True + get_logger(conf, name, log_to_console, log_route='root') + if name is None: + name = conf.get('log_name', 'swift') if conf else 'swift' + if not log_route: + log_route = name + if log_route == 'root': + logger = logging.getLogger() + else: + logger = logging.getLogger(log_route) if hasattr(get_logger, 'handler') and get_logger.handler: - root_logger.removeHandler(get_logger.handler) + logger.removeHandler(get_logger.handler) get_logger.handler.close() get_logger.handler = None if log_to_console: # check if a previous call to get_logger already added a console logger if hasattr(get_logger, 'console') and get_logger.console: - root_logger.removeHandler(get_logger.console) + logger.removeHandler(get_logger.console) get_logger.console = logging.StreamHandler(sys.__stderr__) - root_logger.addHandler(get_logger.console) + logger.addHandler(get_logger.console) if conf is None: - root_logger.setLevel(logging.INFO) - adapted_logger = LogAdapter(root_logger) + logger.setLevel(logging.INFO) + adapted_logger = LogAdapter(logger) return adapted_logger - if name is None: - name = conf.get('log_name', 'swift') get_logger.handler = SysLogHandler(address='/dev/log', facility=getattr(SysLogHandler, conf.get('log_facility', 'LOG_LOCAL0'), SysLogHandler.LOG_LOCAL0)) - root_logger.addHandler(get_logger.handler) - root_logger.setLevel( + logger.addHandler(get_logger.handler) + logger.setLevel( getattr(logging, conf.get('log_level', 'INFO').upper(), logging.INFO)) - adapted_logger = LogAdapter(root_logger) + adapted_logger = LogAdapter(logger) formatter = NamedFormatter(name, adapted_logger) get_logger.handler.setFormatter(formatter) if hasattr(get_logger, 'console'): From 9c3f4a17cf94771b98364e82cd230c3dc1f2afb0 Mon Sep 17 00:00:00 2001 From: gholt Date: Sun, 23 Jan 2011 12:58:54 -0800 Subject: [PATCH 4/5] get_logger now separates different log_levels --- swift/common/utils.py | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/swift/common/utils.py b/swift/common/utils.py index 661e23eeb3..8524ffb711 100644 --- a/swift/common/utils.py +++ b/swift/common/utils.py @@ -396,41 +396,41 @@ def get_logger(conf, name=None, log_to_console=False, log_route=None): :param name: Name of the logger :param log_to_console: Add handler which writes to console on stderr """ + if not conf: + conf = {} if not hasattr(get_logger, 'root_logger_configured'): get_logger.root_logger_configured = True get_logger(conf, name, log_to_console, log_route='root') if name is None: - name = conf.get('log_name', 'swift') if conf else 'swift' + name = conf.get('log_name', 'swift') if not log_route: log_route = name if log_route == 'root': logger = logging.getLogger() else: logger = logging.getLogger(log_route) - if hasattr(get_logger, 'handler') and get_logger.handler: - logger.removeHandler(get_logger.handler) - get_logger.handler.close() - get_logger.handler = None + if not hasattr(get_logger, 'handlers'): + get_logger.handlers = {} + facility = getattr(SysLogHandler, conf.get('log_facility', 'LOG_LOCAL0'), + SysLogHandler.LOG_LOCAL0) + if facility in get_logger.handlers: + logger.removeHandler(get_logger.handlers[facility]) + get_logger.handlers[facility].close() + del get_logger.handlers[facility] if log_to_console: # check if a previous call to get_logger already added a console logger if hasattr(get_logger, 'console') and get_logger.console: logger.removeHandler(get_logger.console) get_logger.console = logging.StreamHandler(sys.__stderr__) logger.addHandler(get_logger.console) - if conf is None: - logger.setLevel(logging.INFO) - adapted_logger = LogAdapter(logger) - return adapted_logger - get_logger.handler = SysLogHandler(address='/dev/log', - facility=getattr(SysLogHandler, - conf.get('log_facility', 'LOG_LOCAL0'), - SysLogHandler.LOG_LOCAL0)) - logger.addHandler(get_logger.handler) + get_logger.handlers[facility] = \ + SysLogHandler(address='/dev/log', facility=facility) + logger.addHandler(get_logger.handlers[facility]) logger.setLevel( getattr(logging, conf.get('log_level', 'INFO').upper(), logging.INFO)) adapted_logger = LogAdapter(logger) formatter = NamedFormatter(name, adapted_logger) - get_logger.handler.setFormatter(formatter) + get_logger.handlers[facility].setFormatter(formatter) if hasattr(get_logger, 'console'): get_logger.console.setFormatter(formatter) return adapted_logger From 4905c71669b270654aa02b69fdb4bc4d15ade7a4 Mon Sep 17 00:00:00 2001 From: gholt Date: Sun, 23 Jan 2011 13:18:28 -0800 Subject: [PATCH 5/5] More doc updates for logger stuff --- doc/source/deployment_guide.rst | 36 ++++++++++---------- etc/account-server.conf-sample | 24 ++++++++++++-- etc/auth-server.conf-sample | 13 +++++--- etc/container-server.conf-sample | 21 ++++++++++-- etc/object-server.conf-sample | 22 +++++++++--- etc/proxy-server.conf-sample | 57 +++++++++++++++++++++++++++----- 6 files changed, 133 insertions(+), 40 deletions(-) diff --git a/doc/source/deployment_guide.rst b/doc/source/deployment_guide.rst index 900cb8577b..59a6e8d76c 100644 --- a/doc/source/deployment_guide.rst +++ b/doc/source/deployment_guide.rst @@ -241,10 +241,10 @@ Option Default Description use paste.deploy entry point for the object server. For most cases, this should be `egg:swift#object`. -log_name object-server Label used when logging -log_facility LOG_LOCAL0 Syslog log facility -log_level INFO Logging level -log_requests True Whether or not to log each request +set log_name object-server Label used when logging +set log_facility LOG_LOCAL0 Syslog log facility +set log_level INFO Logging level +set log_requests True Whether or not to log each request user swift User to run as node_timeout 3 Request timeout to external services conn_timeout 0.5 Connection timeout to external services @@ -341,9 +341,9 @@ Option Default Description use paste.deploy entry point for the container server. For most cases, this should be `egg:swift#container`. -log_name container-server Label used when logging -log_facility LOG_LOCAL0 Syslog log facility -log_level INFO Logging level +set log_name container-server Label used when logging +set log_facility LOG_LOCAL0 Syslog log facility +set log_level INFO Logging level node_timeout 3 Request timeout to external services conn_timeout 0.5 Connection timeout to external services ================== ================ ======================================== @@ -428,9 +428,9 @@ Option Default Description use Entry point for paste.deploy for the account server. For most cases, this should be `egg:swift#account`. -log_name account-server Label used when logging -log_facility LOG_LOCAL0 Syslog log facility -log_level INFO Logging level +set log_name account-server Label used when logging +set log_facility LOG_LOCAL0 Syslog log facility +set log_level INFO Logging level ================== ============== ========================================== [account-replicator] @@ -509,10 +509,10 @@ use Entry point for paste.deploy for the proxy server. For most cases, this should be `egg:swift#proxy`. -log_name proxy-server Label used when logging -log_facility LOG_LOCAL0 Syslog log facility -log_level INFO Log level -log_headers True If True, log headers in each +set log_name proxy-server Label used when logging +set log_facility LOG_LOCAL0 Syslog log facility +set log_level INFO Log level +set log_headers True If True, log headers in each request recheck_account_existence 60 Cache timeout in seconds to send memcached for account @@ -570,10 +570,10 @@ use Entry point for auth. To use the swauth set to: `egg:swift#swauth` -log_name auth-server Label used when logging -log_facility LOG_LOCAL0 Syslog log facility -log_level INFO Log level -log_headers True If True, log headers in +set log_name auth-server Label used when logging +set log_facility LOG_LOCAL0 Syslog log facility +set log_level INFO Log level +set log_headers True If True, log headers in each request reseller_prefix AUTH The naming scope for the auth service. Swift diff --git a/etc/account-server.conf-sample b/etc/account-server.conf-sample index e48650f4a5..1fac948619 100644 --- a/etc/account-server.conf-sample +++ b/etc/account-server.conf-sample @@ -7,18 +7,27 @@ # swift_dir = /etc/swift # devices = /srv/node # mount_check = true +# You can specify default log routing here if you want: +# log_name = swift +# log_facility = LOG_LOCAL0 +# log_level = INFO [pipeline:main] pipeline = account-server [app:account-server] use = egg:swift#account -# log_name = account-server -# log_facility = LOG_LOCAL0 -# log_level = INFO +# You can override the default log routing for this app here: +# set log_name = account-server +# set log_facility = LOG_LOCAL0 +# set log_level = INFO +# set log_requests = True [account-replicator] +# You can override the default log routing for this app here (don't use set!): # log_name = account-replicator +# log_facility = LOG_LOCAL0 +# log_level = INFO # vm_test_mode = no # log_facility = LOG_LOCAL0 # log_level = INFO @@ -36,7 +45,10 @@ use = egg:swift#account # reclaim_age = 86400 [account-stats] +# You can override the default log routing for this app here (don't use set!): # log_name = account-stats +# log_facility = LOG_LOCAL0 +# log_level = INFO # cf_account = AUTH_7abbc116-8a07-4b63-819d-02715d3e0f31 # container_name = account_stats # proxy_server_conf = /etc/swift/proxy-server.conf @@ -44,14 +56,20 @@ use = egg:swift#account # log_level = INFO [account-auditor] +# You can override the default log routing for this app here (don't use set!): # log_name = account-auditor +# log_facility = LOG_LOCAL0 +# log_level = INFO # Will audit, at most, 1 account per device per interval # interval = 1800 # log_facility = LOG_LOCAL0 # log_level = INFO [account-reaper] +# You can override the default log routing for this app here (don't use set!): # log_name = account-reaper +# log_facility = LOG_LOCAL0 +# log_level = INFO # concurrency = 25 # interval = 3600 # node_timeout = 10 diff --git a/etc/auth-server.conf-sample b/etc/auth-server.conf-sample index 27b6cf3e14..711f48d564 100644 --- a/etc/auth-server.conf-sample +++ b/etc/auth-server.conf-sample @@ -7,6 +7,10 @@ # swift_dir = /etc/swift # cert_file = Default is no cert; format is path like /etc/swift/auth.crt # key_file = Default is no key; format is path like /etc/swift/auth.key +# You can specify default log routing here if you want: +# log_name = swift +# log_facility = LOG_LOCAL0 +# log_level = INFO [pipeline:main] pipeline = auth-server @@ -15,11 +19,12 @@ pipeline = auth-server use = egg:swift#auth # Highly recommended to change this. super_admin_key = devauth -# log_name = auth-server -# log_facility = LOG_LOCAL0 -# log_level = INFO +# You can override the default log routing for this app here: +# set log_name = proxy-server +# set log_facility = LOG_LOCAL0 +# set log_level = INFO +# set log_headers = False # reseller_prefix = AUTH # default_cluster_url = http://127.0.0.1:8080/v1 # token_life = 86400 -# log_headers = False # node_timeout = 10 diff --git a/etc/container-server.conf-sample b/etc/container-server.conf-sample index fb250708fe..183f20c6a0 100644 --- a/etc/container-server.conf-sample +++ b/etc/container-server.conf-sample @@ -7,20 +7,29 @@ # swift_dir = /etc/swift # devices = /srv/node # mount_check = true +# You can specify default log routing here if you want: +# log_name = swift +# log_facility = LOG_LOCAL0 +# log_level = INFO [pipeline:main] pipeline = container-server [app:container-server] use = egg:swift#container -# log_name = container-server -# log_facility = LOG_LOCAL0 -# log_level = INFO +# You can override the default log routing for this app here: +# set log_name = container-server +# set log_facility = LOG_LOCAL0 +# set log_level = INFO +# set log_requests = True # node_timeout = 3 # conn_timeout = 0.5 [container-replicator] +# You can override the default log routing for this app here (don't use set!): # log_name = container-replicator +# log_facility = LOG_LOCAL0 +# log_level = INFO # vm_test_mode = no # per_diff = 1000 # concurrency = 8 @@ -31,7 +40,10 @@ use = egg:swift#container # reclaim_age = 604800 [container-updater] +# You can override the default log routing for this app here (don't use set!): # log_name = container-updater +# log_facility = LOG_LOCAL0 +# log_level = INFO # interval = 300 # concurrency = 4 # node_timeout = 3 @@ -40,6 +52,9 @@ use = egg:swift#container # slowdown = 0.01 [container-auditor] +# You can override the default log routing for this app here (don't use set!): # log_name = container-auditor +# log_facility = LOG_LOCAL0 +# log_level = INFO # Will audit, at most, 1 container per device per interval # interval = 1800 diff --git a/etc/object-server.conf-sample b/etc/object-server.conf-sample index cc80c18c07..1fadffb941 100644 --- a/etc/object-server.conf-sample +++ b/etc/object-server.conf-sample @@ -7,16 +7,21 @@ # swift_dir = /etc/swift # devices = /srv/node # mount_check = true +# You can specify default log routing here if you want: +# log_name = swift +# log_facility = LOG_LOCAL0 +# log_level = INFO [pipeline:main] pipeline = object-server [app:object-server] use = egg:swift#object -# log_name = object-server -# log_facility = LOG_LOCAL0 -# log_level = INFO -# log_requests = True +# You can override the default log routing for this app here: +# set log_name = object-server +# set log_facility = LOG_LOCAL0 +# set log_level = INFO +# set log_requests = True # node_timeout = 3 # conn_timeout = 0.5 # network_chunk_size = 65536 @@ -27,7 +32,10 @@ use = egg:swift#object # mb_per_sync = 512 [object-replicator] +# You can override the default log routing for this app here (don't use set!): # log_name = object-replicator +# log_facility = LOG_LOCAL0 +# log_level = INFO # vm_test_mode = no # daemonize = on # run_pause = 30 @@ -45,7 +53,10 @@ use = egg:swift#object # reclaim_age = 604800 [object-updater] +# You can override the default log routing for this app here (don't use set!): # log_name = object-updater +# log_facility = LOG_LOCAL0 +# log_level = INFO # interval = 300 # concurrency = 1 # node_timeout = 10 @@ -54,6 +65,9 @@ use = egg:swift#object # slowdown = 0.01 [object-auditor] +# You can override the default log routing for this app here (don't use set!): # log_name = object-auditor +# log_facility = LOG_LOCAL0 +# log_level = INFO # files_per_second = 20 # bytes_per_second = 10000000 diff --git a/etc/proxy-server.conf-sample b/etc/proxy-server.conf-sample index a3f64f8415..cca2f8c620 100644 --- a/etc/proxy-server.conf-sample +++ b/etc/proxy-server.conf-sample @@ -7,6 +7,10 @@ # user = swift # cert_file = /etc/swift/proxy.crt # key_file = /etc/swift/proxy.key +# You can specify default log routing here if you want: +# log_name = swift +# log_facility = LOG_LOCAL0 +# log_level = INFO [pipeline:main] # For DevAuth: @@ -16,10 +20,11 @@ pipeline = catch_errors healthcheck cache ratelimit auth proxy-server [app:proxy-server] use = egg:swift#proxy -# log_name = proxy-server -# log_facility = LOG_LOCAL0 -# log_level = INFO -# log_headers = False +# You can override the default log routing for this app here: +# set log_name = proxy-server +# set log_facility = LOG_LOCAL0 +# set log_level = INFO +# set log_headers = False # recheck_account_existence = 60 # recheck_container_existence = 60 # object_chunk_size = 8192 @@ -39,6 +44,11 @@ use = egg:swift#proxy # Only needed for DevAuth [filter:auth] use = egg:swift#auth +# You can override the default log routing for this filter here: +# set log_name = auth-server +# set log_facility = LOG_LOCAL0 +# set log_level = INFO +# set log_headers = False # The reseller prefix will verify a token begins with this prefix before even # attempting to validate it with the external authentication server. Also, with # authorization, only Swift storage accounts with this prefix will be @@ -54,10 +64,11 @@ use = egg:swift#auth # Only needed for Swauth [filter:swauth] use = egg:swift#swauth -# log_name = auth-server -# log_facility = LOG_LOCAL0 -# log_level = INFO -# log_headers = False +# You can override the default log routing for this filter here: +# set log_name = auth-server +# set log_facility = LOG_LOCAL0 +# set log_level = INFO +# set log_headers = False # The reseller prefix will verify a token begins with this prefix before even # attempting to validate it. Also, with authorization, only Swift storage # accounts with this prefix will be authorized by this middleware. Useful if @@ -82,15 +93,30 @@ super_admin_key = swauthkey [filter:healthcheck] use = egg:swift#healthcheck +# You can override the default log routing for this filter here: +# set log_name = auth-server +# set log_facility = LOG_LOCAL0 +# set log_level = INFO +# set log_headers = False [filter:cache] use = egg:swift#memcache +# You can override the default log routing for this filter here: +# set log_name = auth-server +# set log_facility = LOG_LOCAL0 +# set log_level = INFO +# set log_headers = False # Default for memcache_servers is below, but you can specify multiple servers # with the format: 10.1.2.3:11211,10.1.2.4:11211 # memcache_servers = 127.0.0.1:11211 [filter:ratelimit] use = egg:swift#ratelimit +# You can override the default log routing for this filter here: +# set log_name = auth-server +# set log_facility = LOG_LOCAL0 +# set log_level = INFO +# set log_headers = False # clock_accuracy should represent how accurate the proxy servers' system clocks # are with each other. 1000 means that all the proxies' clock are accurate to # each other within 1 millisecond. No ratelimit should be higher than the @@ -116,14 +142,29 @@ use = egg:swift#ratelimit [filter:domain_remap] use = egg:swift#domain_remap +# You can override the default log routing for this filter here: +# set log_name = auth-server +# set log_facility = LOG_LOCAL0 +# set log_level = INFO +# set log_headers = False # storage_domain = example.com # path_root = v1 [filter:catch_errors] use = egg:swift#catch_errors +# You can override the default log routing for this filter here: +# set log_name = auth-server +# set log_facility = LOG_LOCAL0 +# set log_level = INFO +# set log_headers = False [filter:cname_lookup] # Note: this middleware requires python-dnspython use = egg:swift#cname_lookup +# You can override the default log routing for this filter here: +# set log_name = auth-server +# set log_facility = LOG_LOCAL0 +# set log_level = INFO +# set log_headers = False # storage_domain = example.com # lookup_depth = 1