Commit Graph

21 Commits

Author SHA1 Message Date
Alistair Coles
6e394bba0a Add request_tries option to object-expirer.conf-sample
...and update the object-expirer man page.

Change-Id: Idca1b8e3b7d5b40481af0d60477510e2557b88c0
2018-01-15 15:29:11 +00:00
Peter Lisák
8bf2233b40 Documantation enhancements of nice/ionice feature
Based on comments from patch #238799.

Change-Id: I9455cf6dc7fd12fee62439ff3c5f3255287ab1be
2016-08-19 07:39:49 +02:00
Peter Lisák
ed772236c7 Change schedule priority of daemon/server in config
The goal is to modify schedule priority and I/O scheduling class and
priority of daemon/server via configuration.
Setting is optional, default keeps current behaviour.

Use case:
Prioritize object-server to object-auditor, because all user's requests
needed to be served in peak hours and audit could wait.

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
DocImpact
Change-Id: I1018a18f4706daabdb84574ffd9a58d831e68396
2016-08-10 23:56:15 +02:00
gh159m
b5311f63db Removed default value for log_statsd_host
Multiple files and documents showed that log_statsd_host had
a default value, usually localhost.  This was incorrect, instead
setting a value for log_statsd_host enables statsd logging.

Removed any reference of log_statsd_host having a default value.

Also changed descriptions to show setting a value enables logging.

Change-Id: I3ca5c0e8b8e4981de3aa6db0c476072b5a59723d
Closes-Bug: #1542227
2016-02-10 10:36:59 -06:00
Zhao Lei
4ac1fea5d1 Fix some spelling typo in comments
s/overide/override for object-expirer.conf and sample.
s/automaticaly/automatically for swift/proxy/controllers/obj.py

Change-Id: Ife107c7a1005a5d4959288db50a7f8f33c522dd4
Signed-off-by: Zhao Lei <zhaolei@cn.fujitsu.com>
2015-08-07 22:21:37 +08:00
Jenkins
b681e74161 Merge "Swift configuration parameter audit" 2014-08-06 16:18:02 +00:00
Matthew Oliver
090baa1fa9 Swift configuration parameter audit
This change is the result of an audit through the config parameters
provided by swift and how/if they are addressed in the swift
documentation. The documentation being the sample config files in
the /etc directory or the documentation.

This change is only concerned with the config files in etc/ next
I will look at the documentation in the doc/ folder.

This change makes the following assumptions:
  - Unless stated otherwise, the commented out parameter in the
    sample configuration is the default for swift.

  - When the default in the code differs from that of the sample
    configuration, the default in the code is correct.

Container reconciler:
  Parameter: interval
    - code: 30
    - config: 300
  Result: config = 30

Object Expirer:
  Parameter: recon_cache_path
    - code: /var/cache/swift
    - config: Parameter missing
  Result: Add parameter

swift-dispersion-populate && swift-dispersion-report
  Parameter: auth_version
    - code: 1.0
    - config: 2.0 (due to being a confusing example of how to setup
                   version 2.0).
  Result: Added 'auth_version = 1.0' to the right section (showing
          default and make the sample configuration for auth version
          2.0 easier to understand.

swift-drive-audit:
  Parameter: log_file_pattern
    - code: /var/log/kern.*[!.][!g][!z]
    - config: /var/log/kern*
  Result: config = /var/log/kern.*[!.][!g][!z]

  NOTE: swift-drive-audit uses a parameter called device_dir which
        defaults to '/srv/node'. In all other swift binaries/services
        there is a similar parameter called devices which stores the
        same thing. This is an inconsistency which I haven't fixed
        as this could break existing swift clusters out in the wild.

Proxy Server:
  Parameter: object_chunk_size
    - code: 65536
    - config: 8192
  Result: config = 65536

  Parameter: client_chunk_size
    - code: 65536
    - config: 8192
  Result: config = 65536

  Parameter: strict_cors_mode
    - code: True
    - config: No parameter
  Result: config = True

Account and Container replicator configuration confusion:
  NOTES:
    The account and container replicators have parameters:
      - interval
      - run_pause

    Both of these are loaded into the same variable in code:
      self.interval = int(conf.get('interval') or
                          conf.get('run_pause') or 30)

    If a user sets both to different values then interval is used.
  Result: Update the configuration to make this more clear.

DocImpact
Change-Id: Iaadbb1a6284f8b3e0801bc343b29772f70f4bf6e
2014-08-06 11:12:14 +10:00
Samuel Merritt
5a84b1cde1 Put proxy-logging in object expirer's sample config
It's generally better to have logs for something than to not have
logs. This way, the object expirer (if using the sample config as a
starting point) will log what it does.

Note that the container reconciler's sample config already contains
proxy-logging, as does the proxy server's. The object expirer is the
odd man out.

Change-Id: I32aac99131746501820319b94405440c1934a694
2014-07-31 12:18:47 -07:00
Clay Gerrard
8d20e0e927 Fix object-expirer for missing objects
Currently if the object-expirer goes to delete an object and the primary nodes
are unavailable, or the object is on handoffs - the object servers are unable
to verify the x-if-delete-at timestamp and return 412, without writing a
tombstone or updating the containers.  The expirer treats 412 as success and
the dark data is not removed form the object servers nor the object removed in
the listing.

As a side effect of this bug, if the expirer encounters split brain the delete
would never get processed in the correct storage policy.

It seems it's just not correct to treat the lack of data as success.  Now the
object server will treat x-if-delete at against a non-existent object as a
404, and to distinguish from a successfull process of an x-if-delete-at
request, will return 204.

The expirer will treat a 404 response from swift as a failure, and will
continue to attempt to expire the object until it is older that it's
configurable reclaim age.  However swift will only return 404 if the majority
of nodes are able to return success, or if only even a single node is able to
accept the x-if-delete-at request the containers will get updated and
replicaiton will settle the tombstone - the subsequent x-if-delete-at request
will 412 and be removed from the queue.

It's worth noting that if an object with x-delete-at meta is DELETED (by a
client request) an async update for the expiring update containers will be
processed to remove the queue entry - but if no primary nodes handle the
DELETE request replication will never remove the expiring entry and assuming
it's scheduled for beyond the tombstones reclaim age - the queue entry will
not be processable.  In this case the expirer will attempt to DELETE the
object (and get 404s) in vain until the queue entry passes the configurable
reclaim age.

DocImpact
Implements: blueprint storage-policies
Change-Id: I66260e99fda37e97d6d2470971b6f811ee9e01be
2014-06-18 21:09:54 -07:00
gholt
2d00f7b7ba New log_max_line_length option.
Log lines can get quite large, as we previously noticed with rsync error
log lines. We added a setting to cap those, but it really looks like we
should have just done this overall limit. We noticed the issue when we
switched to UDP syslogging and it would occasionally blow past the 16436
lo MTU! This causes Python's logging code to get an error and hilarity
ensues.

Change-Id: I44bdbe68babd58da58c14360379e8fef8a6b75f7
2014-05-22 20:30:34 +00:00
Prashanth Pai
aad81528d4 Make .expiring_objects account name configurable
The account which tracks objects scheduled for deletion had its name
hard-coded to 'expiring_objects'. This is made configurable via
expiring_objects_account_name option. Backend file-systems
integration efforts may want to treat these "special" accounts in a
different way.

This would still go undocumented, hence 'pseudo-hidden'.

UpgradeImpact: None as the default value would continue to be the same
               which is '.expiring_objects'.

Change-Id: I1a093b0d0e2bdd0c3d723090af03fc0adf2ad7e3
Signed-off-by: Prashanth Pai <ppai@redhat.com>
2014-02-06 16:38:34 +05:30
Jenkins
5bfd2d798d Merge "Add parallelism to object expirer daemon." 2013-06-11 22:48:24 +00:00
Greg Lange
209c5ec418 Add parallelism to object expirer daemon.
Two types of parallelism are added:

- concurrency to speed up what a single process does
- a way to run multiple daemons to work on different parts of the work

DocImpact

Change-Id: I48997f68eb2fd8de19a5ee8b9fcdf76dde2ba0ab
2013-06-07 20:49:47 +00:00
Samuel Merritt
efdb0e3681 Make sample configs more readable.
Inject some empty lines to avoid the wall-of-text effect and to make
it a little clearer which descriptions go with which options.

Change-Id: I58914b83dad76ea5ca330903a246bee7ffaeba83
2013-06-06 15:35:19 -07:00
Darrell Bishop
8801b74090 Make statsd sample rate behave better.
As Dieter pointed out in bug 1090495
(https://bugs.launchpad.net/swift/+bug/1090495), the volume of metrics
can vary wildly between StatsD metrics.

This patch implements a partial solution by reducing the sample_rate
used for known high-volume metrics (operational experience will need to
inform this over time) and introducing a new tunable,
log_statsd_sample_rate_factor which is multiplied by the sample_rate for
every statsd stat.  This tunable can be used to reduce StatsD traffic
proportionally for all metrics and is intended to replace
log_statsd_default_sample_rate, which is left alone for
backward-compatibility, should anyone be using it.

This patch also includes a drive-by fix for log_udp_port which wasn't
being converted to an int (I didn't verify that actually causes trouble
in SysLogHandler(), but it's definitely an improvement regardles).

Change-Id: Id404636e3629f6431cf1c4e64a143959750a3c23
2013-01-19 15:25:27 -08:00
Florian Hines
92826d0602 add support for custom log handlers
Add a hook to get_logger to run custom functions to add custom log
handlers or the like.

Change-Id: Ib04b12939dcac7e4ad6453dea9795682044c6ae0
2012-10-05 15:56:34 -05:00
Darrell Bishop
4a2ae2b460 Upating proxy-server StatsD logging.
Removed many StatsD logging calls in proxy-server and added
swift-informant-style catch-all logging in the proxy-logger middleware.
Many errors previously rolled into the "proxy-server.<type>.errors"
counter will now appear broken down by response code and with timing
data at: "proxy-server.<type>.<verb>.<status>.timing".  Also, bytes
transferred (sum of in + out) will be at:
"proxy-server.<type>.<verb>.<status>.xfer".  The proxy-logging
middleware can get its StatsD config from standard vars in [DEFAULT] or
from access_log_statsd_* config vars in its config section.

Similarly to Swift Informant, request methods ("verbs") are filtered
using the new proxy-logging config var, "log_statsd_valid_http_methods"
which defaults to GET, HEAD, POST, PUT, DELETE, and COPY.  Requests with
methods not in this list use "BAD_METHOD" for <verb> in the metric name.
To avoid user error, access_log_statsd_valid_http_methods is also
accepted.

Previously, proxy-server metrics used "Account", "Container", and
"Object" for the <type>, but these are now all lowercase.

Updated the admin guide's StatsD docs to reflect the above changes and
also include the "proxy-server.<type>.handoff_count" and
"proxy-server.<type>.handoff_all_count" metrics.

The proxy server now saves off the original req.method and proxy_logging
will use this if it can (both for request logging and as the "<verb>" in
the statsd timing metric).  This fixes bug 1025433.

Removed some stale access_log_* related code in proxy/server.py.  Also
removed the BaseApplication/Application distinction as it's no longer
necessary.

Fixed up the sample config files a bit (logging lines, mostly).

Fixed typo in SAIO development guide.

Got proxy_logging.py test coverage to 100%.

Fixed proxy_logging.py for PEP8 v1.3.2.

Enhanced test.unit.FakeLogger to track more calls to enable testing
StatsD metric calls.

Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576
2012-08-29 16:08:30 -07:00
Victor Rodionov
13e4de1899 Patch for Swift Solaris (Illumos) compability.
* Add new configuration option log_address.

Change-Id: I636bd4116687629c997b70a0d804b7ed4bc46032
2012-06-19 15:38:56 +04:00
Darrell Bishop
3d3ed34f44 Adding StatsD logging to Swift.
Documentation, including a list of metrics reported and their semantics,
is in the Admin Guide in a new section, "Reporting Metrics to StatsD".
An optional "metric prefix" may be configured which will be prepended to
every metric name sent to StatsD.

Here is the rationale for doing a deep integration like this versus only
sending metrics to StatsD in middleware.  It's the only way to report
some internal activities of Swift in a real-time manner. So to have one
way of reporting to StatsD and one place/style of configuration, even
some things (like, say, timing of PUT requests into the proxy-server)
which could be logged via middleware are consistently logged the same
way (deep integration via the logger delegate methods).

When log_statsd_host is configured, get_logger() injects a
swift.common.utils.StatsdClient object into the logger as
logger.statsd_client.  Then a set of delegate methods on LogAdapter
either pass through to the StatsdClient object or become no-ops. This
allows StatsD logging to look like:
    self.logger.increment('some.metric.here')
and do the right thing in all cases and with no messy conditional logic.

I wanted to use the pystatsd module for the StatsD client, but the
version on PyPi is lagging the git repo (and is missing both the prefix
functionality and timing_since() method).  So I wrote my
swift.common.utils.StatsdClient.  The interface is the same as
pystatsd.Client, but the code was written from scratch.  It's pretty
simple, and the tests I added cover it.  This also frees Swift from an
optional dependency on the pystatsd module, making this feature easier
to enable.

There's test coverage for the new code and all existing tests continue
to pass.

Refactored out _one_audit_pass() method in swift/account/auditor.py and
swift/container/auditor.py.

Fixed some misc. PEP8 violations.

Misc test cleanups and refactorings (particularly the way "fake logging"
is handled).

Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
2012-05-11 15:25:38 -07:00
Tom Fifield
9920aeb7d4 bug 661267 adding config eastereggs, fixing defaults
Change-Id: I41356ee250c9088a2387b0d493586dd990a04ac3
2012-04-30 08:22:42 +10:00
gholt
872420efdb Expiring Objects Support
Please see the doc/source/overview_expiring_objects.rst for
more detail.

Change-Id: I4ab49e731248cf62ce10001016e0c819cc531738
2011-11-01 15:49:00 +00:00