It's not sufficient to just look at swift.object-updater.successes to
see the async_pending unlink rate. There are two different spots where
unlinks happen: one when an async_pending has been successfully
processed, and another when the updater notices multiple
async_pendings for the same object. Both events are now tracked under
the same name: swift.object-updater.unlinks.
FakeLogger has now sprouted a couple of convenience methods for
testing logged metrics.
Fixed pep8 1.3.3's complaints in the files this diff touches.
Also: bonus speling and, grammar fixes in the admin guide.
Change-Id: I8c1493784adbe24ba2b5512615e87669b3d94505
This patch introduces a new user format to tempauth which starts with
"user64_" and in which the account and user are base64 encoded (sans
padding equal signs):
user64_<account_b64>_<user_b64> = <key> [group] [group] [...] [storage_url]
This allows unambiguous parsing and adds the ability to have a user or
account name containing underscores.
The handling of tempauth configuration is fully backward compatible and
this only adds new, optional functionality.
Also beefed up unit tests to get full coverage of the user parsing code
in tempauth's __init__().
Change-Id: Id06304934ab8055a28921f2e1aa3c58a2036d8f8
This change replaces WebOb with a mostly compatible local library,
swift.common.swob. Subtle changes to WebOb's API over the years have been a
huge headache. Swift doesn't even run on the current version.
There are a few incompatibilities to simplify the implementation/interface:
* It only implements the header properties we use. More can be easily added.
* Casts header values to str on assignment.
* Response classes ("HTTPNotFound") are no longer subclasses, but partials
on Response, so things like isinstance no longer work on them.
* Unlike newer webob versions, will never return unicode objects.
Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f
This moves search_devs into RingBuilder to make it accessible to other utils
that need to search the builder. Along the same lines this also adds a
load() call to get a RingBuilder instance when working with the builder files.
- This adds python-mock >= 0.7 as a dependency for unittests. On Ubuntu
10.04 you'll have to pip install it, on 12.04 you can apt-get install
it. Fedora 17+ should be able to yum install it.
- new pep8 compliance
- Fixed a small issue (undefined var) in swift-ring-builder when remove was
called but failed to find a match.
Change-Id: I2e02684235aa2f4e901a00858ae037091594c545
Removed many StatsD logging calls in proxy-server and added
swift-informant-style catch-all logging in the proxy-logger middleware.
Many errors previously rolled into the "proxy-server.<type>.errors"
counter will now appear broken down by response code and with timing
data at: "proxy-server.<type>.<verb>.<status>.timing". Also, bytes
transferred (sum of in + out) will be at:
"proxy-server.<type>.<verb>.<status>.xfer". The proxy-logging
middleware can get its StatsD config from standard vars in [DEFAULT] or
from access_log_statsd_* config vars in its config section.
Similarly to Swift Informant, request methods ("verbs") are filtered
using the new proxy-logging config var, "log_statsd_valid_http_methods"
which defaults to GET, HEAD, POST, PUT, DELETE, and COPY. Requests with
methods not in this list use "BAD_METHOD" for <verb> in the metric name.
To avoid user error, access_log_statsd_valid_http_methods is also
accepted.
Previously, proxy-server metrics used "Account", "Container", and
"Object" for the <type>, but these are now all lowercase.
Updated the admin guide's StatsD docs to reflect the above changes and
also include the "proxy-server.<type>.handoff_count" and
"proxy-server.<type>.handoff_all_count" metrics.
The proxy server now saves off the original req.method and proxy_logging
will use this if it can (both for request logging and as the "<verb>" in
the statsd timing metric). This fixes bug 1025433.
Removed some stale access_log_* related code in proxy/server.py. Also
removed the BaseApplication/Application distinction as it's no longer
necessary.
Fixed up the sample config files a bit (logging lines, mostly).
Fixed typo in SAIO development guide.
Got proxy_logging.py test coverage to 100%.
Fixed proxy_logging.py for PEP8 v1.3.2.
Enhanced test.unit.FakeLogger to track more calls to enable testing
StatsD metric calls.
Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576
To tell when replication for a device has finished, it's important to
know when the replicator is removing objects. This was previously
handled for the object-replicator
(object-replicator.partition.delete.count.<device> and
object-replicator.partition.update.count.<device> metrics) but not the
account and container replicators.
This patch extends the existing DB removal count metrics to make them
per-device. The new metrics are:
account-replicator.removes.<device>
container-replicator.removes.<device>
There's also a bonus refactoring and increased test coverage of the DB
replicator code.
Change-Id: I2067317d4a5f8ad2a496834147954bdcdfc541c1
Added "-p" to mkdir cmds in a "set -e" script (/etc/rc.local).
/var/log/swift gets owned by the group "adm", but isn't group writeable
by default, so the find/rm in "resetswift" fails. So I added a chmod to
add group-write for that directory tree.
Added commands for installing tox.
Change-Id: Ic141491272a83a9233cd6a6a70c3ef74411f3f51
The Admin Guide now contains information about the ring serialization
change (and importantly, how to downgrade, if necessary).
Also added container-server conf var, "allow_versions" to the Deployment
Guide.
Also changed description of proxy-server conf var,
"max_containers_whitelist" to say it contains "account names" not
"account hashes".
Change-Id: Ib23c6118cc5195cc04765afd28e442e4c735f0d4
bug fixes for swift 1033089
Swift 1.6.0 All in One setup uses rc.local to create a set of
directories like the following:
/var/cache/swift /var/cache/swift2 /var/cache/swift3
/var/cache/swift4
/var/run/swift
without -p for mkdir command, when some of these directories
already exist, the rc.local fails so that the next set of
command won't execute. This causes swift fail to start because
of /var/run/swift does not exist. Adding -p to both mkdir
command solves this problem.
Change-Id: Ia26c9a857723747008293c0631c9acc1b58eca7e
A warning log line is emitted whenever the proxy has to use a handoff
node. Monitoring these warnings can indicate a problem within your
cluster; however, you can disable these log lines by setting the
proxy conf's log_handoffs to false.
While working on this, I also noticed why many proxy log lines did
not have txn_id and client_ip -- subcoroutines. Now the logger thread
locals are copied to the subcoroutines.
Change-Id: Ibac086e1b985f566c068d083620287509de35da8
We're still using saio:11000 in a few spots so a few things
don't work out of the box on the saio. Fixes bug #1024561
Change-Id: I226de54c2785b0d0b681c8d0cc24260adbd3d663
The object server had a constant KEEP_CACHE_SIZE = 5*1024*1024;
unauthenticated GET requests for files smaller than KEEP_CACHE_SIZE
would not evict the file from the kernel's buffer cache after it was
read from disk.
Now that hardcoded constant is a configuration parameter
("keep_cache_size"), and now there is also another parameter called
"keep_cache_private". If set, then both authenticated and
unauthenticated GET requests for small files will not evict the data
from the buffer cache. The default values are 5 MiB and False,
respectively, so the default behavior is the same.
Bonus: the "mb_per_sync" parameter is now documented in the deployment
guide.
Change-Id: I9a11dbe861f4c23535c6aa82a9111a6fe2db2a59
- It has been to its own gerrit project.
- direct_client should follow next.
- Implements blueprint clientbindings.
Change-Id: I3bb50c95eba81302bfec71cb7ce5288b85a41dc0
Expand recon middleware to include support for account and container
servers in addition to the existing object servers. Also add support
for retrieving recent information from auditors, replicators, and
updaters. In the case of certain checks (such as container auditors)
the stats returned are only for the most recent path processed.
The middleware has also been refactored and should now also handle
errors better in cases where stats are unavailable.
While new check's have been added the output from pre-existing
check's has not changed. This should allow existing 3rd party
utilities such as the Swift ZenPack to continue to function.
Change-Id: Ib9893a77b9b8a2f03179f2a73639bc4a6e264df7
Long explanation, but hopefully answers any questions.
We don't like changing the default behavior of Swift unless there's a
really good reason and, up until now, I've tried doing this with this
new db_preallocation setting.
For clusters with dedicated account/container servers that usually
have fewer disks overall but SSD for speed, having db_preallocation
on will gobble up disk space quite quickly and the fragmentation it's
designed to fight isn't that big a speed impact to SSDs anyway.
For clusters with account/container servers spread across all servers
along with object servers usually having standard disks for cost,
having db_preallocation off will cause very fragmented database files
impacting speed, sometimes dramatically.
Weighing these two negatives, it seems the second is the lesser evil.
The first can cause disks to fill up and disable the cluster. The
second will cause performance degradation, but the cluster will still
function.
Furthermore, if just one piece of code that touches all databases
runs with db_preallocation on, it's effectively on for the whole
cluster. We discovered this most recently when we finally configured
everything within the Swift codebase to have db_preallocation off,
only to find out Slogging didn't know about the new setting and so
ran with it on and starting filling up SSDs.
So that's why I'm proposing this change to the default behavior.
We will definitely need to post a prominent notice of this change
with the next release.
Change-Id: I48a43439264cff5d03c14ec8787f718ee44e78ea
Documentation, including a list of metrics reported and their semantics,
is in the Admin Guide in a new section, "Reporting Metrics to StatsD".
An optional "metric prefix" may be configured which will be prepended to
every metric name sent to StatsD.
Here is the rationale for doing a deep integration like this versus only
sending metrics to StatsD in middleware. It's the only way to report
some internal activities of Swift in a real-time manner. So to have one
way of reporting to StatsD and one place/style of configuration, even
some things (like, say, timing of PUT requests into the proxy-server)
which could be logged via middleware are consistently logged the same
way (deep integration via the logger delegate methods).
When log_statsd_host is configured, get_logger() injects a
swift.common.utils.StatsdClient object into the logger as
logger.statsd_client. Then a set of delegate methods on LogAdapter
either pass through to the StatsdClient object or become no-ops. This
allows StatsD logging to look like:
self.logger.increment('some.metric.here')
and do the right thing in all cases and with no messy conditional logic.
I wanted to use the pystatsd module for the StatsD client, but the
version on PyPi is lagging the git repo (and is missing both the prefix
functionality and timing_since() method). So I wrote my
swift.common.utils.StatsdClient. The interface is the same as
pystatsd.Client, but the code was written from scratch. It's pretty
simple, and the tests I added cover it. This also frees Swift from an
optional dependency on the pystatsd module, making this feature easier
to enable.
There's test coverage for the new code and all existing tests continue
to pass.
Refactored out _one_audit_pass() method in swift/account/auditor.py and
swift/container/auditor.py.
Fixed some misc. PEP8 violations.
Misc test cleanups and refactorings (particularly the way "fake logging"
is handled).
Change-Id: Ie968a9ae8771f59ee7591e2ae11999c44bfe33b2
This commit introduces a new algorithm for assigning partition
replicas to devices. Basically, the ring builder organizes the devices
into tiers (first zone, then IP/port, then device ID). When placing a
replica, the ring builder looks for the emptiest device (biggest
parts_wanted) in the furthest-away tier.
In the case where zone-count >= replica-count, the new algorithm will
give the same results as the one it replaces. Thus, no migration is
needed.
In the case where zone-count < replica-count, the new algorithm
behaves differently from the old algorithm. The new algorithm will
distribute things evenly at each tier so that the replication is as
high-quality as possible, given the circumstances. The old algorithm
would just crash, so again, no migration is needed.
Handoffs have also been updated to use the new algorithm. When
generating handoff nodes, first the ring looks for nodes in other
zones, then other ips/ports, then any other drive. The first handoff
nodes (the ones in other zones) will be the same as before; this
commit just extends the list of handoff nodes.
The proxy server and replicators have been altered to avoid looking at
the ring's replica count directly. Previously, with a replica count of
C, RingData.get_nodes() and RingData.get_part_nodes() would return
lists of length C, so some other code used the replica count when it
needed the number of nodes. If two of a partition's replicas are on
the same device (e.g. with 3 replicas, 2 devices), then that
assumption is no longer true. Fortunately, all the proxy server and
replicators really needed was the number of nodes returned, which they
already had. (Bonus: now the only code that mentions replica_count
directly is in the ring and the ring builder.)
Change-Id: Iba2929edfc6ece89791890d0635d4763d821a3aa
The code has moved to https://github.com/notmyname/swift-cnamelookup.
For current users of cname lookup, this will require installing the new
package and changing the "use" line of the cname lookup conf section's
to:
[filter:cname_lookup]
use = egg:swift_cnamelookup#swift_cnamelookup
And then 'swift-init proxy reload'.
Change-Id: If622486ddb04a53251244c9840aa3cfe72168fc5
Rate Limit middleware is now at
http://dpgoetz.github.com/swift-ratelimit/
For current users of Rate Limit, this will require installing the new
package and changing the "use" line of the ratelimit conf section to:
[filter:ratelimit]
use = egg:swiftratelimit#middleware
And then 'swift-init proxy reload'.
Change-Id: I2ab774e9cee9fba4103c1be4bea6d52d1adb29f7
The code has moved to https://github.com/notmyname/swift-domainremap.
For current users of domain remap, this will require installing the new
package and changing the "use" line of the domain remap conf section's
to:
[filter:domain_remap]
use = egg:swift_domainremap#swift_domainremap
And then 'swift-init proxy reload'.
Change-Id: I710caf9b991f9d37df36b826ae4338086d0ec36d
