If invoked as 'swift-ring-builder-safe' the directory containing the builder
file provided will be locked (via lock_parent_directory()). This provides a
small safe guard against multiple instances of the swift-ring-builder (or
other utilities that observe this lock) from attempting to write to or read
the builder/ring files while operations are in progress.
This is particularly useful in environments where ring management has been
automated (via Chef or custom solutions) but the operator still occasionally
needs to manually interact with the ring.
DocImpact
Change-Id: Ia362744a8151a91bfb586d01da582906726852e6
Fix for bug 1095130
* Added a wrapper function around public methods to handle
CORS actual requests. These requests need to return some
extra headers to be valid responses to a CORS request.
Access-Control-Expose-Headers and Access-Control-Allow-Origin.
* Added support for the CORS header Access-Control-Expose-Headers.
* Some refactoring of the OPTIONS method so the
"is_origin_allowed" logic can be reused.
* Added a little extra detail to the CORS documentation.
DocImpact
Change-Id: I68538e472a900775427f21a8a59e738a83dcc8bc
Removed sidebar with broken (static) links referencing out-of-date docs.
Added an external link to the Swift API docs
fixes bug #1025099
Change-Id: I7f3106175b84b1063f74aa6c5693ab1e422cdb59
Changing that complex dd to a functionally equivalent truncate that's easier
to read and modify.
DocImpact
Change-Id: I64404318364608e62a1d80f6a0550271eb4cd03a
By default, this will be turned off. This will cause eventlet to not
print stack traces to stderr which can be very annoying on production
systems. It is still recommended to turn it on for development or
debuging purposes.
DocImpact
Change-Id: I5e5b902d3d9ed85f784549e53f2ee2fc87cbe2e5
- Add two optional flags that let you limit swift-dispersion-report to only
reporting on containers OR objects.
- Also make dispersion.conf and swift-dispersion-report manpages
current.
DocImpact
Change-Id: Iad56133cad261241db27d0e2103098e3c2f3c245
Currently the timeout for a wsgi server successfully binding to a port
and for a probetest background service to finish starting are hard coded
to 30 seconds. While a reasonable default for most configurations, a
small virtualized environment may need a little more time in order for
probe tests to complete successfully.
This patch adds a 'bind_timeout' option to the DEFAULT section of the
main wsgi servers' config. Also a new [probe_test] section and
'check_server_timeout' option to test.conf
DocImpact
Change-Id: Ibcaff153c7633bbf32e460fd9dbf04932eddb56f
Updates the proxy-server.conf-sample and docs to use
the new Keystoneclient middleware class name.
Change-Id: I3727f7b7328a2513347b8ef257c270126df36d7b
Two improvements: first, document that the container-sync process
connects to the remote cluster's proxy server, so outbound
connectivity is required.
Second, rewrite the behind-the-scenes container-sync example and add
some ASCII-art diagrams.
Fixes bug 1068430.
Bonus fix of docstring in wsgi.py to squelch a sphinx warning.
Change-Id: I85bd56c2bd14431e13f7c57a43852777f14014fb
The determination of the client IP looked at the X-Cluster-Client-Ip
and X-Forwarded-For headers in the incoming HTTP request. This is
trivially spoofable by a malicious client, so there's no security
gained by having the check there.
Worse, having the check there provides a false sense of security to
cluster operators. It sounds like it's based on the client IP, so an
attacker would have to do IP spoofing to defeat it. However, it's
really just a shared secret, and there's already a secret key set
up. Basically, it looks like 2-factor auth (IP+key), but it's really
1-factor (key).
Now, the one case where this might provide some security is where the
Swift cluster is behind an external load balancer that strips off the
X-Cluster-Client-Ip and X-Forwarded-For headers and substitutes its
own. I don't think it's worth the tradeoff, hence this commit.
Fixes bug 1068420 for very small values of "fixes".
DocImpact
Change-Id: I2bef64c2e1e4df8a612a5531a35721202deb6964
There are 3 sections in there, all useless.
Section 1 tells you how to install Swift packages from the swift-core
PPA. However, the latest version there is ancient.
Section 2 tells you how to build your own Swift packages. However, it
talks about getting the source code from the "debian" branch in bzr,
which is obviously really old.
Section 3 tells you how to take the packages from section 2 and
install them. This isn't too out-of-date, but since section 2 doesn't
work any more, section 3 is useless.
Since stale docs are worse than no docs, there's no current
information in this document, and bringing it up-to-date requires a
whole pile of work, I've chosen to delete it entirely.
Also pulled out a couple references to the PPA elsewhere.
Fixes bug 917385.
Fixes bug 1026145.
Change-Id: I510bd8619531fe110419e5488bd20d3602868d66
I know it's just TempAuth, but bug #959953 just caught my eye as
something interesting to solve.
This does a best guess on the storage URL to return for a given
request. It allows $HOST to be used in the storage URL configuration,
where $HOST will resolve to scheme://host:port. It bases the scheme
on how the server is running or on storage_url_scheme if set. The
host:port comes from the request's Host header if it exists, and
falls back to the WSGI SERVER_NAME:SERVER_PORT otherwise.
Fixes: bug #959953
DocImpact
Change-Id: Ia494bcb99a04490911ee8d2cb8b12a94e77820c5
It's not sufficient to just look at swift.object-updater.successes to
see the async_pending unlink rate. There are two different spots where
unlinks happen: one when an async_pending has been successfully
processed, and another when the updater notices multiple
async_pendings for the same object. Both events are now tracked under
the same name: swift.object-updater.unlinks.
FakeLogger has now sprouted a couple of convenience methods for
testing logged metrics.
Fixed pep8 1.3.3's complaints in the files this diff touches.
Also: bonus speling and, grammar fixes in the admin guide.
Change-Id: I8c1493784adbe24ba2b5512615e87669b3d94505
This patch introduces a new user format to tempauth which starts with
"user64_" and in which the account and user are base64 encoded (sans
padding equal signs):
user64_<account_b64>_<user_b64> = <key> [group] [group] [...] [storage_url]
This allows unambiguous parsing and adds the ability to have a user or
account name containing underscores.
The handling of tempauth configuration is fully backward compatible and
this only adds new, optional functionality.
Also beefed up unit tests to get full coverage of the user parsing code
in tempauth's __init__().
Change-Id: Id06304934ab8055a28921f2e1aa3c58a2036d8f8
This change replaces WebOb with a mostly compatible local library,
swift.common.swob. Subtle changes to WebOb's API over the years have been a
huge headache. Swift doesn't even run on the current version.
There are a few incompatibilities to simplify the implementation/interface:
* It only implements the header properties we use. More can be easily added.
* Casts header values to str on assignment.
* Response classes ("HTTPNotFound") are no longer subclasses, but partials
on Response, so things like isinstance no longer work on them.
* Unlike newer webob versions, will never return unicode objects.
Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f
This moves search_devs into RingBuilder to make it accessible to other utils
that need to search the builder. Along the same lines this also adds a
load() call to get a RingBuilder instance when working with the builder files.
- This adds python-mock >= 0.7 as a dependency for unittests. On Ubuntu
10.04 you'll have to pip install it, on 12.04 you can apt-get install
it. Fedora 17+ should be able to yum install it.
- new pep8 compliance
- Fixed a small issue (undefined var) in swift-ring-builder when remove was
called but failed to find a match.
Change-Id: I2e02684235aa2f4e901a00858ae037091594c545
Removed many StatsD logging calls in proxy-server and added
swift-informant-style catch-all logging in the proxy-logger middleware.
Many errors previously rolled into the "proxy-server.<type>.errors"
counter will now appear broken down by response code and with timing
data at: "proxy-server.<type>.<verb>.<status>.timing". Also, bytes
transferred (sum of in + out) will be at:
"proxy-server.<type>.<verb>.<status>.xfer". The proxy-logging
middleware can get its StatsD config from standard vars in [DEFAULT] or
from access_log_statsd_* config vars in its config section.
Similarly to Swift Informant, request methods ("verbs") are filtered
using the new proxy-logging config var, "log_statsd_valid_http_methods"
which defaults to GET, HEAD, POST, PUT, DELETE, and COPY. Requests with
methods not in this list use "BAD_METHOD" for <verb> in the metric name.
To avoid user error, access_log_statsd_valid_http_methods is also
accepted.
Previously, proxy-server metrics used "Account", "Container", and
"Object" for the <type>, but these are now all lowercase.
Updated the admin guide's StatsD docs to reflect the above changes and
also include the "proxy-server.<type>.handoff_count" and
"proxy-server.<type>.handoff_all_count" metrics.
The proxy server now saves off the original req.method and proxy_logging
will use this if it can (both for request logging and as the "<verb>" in
the statsd timing metric). This fixes bug 1025433.
Removed some stale access_log_* related code in proxy/server.py. Also
removed the BaseApplication/Application distinction as it's no longer
necessary.
Fixed up the sample config files a bit (logging lines, mostly).
Fixed typo in SAIO development guide.
Got proxy_logging.py test coverage to 100%.
Fixed proxy_logging.py for PEP8 v1.3.2.
Enhanced test.unit.FakeLogger to track more calls to enable testing
StatsD metric calls.
Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576
To tell when replication for a device has finished, it's important to
know when the replicator is removing objects. This was previously
handled for the object-replicator
(object-replicator.partition.delete.count.<device> and
object-replicator.partition.update.count.<device> metrics) but not the
account and container replicators.
This patch extends the existing DB removal count metrics to make them
per-device. The new metrics are:
account-replicator.removes.<device>
container-replicator.removes.<device>
There's also a bonus refactoring and increased test coverage of the DB
replicator code.
Change-Id: I2067317d4a5f8ad2a496834147954bdcdfc541c1
Added "-p" to mkdir cmds in a "set -e" script (/etc/rc.local).
/var/log/swift gets owned by the group "adm", but isn't group writeable
by default, so the find/rm in "resetswift" fails. So I added a chmod to
add group-write for that directory tree.
Added commands for installing tox.
Change-Id: Ic141491272a83a9233cd6a6a70c3ef74411f3f51
The Admin Guide now contains information about the ring serialization
change (and importantly, how to downgrade, if necessary).
Also added container-server conf var, "allow_versions" to the Deployment
Guide.
Also changed description of proxy-server conf var,
"max_containers_whitelist" to say it contains "account names" not
"account hashes".
Change-Id: Ib23c6118cc5195cc04765afd28e442e4c735f0d4
bug fixes for swift 1033089
Swift 1.6.0 All in One setup uses rc.local to create a set of
directories like the following:
/var/cache/swift /var/cache/swift2 /var/cache/swift3
/var/cache/swift4
/var/run/swift
without -p for mkdir command, when some of these directories
already exist, the rc.local fails so that the next set of
command won't execute. This causes swift fail to start because
of /var/run/swift does not exist. Adding -p to both mkdir
command solves this problem.
Change-Id: Ia26c9a857723747008293c0631c9acc1b58eca7e
