After a container database is replicated, a _post_replicate_hook will enqueue
misplaced objects for the container-reconciler into the .misplaced_objects
containers. Items to be reconciled are "batch loaded" into the reconciler
queue and the end of a container replication cycle by levering container
replication itself.
DocImpact
Implements: blueprint storage-policies
Change-Id: I3627efcdea75403586dffee46537a60add08bfda
* base implementation of is_deleted phrased to use _is_deleted
* wrap pre-conn coded _is_deleted inside a transation for merge_timestamps
Implements: blueprint storage-policies
Change-Id: I6a948908c3e45b70707981d87171cb2cb910fe1e
Keep status_changed_at in container databases current with status changes that
occur as a result of container creation, deletion, or re-creation.
Merge container put/delete/created timestamps when handling replicate
responses from remote servers in addition to during the handling of the
REPLICATE request.
When storage policies are configured on a cluster send status_changed_at,
object_count and storage_policy_index as part of container replication sync
args.
Use status_changed_at during replication to determine the oldest active
container and merge storage_policy_index.
DocImpact
Implements: blueprint storage-policies
Change-Id: Ib9a0dd42c271145e641437dc04d0ebea1e11fc47
This daemon will take objects that are in the wrong storage policy and
move them to the right ones, or delete requests that went to the wrong
storage policy and apply them to the right ones. It operates on a
queue similar to the object-expirer's queue.
Discovering that the object is in the wrong policy will be done in
subsequent commits by the container replicator; this is the daemon
that handles them once they happen.
Like the object expirer, you only need to run one of these per cluster
see etc/container-reconciler.conf.
DocImpact
Implements: blueprint storage-policies
Change-Id: I5ea62eb77ddcbc7cfebf903429f2ee4c098771c9
Rework header handling and add some methods needed by the reconciler.
* response headers are case insensitive HeaderKeyDicts
* add direct client container obj put and delete
* add headers param to direct head object
* add headers to DirectClientException
DirectClientException is a subclass of ClientException with a convience
constructor. ClientException now supports an http_headers kwarg.
Exceptions raised from direct_client will include headers.
DocImpact
Implements: blueprint storage-policies
Change-Id: Ia484d569619df0bf85f973e4e916de2ac6401d5e
* add get_object
* allow extra headers passthrough on HEAD/metadata reqeusts
* expose (account|container|get_object)_ring properties
Pipeline propety access to the auto_create_account_prefix also allows us to
bypass the early exit on a container HEAD for auto_create_accounts if the
container-updater hasn't cycled yet.
Allow overriding of storage policy index.
This is something the reconciler will need so that it can GET from one
policy, PUT in another, and then DELETE from the first one again.
DocImpact
Implements: blueprint storage-policies
Change-Id: I9b287d15f2426022d669d1186c9e22dd8ca13fb9
Objects now have a storage policy index associated with them as well;
this is determined by their filesystem path. Like before, objects in
policy 0 are in /srv/node/$disk/objects; this provides compatibility
on upgrade. (Recall that policy 0 is given to all existing data when a
cluster is upgraded.) Objects in policy 1 are in
/srv/node/$disk/objects-1, objects in policy 2 are in
/srv/node/$disk/objects-2, and so on.
* 'quarantined' dir already created 'objects' subdir so now there
will also be objects-N created at the same level
This commit does not address replicators, auditors, or updaters except
where method signatures changed. They'll still work if your cluster
has only one storage policy, though.
DocImpact
Implements: blueprint storage-policies
Change-Id: I459f3ed97df516cb0c9294477c28729c30f48e09
Containers now have a storage policy index associated with them,
stored in the container_stat table. This index is only settable at
container creation time (PUT request), and cannot be changed without
deleting and recreating the container. This is because a container's
policy index will apply to all its objects, so changing a container's
policy index would require moving large amounts of object data
around. If a user wants to change the policy for data in a container,
they must create a new container with the desired policy and move the
data over.
Keep status_changed_at up-to-date with status changes.
In particular during container recreation and replication.
When a container-server receives a PUT for a deleted database an extra UPDATE
is issued against the container_stat table to notate the x-timestamp of the
request.
During replication if merge_timestamps causes a container's status to change
(from DELETED to ACTIVE or vice-versa) the status_changed_at field is set to
the current time.
Accurate reporting of status_changed_at is useful for container replication
forensics and allows resolution of "set on create" attributes like the
upcoming storage_policy_index.
Expose Backend container info on deleted containers.
Include basic container info in backend headers on 404 responses from the
container server. Default empty values are used as placeholders if the
database does not exist.
Specifically the X-Backend-Status-Changed-At, X-Backend-DELETE-Timestamp and
the X-Backend-Storage-Policy-Index value will be needed by the reconciler to
deal with reconciling out of order object writes in the face of recently
deleted containers.
* Add "status_changed_at" key to the response from ContainerBroker.get_info.
* Add "Status Timestamp" field to swift.cli.info.print_db_info_metadata.
* Add "status_changed_at" key to the response from AccountBroker.get_info.
DocImpact
Implements: blueprint storage-policies
Change-Id: Ie6d388f067f5b096b0f96faef151120ba23c8748
FakeLogger gets better log level handling
Parameterize logger on some daemons which were previously
unparameterized and try and use the interface in tests.
FakeRing use more real code
The existing FakeRing mock's implementation bit me on some pretty subtle
character encoding issue by-passing the hash_path code that is normally
part of get_part_nodes. This change tries to exercise more of the real
ring code paths when it makes sense and provide a better Fake for use in
testing.
Add write_fake_ring helper to test.unit for when you need a real ring.
DocImpact
Implements: blueprint storage-policies
Change-Id: Id2e3740b1dd569050f4e083617e7dd6a4249027e
The basic idea here is to replace the use of a single object ring in
the Application class with a collection of object rings. The
collection includes not only the Ring object itself but the policy
name associated with it, the filename for the .gz and any other
metadata associated with the policy that may be needed. When
containers are created, a policy (thus a specific obj ring) is
selected allowing apps to specify policy at container creation time
and leverage policies simply by using different containers for object
operations.
The policy collection is based off of info in the swift.conf file.
The format of the sections in the .conf file is as follows:
swift.conf format:
[storage-policy:0]
name = chicken
[storage-policy:1]
name = turkey
default = yes
With the above format:
- Policy 0 will always be used for access to existing containers
without the policy specified. The ring name for policy 0 is always
'object', assuring backwards compatiblity. The parser will always
create a policy 0 even if not specified
- The policy with 'default=yes' is the one used for new container
creation. This allows the admin to specify which policy is used without
forcing the application to add the metadata.
This commit simply introduces storage policies and the loading
thereof; nobody's using it yet. That will follow in subsequent
commits.
Expose storage policies in /info
DocImpact
Implements: blueprint storage-policies
Change-Id: Ica05f41ecf3adb3648cc9182f11f1c8c5c678985
HTTP header values should be quoted. Since the WWW-Authenticate
header value contains user-supplied strings, it's important to
ensure it's properly quoted to ensure the integrity of the protocol.
Previous to this patch, the URL was unquoted and then the unquoted
value was returned in the header. This patch re-quotes the value
when it is set on the response.
This is filed as CVS-2014-3497
Fixes bug 1327414
Change-Id: If8bd8842f2ce821756e9b4461a18a8ac8d42fb8c
Just put SLO and DLO after any auth middleware. This works because when
the request goes through that middleware in the pipeline the
authentication takes place: validation of the token, setting up who the
user is, and setting the authorization call back. Each subrequest made
for the segments will be subjected to that authorization call back which
verifies the user has access to the individual segments.
To get this to work with keystone, the keystone identity is set up
during __call__ and applied to the authorize function using a
functools.partial. When the authorize function is later called from the
environ by the proxy server the idenity that was set up when the request
passed through the auth middleware is used, not what can be pulled out
of the possibly altered state of the request's environment.
DocImpact
fixes bug: 1315133
Change-Id: I7827dd2d9dfbb3c6424773fb2891355d47e372ba
Also fixed a bug where SimpleClient would send ?format=json for object
requests, which is not necessary.
Change-Id: If06a7dcebc9de2d7c8b28a046d60b902dae821c1
We previously registered with no parameters, added parms so they
are displayed as follows (example):
"ratelimit": {"max_sleep_time_seconds": 60.0,
"container_listing_ratelimits": [[0, 100.0], [10, 50.0], [50, 20.0]],
"container_ratelimits": [[0, 100.0], [10, 50.0], [50, 20.0]],
"account_ratelimit": 1.0}
Note that not all parameters are exposed (intentionally) via /info
Change-Id: I36c7ef15af17e3eb8ebb93429035bd06d089a945
Closes-Bug: 1308989
Log lines can get quite large, as we previously noticed with rsync error
log lines. We added a setting to cap those, but it really looks like we
should have just done this overall limit. We noticed the issue when we
switched to UDP syslogging and it would occasionally blow past the 16436
lo MTU! This causes Python's logging code to get an error and hilarity
ensues.
Change-Id: I44bdbe68babd58da58c14360379e8fef8a6b75f7
Container sync had a bug where it'd send out the trailing
"; swift_bytes=xxx" part of the content-type header. That trailing part
is just for internal cluster usage by SLO. Since that needed to be
stripped in two places now, I separated it out to a function that both
spots call.
Change-Id: Ibd6035d7a6b78205344bcc9d98bc1b7a9d463427
Based on comments from deployers at the Juno OpenStack summit,
limiting the default logged token length (to, by default, prevent
tokens from being fully logged) is a good idea.
Change-Id: I58980e85329d99de41f1c08f75e85973452317b1
On Lucid at least, xprofile doesn't work. While others try to figure out the
underlying issue, this patch would just skip the tests if the profiling code
can not be imported.
Change-Id: I0fee3f729742cc727edc2adabb4ed782bea7798f
This allows an easier and more explicit way to tell swift-init to run on
specific servers. For example with an SAIO, this allows you to do
something like:
swift-init object-server.1 reload
to reload just the 1st object server. A more real world example is when
you are running separate servers for replication. In this example you
might have an object-server/public.conf and
object-server/replication.conf. With this change you can do something
like:
swift-init object-server.replication reload
to just reload the replication server.
DocImpact
Change-Id: I5c6046b5ee28e17dadfc5fc53d1d872d9bb8fe48
The value of the X-Trans-Id-Extra header on the request (if any) will
now be appended to the transaction ID. This lets users put their own
information into transaction IDs.
For example, Glance folks upload images as large objects, so they'd
like to be able to tie together all the segment PUTs and the manifest
PUT with some operation ID in the logs. This would let them pass in
that operation ID as X-Trans-Id-Extra, and then when things went
wrong, it'd be much easier to find all the requests in Swift's logs.
Also, this aids debuggability when requests fail to receive
responses. If a user is sending in their own X-Trans-Id-Extra strings,
then that gives operators something to search for in the logs. The
normal txid won't work since that's in the response, but the client
didn't receive one.
Swift will only use the first 32 characters of X-Trans-Id-Extra so
that its log lines stay a manageable length. Also, it's URL-quoted so
that users cannot inject double quotes into X-Trans-Id-Extra and screw
up log parsers.
DocImpact
Change-Id: I3c51d0c5ac55697ac230001840da219e73a03157
As seen on #1174809, changes use of mutable types as default
arguments and defaults them within the method. Otherwise, those
defaults can be unexpectedly persisted with the function between
invocations and erupt into mass hysteria on the streets.
There was indeed a test (TestSimpleClient.test_get_with_retries)
that was erroneously relying on this behavior. Since previous tests
had populated their own instantiations with a token, this test only
passed because the modified headers dict from previous tests was
being overridden. As expected, with the mutable defaults fix in
SimpleClient, this test begain to fail since it never specified any
token, yet it has always passed anyway. This change also now provides
the expected token.
Change-Id: If95f11d259008517dab511e88acfe9731e5a99b5
Related-Bug: #1174809
When current code modifies the pipeline, it prints the entry point
names instead of the names used to construct the pipeline. This is
inconvenient because a sysadmin cannot copy and paste from the log.
We already save the pipeline name into contexts in most cases, so
the fix simply reuses that to provide friendly names.
Fixes bug: 1311802
Change-Id: Ic76baf1360cd521f140fa1980029ccbce58f1717
The profile middleware provide a tool to profile Swift
code on the fly and collect statistic data for performance
analysis. An native simple Web UI is also provided to help
query and visualize the data.
Change-Id: I6a1554b2f8dc22e9c8cd20cff6743513eb9acc05
Implements: blueprint profiling-middleware
The behavior of common.utils.cache_from_env
was changed by https://review.openstack.org/#/c/89488/.
This patch adds unit test for that function.
Change-Id: If757e12990c971325f7705731ef529a7e2a9eee7
If an object had a \r or \n in its name, it would end up creating an
invalid HTTP Content-Disposition header. Reviewer consensus was to use
URL encoding.
Fixes bug 1306250
Change-Id: Ibccaaed5152b4d09d6aee4966a1982cc0a0da07d
I've seen several folks recently have problems with their Swift
clusters because they had different hash prefixes on different
nodes. Let's help them out by having recon check that.
Note that MD5-equality is stronger than what we need (which is
ConfigParser-equality for a particular set of keys), but this way we
don't expose the secret hash prefix and suffix across the internal
network, just the MD5 checksum of the file containing them.
Change-Id: I3af984ee45947345891b3c596a88e3464f178cc7
Make account, object, and container servers construct log lines using the
same utility function so they will produce identically formatted lines.
This change reorders the fields logged for the account server.
This change also adds the "additional info" field to the two servers that
didn't log that field. This makes the log lines identical across all 3
servers. If people don't like that, I can take that out. I think it makes
the documentation, parsing of the log lines, and the code a tad cleaner.
DocImpact
Change-Id: I268dc0df9dd07afa5382592a28ea37b96c6c2f44
Closes-Bug: 1280955
internal client.
Internal client uses 'chunked' transfer encoding in it's
upload_object method. If content-length is known in advance,
internal_client should allowed to use content length in the
upload_object method and not the chunked method.
Change-Id: I68d2ebde78e01fa16b7187a2f045ea20f4310722
The use of NamedTemporaryFile creates rings with permissions 0600;
however most installs probably generate the rings as root but the
swift-proxy runs as user swift.
Set the permissions on the generated ring to 0644 prior to rename so
that the swift user can read the rings.
Change-Id: Ia511931f471c5c9840012c3a75b89c1f35b1b245
Closes-Bug: #1302700
Prior to this patch both mainline code and testing modules imported
and used constraints directly into their own namespace, or relied on
the namespace of other modules that were not the constraints
module. This meant that if a unit test wanted to change a constraint
for its operation, it had to know how that module was using the
constraint, instead of referencing the constraint module itself.
This patch unifies the use of constraints so that all constraints are
referenced via the constraints module. In turn, this allows a test to
leverage the re-loadable nature of the constraints in the constraints
module.
It addition, a number of functional tests where using the default
values for constraints, instead of the configured value discovered in
a test.conf or in an existing swift.conf. This patch removes those
direct references in favor of the load_constraint() method from the
test/functional/tests.py module.
Change-Id: Ia5313d653c667dd9ca800786de59b59334c34eaa
It simply makes sense that the definition of DATADIR belongs to
backends. After all, some of them may not even have any.
Coincidentially, a few unnecessary imports are dropped.
By the way, on the object server side, diskfile.py provides DATADIR
in the same way already.
Change-Id: I60bfd522c77c4a0ee13697a2e31141777c7e2398
