This patch increases the test coverage of the following middlewares:
- list_endpoints
- crypto
- crossdomain
Change-Id: I3dec85f61da07bd110bf42220d5ba46e11833a90
The commit a0fcca1e makes reconstructor not sync suffixes when remote
reject reconstructor revert. However, the exact same logic should
be applied to SYNC job as well. REPLICATE requests aren't generally
needed when using SSYC (which the reconstructor always does).
If a ssync_sender fails to finish a sync the reconstructor should skip
the REPLICATE call entirely and move on to the next partition without
causing any useless remote IO.
Change-Id: Ida50539e645ea7e2950ba668c7f031a8d10da787
Closes-Bug: #1665141
If the data in a hashes.pkl is corrupted but still de-serialized without
errors, it will mess up the replication and gets never fixed. This
happens for example if one of the keys is a NULL byte.
This patch checks if the dict keys in hashes.pkl are valid strings and
invalidates it if not.
Closes-Bug: 1830881
Change-Id: I84b062d062ff49935feed0aee3e1963bb72eb5ea
This patch ports the symlink middleware to py3.
The middleware itself seems to be mostly fine and most changes
are in the symlink unit tests.
Change-Id: I973c2e1bb8969cf6bffece8ce68881c393efbaef
Instead of taking a X-Backend-Allow-Method that *must match* the
REQUEST_METHOD, take a truish X-Backend-Allow-Private-Methods and
expand the set of allowed methods. This allows us to also expose
the full list of available private methods when returning a 405.
Drive-By: make async-delete tests a little more robust:
* check that end_marker and prefix are preserved on subsequent
listings
* check that objects with a leading slash are correctly handled
Change-Id: I5542623f16e0b5a0d728a6706343809e50743f73
Previously, we'd preserve the sysmeta that we wrote down with the
original multipart-upload to track its S3-style etag on the new part,
causing it to have an ETag like `<MD5>-<N>`. Later, when the client
tried to complete the new multipart-upload, it would send that etag back
to the server, which would reject the request because the ETag didn't
look like a normal MD5.
Now, have s3api include blank values in the copy request to overwrite
the source sysmeta, and treat a blank etag override the same as a
missing one.
Change-Id: Id33a7ab9d0b8f33fede73eae540d6137708e1218
Closes-Bug: #1829959
I'm pretty sure I botched the intent of the test when I touched it in
https://github.com/openstack/swift/commit/36c4297
As a side-effect, clean up a warning on py2:
UnicodeWarning: Unicode equal comparison failed to convert
both arguments to Unicode - interpreting them as being unequal
Change-Id: I4da8264f85e688bc5109016c197378ca6dfe06a5
Adds a tool, swift-container-deleter, that takes an account/container
and optional prefix, marker, and/or end-marker; spins up an internal
client; makes listing requests against the container; and pushes the
found objects into the object-expirer queue with a special
application/async-deleted content-type.
In order to do this enqueuing efficiently, a new internal-to-the-cluster
container method is introduced: UPDATE. It takes a JSON list of object
entries and runs them through merge_items.
The object-expirer is updated to look for work items with this
content-type and skip the X-If-Deleted-At check that it would normally
do.
Note that the target-container's listing will continue to show the
objects until data is actually deleted, bypassing some of the concerns
raised in the related change about clearing out a container entirely and
then deleting it.
Change-Id: Ia13ee5da3d1b5c536eccaadc7a6fdcd997374443
Related-Change: I50e403dee75585fc1ff2bb385d6b2d2f13653cf8
The default pickle protocol in python3 is version 3. This is not
readable by a python2 interpreter.
Force the use of version 2 in the object server REPLICATE() function,
for compatibility with python 2.
Change-Id: I19d23570ff3a084d288de1308e059cfd8134d6ad
In the diskfile module, there are a couple of cases where we would
quarantine the suffix dir if a single object within is not a directory.
Change the code so that we quarantine only the object.
Change-Id: I35eb16f568c00d973eb089aedd0e5a5619c25a2f
Adds the scaffolding required for tests to use boto3 and converts the
test_bucket.py tests to the new interface. Follow on patches will
convert the other tests to use the boto3 library.
Notable changes: we no longer try to reach for the equivalent of
`boto.make_request()` and instead rely on the boto3/botocore event
system to mutate requests as necessary (or to disable pre-flight
validators).
Partial-Bug: 1557260
Change-Id: I3d77ef4a6b878c49ebfa0c8b8647d7199d87601e
Some middleware, such as that included with ProxyFS, may put
RFC-compliant ETags in container listings; we should leave those
unquoted, so we don't double-up on quotes.
Change-Id: I8b56231aef41076a224dc62cd93c1973355c4670
(Some versions of?) awscli/boto3 will do v4 signatures but send a
Content-MD5 for end-to-end validation. Since a X-Amz-Content-SHA256
is still required to calculate signatures, it uses UNSIGNED-PAYLOAD
similar to how signatures work for pre-signed URLs.
Look for UNSIGNED-PAYLOAD and skip SHA256 validation if set.
Change-Id: I571c16c196dae4e4f8fb41904c8850d0054b1fe9
Related-Change: I61eb12455c37376be4d739eee55a5f439216f0e9
This adds wsgi_to_str(self.path_info) everywhere we forgot it,
not only in the slo module itself.
Dropping the body=''.join(body) after call_slo() is obvious:
the latter only returns strings of bytes, not lists of such.
Change-Id: I6b4d87e4cda4945bc128dbc9c1edd39e736a59d2
S3 supports two metadata operations on object copy: COPY and REPLACE.
When using REPLACE, the Content-Type should be set to the one supplied
by the caller. When using COPY, the existing object's Content-Type value
is used.
Change-Id: Ic7c6278dedef308c9219eb45751abfa5655f144f
Closes-Bug: #1828907
The idea is that we should have a suite of pure-S3 tests that we can
point at AWS to verify that we've written accurate tests, then point at
Swift-with-s3api to verify that we've correctly implemented the S3 api.
As a start, just check GET Service; go ahead and create a few buckets
so we can see them in the service listing.
Change-Id: I283757cd3084b1c83a1e9bf0f46b6ce9d7ee8eb9
Since the change in s3_token_middleware to retrieve the auth info
from keystone directly, now, we don't need to have any tokens provided
by keystone in the request header as X-Auth-Token.
Note that this makes the pipeline ordering change documented in the
related changes mandatory, even when working with a v2 Keystone server.
Change-Id: I7c251a758dfc1fedb3fb61e351de305b431afa79
Related-Change: I21e38884a2aefbb94b76c76deccd815f01db7362
Related-Change: Ic9af387b9192f285f0f486e7171eefb23968007e
Added a test for S3 v1 listings that use URL encoding and have non-ASCII
characters. In the process discovered that the XML schema for
ListBucketResult had a small problem: Delimiter and EncodingType needed
to be reordered.
Change-Id: Ib3124ea079a73a577b86de97657603a64b16f965
