I noticed while poking at the DLO func tests that we don't actually use
non-ascii chars when we set up the test env.
By patching the create name function earlier (in SetUpClass) we can
ensure we get some more interesting characters in our object names.
Change-Id: I9480ddf74463310aeb11ad876b79527888d8c871
- ConfigParser.set() requires that the value be a string
- The stdlib HTTP client responses don't have a body property
- We might raise a ResponseError with response=None
- Bodies should be bytes
- Headers should be strings
- Make containers()/files() return native strings
- file() isn't a thing on py3
- format should be a parm, not a header
- Switch sorted() to use key instead of cmp
- Use integer division explicitly
Change-Id: I99d3eebc9d7ec4e8b295352294b831492135c568
Some tests make use of multiple accounts without checking of they have
been set up. This commit tries to fix some of these situations.
Change-Id: I461679e78e19ce0866c7618c581a8cb573cca7f5
Now that POST-as-COPY is dead and gone, 202 should be the only
successful response code for an object POST. Note that there are
already tests requiring 202s in test/functional/test_object.py.
Change-Id: I33c8d2c031f8dfdf1e789bb8d6e6908bfff4d739
... so we can use storage_url for actual, complete URLs.
Also, expose storage_scheme for easier comparisons.
Change-Id: Iffcfc327bc674681ac51fe75c1d9911fa3a60d54
Apparently that isn't a thing we have to worry about any more? Probably
got fixed when we pulled COPY out to middleware.
Add some tests where we definitely expect 403s across the board, too.
Change-Id: Idb5c0b25969b839cc71c487208447bdd6817c2cf
You've got two test classes: TestContainer and TestContainerUTF8. They
each try to create the same set of containers with names of varying
lengths to make sure the container-name length limit is being honored.
Also, each test class tries to clean up pre-existing data in its
setUpClass method. If TestContainerUTF8 fails to delete a contaienr
that TestContainer made, then its testContainerNameLimit method will
fail because the container PUT response has status 202 instead of 201,
which is because the container still existed from the prior test.
I've made the test consider both 201 and 202 as success. For purposes
of testing the maximum container name length, any 2xx is fine.
Change-Id: I7b343a8ed0d12537659c051ddf29226cefa78a8f
These are better-covered by TestContainer.testContainerNameLimit and
TestFile.testNameLimit in the same file.
Change-Id: Ice48bc6492648613bc743b474d40892d7e4dcc64
Currently, our integrity checking for objects is pretty weak when it
comes to object metadata. If the extended attributes on a .data or
.meta file get corrupted in such a way that we can still unpickle it,
we don't have anything that detects that.
This could be especially bad with encrypted etags; if the encrypted
etag (X-Object-Sysmeta-Crypto-Etag or whatever it is) gets some bits
flipped, then we'll cheerfully decrypt the cipherjunk into plainjunk,
then send it to the client. Net effect is that the client sees a GET
response with an ETag that doesn't match the MD5 of the object *and*
Swift has no way of detecting and quarantining this object.
Note that, with an unencrypted object, if the ETag metadatum gets
mangled, then the object will be quarantined by the object server or
auditor, whichever notices first.
As part of this commit, I also ripped out some mocking of
getxattr/setxattr in tests. It appears to be there to allow unit tests
to run on systems where /tmp doesn't support xattrs. However, since
the mock is keyed off of inode number and inode numbers get re-used,
there's lots of leakage between different test runs. On a real FS,
unlinking a file and then creating a new one of the same name will
also reset the xattrs; this isn't the case with the mock.
The mock was pretty old; Ubuntu 12.04 and up all support xattrs in
/tmp, and recent Red Hat / CentOS releases do too. The xattr mock was
added in 2011; maybe it was to support Ubuntu Lucid Lynx?
Bonus: now you can pause a test with the debugger, inspect its files
in /tmp, and actually see the xattrs along with the data.
Since this patch now uses a real filesystem for testing filesystem
operations, tests are skipped if the underlying filesystem does not
support setting xattrs (eg tmpfs or more than 4k of xattrs on ext4).
References to "/tmp" have been replaced with calls to
tempfile.gettempdir(). This will allow setting the TMPDIR envvar in
test setup and getting an XFS filesystem instead of ext4 or tmpfs.
THIS PATCH SIGNIFICANTLY CHANGES TESTING ENVIRONMENTS
With this patch, every test environment will require TMPDIR to be
using a filesystem that supports at least 4k of extended attributes.
Neither ext4 nor tempfs support this. XFS is recommended.
So why all the SkipTests? Why not simply raise an error? We still need
the tests to run on the base image for OpenStack's CI system. Since
we were previously mocking out xattr, there wasn't a problem, but we
also weren't actually testing anything. This patch adds functionality
to validate xattr data, so we need to drop the mock.
`test.unit.skip_if_no_xattrs()` is also imported into `test.functional`
so that functional tests can import it from the functional test
namespace.
The related OpenStack CI infrastructure changes are made in
https://review.openstack.org/#/c/394600/.
Co-Authored-By: John Dickinson <me@not.mn>
Change-Id: I98a37c0d451f4960b7a12f648e4405c6c6716808
While we're at it, have copy and copy_account raise ResponseErrors
on failure, similar to cluster_info, update_metadata, containers, info,
files, delete, initialize, read, sync_metadata, write, and post.
Related-Change: Ia8b92251718d10b1eb44a456f28d3d2569a30003
Change-Id: I9ef42d922a6b7dbf253f2f8f5df83965d8f47e0f
Currently, EC GET responses from proxy to clients, unlike any other
response, include a "Connection: close" header. If the client has sent
a "Connection: keep-alive" header then eventlet.wsgi appends this to
the client response, so clients can receive a response with both
headers:
Connection: close
Connection: keep-alive
This patch fixes the proxy EC GET path to remove any Connection header
from it's response, but does not change the behaviour of eventlet.wsgi
with respect to returning any client provided 'Connection: keep-alive'
header.
Change-Id: I43cd27c978edb4a1a587f031dbbee26e9acdc920
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Closes-Bug: #1680731
For now, last modified timestamp is supported only on
object listing. (i.e. GET container)
For example:
GET container with json format results in like as:
[{"hash": "d41d8cd98f00b204e9800998ecf8427e", "last_modified":
"2015-06-10T04:58:23.460230", "bytes": 0, "name": "object",
"content_type": "application/octet-stream"}]
However, container listing (i.e. GET account) shows just a dict
consists of ("name", "bytes", "name") for each container.
For example:
GET accounts with json format result in like as:
[{"count": 0, "bytes": 0, "name": "container"}]
This patch is for supporting last_modified key in the container
listing results as well as object listing like as:
[{"count": 0, "bytes": 0, "name": "container", "last_modified":
"2015-06-10T04:58:23.460230"}]
This patch is changing just output for listing. The original
timestamp to show the last modified is already in container table
of account.db as a "put_timestamp" column.
Note that this patch *DOESN'T* change the put_timestamp semantics.
i.e. the last_modified timestamp will be changed only at both PUT
container and POST container.
(PUT object doesn't affect the timestamp)
Note that the tuple format of returning value from
swift.account.backend.AccountBroker.list_containers is now
(name, object_count, bytes_used, put_timestamp, 0)
* put_timestamp is added *
Original discussion was in working session at Vancouver Summit.
Etherpads are around here:
https://etherpad.openstack.org/p/liberty-swift-contributors-meetuphttps://etherpad.openstack.org/p/liberty-container-listing-update
DocImpact
Change-Id: Iba0503916f1481a20c59ae9136436f40183e4c5b
Addresses a TODO in test/functional/test_account.py where
an account metadata test was having to clean up tempurl keys
in the account metadata that were left by another test in
a different module. This cleanup is necessary because tests
in test_account.py fail if there is any pre-existing
account metadata.
This patch:
* makes the tempurl tests clean up their keys from account
metadata.
* makes the test_account.py:TestAccount class remove any
pre-existing metadata before attempting any tests and
replacing that metadata when all the tests in that class
have completed. This is more robust than the existing code
which only removes any tempurl keys that might be in the
account - now you could have x-account-meta-foo = bar in
the test account and test_account.py will still pass.
* consolidates some common setup code currently repeated for
many of the functional test classes into into a BaseEnv class.
Change-Id: I874a9e23dfcdd1caa934945b46089f11b9f6de65
If a user sends a Range header with no satisfiable ranges, we send back
a 416 Requested Range Not Satisfiable response. Previously however,
there would be no indication of the size of the object they were
requesting, so they wouldn't know how to craft a satisfiable range. We
*do* send a Content-Length, but it is (correctly) the length of the
error message.
The RFC [1] has an answer for this:
> A server generating a 416 (Range Not Satisfiable) response to a
> byte-range request SHOULD send a Content-Range header field with an
> unsatisfied-range value, as in the following example:
>
> Content-Range: bytes */1234
>
> The complete-length in a 416 response indicates the current length of
> the selected representation.
Now, we'll send a Content-Range header for all 416 responses, including
those coming from the object server as well as those generated on a
proxy because of the Range mangling required to support EC policies.
[1] RFC 7233, section 4.2, although similar language was used in RFC
2616, sections 10.4.17 and 14.16
Change-Id: I80c7390fc6f84a10a212b0641bb07a64dfccbd45
This patch removes the slo, dlo and tempurl tests from tests.py
As before, this patch does not refactor any code it's only a copy
and paste to make it easy to review
Change-Id: I0b49d4f8bc7dd204d06258f4910cfc9a5ba4bdcb
Signed-off-by: Thiago da Silva <thiago@redhat.com>
tests.py is currently at ~5500 lines of code, it's
time to break it down into smaller files.
I started with an easy middleware set of tests
(i.e., versioned writes, ~600 lines of code ) so I can get
some feedback. There are more complicated tests that cover
multiple middlewares for example, it is not so clear where
those should go.
Change-Id: I2aa6c18ee5b68d0aae73cc6add8cac6fbf7f33da
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Previously in copy middleware, if a user entered an invalid destination
path with an invalid `container/object` path the server would return
a 500 Internal Server Error. However, the correct response should be
a 412 Precondition Failed. This patch updates copy so that it catches
the 412 Precondition Failed exception and returns it to the client.
Closes-Bug: #1641980
Change-Id: Ic4677ae033d05b8730c6ad1041bd9c07268e11a9
Since [1] the functional tests for cross-policy
object versioning fails with an error. These tests are
skipped in all CI test jobs because none of them have
more than one policy enabled. However, when running
tests against a system with multiple policies the tests
will not skip and the error is raised.
[1] Related-Change: Ifebc1c3ce558b1df9e576a58a4100f2219dfc7e7
Related-Bug: #1629234
Change-Id: I48028928bc996252a31cbf3abfd2b5e67a74dc95
This patch includes a couple of small functional test improvement.
A. Change swift_test_client.File.sync_metadata to follow Swift object
metadata semantics:
swift_test_client.File.sync_metadata is designed to post object user
metadata to an object. However, prior to this patch, the
swift_test_client.File instance keeps the existing object metadata as
its member attribute and if sync_metadata is called, it sends both
existing metadata and incomming metadata from caller. It looks to result
in the odd state as if Swift keeps the existing metadata when POST
object requested.
To tell the correct Swift object metadata semantics, when POST object
requested, the existing metadata in the stored object should be gone
even if no metadata is overwritten.
i.e.
if POST object with 'X-Object-Meta-Key: Val' to a stored object with
'X-Object-Meta-foo: bar', it will result in an object with
'X-Object-Meta-Key' (note that X-Object-Meta-Foo will be deleted)
The prior behavior sometimes make us confused in the reviw [1] so that,
this patch fixes it to send only incomming metadata if it's set.
B. Check the response status code more strictly for ObjectVersioning case
This patch fixes test_versioning_check_acl on both TestObjectVersioning and
TestObjectVersioningHistoryMode to assert the response status code
explisitly instead of asserting just "ResponseError". (e.g. 403 when trying
to delete object from other account)
1: https://review.openstack.org/#/c/360933/1/test/functional/tests.py@4142
Change-Id: Ia3e5b40f17dc0f881b695aa4be39c98b91e2bb06
This patch is follow up for [1] and [2] to add new functional
tests for versioned_writes middlware 'history' mode.
(i.e. using X-History-Location header to a container).
The new test class, TestObjectHistoryModeVersioning, will use obvious
setting the mode via new X-History-Location header, since the change [2],
the setting X-Versions-Mode header added since [1] for incomming request has
been deprecated. Hence, since [2], the syntax for stack mode is back to
be same with older Swift than [1] so that the only thing we need now is
just adding a test suite for the new X-History-location.
It means the API has been changing like:
---------------
For stack mode:
---------------
Older than [1]:
X-Versions-Location
[1]~[2]:
X-Vesions-Location (and X-Versions-Mode: 'stack' for obvious)
Newer than [2]:
X-Vesions-Location
-----------------
For history mode:
-----------------
Older than [1]:
(Not supported)
[1]~[2]:
X-Vesions-Location and X-Versions-Mode: 'history'
Newer than [2]:
X-History-Location
Note that this functional tests work on newer swift than [2].
And then, this patch also sets allow_versioned_writes=True
for in-process testing (the container server allow_versions
option was already set, so this is just enabling in the middleware
too). That means that in-process functional tests (such as run by
the tox envs func-in-process-*) because history mode requires the
middleware allow_versioned_writes option to be explicity set to True.
1: https://review.openstack.org/#/c/214922/
2: https://review.openstack.org/#/c/373537/
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Related-Change: I555dc17fefd0aa9ade681aa156da24e018ebe74b
Related-Change: Icfd0f481d4e40dd5375c737190aea7ee8dbc3bf9
Change-Id: Ifebc1c3ce558b1df9e576a58a4100f2219dfc7e7
Since COPY allows a copy of a ranged GET, add some more test
coverage for that in both unit and functional tests.
Drive-by fix to use better test assertion methods.
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Change-Id: I5cb202386df0862f953f7388107c4d3466e2e46c
Currently a versioned write PUT uses a pre-authed request to move
it into the versioned container before checking whether the
user is authorised. This can lead to some interesting behaviour
whereby a user can select a versioned object path that it does not
have access to, request a put on that versioned object, and this
request will execute the copy part of the request before it fails
due to lack of permissions.
This patch changes the behaviour to be the same as versioned DELETE
where the request is authorised before anything is moved.
Change-Id: Ia8b92251718d10b1eb44a456f28d3d2569a30003
Closes-Bug: #1562175
Currently when using fast-post, the manifest is updated with the given
'x-object-manifest' header on a POST. If no such header is supplied,
then the manifest will change to a regular object.
This is not currently true when using post-as-copy.
This patch changes the DLO POST using post-as-copy behavior to match
that of using fast-post. It was also documented that
'x-object-manifest' must be provided on a POST to a manifest file.
Change-Id: Ie1143ab1a2c8f8c21e258a36badbff5d947769d4
Closes-bug: 1612991
Bonus consistency: 416 responses now always have a body. Before, if
you had "swob.HTTPRequestedRangeNotSatisfiable()", you'd get a body,
but if you had "swob.Response(..., conditional_response=True)", then
you'd get a length-0 response body. Now you always get a response
body. It's just the default <html><h1>..., but at it's always there.
Bonus efficiency: do a little caching of sub-SLO manifests to avoid
needless re-fetches. This kicks in when there are multiple references
to the same sub-SLO in a given manifest. The caching only holds 20
sub-SLOs so that a malicious user can't build a giant SLO tree and use
it to run the proxy out of memory (we're already holding up to 10
manifests in memory at a time since a SLO can include another SLO to a
depth of 10; this doesn't make the situation too much worse).
Change-Id: I24716e3271cf3370642e3755447e717fd7d9957c
In the past, a POST to a DLO manifest file when object_post_as_copy
was true resulted in the manifest file contents being replaced
by the concatenation of the DLO segments. This no longer
happens, but tests for this case are missing.
This patch adds a functional test to assert that the manifest
file is preserved in a POST request.
Change-Id: I90546014a7dcc7266f0d0e0ff6339688b7954b96
Related-bug: #1487791
Related-bug: #1514317
Previously, if a container listing produced `subdir` elements
the decrypter would raise a KeyError.
Additionally, update the functests so this sort of thing would
have been caught at the gate.
Closes-Bug: 1609904
Change-Id: Idc1907d19f90af7a086f45f8faecee9fbc3c69c2
I was watching the logs and noticed it creates an object named:
"<bound method type.create_ascii_name of <class 'test.functional.tests.Utils'>>"
Change-Id: I8dcbb40125b84a914e3c01566ae9c3f08dc9ea0f
Fix copy middleware so that all client-defined object
headers that object servers allow to be persisted are copied.
For example, content-encoding and content-disposition will
now be copied.
Fix treatment of x-fresh-metadata header so that, when it is
used, new object sysmeta is applied to the object copy in the
same way as a copy without x-fresh-metadata.
Remove unnecessary passing of original request headers to
sink PUT request constructor: passing the environ is sufficient
to have the new request inherit the original's headers.
Add tests for this change and to verify that content-type
gets either copied or updated if supplied with the copy
request.
Add tests for x-fresh-metadata treatment.
Closes-Bug: #1391826
Closes-Bug: #1600247
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Change-Id: I917fb0b4e831c13e04ade1c5e0b9821802dec967
I changed asserts with more specific assert methods.
e.g.: from assertTrue(sth == None) to assertIsNone(*) or
assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type) or
assertTrue(not sth) to assertFalse(sth).
The code gets more readable, and a better description will be shown on fail.
Change-Id: I80ec96e0b729bef38213a6be4ff4b6eb65c7612d
The varialbe 'size' in TestFile.testMetadataNumberLimit is not used.
This patch remove the variable from the test.
Change-Id: I255a1dcee12bb6b8dec6ff26ed7edf93ab2acf64
Relocates some test infrastructure in preparation for
use with encryption tests, in particular moves the test
server setup code from test/unit/proxy/test_server.py
to a new helpers.py so that it can be re-used, and adds
ability to specify additional config options for the
test servers (used in encryption tests).
Adds unit test coverage for extract_swift_bytes and functional
test coverage for container listings. Adds a check on the content
and metadata of reconciled objects in probe tests.
Change-Id: I9bfbf4e47cb0eb370e7a74d18c78d67b6b9d6645
This patch enables to show a x-put-timestamp as
a last-modified header in container-server.
Note that the last-modified header will be changed only when a
request for container (PUT container or POST container) comes into
Swift. i.e. some requests for objects (e.g. PUT object, POST object)
will never affect the last-modified value but only when using
python-swiftclient like as "swift upload", the last-modified will
be close to the upload time because python-swiftclient will make
a PUT container request for "swift upload" each time.
Change-Id: I9971bf90d24eee8921f67c02b7e2c80fd8995623
When copying an SLO manifest with multipart-manifest=get the actual
manifest content-type should get copied to the destination, rather
than the application/json value that is synthesised by SLO in a GET
response. That way the result of a HEAD on the copied manifest is the
same as a HEAD to the source, and the container listings for the two
are consistent.
This patch also un-skips a functional test and adds functional tests
that verify this patch and also verify that etags and size also get
correctly copied and updated in destination container (bug #1260446).
Closes-Bug: #1260446
Closes-Bug: #1583756
Change-Id: Ie7fa82f70b3ec3ef568f5355c69f6bce460ba25d
The testPUT case is failing when keystone was enabled and
allow_account_management is set to True. There were a few issues
needing addressed.
First the case was renamed to call out what it was actually doing which
is verifying an error scenario for which a PUT on a storage account was
not allowed.
Second the case was running even when allow_account_management is
enabled, which is incorrect. It "accidently" works with TempAuth
because it requires a reseller permission, so the Keystone failure here
has more to do with not requiring a reseller permission to do a PUT on a
storage account for which a user has an operator role on.
The common sense fix here is to not execute this test case when
allow_account_management is enabled.
Change-Id: Id29f5ca48f92cd139535be7064107b8a61b02856
