While we're at it, have copy and copy_account raise ResponseErrors
on failure, similar to cluster_info, update_metadata, containers, info,
files, delete, initialize, read, sync_metadata, write, and post.
Related-Change: Ia8b92251718d10b1eb44a456f28d3d2569a30003
Change-Id: I9ef42d922a6b7dbf253f2f8f5df83965d8f47e0f
Currently, EC GET responses from proxy to clients, unlike any other
response, include a "Connection: close" header. If the client has sent
a "Connection: keep-alive" header then eventlet.wsgi appends this to
the client response, so clients can receive a response with both
headers:
Connection: close
Connection: keep-alive
This patch fixes the proxy EC GET path to remove any Connection header
from it's response, but does not change the behaviour of eventlet.wsgi
with respect to returning any client provided 'Connection: keep-alive'
header.
Change-Id: I43cd27c978edb4a1a587f031dbbee26e9acdc920
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Closes-Bug: #1680731
For now, last modified timestamp is supported only on
object listing. (i.e. GET container)
For example:
GET container with json format results in like as:
[{"hash": "d41d8cd98f00b204e9800998ecf8427e", "last_modified":
"2015-06-10T04:58:23.460230", "bytes": 0, "name": "object",
"content_type": "application/octet-stream"}]
However, container listing (i.e. GET account) shows just a dict
consists of ("name", "bytes", "name") for each container.
For example:
GET accounts with json format result in like as:
[{"count": 0, "bytes": 0, "name": "container"}]
This patch is for supporting last_modified key in the container
listing results as well as object listing like as:
[{"count": 0, "bytes": 0, "name": "container", "last_modified":
"2015-06-10T04:58:23.460230"}]
This patch is changing just output for listing. The original
timestamp to show the last modified is already in container table
of account.db as a "put_timestamp" column.
Note that this patch *DOESN'T* change the put_timestamp semantics.
i.e. the last_modified timestamp will be changed only at both PUT
container and POST container.
(PUT object doesn't affect the timestamp)
Note that the tuple format of returning value from
swift.account.backend.AccountBroker.list_containers is now
(name, object_count, bytes_used, put_timestamp, 0)
* put_timestamp is added *
Original discussion was in working session at Vancouver Summit.
Etherpads are around here:
https://etherpad.openstack.org/p/liberty-swift-contributors-meetuphttps://etherpad.openstack.org/p/liberty-container-listing-update
DocImpact
Change-Id: Iba0503916f1481a20c59ae9136436f40183e4c5b
Addresses a TODO in test/functional/test_account.py where
an account metadata test was having to clean up tempurl keys
in the account metadata that were left by another test in
a different module. This cleanup is necessary because tests
in test_account.py fail if there is any pre-existing
account metadata.
This patch:
* makes the tempurl tests clean up their keys from account
metadata.
* makes the test_account.py:TestAccount class remove any
pre-existing metadata before attempting any tests and
replacing that metadata when all the tests in that class
have completed. This is more robust than the existing code
which only removes any tempurl keys that might be in the
account - now you could have x-account-meta-foo = bar in
the test account and test_account.py will still pass.
* consolidates some common setup code currently repeated for
many of the functional test classes into into a BaseEnv class.
Change-Id: I874a9e23dfcdd1caa934945b46089f11b9f6de65
If a user sends a Range header with no satisfiable ranges, we send back
a 416 Requested Range Not Satisfiable response. Previously however,
there would be no indication of the size of the object they were
requesting, so they wouldn't know how to craft a satisfiable range. We
*do* send a Content-Length, but it is (correctly) the length of the
error message.
The RFC [1] has an answer for this:
> A server generating a 416 (Range Not Satisfiable) response to a
> byte-range request SHOULD send a Content-Range header field with an
> unsatisfied-range value, as in the following example:
>
> Content-Range: bytes */1234
>
> The complete-length in a 416 response indicates the current length of
> the selected representation.
Now, we'll send a Content-Range header for all 416 responses, including
those coming from the object server as well as those generated on a
proxy because of the Range mangling required to support EC policies.
[1] RFC 7233, section 4.2, although similar language was used in RFC
2616, sections 10.4.17 and 14.16
Change-Id: I80c7390fc6f84a10a212b0641bb07a64dfccbd45
This patch removes the slo, dlo and tempurl tests from tests.py
As before, this patch does not refactor any code it's only a copy
and paste to make it easy to review
Change-Id: I0b49d4f8bc7dd204d06258f4910cfc9a5ba4bdcb
Signed-off-by: Thiago da Silva <thiago@redhat.com>
tests.py is currently at ~5500 lines of code, it's
time to break it down into smaller files.
I started with an easy middleware set of tests
(i.e., versioned writes, ~600 lines of code ) so I can get
some feedback. There are more complicated tests that cover
multiple middlewares for example, it is not so clear where
those should go.
Change-Id: I2aa6c18ee5b68d0aae73cc6add8cac6fbf7f33da
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Previously in copy middleware, if a user entered an invalid destination
path with an invalid `container/object` path the server would return
a 500 Internal Server Error. However, the correct response should be
a 412 Precondition Failed. This patch updates copy so that it catches
the 412 Precondition Failed exception and returns it to the client.
Closes-Bug: #1641980
Change-Id: Ic4677ae033d05b8730c6ad1041bd9c07268e11a9
Since [1] the functional tests for cross-policy
object versioning fails with an error. These tests are
skipped in all CI test jobs because none of them have
more than one policy enabled. However, when running
tests against a system with multiple policies the tests
will not skip and the error is raised.
[1] Related-Change: Ifebc1c3ce558b1df9e576a58a4100f2219dfc7e7
Related-Bug: #1629234
Change-Id: I48028928bc996252a31cbf3abfd2b5e67a74dc95
This patch includes a couple of small functional test improvement.
A. Change swift_test_client.File.sync_metadata to follow Swift object
metadata semantics:
swift_test_client.File.sync_metadata is designed to post object user
metadata to an object. However, prior to this patch, the
swift_test_client.File instance keeps the existing object metadata as
its member attribute and if sync_metadata is called, it sends both
existing metadata and incomming metadata from caller. It looks to result
in the odd state as if Swift keeps the existing metadata when POST
object requested.
To tell the correct Swift object metadata semantics, when POST object
requested, the existing metadata in the stored object should be gone
even if no metadata is overwritten.
i.e.
if POST object with 'X-Object-Meta-Key: Val' to a stored object with
'X-Object-Meta-foo: bar', it will result in an object with
'X-Object-Meta-Key' (note that X-Object-Meta-Foo will be deleted)
The prior behavior sometimes make us confused in the reviw [1] so that,
this patch fixes it to send only incomming metadata if it's set.
B. Check the response status code more strictly for ObjectVersioning case
This patch fixes test_versioning_check_acl on both TestObjectVersioning and
TestObjectVersioningHistoryMode to assert the response status code
explisitly instead of asserting just "ResponseError". (e.g. 403 when trying
to delete object from other account)
1: https://review.openstack.org/#/c/360933/1/test/functional/tests.py@4142
Change-Id: Ia3e5b40f17dc0f881b695aa4be39c98b91e2bb06
This patch is follow up for [1] and [2] to add new functional
tests for versioned_writes middlware 'history' mode.
(i.e. using X-History-Location header to a container).
The new test class, TestObjectHistoryModeVersioning, will use obvious
setting the mode via new X-History-Location header, since the change [2],
the setting X-Versions-Mode header added since [1] for incomming request has
been deprecated. Hence, since [2], the syntax for stack mode is back to
be same with older Swift than [1] so that the only thing we need now is
just adding a test suite for the new X-History-location.
It means the API has been changing like:
---------------
For stack mode:
---------------
Older than [1]:
X-Versions-Location
[1]~[2]:
X-Vesions-Location (and X-Versions-Mode: 'stack' for obvious)
Newer than [2]:
X-Vesions-Location
-----------------
For history mode:
-----------------
Older than [1]:
(Not supported)
[1]~[2]:
X-Vesions-Location and X-Versions-Mode: 'history'
Newer than [2]:
X-History-Location
Note that this functional tests work on newer swift than [2].
And then, this patch also sets allow_versioned_writes=True
for in-process testing (the container server allow_versions
option was already set, so this is just enabling in the middleware
too). That means that in-process functional tests (such as run by
the tox envs func-in-process-*) because history mode requires the
middleware allow_versioned_writes option to be explicity set to True.
1: https://review.openstack.org/#/c/214922/
2: https://review.openstack.org/#/c/373537/
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Related-Change: I555dc17fefd0aa9ade681aa156da24e018ebe74b
Related-Change: Icfd0f481d4e40dd5375c737190aea7ee8dbc3bf9
Change-Id: Ifebc1c3ce558b1df9e576a58a4100f2219dfc7e7
Since COPY allows a copy of a ranged GET, add some more test
coverage for that in both unit and functional tests.
Drive-by fix to use better test assertion methods.
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Change-Id: I5cb202386df0862f953f7388107c4d3466e2e46c
Currently a versioned write PUT uses a pre-authed request to move
it into the versioned container before checking whether the
user is authorised. This can lead to some interesting behaviour
whereby a user can select a versioned object path that it does not
have access to, request a put on that versioned object, and this
request will execute the copy part of the request before it fails
due to lack of permissions.
This patch changes the behaviour to be the same as versioned DELETE
where the request is authorised before anything is moved.
Change-Id: Ia8b92251718d10b1eb44a456f28d3d2569a30003
Closes-Bug: #1562175
Currently when using fast-post, the manifest is updated with the given
'x-object-manifest' header on a POST. If no such header is supplied,
then the manifest will change to a regular object.
This is not currently true when using post-as-copy.
This patch changes the DLO POST using post-as-copy behavior to match
that of using fast-post. It was also documented that
'x-object-manifest' must be provided on a POST to a manifest file.
Change-Id: Ie1143ab1a2c8f8c21e258a36badbff5d947769d4
Closes-bug: 1612991
Bonus consistency: 416 responses now always have a body. Before, if
you had "swob.HTTPRequestedRangeNotSatisfiable()", you'd get a body,
but if you had "swob.Response(..., conditional_response=True)", then
you'd get a length-0 response body. Now you always get a response
body. It's just the default <html><h1>..., but at it's always there.
Bonus efficiency: do a little caching of sub-SLO manifests to avoid
needless re-fetches. This kicks in when there are multiple references
to the same sub-SLO in a given manifest. The caching only holds 20
sub-SLOs so that a malicious user can't build a giant SLO tree and use
it to run the proxy out of memory (we're already holding up to 10
manifests in memory at a time since a SLO can include another SLO to a
depth of 10; this doesn't make the situation too much worse).
Change-Id: I24716e3271cf3370642e3755447e717fd7d9957c
In the past, a POST to a DLO manifest file when object_post_as_copy
was true resulted in the manifest file contents being replaced
by the concatenation of the DLO segments. This no longer
happens, but tests for this case are missing.
This patch adds a functional test to assert that the manifest
file is preserved in a POST request.
Change-Id: I90546014a7dcc7266f0d0e0ff6339688b7954b96
Related-bug: #1487791
Related-bug: #1514317
Previously, if a container listing produced `subdir` elements
the decrypter would raise a KeyError.
Additionally, update the functests so this sort of thing would
have been caught at the gate.
Closes-Bug: 1609904
Change-Id: Idc1907d19f90af7a086f45f8faecee9fbc3c69c2
I was watching the logs and noticed it creates an object named:
"<bound method type.create_ascii_name of <class 'test.functional.tests.Utils'>>"
Change-Id: I8dcbb40125b84a914e3c01566ae9c3f08dc9ea0f
Fix copy middleware so that all client-defined object
headers that object servers allow to be persisted are copied.
For example, content-encoding and content-disposition will
now be copied.
Fix treatment of x-fresh-metadata header so that, when it is
used, new object sysmeta is applied to the object copy in the
same way as a copy without x-fresh-metadata.
Remove unnecessary passing of original request headers to
sink PUT request constructor: passing the environ is sufficient
to have the new request inherit the original's headers.
Add tests for this change and to verify that content-type
gets either copied or updated if supplied with the copy
request.
Add tests for x-fresh-metadata treatment.
Closes-Bug: #1391826
Closes-Bug: #1600247
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Change-Id: I917fb0b4e831c13e04ade1c5e0b9821802dec967
I changed asserts with more specific assert methods.
e.g.: from assertTrue(sth == None) to assertIsNone(*) or
assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type) or
assertTrue(not sth) to assertFalse(sth).
The code gets more readable, and a better description will be shown on fail.
Change-Id: I80ec96e0b729bef38213a6be4ff4b6eb65c7612d
The varialbe 'size' in TestFile.testMetadataNumberLimit is not used.
This patch remove the variable from the test.
Change-Id: I255a1dcee12bb6b8dec6ff26ed7edf93ab2acf64
Relocates some test infrastructure in preparation for
use with encryption tests, in particular moves the test
server setup code from test/unit/proxy/test_server.py
to a new helpers.py so that it can be re-used, and adds
ability to specify additional config options for the
test servers (used in encryption tests).
Adds unit test coverage for extract_swift_bytes and functional
test coverage for container listings. Adds a check on the content
and metadata of reconciled objects in probe tests.
Change-Id: I9bfbf4e47cb0eb370e7a74d18c78d67b6b9d6645
This patch enables to show a x-put-timestamp as
a last-modified header in container-server.
Note that the last-modified header will be changed only when a
request for container (PUT container or POST container) comes into
Swift. i.e. some requests for objects (e.g. PUT object, POST object)
will never affect the last-modified value but only when using
python-swiftclient like as "swift upload", the last-modified will
be close to the upload time because python-swiftclient will make
a PUT container request for "swift upload" each time.
Change-Id: I9971bf90d24eee8921f67c02b7e2c80fd8995623
When copying an SLO manifest with multipart-manifest=get the actual
manifest content-type should get copied to the destination, rather
than the application/json value that is synthesised by SLO in a GET
response. That way the result of a HEAD on the copied manifest is the
same as a HEAD to the source, and the container listings for the two
are consistent.
This patch also un-skips a functional test and adds functional tests
that verify this patch and also verify that etags and size also get
correctly copied and updated in destination container (bug #1260446).
Closes-Bug: #1260446
Closes-Bug: #1583756
Change-Id: Ie7fa82f70b3ec3ef568f5355c69f6bce460ba25d
The testPUT case is failing when keystone was enabled and
allow_account_management is set to True. There were a few issues
needing addressed.
First the case was renamed to call out what it was actually doing which
is verifying an error scenario for which a PUT on a storage account was
not allowed.
Second the case was running even when allow_account_management is
enabled, which is incorrect. It "accidently" works with TempAuth
because it requires a reseller permission, so the Keystone failure here
has more to do with not requiring a reseller permission to do a PUT on a
storage account for which a user has an operator role on.
The common sense fix here is to not execute this test case when
allow_account_management is enabled.
Change-Id: Id29f5ca48f92cd139535be7064107b8a61b02856
Rewrite server side copy and 'object post as copy' feature as middleware to
simplify the PUT method in the object controller code. COPY is no longer
a verb implemented as public method in Proxy application.
The server side copy middleware is inserted to the left of dlo, slo and
versioned_writes middlewares in the proxy server pipeline. As a result,
dlo and slo copy_hooks are no longer required. SLO manifests are now
validated when copied so when copying a manifest to another account the
referenced segments must be readable in that account for the manifest
copy to succeed (previously this validation was not made, meaning the
manifest was copied but could be unusable if the segments were not
readable).
With this change, there should be no change in functionality or existing
behavior. This is asserted with (almost) no changes required to existing
functional tests.
Some notes (for operators):
* Middleware required to be auto-inserted before slo and dlo and
versioned_writes
* Turning off server side copy is not configurable.
* object_post_as_copy is no longer a configurable option of proxy server
but of this middleware. However, for smooth upgrade, config option set
in proxy server app is also read.
DocImpact: Introducing server side copy as middleware
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Change-Id: Ic96a92e938589a2f6add35a40741fd062f1c29eb
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Previously, if object versioning was enabled via the old-style
allow_versions container-server setting rather than the new-style
allow_versioned_writes proxy-server setting,
TestCrossPolicyObjectVersioning would skip tests while
TestObjectVersioning and TestObjectVersioningUTF8 would run them.
Additionally, if versioned_writes was explicitly included in the
proxy-server's pipeline and allow_versioned_writes was disabled,
the functional tests would fail with a 412.
Now, all three will use the same logic to check whether versioning is
enabled. Specifically, they will all try to set an X-Versions-Location
header and skip if it doesn't stick.
Additionally, the TestCrossPolicyObjectVersioningEnv will now properly
clean up after itself.
Change-Id: I4c788a0e18587ff17d3c6e346fd22b881495f06d
Previously, attempting to PUT a SLO manifest with `If-None-Match: *`
would include the header when validating the segments, causing the
upload to fail.
Now when SLO validates segments, no conditional headers will be
included in the HEAD request.
Change-Id: I03ad454092d3caa73d29e6d30d8033b45bc96136
Closes-Bug: #1569253
This change removes the use of the COPY request in the versioned
writes middleware. It changes the COPY verb for GETs and PUTs
requests. The main reasoning for this change is to remove any
dependency that versioning had on copy, which will allow for the COPY
functionality to be moved to middleware and to be to the left of the
versioned writes middleware in the proxy pipeline. In this way,
no COPY request will ever arrive at the versioned writes middleware.
A side benefit of this change is that it removes a HEAD request from
the PUT path. Instead of checking if a current version exists, a
GET request is sent, in case of success, a PUT is sent to the
versions container.
A unit test was removed that tested non-default storage policies.
This test is no longer necessary, since it was used to test
specific policy handling code in the COPY method in the proxy
object controller.
Closes-Bug: #1365862
Change-Id: Idf34fa8d04ff292df7134b6d4aa94ff40887b3a4
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Currently, the multipart-manifest=get call returns output in json
format that is inconsistent with the format that is used for the
multipart-manifest=put.
This in turn introduces a new call: ?multipart-manifest=get&format=raw
Change-Id: I2242943a738f667cbda6363bcb6a017f341e834f
Closes-Bug: 1252482
When using an EC policy, 304 responses to conditional GETs
are missing the Accept-Ranges header and have the wrong ETag
value. 412 responses also have the wrong etag.
416 responses to ranged GETs also have the wrong ETag.
This patch ensures behaviour with EC policy is consistent
with replication policy:
- 304 and 416 responses have correct etag and Accept-Ranges
- 412 responses have correct Etag but no Accept-Ranges
Co-Authored-By: Mahati Chamarthy <mahati.chamarthy@gmail.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Closes-Bug: #1496234
Closes-Bug: #1558197
Closes-Bug: #1558193
Change-Id: Ic21317b9e4f632f0751133a3383eb5487379e11f
This patch makes a number of changes to enable content-type
metadata to be updated when using the fast-POST mode of
operation, as proposed in the associated spec [1].
* the object server and diskfile are modified to allow
content-type to be updated by a POST and the updated value
to be stored in .meta files.
* the object server accepts PUTs and DELETEs with older
timestamps than existing .meta files. This is to be
consistent with replication that will leave a later .meta
file in place when replicating a .data file.
* the diskfile interface is modified to provide accessor
methods for the content-type and its timestamp.
* the naming of .meta files is modified to encode two
timestamps when the .meta file contains a content-type value
that was set prior to the latest metadata update; this
enables consistency to be achieved when rsync is used for
replication.
* ssync is modified to sync meta files when content-type
differs between local and remote copies of objects.
* the object server issues container updates when handling
POST requests, notifying the container server of the current
immutable metadata (etag, size, hash, swift_bytes),
content-type with their respective timestamps, and the
mutable metadata timestamp.
* the container server maintains the most recently reported
values for immutable metadata, content-type and mutable
metadata, each with their respective timestamps, in a single
db row.
* new probe tests verify that replication achieves eventual
consistency of containers and objects after discrete updates
to content-type and mutable metadata, and that container-sync
sync's objects after fast-post updates.
[1] spec change-id: I60688efc3df692d3a39557114dca8c5490f7837e
Change-Id: Ia597cd460bb5fd40aa92e886e3e18a7542603d01
test/functional/tests.py:TestObjectVersioningUTF8 does not clean
up the versions files it creates because the class's multiple
inheritance is such that it does not call the tearDown method in
TestObjectVersioning.
As a result, any attempt to clean up account containers wastes
time retrying container delete requests. This occurs either in
the setUp for TestSloEnv, if the TestSlo class is included in a
test run, or in the tests.py package tearDown method.
On the author's dev machine this patch reduces the execution
time of functional tests in tests.py by approx 30% or 1 minute.
Change-Id: I8194672bf2ca82435df5868720b6a55a79b94413
It feels silly, and we don't do it for any of the other headers in
headers_to_container_info.
While we're at it, clean up a stray '
Change-Id: I0745038cc3832a77d064e515c37cacbdcb97c4d9
Related-Change: Iea3d06de80210e9e504e296d4572583d7ffabeac
