This patch is follow up for [1] and [2] to add new functional
tests for versioned_writes middlware 'history' mode.
(i.e. using X-History-Location header to a container).
The new test class, TestObjectHistoryModeVersioning, will use obvious
setting the mode via new X-History-Location header, since the change [2],
the setting X-Versions-Mode header added since [1] for incomming request has
been deprecated. Hence, since [2], the syntax for stack mode is back to
be same with older Swift than [1] so that the only thing we need now is
just adding a test suite for the new X-History-location.
It means the API has been changing like:
---------------
For stack mode:
---------------
Older than [1]:
X-Versions-Location
[1]~[2]:
X-Vesions-Location (and X-Versions-Mode: 'stack' for obvious)
Newer than [2]:
X-Vesions-Location
-----------------
For history mode:
-----------------
Older than [1]:
(Not supported)
[1]~[2]:
X-Vesions-Location and X-Versions-Mode: 'history'
Newer than [2]:
X-History-Location
Note that this functional tests work on newer swift than [2].
And then, this patch also sets allow_versioned_writes=True
for in-process testing (the container server allow_versions
option was already set, so this is just enabling in the middleware
too). That means that in-process functional tests (such as run by
the tox envs func-in-process-*) because history mode requires the
middleware allow_versioned_writes option to be explicity set to True.
1: https://review.openstack.org/#/c/214922/
2: https://review.openstack.org/#/c/373537/
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Related-Change: I555dc17fefd0aa9ade681aa156da24e018ebe74b
Related-Change: Icfd0f481d4e40dd5375c737190aea7ee8dbc3bf9
Change-Id: Ifebc1c3ce558b1df9e576a58a4100f2219dfc7e7
It has valid_timestamp() to check timestamps in /POST/PUT/DELETE methods.
It has test_POST_bad_timestamps in Post, but it has no test_PUT/DELETE_bad_timestamps
int PUT/DELETE. Therefore,I add them
Change-Id: Ib531f3b7819c1c8dc69a7b51d990581bf1e24dab
Cleans up some auditor tests added in the Related-Change.
Make auditor NOT call invalidate_hash for a reclaimable
tombstone in a zero bytes file process, so that
invalidate_hash is only called once per reclaimable
tombstone per auditor cycle.
Previously each execution of 'swift-init object-auditor once'
would result in two identical entries being appended to
hashes.invalid for each reclaimable tombstone. With this change
that unnecessary duplication is removed.
Related-Change: I3e99dc702d55a7424c6482969e03cb4afac854a4
Change-Id: I5dfaa8d74c07ca8a494c29159c1a2bed39499613
I don't think this is a real bug - just that the mocked iter wasn't
closing it subiters like the real iter does.
Change-Id: I44c8159f9eea8737bc86b6c7eb59a512e57e86c1
As mentioned in link[1], if we need filter on python3,
Raplace filter(lambda obj: test(obj), data) with:
[obj for obj in data if test(obj)].
[1] https://wiki.openstack.org/wiki/Python3
Change-Id: Ia1ea2ec89e4beb957a4cb358b0d0cef970f23e0a
Now, instead of saying
X-Versions-Location: <container>
X-Versions-Mode: history
clients should just say
X-History-Location: <container>
Since we've never had a release featuring a user-settable
X-Versions-Mode header, support may be dropped and that is now ignored.
Change-Id: Icfd0f481d4e40dd5375c737190aea7ee8dbc3bf9
- Call invalidate_hash in auditor for reclaimable tombstones
- assert changed auditor behavior with a unit test
- driveby test: assert get_hashes behavior with a unit test
Co-Authored-By: Pete Zaitcev <zaitcev@redhat.com>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Closes-Bug: #1301728
Change-Id: I3e99dc702d55a7424c6482969e03cb4afac854a4
test_GET_HEAD_with_fragment_preferences seemed to fail on ocassion on
my machine; the test seemed to be assuming the order of dictionary
keys in json output sent in headers; the fix was convert back to a
dictionary when making the assertion on the context of the json.
Change-Id: I1ea1b734c2a9fb12b8f59262bb1229421803e48e
Closes-Bug: #1625956
Clarify Content-Type header definition for listings.
Distinguish between request and response definitions for
X-Account-Meta-Temp-URL-Key* headers.
Insert X-Container-Meta-Quota-* headers missing from some
request/response definitions.
Improve X-Container-Meta-Access-Control-Expose-Headers
parameter formatting.
Distinguish between request and response definitions for
X-Container-Meta-Temp-URL-Key* headers.
Co-Authored-By: Christian Schwede <cschwede@redhat.com>
Change-Id: I8fc7610395973b520aa9ddd72c94e1eb75f602cd
Related-Change: I315b4e550b7d10880fbc00fce9311127ba609c2d
If we're going to talk about replacing an object, we should use the
same object name as the previous example.
Including a non-zero content-length on PUT but not providing a body
will timeout.
Not including the '-' in '-H' will make curl complain:
Rebuilt URL to: H/
* Could not resolve host: H
* Closing connection 1
201 Created responses have content-length of zero, not 116.
Change-Id: Ifd878559ee4036e4893221c7968f53021f38e236
Move the account listing sample responses to follow the sample
requests, and to precede the request/response parameter definitions to
be consistent with other parts of the doc.
Related-Change: I315b4e550b7d10880fbc00fce9311127ba609c2d
Change-Id: Ia20acacd238db4a423b8cd89af1658222b4c5828
Move repeated test re metadata header syntax to an include
file and make it be rendered as a note.
Also make already included text about metadata header value
encoding be a note.
Change-Id: I4795836587492954ad24dd5baaa5d668746d6040
This is *only* sysmeta, to ensure we neither change the semantics of the
response (which could happen if we included allowed_headers) nor reveal
previously-written metadata to write-only clients (which could happen if
we ever get updateable object metadata and sent back user meta).
We could conceivably send back transient sysmeta, though it seems
better to wait for a use-case that demands it.
Change-Id: I04fe0b36b85e546b5379bed12089ffd1bce01fcf
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
F812 list comprehension redefines <variable> from line ...
While the current violations were benign, this sort of code can easily
lead to subtle bugs. Seems worth checking, especially given how cheap it
is to bring existing code in line with it.
Change-Id: Ibdcf9f93b85a1f1411198001df6bdbfa8f92d114
This commit updates the Swift Bandit file to the new style
introduced in Bandit 1.0. In response to the struggle with
getting a Bandit config file working and kept up to date we
introduced a simplified version in Bandit 1.0.
This commit updates Swift's bandit.yaml to use the new version.
Change-Id: Ida5dd08f4ea72a377346f2159caeb2f3741d4980
This patch improves EC GET response handling:
- The proxy no longer requires all object servers to have a
durable file for the fragment archive that they return in
response to a GET. The proxy will now be satisfied if just
one object server has a durable file at the same timestamp
as fragments from other object servers.
This means that the proxy can now successfully GET an
object that had missing durable files when it was PUT.
- The proxy will now ensure that it has a quorum of *unique*
fragment indexes from object servers before considering a
GET to be successful.
- The proxy is now able to fetch multiple fragment archives
having different indexes from the same node. This enables
the proxy to successfully GET an object that has some
fragments that have landed on the same node, for example
after a rebalance.
This new behavior is facilitated by an exchange of new
headers on a GET request and response between the proxy and
object servers.
An object server now includes with a GET (or HEAD) response:
- X-Backend-Fragments: the value of this describes all
fragment archive indexes that the server has for the
object by encoding a map of the form: timestamp -> <list
of fragment indexes>
- X-Backend-Durable-Timestamp: the value of this is the
internal form of the timestamp of the newest durable file
that was found, if any.
- X-Backend-Data-Timestamp: the value of this is the
internal form of the timestamp of the data file that was
used to construct the diskfile.
A proxy server now includes with a GET request:
- X-Backend-Fragment-Preferences: the value of this
describes the proxy's current preference with respect to
those fragments that it would have object servers
return. It encodes a list of timestamp, and for each
timestamp a list of fragment indexes that the proxy does
NOT require (because it already has them).
The presence of a X-Backend-Fragment-Preferences header
(even one with an empty list as its value) will cause the
object server to search for the most appropriate fragment
to return, disregarding the existence or not of any
durable file. The object server assumes that the proxy
knows best.
Closes-Bug: 1469094
Closes-Bug: 1484598
Change-Id: I2310981fd1c4622ff5d1a739cbcc59637ffe3fc3
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
