This change allows individual SLO segments to be downloaded by adding
an extra 'part-number' query parameter to the GET request. You can
also retrieve the Content-Length of an individual segment with a HEAD
request.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I7af0dc9898ca35f042b52dd5db000072f2c7512e
md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.
md5 is allowed when in a non-security context. There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.
In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.
Some downstream python versions already support this parameter. To
support these versions, a new encapsulation of md5() is added to
swift/common/utils.py. This encapsulation is identical to the one being
added to oslo.utils, but is recreated here to avoid adding a dependency.
This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.
While this patch seems large, it is really just the same change over and
again. Reviewers need to pay particular attention as to whether the
keyword parameter (usedforsecurity) is set correctly. Right now, all
of them appear to be not used in a security context.
Now that all the instances have been converted, we can update the bandit
run to look for these instances and ensure that new invocations do not
creep in.
With this latest patch, the functional and unit tests all pass
on a FIPS enabled system.
Co-Authored-By: Pete Zaitcev
Change-Id: Ibb4917da4c083e1e094156d748708b87387f2d87
When client sends a '?multipart-manifest=get&format=raw' request
middleware will change the manifest returned from object server.
This patch makes sure the response etag is updated to reflect
changes to manifest content
Change-Id: I0ac6dd0808fb041ba7663f4a472a06ee3f1d9a71
Some tests make use of multiple accounts without checking of they have
been set up. This commit tries to fix some of these situations.
Change-Id: I461679e78e19ce0866c7618c581a8cb573cca7f5
Container servers will store an etag like
<MD5 of manifest on disk>; slo_etag=<MD5 on concatenated ETags>
which the SLO middleware will break out into separate
"hash": "<MD5 of manifest on disk",
"slo_etag": "\"<MD5 of concatenated ETags\"",
keys for JSON listings. Text and XML listings are unaffected.
If a middleware left of SLO already specified a container update
override, the slo_etag parameter will be appended. If the base header
value was blank, the MD5 of the manifest will be inserted.
SLOs that were created on previous versions of Swift will continue to
just have the MD5 of the manifest in container listings.
Closes-Bug: 1618573
Change-Id: I67478923619b00ec1a37d56b6fec6a218453dafc
This patch updates the SLO middleware and SegmentedIterable to add
support for user-specified inlined-data segments. Such segments will
contain base64-encoded data to be added before/after an object-backed
segment within an SLO. To accommodate the potential extra data we
increase the default SLO maximum manifest size from 2MiB to 8MiB.
The default maximum number of segments remains 1000, but this will
only be enforced for object-backed segments.
This patch is a prerequisite for a future patch enabling the
download of large objects as tarballs. The TLO patch will be added
as a dependent patch later.
UpgradeImpact
=============
During a rolling upgrade, an updated proxy may write a manifest that
out-of-date proxies will not be able to read. This will resolve itself
once the upgrade completes on all nodes.
Change-Id: Ib8dc216a84d370e6da7d6b819af79582b671d699
Why weren't we doing that before?? The etag should be the same as for
GET/HEAD, and by sending it, we can assure resuming clients that they're
downlading the same object even if they didn't include an If-Match
header.
Change-Id: I4ccbd1ae3a909ecb4606ef18211d1b868f5cad86
Related-Change: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1
An SLO PUT requires that we HEAD every referenced object; as a result, it
can be a very time-intensive operation. This makes it difficult as a
client to differentiate between a proxy-server that's still doing work and
one that's crashed but left the socket open.
Now, clients can opt-in to receiving heartbeats during long-running PUTs
by including the query parameter
heartbeat=on
With heartbeating turned on, the proxy will start its response immediately
with 202 Accepted then send a single whitespace character periodically
until the request completes. At that point, a final summary chunk will be
sent which includes a "Response Status" key indicating success or failure
and (if successful) an "Etag" key indicating the Etag of the resulting SLO.
This mechanism is very similar to the way bulk extractions and deletions
work, and even the way SLO behaves for ?multipart-manifest=delete requests.
Note that this is opt-in: this prevents us from sending the 202 response
to existing clients that may mis-interpret it as an immediate indication
of success.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Related-Bug: 1718811
Change-Id: I65cee5f629c87364e188aa05a06d563c3849c8f3
While we're at it, have copy and copy_account raise ResponseErrors
on failure, similar to cluster_info, update_metadata, containers, info,
files, delete, initialize, read, sync_metadata, write, and post.
Related-Change: Ia8b92251718d10b1eb44a456f28d3d2569a30003
Change-Id: I9ef42d922a6b7dbf253f2f8f5df83965d8f47e0f
SLOs created after commit 2d25fe6ad3573b2a06b6b3e5e66493d7b0c55693
have an explicit X-Object-Sysmeta-SLO-Size attribute. Fetching one of
those with a range like "bytes=100-" would crash and cause a 500. This
commit fixes the crash.
Closes-Bug: #1656147
Change-Id: I174ed98b4f01a5943af12177d6f8c3586f049778
With a multipart-manifest PUT request, if client sends the md5 of the
segments' etags, a 422 Unprocessable Entity response is returned. This
patch fixes that and confirms the etag
Change-Id: I4598a2a3f16ca8727bb07bbb6d8efcfcae777796
Closes-Bug: #1213200
Co-Authored-By: Tim Burke <tim@swiftstack.com>
Addresses a TODO in test/functional/test_account.py where
an account metadata test was having to clean up tempurl keys
in the account metadata that were left by another test in
a different module. This cleanup is necessary because tests
in test_account.py fail if there is any pre-existing
account metadata.
This patch:
* makes the tempurl tests clean up their keys from account
metadata.
* makes the test_account.py:TestAccount class remove any
pre-existing metadata before attempting any tests and
replacing that metadata when all the tests in that class
have completed. This is more robust than the existing code
which only removes any tempurl keys that might be in the
account - now you could have x-account-meta-foo = bar in
the test account and test_account.py will still pass.
* consolidates some common setup code currently repeated for
many of the functional test classes into into a BaseEnv class.
Change-Id: I874a9e23dfcdd1caa934945b46089f11b9f6de65
Previously, we still required that clients send "etag" and "size_bytes"
keys in their segment definitions. This was done as a way to guard
against typos leading to an accidental lack of verification.
However, typos should already be caught when we check for extra keys. As
a result, the only truly required key is "path".
Change-Id: Ie1d8691115f8c68b5a3f3b59317cdab59f9a3fca
This patch removes the slo, dlo and tempurl tests from tests.py
As before, this patch does not refactor any code it's only a copy
and paste to make it easy to review
Change-Id: I0b49d4f8bc7dd204d06258f4910cfc9a5ba4bdcb
Signed-off-by: Thiago da Silva <thiago@redhat.com>
