When using the copy-part API it is expected for s3api to write down an
empty value for X-Object-Sysmeta-S3Api-Etag on segments. This was
ostensibly to prevent writing down an unrelated S3Api-Etag when copying
a part from another MPU the copy transfers object sysmeta. We should
assume a S3Api-Etag w/o X-Static-Large-Object is non-sense, and SLO
should forever expect empty values for it's sysmeta.
Drive-By: consolidate handling of boto2 sigv4 skips
Related-Bug: #2035158
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ic6f04a5a6af8a3e65b226cff2ed6c9fce8ce1fa2
We need to support the aforementioned headers in our s3 apis
and raise an InvalidArgumentError if a s3 client makes a request
Change-Id: I2c5b18e52da7f33b31ba386cdbd042f90b69ef97
Previously, this would trip TypeErrors on py310:
TypeError: 'S3Timestamp' object cannot be interpreted as an integer
Change-Id: I124c1957264c80d28a6b3e852d042cbc8468939c
s3api bucket listing elements currently have LastModified values with
millisecond precision. This is inconsistent with the value of the
Last-Modified header returned with an object GET or HEAD response
which has second precision. This patch reduces the precision to
seconds in bucket listings and upload part listings. This is also
consistent with observation of an aws listing response.
The last modified values in the swift native listing *up* to
the nearest second to be consistent with the seconds-precision
Last-Modified time header that is returned with an object GET or HEAD.
However, we continue to include millisecond digits set to 0 in the
last-modified string, e.g.: '2014-06-10T22:47:32.000Z'.
Also, fix the last modified time returned in an object copy response
to be consistent with the last modified time of the object that was
created. Previously it was rounded down, but it should be rounded up.
Change-Id: I8c98791a920eeedfc79e8a9d83e5032c07ae86d3
Trying to copy an object with non-ASCII characters in its name results
in, depending on the pipeline:
- an error code 412 because of a badly urlencoded path
- an error code 500 "TypeError: Expected a WSGI string"
This commit fixes the problem by calling str_to_wsgi on the object name
after it has been urldecoded. We do not need to call this on the
container name because it is supposed to contain only ASCII characters.
Change-Id: If837d4e55735b10a783c85d91f37fbea5e3baf1d
md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.
md5 is allowed when in a non-security context. There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.
In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.
Some downstream python versions already support this parameter. To
support these versions, a new encapsulation of md5() is added to
swift/common/utils.py. This encapsulation is identical to the one being
added to oslo.utils, but is recreated here to avoid adding a dependency.
This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.
While this patch seems large, it is really just the same change over and
again. Reviewers need to pay particular attention as to whether the
keyword parameter (usedforsecurity) is set correctly. Right now, all
of them appear to be not used in a security context.
Now that all the instances have been converted, we can update the bandit
run to look for these instances and ensure that new invocations do not
creep in.
With this latest patch, the functional and unit tests all pass
on a FIPS enabled system.
Co-Authored-By: Pete Zaitcev
Change-Id: Ibb4917da4c083e1e094156d748708b87387f2d87
unittest2 was needed for Python version <= 2.6, so it hasn't been needed
for quite some time. See unittest2 note one:
https://docs.python.org/2.7/library/unittest.html
This drops unittest2 in favor of the standard unittest module.
Change-Id: I2e787cfbf1709b7f9c889230a10c03689e032957
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Previously, the 'x-amz-metadata-directive' header was ignored except to
ensure that it had a valid value if it existed. In all cases any
metadata specified was applied to the copied object, while
non-conflicting metadata remained.
This patch fixes this behaviour.
Now, if the 'x-amz-metadata-directive' header is set to 'REPLACE'
during a copy operation, the s3api middleware sets the
'x-fresh-metadata' header to 'True' to replace valid metadata values.
If the 'x-amz-metadata-directive' header is set to 'COPY' or if it is
omited during a copy operation, then the s3api middleware removes all
metadata (custom or not) from the request to prevent it from being
changed.
Content-Type can never be set on an S3 copy operation, even if the
metadata directive is set to 'REPLACE', so it is specifically filtered
out on copy.
Change-Id: I333e46758bd2b7a29f672c098af267849232c911
Closes-Bug: #1433875
On vanilla Swift, deleting an object that doesn't exist will 404.
On AWS, deleting a key that doesn't exist will either 404 if the bucket
doesn't exist (with a NoSuchBucket code) or 204 (because yep, that's not
accessible).
Change-Id: Ied2a78b56522316bb374f23961621641af3adc83
Related-Change: I6e154594dfda6c3065774c23b24f728625a842bc
This attempts to import openstack/swift3 package into swift upstream
repository, namespace. This is almost simple porting except following items.
1. Rename swift3 namespace to swift.common.middleware.s3api
1.1 Rename also some conflicted class names (e.g. Request/Response)
2. Port unittests to test/unit/s3api dir to be able to run on the gate.
3. Port functests to test/functional/s3api and setup in-process testing
4. Port docs to doc dir, then address the namespace change.
5. Use get_logger() instead of global logger instance
6. Avoid global conf instance
Ex. fix various minor issue on those steps (e.g. packages, dependencies,
deprecated things)
The details and patch references in the work on feature/s3api are listed
at https://trello.com/b/ZloaZ23t/s3api (completed board)
Note that, because this is just a porting, no new feature is developed since
the last swift3 release, and in the future work, Swift upstream may continue
to work on remaining items for further improvements and the best compatibility
of Amazon S3. Please read the new docs for your deployment and keep track to
know what would be changed in the future releases.
Change-Id: Ib803ea89cfee9a53c429606149159dd136c036fd
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
