md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.
md5 is allowed when in a non-security context. There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.
In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.
Some downstream python versions already support this parameter. To
support these versions, a new encapsulation of md5() is added to
swift/common/utils.py. This encapsulation is identical to the one being
added to oslo.utils, but is recreated here to avoid adding a dependency.
This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.
While this patch seems large, it is really just the same change over and
again. Reviewers need to pay particular attention as to whether the
keyword parameter (usedforsecurity) is set correctly. Right now, all
of them appear to be not used in a security context.
Now that all the instances have been converted, we can update the bandit
run to look for these instances and ensure that new invocations do not
creep in.
With this latest patch, the functional and unit tests all pass
on a FIPS enabled system.
Co-Authored-By: Pete Zaitcev
Change-Id: Ibb4917da4c083e1e094156d748708b87387f2d87
unittest2 was needed for Python version <= 2.6, so it hasn't been needed
for quite some time. See unittest2 note one:
https://docs.python.org/2.7/library/unittest.html
This drops unittest2 in favor of the standard unittest module.
Change-Id: I2e787cfbf1709b7f9c889230a10c03689e032957
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
... by embedding something like `If-Match: <etag>` semantics in the
symlink.
When creating a symlink, users may now specify an optional
X-Symlink-Target-Etag header. If present, the etag of the final object
returned to the client will be checked; if it does not match the
X-Symlink-Target-Etag header, a 409 Conflict error will be returned to
the client.
Note that, unlike "dynamic symlink" behavior, the target object must
exist with the matching Etag for the "static symlink" to be created.
Since we're validating the Etag anyway, we also set the content-type of
the symlink to match if the client didn't otherwise specifiy and send
the etag & content-length along to the container listing as well.
Bonus goodness:
- Tighten assertions on Content-Location
- Get rid of swift.source-sniffing by making versioned_writes
symlink-aware ('cause I'm going to want to make it symlink-aware
later anyway)
- Allow middlewares left of symlink to set their own
Container-Update-Override-Etag when creating a symlink
- Set dynamic symlink content type if client doesn't supply something
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I179ea6180d31146bb947061c69b1807c59529ac8
Also, ensure that the stored symlink headers really *are* url-encoded as
we've been assuming.
Change-Id: I1f300d69bec43f0deb430294da05a4ec04308040
Related-Bug: 1774238
Closes-Bug: #1821240
Some tests make use of multiple accounts without checking of they have
been set up. This commit tries to fix some of these situations.
Change-Id: I461679e78e19ce0866c7618c581a8cb573cca7f5
Container servers will store an etag like
<MD5 of manifest on disk>; slo_etag=<MD5 on concatenated ETags>
which the SLO middleware will break out into separate
"hash": "<MD5 of manifest on disk",
"slo_etag": "\"<MD5 of concatenated ETags\"",
keys for JSON listings. Text and XML listings are unaffected.
If a middleware left of SLO already specified a container update
override, the slo_etag parameter will be appended. If the base header
value was blank, the MD5 of the manifest will be inserted.
SLOs that were created on previous versions of Swift will continue to
just have the MD5 of the manifest in container listings.
Closes-Bug: 1618573
Change-Id: I67478923619b00ec1a37d56b6fec6a218453dafc
... and add support for SHA-256 and SHA-512 by default. This allows us
to start moving toward replacing SHA-1-based signatures. We've known
this would eventually be necessary for a while [1], and earlier this
year we've seen SHA-1 collisions [2].
Additionally, allow signatures to be base64-encoded, provided they start
with a digest name followed by a colon. Trailing padding is optional for
base64-encoded signatures, and both normal and "url-safe" modes are
supported. For example, all of the following SHA-1 signatures are
equivalent:
da39a3ee5e6b4b0d3255bfef95601890afd80709
sha1:2jmj7l5rSw0yVb/vlWAYkK/YBwk=
sha1:2jmj7l5rSw0yVb/vlWAYkK/YBwk
sha1:2jmj7l5rSw0yVb_vlWAYkK_YBwk=
sha1:2jmj7l5rSw0yVb_vlWAYkK_YBwk
(Note that "normal" base64 encodings will require that you url encode
all "+" characters as "%2B" so they aren't misinterpretted as spaces.)
This was done for two reasons:
1. A hex-encoded SHA-512 is rather lengthy at 128 characters -- 88
isn't *that* much better, but it's something.
2. This will allow us to more-easily add support for different
digests with the same bit length in the future.
Base64-encoding is required for SHA-512 signatures; hex-encoding is
supported for SHA-256 signatures so we aren't needlessly breaking from
what Rackspace is doing.
[1] https://www.schneier.com/blog/archives/2012/10/when_will_we_se.html
[2] https://security.googleblog.com/2017/02/announcing-first-sha1-collision.html
Change-Id: Ia9dd1a91cc3c9c946f5f029cdefc9e66bcf01046
Related-Bug: #1733634
The functional test for versioning symlinks is better located in
test_versioned_writes where it can be added to
TestObjectVersioning. This saves duplicated versioned_writes specific
setup code in test_symlink, and has the benefit of the test being
repeated for each of the versioned writes test subclasses. With a
small refactor this includes the test now running with
x-history-location mode as well as x-versions-location mode.
Related-Change: I838ed71bacb3e33916db8dd42c7880d5bb9f8e18
Change-Id: If215446c558b61c1a8aea37ce6be8fcb5a9ea2f4
Add a symbolic link ("symlink") object support to Swift. This
object will reference another object. GET and HEAD
requests for a symlink object will operate on the referenced object.
DELETE and PUT requests for a symlink object will operate on the
symlink object, not the referenced object, and will delete or
overwrite it, respectively.
POST requests are *not* forwarded to the referenced object and should
be sent directly. POST requests sent to a symlink object will
result in a 307 Error.
Historical information on symlink design can be found here:
https://github.com/openstack/swift-specs/blob/master/specs/in_progress/symlinks.rst.
https://etherpad.openstack.org/p/swift_symlinks
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Co-Authored-By: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Change-Id: I838ed71bacb3e33916db8dd42c7880d5bb9f8e18
Signed-off-by: Thiago da Silva <thiago@redhat.com>
