Depending on how the key was stored in Barbican, it may come out of
Castellan as a native string, which would not be suitable on Python 3.
Now, check that the secret is a byte string, and if it isn't, encode as
UTF-8 (to match Barbican's internal encoding).
Change-Id: I6da047716c05e4f2a9e1e74ca19afb62e812d172
Closes-Bug: #1847755
Several tools are returning a misleading error message if swift.conf is
missing or not readable by the user, stating that the hash pre-/suffixes
are missing. Let's fix this by catching the real issue down below.
Change-Id: I7a47e6260ed51a3b7d9665b3a4510520429ae158
When users upload an MPU object, s3api will automatically
create a segment container if one doesn't already exist.
Currently, s3api will create the segment bucket using the
cluster's default storage policy. This patch changes that
behavior to use the same storage policy as the primary bucket.
Change-Id: Ib64a06868bd3670a1d4a1860ac29122e1ede7c39
Closes-Bug: 1832390
Currently one has to remove and re-add devices when the region or zone needs to
be changed. Adding the subcommands set_region and set_zone simplifies this, now
it is possible to change the region and/or zone easily. Note that there is no
change to the required rebalancing; it is likely that data still needs to be
moved within the cluster.
This is mostly copy-n-paste of the existing set_weight subcommand and adopting
tests accordingly. Some duplicated code in the tests has been aggregated as
well.
Change-Id: I37dd8e8ac24e2b0bb196758449a1d91a769b2e96
The test test_PUT_ec_fragment_quorum_archive_etag_mismatch
busts the md5 in server.py, so it ends damaging the md5 of
footers instead of the fragment archive. It appears that the
intention of the test was to check the integrity verification
for fragment archive, so change the test to bust diskfile.py
instead.
Change-Id: I54a203bb637d5f5814e8df2b4297758b0b72adac
A Python 3 bug causes us to abort header parsing in some cases. We
mostly worked around that in the related change, but that was *after*
eventlet used the parsed headers to determine things like message
framing. As a result, a client sending a malformed request (for example,
sending both Content-Length *and* Transfer-Encoding: chunked headers)
might have that request parsed properly and authorized by a proxy-server
running Python 2, but the proxy-to-backend request could get misparsed
if the backend is running Python 3. As a result, the single client
request could be interpretted as multiple requests by an object server,
only the first of which was properly authorized at the proxy.
Now, after we find and parse additional headers that weren't parsed by
Python, fix up eventlet's wsgi.input to reflect the message framing we
expect given the complete set of headers. As an added precaution, if the
client included Transfer-Encoding: chunked *and* a Content-Length,
ensure that the Content-Length is not forwarded to the backend.
Change-Id: I70c125df70b2a703de44662adc66f740cc79c7a9
Related-Change: I0f03c211f35a9a49e047a5718a9907b515ca88d7
Closes-Bug: 1840507
Previously, configs like
fallocate_reserve = 1%
would cause a py3 backend server to fail to start, complaining like
configparser.InterpolationSyntaxError: Error in file
/etc/swift/object-server/1.conf.d: '%' must be followed
by '%' or '(', found: '%'
This could also come up in proxy-server configs, with things like
percent signs in tempauth password.
In general, we haven't really thought much about interpolation in
configs. Python's default ConfigParser has always supported it, though,
so we got it "for free". On py2, we didn't really have to think about
it, since values like "1%" would pass through just fine. (It would blow
up a SafeConfigParser, but a normal ConfigParser only does replacements
when there's something like a "%(opt)s" in the value.)
On py3, SafeConfigParser became ConfigParser, and the old interpolation
mode (AFAICT) doesn't exist.
Unfortunatley, since we "supported" interpolation, we have to assume
there are deployments in the wild that use it, and try not to break
them. So, do what we can to mimic the py2 behavior.
Change-Id: I0f9cecd11f00b522a8486972551cb30af151ce32
Closes-Bug: #1844368
When a container is being cleaved there is a possiblity that we're
dealing with an empty or near empty container created on a handoff node.
These containers may have a valid list of shard ranges, so would need
to cleave to the new shards.
Currently, when using a `cleave_batch_size` that is smaller then the
number of shard ranges on the cleaving container, these containers will
have to take a few shard passes to shard, even though there maybe
nothing in them.
This is worse if a really large container is sharding, and due to being
slow, error limitted a node causing a new container on a handoff
location. This empty container would have a large number of shard ranges
and could take a _very_ long time to shard away, slowing the process
down.
This patch eliminates the issue by detecting when no objects are
returned for a shard range. The `_cleave_shard_range` method now
returns 3 possible results:
- CLEAVE_SUCCESS
- CLEAVE_FAILED
- CLEAVE_EMPTY
They all are pretty self explanitory. When `CLEAVE_EMPTY` is returned
the code will:
- Log
- Not replicate the empty temp shard container sitting in a
handoff location
- Not count the shard range in the `cleave_batch_size` count
- Update the cleaving context so sharding can move forward
If there already is a shard range DB existing on a handoff node to use
then the sharder wont skip it, even if there are no objects, it'll
replicate it and treat it as normal, including using a `cleave_batch_size`
slot.
Change-Id: Id338f6c3187f93454bcdf025a32a073284a4a159
Closes-Bug: #1839355
Users are able to change versioning in a container
from X-Versions-Location to X-History-Location, which affects
how DELETEs are handled. We have some unit tests that check this
behavior, but no functional tests.
This patch adds a functional test that helps us understand and
document how changing modes affects the handling of DELETE
requests.
Change-Id: I5dbe5bdca17e624963cb3a3daba3b240cbb4bec4
This is a follow up patch from the cleaning up cleave context's patch
(patch 681970). Instead of tracking a last_modified timestamp, and storing
it in the context metadata, use the timestamp we use when storing any
metadata.
Reducing duplication is nice, but there's a more significant reason to
do this: affected container DBs can start getting cleaned up as soon as
they're running the new code rather than needing to wait for an
additional reclaim_age.
Change-Id: I2cdbe11f06ffb5574e573c4a60ba4e5d41a00c50
Previously we'd
- complain that a client disconnected even though they finished their
chunked transfer just fine, and
- on EC, send a X-Backend-Obj-Content-Length for pre-allocation even
though Content-Length doesn't determine request body size.
Change-Id: Ia80e595f713695cbb41dab575963f2cb9bebfa09
Related-Bug: 1840507
There is a sharding edge case where more CleaveContext are generated and
stored in the sharding container DB. If this number get's high enough,
like in the linked bug. If enough CleaveContects build up in the DB then
this can lead to the 503's when attempting to list the container due to
all the `X-Container-Sysmeta-Shard-Context-*` headers.
This patch resolves this by tracking the a CleaveContext's last
modified. And during the sharding audit, any context's that hasn't been
touched after reclaim_age are deleted.
This plus the skip empty ranges patches should improve these handoff
shards.
Change-Id: I1e502c328be16fca5f1cca2186b27a0545fecc16
Closes-Bug: #1843313
Previously, versioned_writes middleware copy an already existing
object using PUT. However, SLO requires the additional query
to properly update the object size when listing.
Propose fix: In _put_versioned_obj - which is called when on
creating version obj and also on restoring obj,
if 'X-Object-Sysmeta-Slo-Size' header is present it will
add needed headers for container to update obj size
Added a new functional test case with size assertion for slo
Change-Id: I47e0663e67daea8f1cf4eaf3c47e7c8429fd81bc
Closes-Bug: #1840322
Recent versions of py27 [1] have begun raising InvalidURL if you try to
include non-ASCII characters in the request path. This was observed
recently in the periodic checks of stable/ocata and stable/pike. In
particular, we would spin up some in-process servers in
test.unit.proxy.test_server.TestSocketObjectVersions and do a container
listing with a prefix param that included raw (unquoted) UTF-8. This
query string would pass unmolested through the proxy, tripping the
InvalidURL error when bufferedhttp called putrequest.
More recent versions of Swift would not exhibit this particular failure,
as the listing_formats middleware would force a decoding/re-encoding of
the query string for account and container requests. However, object
requests with errant query strings would likely be able to trip the same
error.
Swift on py3 should not exhibit this behavior, as we so
thoroughly re-write the request line to avoid hitting
https://bugs.python.org/issue33973.
Now, always parse and re-encode the query string in bufferedhttp. This
prevents any errors on object requests and cleans up any callers that
might use bufferedhttp directly.
[1] Anything after https://github.com/python/cpython/commit/bb8071a;
see https://bugs.python.org/issue30458
Closes-Bug: 1843816
Change-Id: I73f84b96f164e6fc5d3cb890355871c26ed271a6
Related-Change: Id3ce37aa0402e2d8dd5784ce329d7cb4fbaf700d
Related-Change: Ie648f5c04d4415f3b620fb196fa567ce7575d522
AWS seems to support this, so let's allow s3api to do it, too.
Previously, S3 clients trying to use multi-character delimiters would
get 500s back, because s3api didn't know how to handle the 412s that the
container server would send.
As long as we're adding support for container listings, may as well do
it for accounts, too.
Change-Id: I62032ddd50a3493b8b99a40fb48d840ac763d0e7
Co-Authored-By: Thiago da Silva <thiagodasilva@gmail.com>
Closes-Bug: #1797305
There's still one problem, though: since swiftclient on py3 doesn't
support non-ASCII characters in metadata names, none of the tests in
TestReconstructorRebuildUTF8 will pass.
Change-Id: I4ec879ade534e09c3a625414d8aa1f16fd600fa4
Previously, our unit tests with socket servers would let eventlet
capitalize headers on the way out, which
- isn't something we want to have eventlet do, because it
- breaks unicode-in-header-names on py3, so it
- is already disabled in swift.common.wsgi.run_server() for real servers.
Include a test to make sure we don't forget about it in the future.
Change-Id: I0156d0059092ed414b296c65fb70fc18533b074a
Whether the frag is durable or non-durable greatly affects how much I
care whether I can reconstruct it.
Change-Id: Ie6f46267d4bb567ecc0cc195d1fd7ce55c8cb325
