We were (indirectly) importing swiftclient (and therefore requests and
urllib3) before doing our eventlet monkey-patching. This would lead
boto3 (which digs an SSLContext out of urllib3) to trip RecursionErrors
on py3 similar to
>>> from ssl import SSLContext, PROTOCOL_SSLv23
>>> import eventlet
>>> eventlet.monkey_patch(socket=True)
>>> SSLContext(PROTOCOL_SSLv23).options |= 0
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/ssl.py", line 465, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "/usr/lib/python3.6/ssl.py", line 465, in options
super(SSLContext, SSLContext).options.__set__(self, value)
File "/usr/lib/python3.6/ssl.py", line 465, in options
super(SSLContext, SSLContext).options.__set__(self, value)
[Previous line repeated 330 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
Change-Id: I4bb59edd87336597791416c4f2a096efe0e72fe3
We already had a unit test for this, but it was inadvertently retesting
what test_get_suffix_range_many_segments tested.
Change-Id: I43eee7029ca884268c3f2ad74300b94b299fd8d2
Closes-Bug: #1831790
... by embedding something like `If-Match: <etag>` semantics in the
symlink.
When creating a symlink, users may now specify an optional
X-Symlink-Target-Etag header. If present, the etag of the final object
returned to the client will be checked; if it does not match the
X-Symlink-Target-Etag header, a 409 Conflict error will be returned to
the client.
Note that, unlike "dynamic symlink" behavior, the target object must
exist with the matching Etag for the "static symlink" to be created.
Since we're validating the Etag anyway, we also set the content-type of
the symlink to match if the client didn't otherwise specifiy and send
the etag & content-length along to the container listing as well.
Bonus goodness:
- Tighten assertions on Content-Location
- Get rid of swift.source-sniffing by making versioned_writes
symlink-aware ('cause I'm going to want to make it symlink-aware
later anyway)
- Allow middlewares left of symlink to set their own
Container-Update-Override-Etag when creating a symlink
- Set dynamic symlink content type if client doesn't supply something
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I179ea6180d31146bb947061c69b1807c59529ac8
We were playing a little fast & loose with types before; as a result,
marker/end_marker weren't quite working right. In particular, we were
checking whether a WSGI string was contained in a shard range, while
ShardRange assumes all comparisons are against native strings.
Now, get everything to native strings before making comparisons, and
get them back to wsgi when we shove them in the params dict.
Change-Id: Iddf9e089ef95dc709ab76dc58952a776246991fd
If you set SLO's max_manifest_segments to a value larger than 10000,
then clients are able to create manifests with that many segments, but
unable to use "?multipart-manifest=delete" to delete them.
This is because the SLO middleware has its very own bulk-deleter that
it uses to handle such requests, and that bulk-deleter only allows
10000 deletions per request by default. This commit removes the
limitation so that any SLO manifest can be deleted along with its
segments.
I considered setting max-deletes-per-request to be equal to SLO's
max_manifest_segments, but that only works if max_manifest_segments
has never been decreased.
Note that this commit does not increase max_manifest_segments. Clients
cannot make SLOs any bigger than they could before. Also note that
this commit does not affect user-initiated bulk deletes, i.e. POST
requests with "?bulk-delete=true" set. Those requests are still
limited in their size, and those limits are not changed.
Change-Id: I6a35937e8418f4f2b8e29825fc9c40415e34742f
Closes-Bug: 1746685
Proxies and LBs parse the Connection: header and re-insert
the value with random casing. This occurs when we run functional
tests against realistic clusters.
Change-Id: Icb0d3e3708949127e9750f2022de8c74ce51ebe9
This matches the ETag of the underlying swift object, as opposed to the
MD5-of-MD5s that is the large object's ETag.
Change-Id: Ifab726f63739f62aeef495c970939410341694d1
Drive-by: fix an issue where tf.config wasn't initialized when we decide
whether to skip test_dlo_referer_on_segment_container.
Change-Id: Iad8b3ab40dccb23847cde5c09f55627da0963fe0
I saw an encryption test fail with 10s node timeout when the py36
unittests ran in the gate. I believe there is an outsized potential for
a beneficial increase in test reliability when run in the gate compared
to relatively small chance of any negative side-effect in test failure
responsiveness.
Change-Id: Ia31912828d416d84c39782222e4636a97a8bfe44
It's probably weird that StreamingPile has this interfaces that swallows
exceptions, but this seems better than hanging.
Change-Id: I8fe45c0f0d291efc84f3edf5d6b7cd116b5c7835
Even when your cluster's configured funny, like your
container_listing_limit is too low, or your max_manifest_segments and
max_upload_part_num are too high, an abort should (attempt to) clean up
*all* segments.
Change-Id: I5a57f919cc74ddb08bbb35a7d852fbc1457185e8
We previously realized we needed to do that for accounts and containers
where the consequences of treating the 404 as authoritative were more
obvious: we'd cache the non-existence which prevented writes until it
fell out of cache.
The same basic logic applies for objects, though: if we see
(Timeout, Timeout, Timeout, 404, 404, 404)
on a triple-replica policy, we don't really have any reason to think
that a 404 is appropriate. In fact, it seems reasonably likely that
there's a thundering-herd problem where there are too many concurrent
requests for data that *definitely is there*. By responding with a 503,
we apply some back-pressure to clients, who hopefully have some
exponential backoff in their retries.
The situation gets a bit more complicated with erasure-coded data, but
the same basic principle applies. We're just more likely to have
confirmation that there *is* data out there, we just can't reconstruct
it (right now).
Note that we *still want to check* those handoffs, of course. Our
fail-in-place strategy has us replicate (and, more recently,
reconstruct) to handoffs to maintain durability; it'd be silly *not* to
look.
UpgradeImpact:
--------------
Be aware that this may cause an increase in 503 Service Unavailable
responses served by proxy-servers. However, this should more accurately
reflect the state of the system.
Co-Authored-By: Thiago da Silva <thiagodasilva@gmail.com>
Change-Id: Ia832e9bab13167948f01bc50aa8a61974ce189fb
Closes-Bug: #1837819
Related-Bug: #1833612
Related-Change: I53ed04b5de20c261ddd79c98c629580472e09961
Related-Change: Ief44ed39d97f65e4270bf73051da9a2dd0ddbaec
Previously, we were storing the WSGI-style UTF-8-bytes-decoded-as-Latin-1
strings in the JSON field, and sending them back to eventlet directly.
If running in a mixed py2/py3 cluster, replication would eventually get
that back to the py2 server, and worse, the native-string version would
get back to the py3 server! Then on GET or HEAD, eventlet would barf
if any characters were outside the Latin-1 range.
Closes-Bug: #1837805
Change-Id: I31d942e72fd7bfbb1db4dbb1dd522dff69969e5d
Previously, we never checked whether the response we get when refetching
is even successful, much less whether it's still coming from an SLO.
Now, if the refetched data is newer, act on it. If it's older, 503.
Closes-Bug: #1837270
Change-Id: I106b94c77da220c762869aa800c31b87c3dffeeb
