The connect method of ssync_sender tells the remote connection that it's
going to send a valid HTTP chunked request, but if the remote end needs
to respond with an error of any kind sender throws HTTP right out the
window, picks up his ball, and closes the socket down hard - much to the
surprise of the eventlet.wsgi server who up to this point had been
playing along quite nicely with this 'SSYNC' nonsense assuming that
everyone here is consenting mature adults.
If you're going to make a "Transfer-Encoding: chunked" request have the
good decency to finish the job with a proper '0\r\n\r\n'. [1]
N.B. It might be possible to handle an error status during the
initialize_request phase with some sort of 100-continue support, but
honestly it's not entirely clear to me when the server isn't going to
close the connection if the client is still expected to send the body
[2] - further if the error comes later during missing_check or updates
we'll for sure want to send the chunk transfer termination line before
we close down the socket and this way we cover both.
1. Really, eventlet.wsgi shouldn't be so blasted brittle about this [3]
2. https://lists.w3.org/Archives/Public/ietf-http-wg/2005AprJun/0007.html
3. c3ce3eef0b
Closes-Bug #1489587
Change-Id: Ic17c6c3075553f8cf6ef6213e62a00282f0d01cf
There is duplicate 'X-Backend-Storage-Policy-Index' dictionary key in unit.obj.test_server.py.
One key has fixed policy index value, and another has random value.
Unittest should done with random policy index, so remove key which is set fixed value.
Change-Id: Ic91fcf44d48297d0feee33c928ca682def9790a3
It used to be that a GET of a tempurl referencing a large object would
let you download that large object regardless of where its segments
lived. However, this led to some violated user expectations around
container tempurls.
(Note on shorthand: all tempurls reference objects. However, "account
tempurl" and "container tempurl" are shorthand meaning tempurls
generated using a key on the account or container, respectively.)
Let's say an application is given tempurl keys to a particular
container, and it does all its work therein using those keys. The user
expects that, if the application is compromised, then the attacker
only gains access to the "compromised-container". However, with the old
behavior, the attacker could read data from *any* container like so:
1) Choose a "victim-container" to download
2) Create PUT and GET tempurl for any object name within the
"compromised-container". The object doesn't need to exist;
we'll create it.
3) Using the PUT tempurl, upload a DLO manifest with
"X-Object-Manifest: /victim-container/"
4) Using the GET tempurl, download the object created in step 3. The
result will be the concatenation of all objects in the
"victim-container".
Step 3 need not be for all objects in the "victim-container"; for
example, a value "X-Object-Manifest: /victim-container/abc" would only
be the concatenation of all objects whose names begin with "abc". By
probing for object names in this way, individual objects may be found
and extracted.
A similar bug would exist for manifests referencing other accounts
except that neither the X-Object-Manifest (DLO) nor the JSON manifest
document (SLO) have a way of specifying a different account.
This change makes it so that a container tempurl only grants access to
objects within its container, *including* large-object segments. This
breaks backward compatibility for container tempurls that may have
pointed to cross container *LO's, but (a) there are security
implications, and (b) container tempurls are a relatively new feature.
This works by having the tempurl middleware install an authorization
callback ('swift.authorize' in the WSGI environment) that limits the
scope of any requests to the account or container from which the key
came.
This requires swift.authorize to persist for both the manifest request
and all segment requests; this is done by having the proxy server
restore it to the WSGI environment prior to returning from __call__.
[CVE-2015-5223]
Co-Authored-By: Clay Gerrard <clayg@swiftstack.com>
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Christian Schwede <cschwede@redhat.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: Ie6d52f7a07e87f6fec21ed8b0ec1d84be8b2b11c
Closes-Bug: 1449212
Do not allow PUT tempurls to create pointers to other data. Specifically
disallow the creation of DLO object manifests by returning an error if a
non-safe tempurl request includes an X-Object-Manifest header regardless of
the value of the header.
This prevents discoverability attacks which can use any PUT tempurl to probe
for private data by creating a DLO object manifest and then using the PUT
tempurl to head the object which would 404 if the prefix does not match any
object data or form a valid DLO HEAD response if it does.
This also prevents a tricky and potentially unexpected consequence of PUT
tempurls which would make it unsafe to allow a user to download objects
created by tempurl (even if they just created them) because the result of
reading the object created via tempurl may not be the data which was uploaded.
[CVE-2015-5223]
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Change-Id: I11e68830009d3f6bff44ae4011a41b67139146f6
Closes-Bug: 1453948
v1.0.9 rev of PyECLib replaces Jerasure with a native EC
implementation (liberasurecode_rs_vand) as the default
EC scheme. Going forward, Jerasure will not be bundled
with PyPI version of PyECLib as it used to be, until
v1.0.7.
This is an interim change to Swift requirements until we
get v1.0.9 PyECLib included into global-requirements and
ready patches that change Swift default ec_type (for doc,
config samples and unit tests) from "jerasure_rs_vand"
to "liberasurecode_rs_vand."
Change-Id: Ica4fee2cdea2bc7f5edd0e51ad637a4457faf3b4
In EC case, When GET object requested, proxy-server always makes a log
line "Client disconnected on read" even though the request succeeded.
That is because ECAppIter class doesn't maintain a bunch of backend
stream when closing the app_iter. It will cause unfortunately
GeneratorExit on backend stream ResumingGetter.
This patch fixes to set non_client_disconnected to propagate the state
to the backend streams when the range iteration stopped successful.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I77af9807816bea1444d66534a17e2a210bcf09f8
Closes-Bug: #1472201
Now you can run
$ tox -e pep8 path/to/file.py [path/to/file2.py [...]]
to run pep8 against just those files[1]. This is quite a bit faster
than a full pep8 run, and the faster feedback is nice when you're
fiddling with some formatting to placate pep8.
Of course, you can still run "tox -e pep8" to check the whole source
tree, just as before this commit.
[1] It'll still run against bin/swift* as well, but that's still a lot
faster than running against all our .py files.
Change-Id: I81b4363fb95a34ff0f5c346b2b24f2047154f502
This change cleans up test/unit/obj/test_replicator.py's imports
to use only 1 version of multiline import syntaxes (' \' vs '()').
I don't really mind which, but we should be consistant, at least
in the same file.
This is a follow up for patch 215857.
Change-Id: Ie2d328c25865b19092c493981a803ee246a9d7a5
Previously, account listings that used the delimiter query param could
omit some containers if they ended with the character that follows the
delimiter.
See If196e3075612b121ef8da4a9128167d00a248c27 for the corresponding fix
for container listings.
Change-Id: I57fcb97e51f653f5f4e306a632fcb3a0fb148c4e
Under some concurrency the object-replicator could potentially send the
wrong X-Backed-Storage-Policy-Index header to it's partner nodes during
replication if there were multiple storage policies on the same node
because of a race where multiple jobs being processed concurrently would
mutate some shared state on the ObjectReplicator instance.
Instead of using shared stated on the ObjectReplicator instance when
mutating the default headers send with REPLICATION requests each job
will copy them into a local where they can safely be updated.
Change-Id: I5522db57af7e308b1f9d4181f14ea14e386a71fd
When using fast-post and POST (i.e. metadata update) is requested to
a SLO manifest files, current Swift drops the 'X-Static-Large-Object'
header from the existing metadata. It results in breaking the SLO
state because the manifest missing the 'X-Static-Large-Object' metadata
will be maintained as a normal files.
This patch fixes object-server to keep the existing
'X-Static-Large-Object' flag and then keep the SLO state.
Change-Id: Ib1eb569071372c322dd105c52baeeb094003291e
Closes-bug: #1453807
SAIO Configuration and documentation changes enabling to run the
container sync probe test by default
Change-Id: Iccf59533d0d4fe72549d318339ab125d04dde006
Related-Bug: #1476623
When the object-replicator encounters handoffs_first and
handoff_delete options as enabled it should emit a log
warning indicating that it should be changed back to the
default before the next "normal" rebalance.
Closes-Bug: #1457262
Change-Id: If9dc2796c18ed3cf13da920831e2d5c2ae9f12a0
The proxy was trying to pop a byterange off a Range header that didn't
contain syntactically-valid byteranges. This worked about as well as
you'd expect. Now we detect the bogus value and remove the header
entirely.
Change-Id: I24b92f900d33ec79880c7db2870378489d5a6810
(line 259) parameter(op) of object_update method is 'PUT' or
'DELETE' not 'POST' or 'DELETE'.
[1]: swift/obj/updater.py
Change-Id: I876a620ba8e09e69fba7156b12e69445c229e160
'print' function is compatible with 2.x and 3.x python versions
Link : https://www.python.org/dev/peps/pep-3105/
Python 2.6 has a __future__ import that removes print as language syntax,
letting you use the functional form instead
Change-Id: I416c6ac21ccbfb91ec328ffb1ed21e492ef52d58
Fix the pep8 warning H702 "Formatting operation should be outside of
localization method call".
For the logger, pass parameters as indexed parameters instead of using
the string str%args operator, the logger is more reliable in case of
formatting error.
Change-Id: If418bc155f6a6c0a00f63e3d87ebe4addf4aae55
The TestCase.assert_() has been deprecated in Python 2.7. Replace it
with assertTrue() or even better methods (assertIn, assertNotIn,
assertIsInstance) which provide better error messages.
Change-Id: I21c730351470031a2dabe5238693095eabdb8964
