contextlib.nested() is missing completely in Python 3.
Since 2.7, we can use multiple context managers in a 'with' statement,
like so:
with thing1() as t1, thing2() as t2:
do_stuff()
Now, if we had some code that needed to nest an arbitrary number of
context managers, there's stuff we could do with contextlib.ExitStack
and such... but we don't. We only use contextlib.nested() in tests to
set up bunches of mocks without crazy-deep indentation, and all that
stuff fits perfectly into multiple-context-manager 'with' statements.
Change-Id: Id472958b007948f05dbd4c7fb8cf3ffab58e2681
This patch add a expositive test that closing connection
(fd and socket) causes StopIteration on reader at proxy server
on PUT object case.
And then, if the connection closed and proxy detects a lack of
data length, it will result in HTTPClientDisconnected.
The only reason why I leave test_client_disconnect as current
is Bill is now working for the change at
https://review.openstack.org/#/c/236007/
Change-Id: If68f752a2b0dd6cfad4d007ca5d155e6ffb378e5
assertEquals is deprecated in py3, replacing it.
Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8
Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
The urllib, urllib2 and urlparse modules of Python 2 were reorganized
into a new urllib namespace on Python 3. Replace urllib, urllib2 and
urlparse imports with six.moves.urllib to make the modified code
compatible with Python 2 and Python 3.
The initial patch was generated by the urllib operation of the sixer
tool on: bin/* swift/ test/.
Change-Id: I61a8c7fb7972eabc7da8dad3b3d34bceee5c5d93
... which ensures no Timeouts remain pending after the parent generator
is closed when a client disconnects before being able to read the entire
body.
Also tighten up a few tests that may have left some open ECAppIter
generators lying about after the tests themselves had finished. This
has the side effect of preventing the extraneous printing of the Timeout
errors being raised by the eventlet hub in the background while our
unittests are running.
Change-Id: I156d873c72c19623bcfbf39bf120c98800b3cada
The rfc822 module has been deprecated since Python 2.3, and in
particular is absent from the Python 3 standard library. However, Swift
uses instances of rfc822.Message in a number of places, relying on its
behavior of immediately parsing the headers of a file-like object
without consuming the body, leaving the position of the file at the
start of the body. Python 3's http.client has an undocumented
parse_headers function with the same behavior, which inspired the new
parse_mime_headers utility introduced here. (The HeaderKeyDict returned
by parse_mime_headers doesn't have a `.getheader(key)` method like
rfc822.Message did; the dictionary-like `[key]` or `.get(key)` interface
should be used exclusively.)
The implementation in this commit won't actually work with Python 3, the
email.parser.Parser().parsestr of which expects a Unicode string, but it
is believed that this can be addressed in followup work.
Change-Id: Ia5ee2ead67e36e8c6416183667f64ae255887736
In EC PUT request case, proxy-server may send commits to object-servers
it may make .durable file even though the request failed due to a lack
of quorum number.
For example:
- Considering the case that almost all object-servers fail by 422
Unprocessable Entity
- Using ec scheme 4 + 2
- 5 (quorum size) object-server failed with 422, 1 object-servers
succeeded as 201 created
How it works:
- Client creates a PUT request
- Proxy will open connections to backend object-servers
- Proxy will send whole encoded chunks to object-servers
- Proxy will send content-md5 as footers.
- Proxy will get responses [422, 422, 422, 422, 422, 201] (currently
this list will be regarded as "we have quorum response")
- And then proxy will send commits to object-servers (the only
object-server with 201 will create .durable file)
- Proxy will return 503 because the commits results in no response
statuses from object-servers except the 201 node.
This patch fixes the quorum handling at ObjectController to check
that it has *successful* quorum responses before sending durable commits.
Closes-Bug: #1491748
Change-Id: Icc099993be76bcc687191f332db56d62856a500f
At PUT object request, proxy server makes backend headers (e.g.
X-Container-Partition) which help object-servers to determine
the container-server they should update. In addition, the backend
headers are created as many as the number of container replicas.
(i.e. 3 replica in container ring, 3 backend headers will be created)
On EC case, Swift fans out fragment archives to backend object-servers.
Basically the number of fragment archives will be more than the container
replica number and proxy-server assumes a request as success when quorum
number of object-server succeeded to store. That would cause to make an
orphaned object which is stored but not container updated.
For example, assuming k=10, m=4, container replica=3 case:
Assuming, proxy-server attempts to make 14 backend streams but
unfortunately first 3 nodes returns 507 (disk failure) and then
the Swift doesn't have any other disks.
In the case, proxy keeps 11 backend streams to store and current Swift
assumes it as sufficient because it is more than or equals quorum (right
now k+1 is sufficient i.e. 11 backend streams are enough to store)
However, in the case, the 11 streams doesn't have the container update
header so that the request will succeed but container will be never updated.
This patch allows to extract container updates up to object quorum_size
+ 1 to more nodes to ensure the updates. This approach sacrifices the
container update cost a bit because duplicated updates will be there but
quorum sizes + 1 seems reasonable (even if it's reaplicated case) to pay
to ensure that instead of whole objects incude the update headers.
Now Swift will work like as follows:
For example:
k=10, m=4, qurum_size=11 (k+1), 3 replica for container.
CU: container update
CA: commit ack
That result in like as
CU CU CU CU CU CU CU CU CU CU CU CU
[507, 507, 507, 201, 201, 201, 201, 201, 201, 201, 201, 201, 201, 201]
CA CA CA CA CA
In this case, at least 3 container updates are saved.
For another example:
7 replicated objects, qurum_size=4 (7//2+1), 3 replica for container.
CU: container update
CA: commit ack (201s for successful PUT on replicated)
CU CU CU CU CU
[507, 507, 507, 201, 201, 201, 201]
CA CA CA CA
In this replicated case, at least 2 container updates are saved.
Cleaned up some unit tests so that modifying policies doesn't leak
between tests.
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Sam Merritt <sam@swiftstack.com>
Closes-Bug: #1460920
Change-Id: I04132858f44b42ee7ecf3b7994cb22a19d001d70
The ECObjectController was unconditionally sending down the frag archive
commit document after the client source stream terminated - even if the
client disconnected early.
We can detect early disconnect in two ways:
1. Content-Length and not enough bytes_transfered
When eventlet.wsgi is reading from a Content-Length body the
readable returns the empty string and our iterable raises
StopIteration - but we can check content-length against
bytes_transfered and know if the client disconnected.
2. Transfer-Encoding: chunked - w/o a 0\r\n\r\n
When eventlet.wsgi is reading from a Transfer-Encoding: chunked
body the socket read returns the empty string, eventlet.wsgi's
chunked parser raises ValueError (which we translate to
ChunkReadError*) and we know we know the client disconnected.
... if we detect either of these conditions the proxy should:
1. *not* send down the commit document to object servers
2. disconnect from backend servers
3. log the client disconnect
Oddly the code on master was only messing up the first part. Backend
connections were terminated (gracefully after the commit document), and
then the disconnect was being logged as 499.
So now we only send down the commit document on a successful complete
client HTTP request (either whole Content-Length, or clean
Transfer-Encoding: chunked 0\r\n\r\n).
* To detect the early disconnect on Transfer-Encoding: chunked a new
swift.common.exceptions.ChunkReadError is used to translate
eventlet.wsgi's more general IOError and ValueErrors into something
more appropriate to catch and handle closer to our generic
ChunkReadTimeout handling.
Co-Author: Alistair Coles <alistair.coles@hp.com>
Closes-Bug: #1496205
Change-Id: I028a530aba82d50baa4ee1d05ddce18d4cce4e81
In EC case, When GET object requested, proxy-server always makes a log
line "Client disconnected on read" even though the request succeeded.
That is because ECAppIter class doesn't maintain a bunch of backend
stream when closing the app_iter. It will cause unfortunately
GeneratorExit on backend stream ResumingGetter.
This patch fixes to set non_client_disconnected to propagate the state
to the backend streams when the range iteration stopped successful.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I77af9807816bea1444d66534a17e2a210bcf09f8
Closes-Bug: #1472201
The proxy was trying to pop a byterange off a Range header that didn't
contain syntactically-valid byteranges. This worked about as well as
you'd expect. Now we detect the bogus value and remove the header
entirely.
Change-Id: I24b92f900d33ec79880c7db2870378489d5a6810
The TestCase.assert_() has been deprecated in Python 2.7. Replace it
with assertTrue() or even better methods (assertIn, assertNotIn,
assertIsInstance) which provide better error messages.
Change-Id: I21c730351470031a2dabe5238693095eabdb8964
Rewrite object versioning as middleware to simplify the PUT method
in the object controller.
The functionality remains basically the
same with the only major difference being the ability to now
version slo manifest files. dlo manifests are still not
supported as part of this patch.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
DocImpact
Change-Id: Ie899290b3312e201979eafefb253d1a60b65b837
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Fix pep8 warnings of the E category of hacking 0.10:
* E113: unexpected indentation
* E121: continuation line under-indented for hanging indent
* E122: continuation line missing indentation or outdented
* E123: closing bracket does not match indentation of opening bracket's
line
* E126: continuation line over-indented for hanging indent
* E251: unexpected spaces around keyword / parameter equals
Change-Id: I0b24eebdf1a37dc1b572b6c9a3d3d4832d050237
'print' function is compatible with 2.x and 3.x python versions
Link : https://www.python.org/dev/peps/pep-3105/
Python 2.6 has a __future__ import that removes print as language syntax,
letting you use the functional form instead
Change-Id: I94e1bc6bd83ad6b05695c7ebdf7cbfd8f6d9f9af
The assert_() method is deprecated and can be safely replaced by assertTrue().
This patch makes sure that running the tests does not create undesired
warnings.
Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
wsgi.input is a binary stream (bytes), not a text stream (unicode).
* Replace StringIO with BytesIO for WSGI input
* Replace StringIO('') with StringIO() and replace WsgiStringIO('') with
WsgiStringIO(): an empty string is already the default value
Change-Id: I09c9527be2265a6847189aeeb74a17261ddc781a
* replace "from cStringIO import StringIO"
with "from six.moves import cStringIO as StringIO"
* replace "from StringIO import StringIO"
with "from six import StringIO"
* replace "import cStringIO" and "cStringIO.StringIO()"
with "from six import moves" and "moves.cStringIO()"
* replace "import StringIO" and "StringIO.StringIO()"
with "import six" and "six.StringIO()"
This patch was generated by the stringio operation of the sixer tool:
https://pypi.python.org/pypi/sixer
Change-Id: Iacba77fec3045f96773d1090c0bd48613729a561
Some EC tests in proxy/test_server.py use the replication
DiskFileManager to open diskfiles. That works because it
turns out that DiskFileManager is robust to the frag index
being stitched onto the end of the timestamp in filenames,
but it would be better to test using the ECDiskFileManager
for EC diskfiles.
Also, drive-by deletion of some duplicated lines.
Change-Id: Ie3adb607d8c34a469df7d0f6f542ed5ba9a2dac0
Python 2.7.9+ and 3.2.6+ limits the number of maximum headers in httplib to 100
[1,2,3]. This setting is too low for Swift.
By default the maximum number of allowed headers depends on the number of max
allowed metadata settings plus a default value of 32 for regular http headers.
If for some reason this is not enough (custom middleware for example) it can be
increased with the extra_header_count constraint.
[1] https://bugs.python.org/issue16037
[2] https://hg.python.org/cpython/raw-file/15c95b7d81dc/Misc/NEWS
[3] https://hg.python.org/cpython/raw-file/v3.2.6/Misc/NEWS
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Thomas Herve <therve@redhat.com>
Change-Id: I388fd697ec88476024b0e9f1ae75ba35ff765282
The iteritems() of Python 2 dictionaries has been renamed to items() on
Python 3. According to a discussion on the openstack-dev mailing list,
the overhead of creating a temporary list using dict.items() on Python 2
is very low because most dictionaries are small:
http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Patch generated by the following command:
sed -i 's,iteritems,items,g' \
$(find swift -name "*.py") \
$(find test -name "*.py")
Change-Id: I6070bb6c684be76e8e77222a7d280ec6edd43496
The Python 2 next() method of iterators was renamed to __next__() on
Python 3. Use the builtin next() function instead which works on Python
2 and Python 3.
Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
This commit lets clients receive multipart/byteranges responses (see
RFC 7233, Appendix A) for erasure-coded objects. Clients can already
do this for replicated objects, so this brings EC closer to feature
parity (ha!).
GetOrHeadHandler got a base class extracted from it that treats an
HTTP response as a sequence of byte-range responses. This way, it can
continue to yield whole fragments, not just N-byte pieces of the raw
HTTP response, since an N-byte piece of a multipart/byteranges
response is pretty much useless.
There are a couple of bonus fixes in here, too. For starters, download
resuming now works on multipart/byteranges responses. Before, it only
worked on 200 responses or 206 responses for a single byte
range. Also, BufferedHTTPResponse grew a readline() method.
Also, the MIME response for replicated objects got tightened up a
little. Before, it had some leading and trailing CRLFs which, while
allowed by RFC 7233, provide no benefit. Now, both replicated and EC
multipart/byteranges avoid extraneous bytes. This let me re-use the
Content-Length calculation in swob instead of having to either hack
around it or add extraneous whitespace to match.
Change-Id: I16fc65e0ec4e356706d327bdb02a3741e36330a0
On EC PUT in an M+K scheme, we require M+1 fragment archives to
durably land on disk. If we get that, then we go ahead and ask the
object servers to "commit" the object by writing out .durable
files. We only require 2 of those.
When we got exactly M+1 fragment archives on disk, and then one
connection timed out while writing .durable files, we should still be
okay (provided M is at least 3). However, we'd take our M > 2
remaining successful responses and pass that off to best_response()
with a quorum size of M+1, thus getting a 503 even though everything
worked well enough.
Now we pass 2 to best_response() to avoid that false negative.
There was also a spot where we were getting the quorum size wrong. If
we wrote out 3 fragment archives for a 2+1 policy, we were only
requiring 2 successful backend PUTs. That's wrong; the right number is
3, which is what the policy's .quorum() method says. There was a spot
where the right number wasn't getting plumbed through, but it is now.
Change-Id: Ic658a199e952558db329268f4d7b4009f47c6d03
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Closes-Bug: 1452468
Swift doesn't check if the used API version is valid. Currently there
is only one valid REST API version, but that might change in the
future.
This patch enforces "v1" or "v1.0" as the version string when accessing
account, containers and objects.
The list of accepted version strings can be manually overridden using a
comma-separated list in swift.conf to make this backward-compatible.
The constraint loader has been modified slightly to accept strings as
well as integers.
Any request to an account, container, and object which does not
provide the correct version string will get a 400 BadRequest response.
The allowed api versions are by default excluded from /info.
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: John Dickinson <me@not.mn>
Closes Bug #1437442
Change-Id: I5ab6e236544378abf2eab562ab759513d09bc256
Since the existing check depends on how quickly the test
passes, it causes to the unpredictable behavior and may break
the test.
This commit removes the unnecessary dependency on the
current time and leaves only the check that timestamp is
greater than 0.
Change-Id: I3321e9aba0df52cb78523710b26f310f35801674
Closes-Bug: #1441599
This commit makes it possible to PUT an object into Swift and have it
stored using erasure coding instead of replication, and also to GET
the object back from Swift at a later time.
This works by splitting the incoming object into a number of segments,
erasure-coding each segment in turn to get fragments, then
concatenating the fragments into fragment archives. Segments are 1 MiB
in size, except the last, which is between 1 B and 1 MiB.
+====================================================================+
| object data |
+====================================================================+
|
+------------------------+----------------------+
| | |
v v v
+===================+ +===================+ +==============+
| segment 1 | | segment 2 | ... | segment N |
+===================+ +===================+ +==============+
| |
| |
v v
/=========\ /=========\
| pyeclib | | pyeclib | ...
\=========/ \=========/
| |
| |
+--> fragment A-1 +--> fragment A-2
| |
| |
| |
| |
| |
+--> fragment B-1 +--> fragment B-2
| |
| |
... ...
Then, object server A gets the concatenation of fragment A-1, A-2,
..., A-N, so its .data file looks like this (called a "fragment archive"):
+=====================================================================+
| fragment A-1 | fragment A-2 | ... | fragment A-N |
+=====================================================================+
Since this means that the object server never sees the object data as
the client sent it, we have to do a few things to ensure data
integrity.
First, the proxy has to check the Etag if the client provided it; the
object server can't do it since the object server doesn't see the raw
data.
Second, if the client does not provide an Etag, the proxy computes it
and uses the MIME-PUT mechanism to provide it to the object servers
after the object body. Otherwise, the object would not have an Etag at
all.
Third, the proxy computes the MD5 of each fragment archive and sends
it to the object server using the MIME-PUT mechanism. With replicated
objects, the proxy checks that the Etags from all the object servers
match, and if they don't, returns a 500 to the client. This mitigates
the risk of data corruption in one of the proxy --> object connections,
and signals to the client when it happens. With EC objects, we can't
use that same mechanism, so we must send the checksum with each
fragment archive to get comparable protection.
On the GET path, the inverse happens: the proxy connects to a bunch of
object servers (M of them, for an M+K scheme), reads one fragment at a
time from each fragment archive, decodes those fragments into a
segment, and serves the segment to the client.
When an object server dies partway through a GET response, any
partially-fetched fragment is discarded, the resumption point is wound
back to the nearest fragment boundary, and the GET is retried with the
next object server.
GET requests for a single byterange work; GET requests for multiple
byteranges do not.
There are a number of things _not_ included in this commit. Some of
them are listed here:
* multi-range GET
* deferred cleanup of old .data files
* durability (daemon to reconstruct missing archives)
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I9c13c03616489f8eab7dcd7c5f21237ed4cb6fd2
Adds specific disk file classes for EC policy types.
The new ECDiskFile and ECDiskFileWriter classes are used by the
ECDiskFileManager.
ECDiskFileManager is registered with the DiskFileRouter for use with
EC_POLICY type policies.
Refactors diskfile tests into BaseDiskFileMixin and BaseDiskFileManagerMixin
classes which are then extended in subclasses for the legacy
replication-type DiskFile* and ECDiskFile* classes.
Refactor to prefer use of a policy instance reference over a policy_index
int to refer to a policy.
Add additional verification to DiskFileManager.get_dev_path to validate the
device root with common.constraints.check_dir, even when mount_check is
disabled for use in on a virtual swift-all-in-one.
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I22f915160dc67a9e18f4738c1ddf068344e8ad5d
An authenticated user can delete the most recent version of any
versioned object who's name is known if the user has listing access
to the x-versions-location container. Only Swift setups with
allow_version setting are affected.
This patch closes this bug, tracked as CVE-2015-1856
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Christian Schwede <info@cschwede.de>
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Closes-Bug: 1430645
Change-Id: Ibacc7413afe7cb6f77d92e5941dcfdf4768ffa18
Wikipedia's list of common misspellings [1] has a machine-readable
version. This patch fixes those misspellings mentioned in the list
which don't have multiple right variants (as e.g. "accension", which can
be both "accession" and "ascension"), such misspellings are left
untouched. The list of changes was manually re-checked for false
positives.
[1] https://en.wikipedia.org/wiki/Wikipedia:Lists_of_common_misspellings/For_machines
Change-Id: Ic9a5438629664f7cea216413a28acc0e8992da05
Signed-off-by: Martin Kletzander <mkletzan@redhat.com>
