Previously, we would set eventlet.minimum_write_chunk_size *after* we
emit the XML declaration. As a result, eventlet thinks it should buffer
the entire response, rendering our attempts at heartbeating useless.
Change-Id: Idb58c6cc982cc221fed5fc837afd06aa958d9ad4
Following OpenStack Style Guidelines:
[1] http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H203] Unit test assertions tend to give better messages for more specific
assertions. As a result, assertIsNone(...) is preferred over
assertEqual(None, ...) and assertIs(..., None)
Change-Id: If4db8872c4f5705c1fff017c4891626e9ce4d1e4
Before, server-side deletes of static large objects could take a long
time to complete since the proxy would wait for a response to each
segment DELETE before starting the next DELETE request.
Now, operators can configure a concurrency factor for the slo and bulk
middlewares to allow up to N concurrent DELETE requests. By default, two
DELETE requests will be allowed at a time.
Note that objects and containers are now deleted in separate passes, to
reduce the likelihood of 409 Conflict responses when deleting
containers.
Upgrade Consideration
=====================
If operators have enabled the bulk or slo middlewares and would like to
preserve the prior (single-threaded) DELETE behavior, they must add the
following line to their [filter:slo] and [filter:bulk] proxy config
sections:
delete_concurrency = 1
This may be done prior to upgrading Swift.
UpgradeImpact
Closes-Bug: 1524454
Change-Id: I128374d74a4cef7a479b221fd15eec785cc4694a
There was a function in swift.common.utils that was importing
swob.HeaderKeyDict at call time. It couldn't import it at compilation
time since utils can't import from swob or else it blows up with a
circular import error.
This commit just moves HeaderKeyDict into swift.common.header_key_dict
so that we can remove the inline import.
Change-Id: I656fde8cc2e125327c26c589cf1045cb81ffc7e5
assertEquals is deprecated in py3, replacing it.
Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8
Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
The urllib, urllib2 and urlparse modules of Python 2 were reorganized
into a new urllib namespace on Python 3. Replace urllib, urllib2 and
urlparse imports with six.moves.urllib to make the modified code
compatible with Python 2 and Python 3.
The initial patch was generated by the urllib operation of the sixer
tool on: bin/* swift/ test/.
Change-Id: I61a8c7fb7972eabc7da8dad3b3d34bceee5c5d93
The unicode type was renamed to str in Python 3. Use six.text_type to
make the modified code compatible with Python 2 and Python 3.
The initial patch was generated by the unicode operation of the sixer
tool on: bin/* swift/ test/.
Change-Id: I9e13748ccde36ee8110756202d55d3ae945d4860
The assert_() method is deprecated and can be safely replaced by assertTrue().
This patch makes sure that running the tests does not create undesired
warnings.
Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
wsgi.input is a binary stream (bytes), not a text stream (unicode).
* Replace StringIO with BytesIO for WSGI input
* Replace StringIO('') with StringIO() and replace WsgiStringIO('') with
WsgiStringIO(): an empty string is already the default value
Change-Id: I09c9527be2265a6847189aeeb74a17261ddc781a
* replace "from cStringIO import StringIO"
with "from six.moves import cStringIO as StringIO"
* replace "from StringIO import StringIO"
with "from six import StringIO"
* replace "import cStringIO" and "cStringIO.StringIO()"
with "from six import moves" and "moves.cStringIO()"
* replace "import StringIO" and "StringIO.StringIO()"
with "import six" and "six.StringIO()"
This patch was generated by the stringio operation of the sixer tool:
https://pypi.python.org/pypi/sixer
Change-Id: Iacba77fec3045f96773d1090c0bd48613729a561
The iteritems() of Python 2 dictionaries has been renamed to items() on
Python 3. According to a discussion on the openstack-dev mailing list,
the overhead of creating a temporary list using dict.items() on Python 2
is very low because most dictionaries are small:
http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Patch generated by the following command:
sed -i 's,iteritems,items,g' \
$(find swift -name "*.py") \
$(find test -name "*.py")
Change-Id: I6070bb6c684be76e8e77222a7d280ec6edd43496
Currently, if you PUT a single object, then you can also associate
metadata with it by putting it in the request headers, prefixed with
"X-Object-Meta". However, if you're bulk-uploading objects, then you
have no way to assign any metadata.
The tar file format* allows for arbitrary UTF-8 key/value pairs to be
associated with each file in an archive (as well as with the archive
itself, but we don't care about that here). If a file has extended
attributes, then tar will store those as key/value pairs.
This commit makes bulk upload read those extended attributes, if
present, and convert those to Swift object metadata. Attributes
starting with "user.meta" are converted to object metadata, and
"user.mime_type"** is converted to Content-Type.
For example, if you have a file "setup.py":
$ setfattr -n user.mime_type -v "application/python-setup" setup.py
$ setfattr -n user.meta.lunch -v "burger and fries" setup.py
$ setfattr -n user.meta.dinner -v "baked ziti" setup.py
$ setfattr -n user.stuff -v "whee" setup.py
This will get translated to headers:
Content-Type: application/python-setup
X-Object-Meta-Lunch: burger and fries
X-Object-Meta-Dinner: baked ziti
Swift will handle xattrs stored by both GNU and BSD tar***. Only
xattrs user.mime_type and user.meta.* are processed; others are
ignored.
This brings bulk upload much closer to feature-parity with non-bulk upload.
* The POSIX 1003.1-2001 (pax) format, at least. There are a few
different, mutually-incompatible tar formats out there, because of
course there are. This is the default format on GNU tar 1.27.1 or
later.
** http://standards.freedesktop.org/shared-mime-info-spec/latest/ar01s02.html#idm140622087713936
*** Even with pax-format tarballs, different encoders store xattrs
slightly differently; for example, GNU tar stores the xattr
"user.rubberducky" as pax header "SCHILY.xattr.user.rubberducky",
while BSD tar (which uses libarchive) stores it as
"LIBARCHIVE.xattr.user.rubberducky". One might wonder if this is
some programmer's attempt at job security.
Change-Id: I5e3ce87d31054f5239e86d47c45adbde2bb93640
Prior to this patch both mainline code and testing modules imported
and used constraints directly into their own namespace, or relied on
the namespace of other modules that were not the constraints
module. This meant that if a unit test wanted to change a constraint
for its operation, it had to know how that module was using the
constraint, instead of referencing the constraint module itself.
This patch unifies the use of constraints so that all constraints are
referenced via the constraints module. In turn, this allows a test to
leverage the re-loadable nature of the constraints in the constraints
module.
It addition, a number of functional tests where using the default
values for constraints, instead of the configured value discovered in
a test.conf or in an existing swift.conf. This patch removes those
direct references in favor of the load_constraint() method from the
test/functional/tests.py module.
Change-Id: Ia5313d653c667dd9ca800786de59b59334c34eaa
Bulk middleware now has a mechanism to retry a delete when the
HTTP response code is 409. This happens when the container still
has objects. It can be useful in a bulk delete where you delete
all the objects in a container and then try to delete the container.
It is very likely that at the end it will fail because the replica
objects have not been deleted by the time the middleware got a
successful response.
Change-Id: I1614fcb5cc511be26a9dda90753dd08ec9546a3c
Closes-Bug: #1253478
These will allow clients to perform the minimal number of requests
required to accomplish some bulk tasks. For example, a client with
many objects to delete can learn that the cluster's limit on
deletes-per-request is, say, 128, and then batch up their deletes in
groups of 128. Without this, the client has to either discover the
limit out-of-band somehow (and get notified if it changes), or do some
sort of binary search to figure out the limit.
Similar reasoning applies to the containers-per-request value.
The errors-per-request values are included so that clients may size
their requests such that everything is attempted regardless of
failure.
I split the 'bulk' entry into 'bulk_delete' and 'bulk_upload' because,
from a client's standpoint, they're separate operations. It so happens
that Swift implements both in one piece of middleware, but clients
don't care.
Bonus fix: documented a missing config setting for the bulk middleware.
Change-Id: Ic3549aef79682fd5b798145c3545c1609aa1592b
* ensure all responses are 200 OK
* report missing sub-SLO manifests or other error messages in bulk
delete response
Change-Id: Iaf88c94bc7114ff3c9751f9f31f8f748de911f8a
This fixes bug: https://bugs.launchpad.net/swift/+bug/1203182 (well
it should :) I don't have keystone installed but the same issue
existed with tempauth).
Change-Id: I6f4045f484b27c0153a9244e0dbf2641cbc9e84e
In unit tests, many test class use tempfile.mkdtemp in SetUp and
shutil.rmtree in tearDown.
A common wrong case is putting test data in `testdir = os.join(mkdtemp() +
'some_class')` and `rmmtree(testdir)`. As a result, /tmp/xxx/some_class
was deleted, but /tmp/xxx is still leaved in system. So a proper fix is
`rmtree(os.path.dirname(testdir), ignore_errors=1)`
fixes bug #1212583
Change-Id: Iafbe3e11f16b51bdf49abce9e68eb01f25bc5df2
Turns out if your $TMPDIR is really long
(e.g. /var/folders/vq/n32yszdn4s76z6l8dxklmwdh0000gn/T/, which is 50
characters), then the test that drops stuff into $TMPDIR and adds it
to a tarball and uploads it will fail due to every added file's name
being too long.
Change-Id: I8fcab2d95091f72b831b906bfc87a36d8be12631
This will allow users to more easily match the failed objects in Errors
for bulk delete requests to the object names they provided in the request.
For extract Errors, it reports the failed path from the tar archive.
DocImpact
Change-Id: I084057810fc4fb7fdac05494cc6fec2cbf81bb9d
Added 411 responses in bulk and slo when needed.
Make X-Static-Large-Object an illegal header with slo installed- somehow
that got lost with some refactor I did.
Change-Id: I986c1656658f874172860469624118cc63bff9bc
header to a query parameter.
This is needed because query parameters show up in proxy logs and headers do
not. With this change it will be easy to determine from any log line that gets
created from the original request (of which there is currently none) that the
request was a bulk action.
Note: This is not backwards compatible with the previous method of setting a
header. Because the bulk middleware has not been included in an openstack swift
release this should be fine.
Change-Id: I0297fa2de9e491bf0b8c430c0781e2e12316ed4b
