Rewrite object versioning as middleware to simplify the PUT method
in the object controller.
The functionality remains basically the
same with the only major difference being the ability to now
version slo manifest files. dlo manifests are still not
supported as part of this patch.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
DocImpact
Change-Id: Ie899290b3312e201979eafefb253d1a60b65b837
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Signed-off-by: Prashanth Pai <ppai@redhat.com>
The replacement of assert_ with assertTrue [1] resulted in a number of
tests using calls of the form assertTrue(not <condition>). This patch
replaces those with assertFalse(<condition>).
[1] change I74705c6498249337bfdf955d62e0ad972035bc1f
Change-Id: I78b49558f4425c2335df187b1793d1e4b3c514b1
We have a bunch of assertions like
self.assertTrue(resp.status in (200, 204))
Sometimes we get smart about failure messages and have something like
self.assertTrue(resp.status in (200, 204), resp.status)
so we can see what the status was when it failed.
Since we don't have to support Python 2.6 any more, we can use
assertIn/assertNotIn and get nice failure messages for free.
Change-Id: I2d46c9969d41207a89e01017b4c2bc533c3d744f
The assert_() method is deprecated and can be safely replaced by assertTrue().
This patch makes sure that running the tests does not generate warnings
all over the screen.
Change-Id: I74705c6498249337bfdf955d62e0ad972035bc1f
The first test verifies that a delimiter will trim entries beyond the
first matching instance of delimiter (after the given matching prefix,
if any) and squash duplicates. So, when setting the delimiter
to "-", given blobs "test", "test-foo" and "test-bar-baz", we expect
only "test" (no matching delim) and "test-" (trim all characters after
the first "-", and squash duplicates).
The second test verifies that when a prefix is provided, the delimiter
will trim entries beyond the first matching instance of the delimiter
*after the given prefix*. So "bar, "bazar" which both match the
prefix "ba" will be returned as "bar" (no delimiter after the matching
prefix) and "baza" (after matching the prefix the remainder after the
the *next* matching delimiter "a" is trimmed).
Change-Id: I49a2aa8722f83e87b7d211e5c26827e93963d92a
* replace "from cStringIO import StringIO"
with "from six.moves import cStringIO as StringIO"
* replace "from StringIO import StringIO"
with "from six import StringIO"
* replace "import cStringIO" and "cStringIO.StringIO()"
with "from six import moves" and "moves.cStringIO()"
* replace "import StringIO" and "StringIO.StringIO()"
with "import six" and "six.StringIO()"
This patch was generated by the stringio operation of the sixer tool:
https://pypi.python.org/pypi/sixer
Change-Id: Iacba77fec3045f96773d1090c0bd48613729a561
When functional tests are run in tox and an exception is raised when connecting
to Swift (for example: Swift not running, missing python-keystoneclient package
used by python-swiftclient) 0 tests are executed, but tox returns a success.
An exception is raised during tests, caused by a missing python-keystoneclient
in python-swiftclient. Instead of adding python-keystoneclient as a dependency
in python-swiftclient the package is added to the test-requirements.txt in Swift
itself. Note that adding python-keystoneclient to the test-requirements in
python-swiftclient is not sufficient (it's already in there).
The exception in setup_package() is catched by the openstack.nose_plugin, thus
disabling this plugin for now as well.
Also fixing two test errors seen on the gate regarding the tempurl middleware.
There was also an update to tox, environment variables were no longer passed
with versions >= 2.0 (http://tox.readthedocs.org/en/latest/changelog.html).
Swift test environment variables have been added to the passenv to re-enable the
former behavior, as well as environment variables required to pass proxy
settings.
This also led to skipped tempauth tests, and together with the missing
python-keystoneclient no tests were executed.
Related-Bug: 1461440
Related-Bug: 1455102
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Change-Id: Ideea071017d04912c60ed0bc76532adbb446c31d
While manifests still require 'etag' and 'size_bytes' fields for each
segment (to catch user errors like 'etaf' or 'size_btyes'), an explicit
null for either will skip that particular integrity check and instead
use whatever value is retrieved when HEADing the segment. So, if a user
uploads a manifest like:
[{"path": "/con/obj_seg_1", "etag": null, "size_bytes": 1048576},
{"path": "/con/obj_seg_2", "etag": "etag2", "size_bytes": null},
{"path": "/con/obj_seg_3", "etag": null, "size_bytes": null}]
then the etag will only be verified for the /con/obj_seg_2 segment,
and the segment size will only be verified for the /con/obj_seg_1
segment. However, the manifest that's ultimately stored (and can be
retrieved with a ?multipart-manifest=get query-string) will still look
like:
[{"name": "/con/obj_seg_1", "hash": "etag1", "bytes": 1048576, ...},
{"name": "/con/obj_seg_2", "hash": "etag2", "bytes": 1048576, ...},
{"name": "/con/obj_seg_3", "hash": "etag3", "bytes": 1234, ...}]
This allows the middleware to continue performing integrity checks on
object GET.
Change-Id: I2c4e585221387dd02a8679a50398d6b614407b12
DocImpact
Current post as copy routine (i.e. POST object with post_as_copy option
turned on) on Object Controller uses "multipart-manifest" query string
which is feeded to env['copy_hook'] to decide which data (the manifest or
object pointed by the manifest) should be copied.
However, the way using the query string will confuse operators looking at
logging system (or analyzing the log) because whole POST object requests
have 'multipart-manifest=get' like as:
POST /v1/AUTH_test/d4c816b24d38489082f5118599a67920/manifest-abcde%3Fmultipart-manifest%3Dget
We cannot know whether the query string was added by hand
(from user) or not. In addition, the query isn't needed by the
backend conversation between proxy-server and object-server.
(Just needed by "copy_hook" on the proxy controller!)
To remove the confusable query string and to keep the log to be clean,
this patch introduces new environment variable "swift.post_as_copy"
and changes proxy controller and the copy_hook to use the new env.
This item was originally discussed at
https://review.openstack.org/#/c/177132/
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Change-Id: I0cd37520eea1825a10ebd27ccdc7e9162647233e
An authenticated user can delete the most recent version of any
versioned object who's name is known if the user has listing access
to the x-versions-location container. Only Swift setups with
allow_version setting are affected.
This patch closes this bug, tracked as CVE-2015-1856
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Christian Schwede <info@cschwede.de>
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Closes-Bug: 1430645
Change-Id: Ibacc7413afe7cb6f77d92e5941dcfdf4768ffa18
* Get FakeConn ready for expect 100 continue
* Use debug_logger more and with better interfaces
* Fix patch_policies to be less annoying
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I28c0a3539d994cbb8e6b94d63a23ed4ea6cb956d
Current behavior:
* If data/body is present in manifest file PUT request, the data/body gets
saved onto disk, just like for a normal object.
* Generally, this data in manifest file is never served on a GET response.
However, when the manifest object path itself is part of prefix, GET
response would contain data present in manifest file as well.
* The query param multipart-manifest=get meant to retrieve SLO manifest
also works in case of DLO manifest. Hence a COPY request with the
multipart-manifest=get query param would actually copy DLO manifest.
How things should have been:
* The DLO manifest object is supposed to have no content and only have
X-Object-Manifest metadata header.
* Query param multipart-manifest=get is SLO specific and shouldn't have
any role in DLO.
This change intends to only document current behaviour and not change it,
assuming there are users who have previously saved some content in DLO
manifest file and/or have been using multipart-manifest=get to fetch
and/or COPY the DLO manifest file with it's content.
Change-Id: I0f6e175ad7752169ecf94df949336e0665928df7
Signed-off-by: Prashanth Pai <ppai@redhat.com>
According to documentation dlo manifest files should not
be versioned. This patch fixes this issue and adds
some unit and functional for this scenario.
Change-Id: Ib5b29a19e1d577026deb50fc9d26064a8da81cd7
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Trivial patch to tidy-up change to the functional test for
www-authenticate header and add a comment to explain
that multiple header values might be returned.
Change-Id: If62cb3fd9e11450a2be0cec71e80ecb74a959d04
Related-bug: 1368048
testQuotedWWWAuthenticateHeader functional test started failing
due to a change to keystonemiddleware.auth_token, which now adds
its own www-authenticate header in addition to the one that swift
keystoneauth adds.
This patch changes the functional test to check expected
swift generated header value is in the concatenation of
www-authenticate values.
Verified that functional tests still pass using tempauth.
Closes-Bug: 1368048
Change-Id: I913af077df800a559d259c1622f286ad10eae9df
This adds a sanity check on x-delete headers as
part of check_object_creation method
Change-Id: If5069469e433189235b1178ea203b5c8a926f553
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Adds ability to copy objects between different accounts (on server side)
Adds new header to `PUT` request:
`X-Copy-From-Account: <account name>`
Account name corresponds to the last part of storage URL.
Adds new header to `COPY` request:
`Destination-Account: <account name>`
Account name corresponds to the last part of storage URL.
If your storage URL is: http://server:8080/v1/AUTH_test
Then the account name is `AUTH_test`
These headers should be used alongside `X-Copy-From` and `Destination` headers
The legacy headers should specify `<container name>/<object name>` path as usual.
DocImpact
Change-Id: I0285fe6a47df9e699ac20ae4a83b0bf23829e1e6
Replaced throughout code base & tox'd. Functional as well
as probe tests pass with and without policies defined.
POLICY --> 'X-Storage-Policy'
POLICY_INDEX --> 'X-Backend-Storage-Policy-Index'
Change-Id: Iea3d06de80210e9e504e296d4572583d7ffabeac
* additional container tests
* refactor test cross policy copy
* make functional tests cleanup better
In-process functional tests only define a single ring and will skip some of
the multi-storage policy tests, but have been updated to reload_policies with
the patched swift.conf.
DocImpact
Implements: blueprint storage-policies
Change-Id: If17bc7b9737558d3b9a54eeb6ff3e6b51463f002
HTTP header values should be quoted. Since the WWW-Authenticate
header value contains user-supplied strings, it's important to
ensure it's properly quoted to ensure the integrity of the protocol.
Previous to this patch, the URL was unquoted and then the unquoted
value was returned in the header. This patch re-quotes the value
when it is set on the response.
This is filed as CVS-2014-3497
Fixes bug 1327414
Change-Id: If8bd8842f2ce821756e9b4461a18a8ac8d42fb8c
The tempurl functional tests would fail if you put tempurl.methods in
the proxy server's disallowed_sections. The test assumed that the
presence of the key 'tempurl' implied the presence of the subkey
'methods', but since 9cbf8a3 landed, operators can remove arbitrary
things from /info.
Normally I'd just change x['methods'] to x.get('methods', []), but it
turns out this was setting up a class variable that nobody ever looked
out again, so removing the code also works.
Change-Id: Ie899d146bc6fff81a5fae77815897244e8ec6bec
Provide a way to run the functional tests using a p/a/c/o server setup
in the same process running the nosetests infrastructure.
By setting the environment variable, SWIFT_TEST_IN_PROCESS, to a true
value, the functional test framework will construct a set of proxy,
account, container and object servers running in the same process that
is running the functional tests, ignoring any external swift
service. This in-process swift environment is akin to the one used in
test/unit/proxy/test_server.py.
Setting that same environment variable to a false value will ensure the
in-process servers are not used.
When the above environment variable is not present, and the
/etc/swift/test.conf is _not_ present (or present but empty) on the
system where the functional tests are executing, the in-process
environment will be used. Previously, if no /etc/swift/test.conf file
was found, the tests would just be marked as skipped.
Using this in-process method allows one to gather code coverage using
the functional tests to exercise code paths, in addition to the unit
tests, or more easily debug existing functional tests, or even write new
ones.
There are two constraints that are changed for use with the in-process
functional tests: max_file_size is lowered to roughly 8 MB, and
client_timeout is set to 4s.
Change-Id: I5acd65e3068868d6509feae1d1954237d37fad45
Use constrainst from the new "swift-constraints" section of test.conf,
fall back to those found in the response to the /info API call,
ultimately falling back to the constraints module's effective
constraints.
Change-Id: Iea01c9c4b5148faa10004a240df411cbe7336a6a
The only explicit use of Python threading is found in the
testFileSizeLimit test. Using eventlet seems a bit easier to follow,
accomplishing the same goal, and does not constrain us to a
multi-threaded environment.
The chunks() and timeout() module level functions are only used by one
test each, so we just move them to those tests to indicate they are not
used globally.
Change-Id: I50b9fb798fbfd1d552b3c3f90309f6b86da34853
Merge the swift_testing module into the functional test module itself,
so that we can read the configuration once for all unit tests, sharing
the same constraints.
Change-Id: I9fbbfdade9adca329cd79f7d4291ba009327c842
The unconditional module-level change of the locale collation can
affect other tests run under nose. To avoid this, make the change at
the package level, and restore it when finished.
Change-Id: Iacda42aa7155f1cfa7d0dcadbf73b961dfb4b923
We already supported it for object GET requests, but not for
HEAD. This lets clients keep metadata up-to-date without having to
either fetch the whole object when it's changed or do their own date
parsing. They can just treat Last-Modified as opaque and update their
idea of metadata when they get a 200.
Change-Id: Iff25d8989a93d651fd2c327e1e58036e79e1bde1
Looks like we are keeping at least four lists of the default
constraints in various places, and when we added the max_header_size
in initial Static Large Object Support commit (5d73da15) we failed to
add it to the two locations in the test tree (test/sample.conf and
test/functional/tests.py).
With this commit we take a small step to consolidate the list of
constraints to the constraints module by removing the list from the
tests module (note we still have them listed in the sample swift.conf
file and the test module's sample.conf file), document the missing
reference in the test module's sample.conf file, and, while we're at
it, use the SWIFT_CONF_FILE reference where possible.
Change-Id: Ic4fa5f1aedec9bc7a7b43ce1a2ca3cdebce6171f
If a client issues a COPY request for a DLO with the query param
multipart-manifest=get, then the expectation is a new
manifest. However, this wasn't the case; X-Object-Manifest wasn't
being set on the new object, so the result was a normal object with
the same contents as the old manifest (typically 0 bytes).
There was already a mechanism by which middlewares could modify COPY
requests; this commit extends that so they can set headers on the new
object.
Note that this has nothing to do with a "normal" DLO copy, i.e. one
without multipart-manifest=get. That one makes a new object that's the
concatenation of the segments, and it was working just fine.
Change-Id: I1073af9fee6e34ebdfad7b1a89aeb05e4523a151
