Previously, if object versioning was enabled via the old-style
allow_versions container-server setting rather than the new-style
allow_versioned_writes proxy-server setting,
TestCrossPolicyObjectVersioning would skip tests while
TestObjectVersioning and TestObjectVersioningUTF8 would run them.
Additionally, if versioned_writes was explicitly included in the
proxy-server's pipeline and allow_versioned_writes was disabled,
the functional tests would fail with a 412.
Now, all three will use the same logic to check whether versioning is
enabled. Specifically, they will all try to set an X-Versions-Location
header and skip if it doesn't stick.
Additionally, the TestCrossPolicyObjectVersioningEnv will now properly
clean up after itself.
Change-Id: I4c788a0e18587ff17d3c6e346fd22b881495f06d
Previously, attempting to PUT a SLO manifest with `If-None-Match: *`
would include the header when validating the segments, causing the
upload to fail.
Now when SLO validates segments, no conditional headers will be
included in the HEAD request.
Change-Id: I03ad454092d3caa73d29e6d30d8033b45bc96136
Closes-Bug: #1569253
This change removes the use of the COPY request in the versioned
writes middleware. It changes the COPY verb for GETs and PUTs
requests. The main reasoning for this change is to remove any
dependency that versioning had on copy, which will allow for the COPY
functionality to be moved to middleware and to be to the left of the
versioned writes middleware in the proxy pipeline. In this way,
no COPY request will ever arrive at the versioned writes middleware.
A side benefit of this change is that it removes a HEAD request from
the PUT path. Instead of checking if a current version exists, a
GET request is sent, in case of success, a PUT is sent to the
versions container.
A unit test was removed that tested non-default storage policies.
This test is no longer necessary, since it was used to test
specific policy handling code in the COPY method in the proxy
object controller.
Closes-Bug: #1365862
Change-Id: Idf34fa8d04ff292df7134b6d4aa94ff40887b3a4
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Currently, the multipart-manifest=get call returns output in json
format that is inconsistent with the format that is used for the
multipart-manifest=put.
This in turn introduces a new call: ?multipart-manifest=get&format=raw
Change-Id: I2242943a738f667cbda6363bcb6a017f341e834f
Closes-Bug: 1252482
When using an EC policy, 304 responses to conditional GETs
are missing the Accept-Ranges header and have the wrong ETag
value. 412 responses also have the wrong etag.
416 responses to ranged GETs also have the wrong ETag.
This patch ensures behaviour with EC policy is consistent
with replication policy:
- 304 and 416 responses have correct etag and Accept-Ranges
- 412 responses have correct Etag but no Accept-Ranges
Co-Authored-By: Mahati Chamarthy <mahati.chamarthy@gmail.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Closes-Bug: #1496234
Closes-Bug: #1558197
Closes-Bug: #1558193
Change-Id: Ic21317b9e4f632f0751133a3383eb5487379e11f
This patch makes a number of changes to enable content-type
metadata to be updated when using the fast-POST mode of
operation, as proposed in the associated spec [1].
* the object server and diskfile are modified to allow
content-type to be updated by a POST and the updated value
to be stored in .meta files.
* the object server accepts PUTs and DELETEs with older
timestamps than existing .meta files. This is to be
consistent with replication that will leave a later .meta
file in place when replicating a .data file.
* the diskfile interface is modified to provide accessor
methods for the content-type and its timestamp.
* the naming of .meta files is modified to encode two
timestamps when the .meta file contains a content-type value
that was set prior to the latest metadata update; this
enables consistency to be achieved when rsync is used for
replication.
* ssync is modified to sync meta files when content-type
differs between local and remote copies of objects.
* the object server issues container updates when handling
POST requests, notifying the container server of the current
immutable metadata (etag, size, hash, swift_bytes),
content-type with their respective timestamps, and the
mutable metadata timestamp.
* the container server maintains the most recently reported
values for immutable metadata, content-type and mutable
metadata, each with their respective timestamps, in a single
db row.
* new probe tests verify that replication achieves eventual
consistency of containers and objects after discrete updates
to content-type and mutable metadata, and that container-sync
sync's objects after fast-post updates.
[1] spec change-id: I60688efc3df692d3a39557114dca8c5490f7837e
Change-Id: Ia597cd460bb5fd40aa92e886e3e18a7542603d01
test/functional/tests.py:TestObjectVersioningUTF8 does not clean
up the versions files it creates because the class's multiple
inheritance is such that it does not call the tearDown method in
TestObjectVersioning.
As a result, any attempt to clean up account containers wastes
time retrying container delete requests. This occurs either in
the setUp for TestSloEnv, if the TestSlo class is included in a
test run, or in the tests.py package tearDown method.
On the author's dev machine this patch reduces the execution
time of functional tests in tests.py by approx 30% or 1 minute.
Change-Id: I8194672bf2ca82435df5868720b6a55a79b94413
It feels silly, and we don't do it for any of the other headers in
headers_to_container_info.
While we're at it, clean up a stray '
Change-Id: I0745038cc3832a77d064e515c37cacbdcb97c4d9
Related-Change: Iea3d06de80210e9e504e296d4572583d7ffabeac
This patch adds test cases for PUT, DELETE, GET, HEAD, POST and OPTIONS
requests to accounts, containers and objects using various combinations
of users/projects, roles and/or service tokens.
Change-Id: Iea8141ac74ad949a3ae7fa47fda3135d0f2612f6
Multiple etags can be provided on an if-match or if-none-match
request. This is currently being tested in the unit tests, but not
in the functional tests. Since these etags can be modified by
middleware, we need functional tests to assert multiple-etag
requests are handled correctly.
Change-Id: Idc409c85e8aa82b59dc2bc28af6ca2617de82699
When processing keys where the names start with the delimiter
character, swift should list only the delimiter character. To get the
list of nested keys, the caller should also supply the prefix which is
equal to the delimiter.
Added a functional test and unit tests to verify this behavior.
Fixes Bug: 1475018
Change-Id: I27701a31bfa22842c272b7781738e8c546b82cbc
Currently a HTTP_REFERER (Referer) header isn't passed down to
subrequests. This means *LO subrequests to segment containers
return a 403 on a *LO GET when accessed by requests using referer
ACLs.
Currently the only way around referer access to *LO's is to make the
segments container world readable.
This change makes sure the referer header is passed into subrequests
allowing a segments container to only need to be locked down with
the same referer as the *LO container.
This is a 1 line change to code, but also adds a unit and 2 functional
functional tests (one for DLO and one for SLO).
Change-Id: I1fa5328979302d9c8133aa739787c8dae6084f54
Closes-Bug: #1526575
This patch changes functional test classes to subclass
unittest2.TestCase rather than unittest.TestCase.
This fixes errors when attempting to use
tox -e func -- -n <test_path_including_test_method>
and
tox -e func -- --until-failure
Also migrate from using nose.SkipTest to unittest2.SkipTest
Change-Id: I903033f5e01833550b2f2b945894edca4233c4a2
Closes-Bug: 1526725
Co-Authored-By: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
Defcore uses Tempest, which uses Test Repository.
This change makes it easier for Defcore to pull functional
tests from Swift and run them. Additionally, using testr
allows tests to be run in parallel.
Concurrency set to 1 for now, >1 causes failures for
reasons that are still TBD.
With switch to ostestr all the server logs are being sent to stdout
which makes it completely unreadable. Suppressing the logs by default
now with a flag to enable it if desired.
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Robert Collins <rbtcollins@hpe.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
Change-Id: I53ef4a116996a772cf1f3abc2eb0ad60047322d5
Related-Bug: 1177924
Updates the functional test to verify the fix applied by
change Iff7274aa631a92cd7332212ed8b4378c27da4a1f
Change-Id: Iae63ac027e4f4acfe46a36dc1325888b1f834ea4
This change adds the ability to tell the container or account server to
reverse their listings. This is done by sending a reverse=TRUE_VALUE,
Where TRUE_VALUE is one of the values true can be in common/utils:
TRUE_VALUES = set(('true', '1', 'yes', 'on', 't', 'y'))
For example:
curl -i -X GET -H "X-Auth-Token: $TOKEN" $STORAGE_URL/c/?reverse=on
I borrowed the swapping of the markers code from Kevin's old change,
thanks Kevin. And Tim Burke added some real nuggets of awesomeness.
DocImpact
Co-Authored-By: Kevin McDonald <kmcdonald@softlayer.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Implements: blueprint reverse-object-listing
Change-Id: I5eb655360ac95042877da26d18707aebc11c02f6
This commit tries to give the user a reason that their SLO manifest
was invalid instead of just saying "Invalid SLO Manifest File". It
doesn't get every error condition, but it's better than before.
Examples of things that now have real error messages include:
* bad keys in manifest (e.g. using "name" instead of "path")
* bogus range (e.g. "bytes=123-taco")
* multiple ranges (e.g. "bytes=10-20,30-40")
* bad JSON structure (i.e. not a list of objects)
* non-integer size_bytes
Also fixed an annoyance with unspecified-size segments that are too
small. Previously, if you uploaded a segment reference with
'{"size_bytes": null, ...}' in it and the referenced segment was less
than 1 MiB, you'd get a response that looked like this:
HTTP/1.1 400 Bad Request
Content-Length: 62
Content-Type: text/html; charset=UTF-8
X-Trans-Id: txd9ee3b25896642098e4d9-0055dd095a
Date: Wed, 26 Aug 2015 00:33:30 GMT
Each segment, except the last, must be at least 1048576 bytes.
This is true, but not particularly helpful, since it doesn't tell you
which of your segments violated the rule.
Now you get something more like this:
HTTP/1.1 400 Bad Request
Content-Length: 49
Content-Type: text/plain
X-Trans-Id: tx586e52580bac4956ad8e2-0055dd09c2
Date: Wed, 26 Aug 2015 00:35:14 GMT
Errors:
/segs/small, Too Small; each segment, except the last...
It's not exactly a tutorial on SLO manifests, but at least it names
the problematic segment.
This also changes the status code for a self-referential manifest from
409 to 400. The rest of the error machinery was using 400, and
special-casing self-reference would be really annoying. Besides, now
that we're showing more than one error message at a time, what would
the right status code be for a manifest with a self-referential
segment *and* a segment with a bad range? 400? 409? 404.5? It's much
more consistent to just say invalid manifest --> 400.
Change-Id: I2275683230b36bc273319254e37c16b9e9b9d69c
assertEquals is deprecated in py3, replacing it.
Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8
Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
The urllib, urllib2 and urlparse modules of Python 2 were reorganized
into a new urllib namespace on Python 3. Replace urllib, urllib2 and
urlparse imports with six.moves.urllib to make the modified code
compatible with Python 2 and Python 3.
The initial patch was generated by the urllib operation of the sixer
tool on: bin/* swift/ test/.
Change-Id: I61a8c7fb7972eabc7da8dad3b3d34bceee5c5d93
The unicode type was renamed to str in Python 3. Use six.text_type to
make the modified code compatible with Python 2 and Python 3.
The initial patch was generated by the unicode operation of the sixer
tool on: bin/* swift/ test/.
Change-Id: I9e13748ccde36ee8110756202d55d3ae945d4860
Users can now include an optional 'range' field in segment descriptions
to specify which bytes from the underlying object should be used for the
segment data. Only one range may be specified per segment. Note that the
'etag' and 'size_bytes' fields still describe the backing object as a
whole. So, if a user uploads a manifest like:
[{"path": "/con/obj_seg_1", "etag": null, "size_bytes": 1048576,
"range": "0-1023"},
{"path": "/con/obj_seg_2", "etag": null, "size_bytes": 1048576,
"range": "512-4095"},
{"path": "/con/obj_seg_1", "etag": null, "size_bytes": 1048576,
"range": "-2048"}]
then the segment will consist of the first 1024 bytes of /con/obj_seg_1,
followed by bytes 513 through 4096 (inclusive) of /con/obj_seg_2, and
finally bytes 1046528 through 1048576 (i.e., the last 2048 bytes) of
/con/obj_seg_1.
ETag generation for SLOs had been updated to prevent collisions when
using different ranges for the same set of objects.
Additionally, there are two performance enhancements:
* On download, multiple sequential requests for segments from the same
underlying object will be coalesced into a single ranged request,
provided it still does not meet Swift's "egregious range requests"
critieria.
* On upload, multiple sequential segments referencing the same object
will be validated against the response from a single HEAD request.
Change-Id: Ia21d51c2cef4e2ee5162161dd2c1d3069009b52c
DocImpact
Follow up to [1] to add tests for tempurl POSTs not being allowed
to set a DLO manifest header.
[1] I11e68830009d3f6bff44ae4011a41b67139146f6
Change-Id: I7c0ad5a936f71e56c599b8495a586913d3334422
Related-Bug: 1453948
It used to be that a GET of a tempurl referencing a large object would
let you download that large object regardless of where its segments
lived. However, this led to some violated user expectations around
container tempurls.
(Note on shorthand: all tempurls reference objects. However, "account
tempurl" and "container tempurl" are shorthand meaning tempurls
generated using a key on the account or container, respectively.)
Let's say an application is given tempurl keys to a particular
container, and it does all its work therein using those keys. The user
expects that, if the application is compromised, then the attacker
only gains access to the "compromised-container". However, with the old
behavior, the attacker could read data from *any* container like so:
1) Choose a "victim-container" to download
2) Create PUT and GET tempurl for any object name within the
"compromised-container". The object doesn't need to exist;
we'll create it.
3) Using the PUT tempurl, upload a DLO manifest with
"X-Object-Manifest: /victim-container/"
4) Using the GET tempurl, download the object created in step 3. The
result will be the concatenation of all objects in the
"victim-container".
Step 3 need not be for all objects in the "victim-container"; for
example, a value "X-Object-Manifest: /victim-container/abc" would only
be the concatenation of all objects whose names begin with "abc". By
probing for object names in this way, individual objects may be found
and extracted.
A similar bug would exist for manifests referencing other accounts
except that neither the X-Object-Manifest (DLO) nor the JSON manifest
document (SLO) have a way of specifying a different account.
This change makes it so that a container tempurl only grants access to
objects within its container, *including* large-object segments. This
breaks backward compatibility for container tempurls that may have
pointed to cross container *LO's, but (a) there are security
implications, and (b) container tempurls are a relatively new feature.
This works by having the tempurl middleware install an authorization
callback ('swift.authorize' in the WSGI environment) that limits the
scope of any requests to the account or container from which the key
came.
This requires swift.authorize to persist for both the manifest request
and all segment requests; this is done by having the proxy server
restore it to the WSGI environment prior to returning from __call__.
[CVE-2015-5223]
Co-Authored-By: Clay Gerrard <clayg@swiftstack.com>
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Christian Schwede <cschwede@redhat.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: Ie6d52f7a07e87f6fec21ed8b0ec1d84be8b2b11c
Closes-Bug: 1449212
Do not allow PUT tempurls to create pointers to other data. Specifically
disallow the creation of DLO object manifests by returning an error if a
non-safe tempurl request includes an X-Object-Manifest header regardless of
the value of the header.
This prevents discoverability attacks which can use any PUT tempurl to probe
for private data by creating a DLO object manifest and then using the PUT
tempurl to head the object which would 404 if the prefix does not match any
object data or form a valid DLO HEAD response if it does.
This also prevents a tricky and potentially unexpected consequence of PUT
tempurls which would make it unsafe to allow a user to download objects
created by tempurl (even if they just created them) because the result of
reading the object created via tempurl may not be the data which was uploaded.
[CVE-2015-5223]
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Change-Id: I11e68830009d3f6bff44ae4011a41b67139146f6
Closes-Bug: 1453948
When using fast-post and POST (i.e. metadata update) is requested to
a SLO manifest files, current Swift drops the 'X-Static-Large-Object'
header from the existing metadata. It results in breaking the SLO
state because the manifest missing the 'X-Static-Large-Object' metadata
will be maintained as a normal files.
This patch fixes object-server to keep the existing
'X-Static-Large-Object' flag and then keep the SLO state.
Change-Id: Ib1eb569071372c322dd105c52baeeb094003291e
Closes-bug: #1453807
The TestCase.assert_() has been deprecated in Python 2.7. Replace it
with assertTrue() or even better methods (assertIn, assertNotIn,
assertIsInstance) which provide better error messages.
Change-Id: I21c730351470031a2dabe5238693095eabdb8964
Rewrite object versioning as middleware to simplify the PUT method
in the object controller.
The functionality remains basically the
same with the only major difference being the ability to now
version slo manifest files. dlo manifests are still not
supported as part of this patch.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
DocImpact
Change-Id: Ie899290b3312e201979eafefb253d1a60b65b837
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Signed-off-by: Prashanth Pai <ppai@redhat.com>
The replacement of assert_ with assertTrue [1] resulted in a number of
tests using calls of the form assertTrue(not <condition>). This patch
replaces those with assertFalse(<condition>).
[1] change I74705c6498249337bfdf955d62e0ad972035bc1f
Change-Id: I78b49558f4425c2335df187b1793d1e4b3c514b1
We have a bunch of assertions like
self.assertTrue(resp.status in (200, 204))
Sometimes we get smart about failure messages and have something like
self.assertTrue(resp.status in (200, 204), resp.status)
so we can see what the status was when it failed.
Since we don't have to support Python 2.6 any more, we can use
assertIn/assertNotIn and get nice failure messages for free.
Change-Id: I2d46c9969d41207a89e01017b4c2bc533c3d744f
The assert_() method is deprecated and can be safely replaced by assertTrue().
This patch makes sure that running the tests does not generate warnings
all over the screen.
Change-Id: I74705c6498249337bfdf955d62e0ad972035bc1f
The first test verifies that a delimiter will trim entries beyond the
first matching instance of delimiter (after the given matching prefix,
if any) and squash duplicates. So, when setting the delimiter
to "-", given blobs "test", "test-foo" and "test-bar-baz", we expect
only "test" (no matching delim) and "test-" (trim all characters after
the first "-", and squash duplicates).
The second test verifies that when a prefix is provided, the delimiter
will trim entries beyond the first matching instance of the delimiter
*after the given prefix*. So "bar, "bazar" which both match the
prefix "ba" will be returned as "bar" (no delimiter after the matching
prefix) and "baza" (after matching the prefix the remainder after the
the *next* matching delimiter "a" is trimmed).
Change-Id: I49a2aa8722f83e87b7d211e5c26827e93963d92a
* replace "from cStringIO import StringIO"
with "from six.moves import cStringIO as StringIO"
* replace "from StringIO import StringIO"
with "from six import StringIO"
* replace "import cStringIO" and "cStringIO.StringIO()"
with "from six import moves" and "moves.cStringIO()"
* replace "import StringIO" and "StringIO.StringIO()"
with "import six" and "six.StringIO()"
This patch was generated by the stringio operation of the sixer tool:
https://pypi.python.org/pypi/sixer
Change-Id: Iacba77fec3045f96773d1090c0bd48613729a561
When functional tests are run in tox and an exception is raised when connecting
to Swift (for example: Swift not running, missing python-keystoneclient package
used by python-swiftclient) 0 tests are executed, but tox returns a success.
An exception is raised during tests, caused by a missing python-keystoneclient
in python-swiftclient. Instead of adding python-keystoneclient as a dependency
in python-swiftclient the package is added to the test-requirements.txt in Swift
itself. Note that adding python-keystoneclient to the test-requirements in
python-swiftclient is not sufficient (it's already in there).
The exception in setup_package() is catched by the openstack.nose_plugin, thus
disabling this plugin for now as well.
Also fixing two test errors seen on the gate regarding the tempurl middleware.
There was also an update to tox, environment variables were no longer passed
with versions >= 2.0 (http://tox.readthedocs.org/en/latest/changelog.html).
Swift test environment variables have been added to the passenv to re-enable the
former behavior, as well as environment variables required to pass proxy
settings.
This also led to skipped tempauth tests, and together with the missing
python-keystoneclient no tests were executed.
Related-Bug: 1461440
Related-Bug: 1455102
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Change-Id: Ideea071017d04912c60ed0bc76532adbb446c31d
While manifests still require 'etag' and 'size_bytes' fields for each
segment (to catch user errors like 'etaf' or 'size_btyes'), an explicit
null for either will skip that particular integrity check and instead
use whatever value is retrieved when HEADing the segment. So, if a user
uploads a manifest like:
[{"path": "/con/obj_seg_1", "etag": null, "size_bytes": 1048576},
{"path": "/con/obj_seg_2", "etag": "etag2", "size_bytes": null},
{"path": "/con/obj_seg_3", "etag": null, "size_bytes": null}]
then the etag will only be verified for the /con/obj_seg_2 segment,
and the segment size will only be verified for the /con/obj_seg_1
segment. However, the manifest that's ultimately stored (and can be
retrieved with a ?multipart-manifest=get query-string) will still look
like:
[{"name": "/con/obj_seg_1", "hash": "etag1", "bytes": 1048576, ...},
{"name": "/con/obj_seg_2", "hash": "etag2", "bytes": 1048576, ...},
{"name": "/con/obj_seg_3", "hash": "etag3", "bytes": 1234, ...}]
This allows the middleware to continue performing integrity checks on
object GET.
Change-Id: I2c4e585221387dd02a8679a50398d6b614407b12
DocImpact
