Since JSON object listings deserialize as unicode, obj['name'] would
hoist *everything* to unicode. If the account or container name was a
byte string, though, it would trip a UnicodeDecodeError.
Change-Id: I2c1932143b78521c6bdcfa48182b475528fc1bb3
The existing test works fine if you're running the tests on an
all-in-one, but is pretty brittle if you aren't running them on the
one and only proxy-server they're hitting.
Add 0.1s sleep to allow *some* clock slippage between client and server.
Change-Id: Iacd08e9f703d08d0092b5e8eb53fe287ba1d1596
Currently, our integrity checking for objects is pretty weak when it
comes to object metadata. If the extended attributes on a .data or
.meta file get corrupted in such a way that we can still unpickle it,
we don't have anything that detects that.
This could be especially bad with encrypted etags; if the encrypted
etag (X-Object-Sysmeta-Crypto-Etag or whatever it is) gets some bits
flipped, then we'll cheerfully decrypt the cipherjunk into plainjunk,
then send it to the client. Net effect is that the client sees a GET
response with an ETag that doesn't match the MD5 of the object *and*
Swift has no way of detecting and quarantining this object.
Note that, with an unencrypted object, if the ETag metadatum gets
mangled, then the object will be quarantined by the object server or
auditor, whichever notices first.
As part of this commit, I also ripped out some mocking of
getxattr/setxattr in tests. It appears to be there to allow unit tests
to run on systems where /tmp doesn't support xattrs. However, since
the mock is keyed off of inode number and inode numbers get re-used,
there's lots of leakage between different test runs. On a real FS,
unlinking a file and then creating a new one of the same name will
also reset the xattrs; this isn't the case with the mock.
The mock was pretty old; Ubuntu 12.04 and up all support xattrs in
/tmp, and recent Red Hat / CentOS releases do too. The xattr mock was
added in 2011; maybe it was to support Ubuntu Lucid Lynx?
Bonus: now you can pause a test with the debugger, inspect its files
in /tmp, and actually see the xattrs along with the data.
Since this patch now uses a real filesystem for testing filesystem
operations, tests are skipped if the underlying filesystem does not
support setting xattrs (eg tmpfs or more than 4k of xattrs on ext4).
References to "/tmp" have been replaced with calls to
tempfile.gettempdir(). This will allow setting the TMPDIR envvar in
test setup and getting an XFS filesystem instead of ext4 or tmpfs.
THIS PATCH SIGNIFICANTLY CHANGES TESTING ENVIRONMENTS
With this patch, every test environment will require TMPDIR to be
using a filesystem that supports at least 4k of extended attributes.
Neither ext4 nor tempfs support this. XFS is recommended.
So why all the SkipTests? Why not simply raise an error? We still need
the tests to run on the base image for OpenStack's CI system. Since
we were previously mocking out xattr, there wasn't a problem, but we
also weren't actually testing anything. This patch adds functionality
to validate xattr data, so we need to drop the mock.
`test.unit.skip_if_no_xattrs()` is also imported into `test.functional`
so that functional tests can import it from the functional test
namespace.
The related OpenStack CI infrastructure changes are made in
https://review.openstack.org/#/c/394600/.
Co-Authored-By: John Dickinson <me@not.mn>
Change-Id: I98a37c0d451f4960b7a12f648e4405c6c6716808
In test.functional.test_object.TestObject.setUp, we create a container
in account 2. However, if we've only got one account, we don't skip
this class, resulting in a TypeError down in requests somewhere and a
stack trace. Since we're using account 2 in setup, we should skip the
tests if account 2 is not configured.
Change-Id: I569d98baf071d2dce7cf34a9538070f00afda388
This looks like a case of copy-paste-itis. The cross-account-copy
functest is skipped if we have no test accounts configured, but not if
we have only one.
Change-Id: Ifbefdd9aeb98e3d02c536e9d29759f86ec9af6a1
X-Delete-After: 1 is known to be flakey; use 2 instead.
When the proxy receives an X-Delete-After header, it automatically
converts it to an X-Delete-At header based on the current time. So far,
so good. But in normalize_delete_at_timestamp we convert our
time.time() + int(req.headers['X-Delete-After'])
to a string representation of an integer and in the process always round
*down*. As a result, we lose up to a second worth of object validity,
meaning the object server can (rarely) respond 400, complaining that the
X-Delete-At is in the past.
Change-Id: Ib5e5a48f5cbed0eade8ba3bca96b26c82a9f9d84
Related-Change: I643be9af8f054f33897dd74071027a739eaa2c5c
Related-Change: I10d3b9fcbefff3c415a92fa284a1ea1eda458581
Related-Change: Ifdb1920e5266aaa278baa0759fc0bfaa1aff2d0d
Related-Bug: #1597520
Closes-Bug: #1699114
When an object is deleted, we retrieve its metadata on the next PUT
requests, which make if-none-match requests fail while the object
shouldn't be here. It seems we're only interested in the timestamp of
the deleted object, so get that but forget the metadata.
Change-Id: I4a4a8be9b631598ca1cd52c53885c68c3fbdfc4a
Closes-Bug: #1640448
I changed asserts with more specific assert methods.
e.g.: from assertTrue(sth == None) to assertIsNone(*) or
assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type) or
assertTrue(not sth) to assertFalse(sth).
The code gets more readable, and a better description will be shown on fail.
Change-Id: I80ec96e0b729bef38213a6be4ff4b6eb65c7612d
With the X-Timestamp validation added in commit e619411, end users
could upload objects with
X-Timestamp: 9999999999.99999_ffffffffffffffff
(the maximum value) and Swift would be unable to delete them.
Now, inbound X-Timestamp headers will be moved to
X-Backend-Inbound-X-Timestamp, effectively rendering them harmless.
The primary reason to allow X-Timestamp before was to prevent
Last-Modified changes for objects coming from either:
* container_sync or
* a migration from another storage system.
To enable the former use-case, the container_sync middleware will now
translate X-Backend-Inbound-X-Timestamp headers back to X-Timestamp
after verifying the request.
Additionally, a new option is added to the gatekeeper filter config:
# shunt_inbound_x_timestamp = true
To enable the latter use-case (or any other use-case not mentioned), set
this to false.
Upgrade Consideration
=====================
If your cluster workload requires that clients be allowed to specify
objects' X-Timestamp values, disable the shunt_inbound_x_timestamp
option before upgrading.
UpgradeImpact
Change-Id: I8799d5eb2ae9d795ba358bb422f69c70ee8ebd2c
Previously, if a user could write to (but not read from) a container,
the behavior for object POST would vary depending on whether
object_post_as_copy was enabled (403 response) or disabled (202
response).
Now, POSTs will consistently be allowed, regardless of whether fast-POST
is enabled.
Change-Id: I1d6dcbc4f5034a322a1073850fc3b059ebb1c0fa
This patch adds test cases for PUT, DELETE, GET, HEAD, POST and OPTIONS
requests to accounts, containers and objects using various combinations
of users/projects, roles and/or service tokens.
Change-Id: Iea8141ac74ad949a3ae7fa47fda3135d0f2612f6
This patch changes functional test classes to subclass
unittest2.TestCase rather than unittest.TestCase.
This fixes errors when attempting to use
tox -e func -- -n <test_path_including_test_method>
and
tox -e func -- --until-failure
Also migrate from using nose.SkipTest to unittest2.SkipTest
Change-Id: I903033f5e01833550b2f2b945894edca4233c4a2
Closes-Bug: 1526725
Co-Authored-By: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
Defcore uses Tempest, which uses Test Repository.
This change makes it easier for Defcore to pull functional
tests from Swift and run them. Additionally, using testr
allows tests to be run in parallel.
Concurrency set to 1 for now, >1 causes failures for
reasons that are still TBD.
With switch to ostestr all the server logs are being sent to stdout
which makes it completely unreadable. Suppressing the logs by default
now with a flag to enable it if desired.
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Robert Collins <rbtcollins@hpe.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
Change-Id: I53ef4a116996a772cf1f3abc2eb0ad60047322d5
Related-Bug: 1177924
We have a bunch of assertions like
self.assertTrue(resp.status in (200, 204))
Sometimes we get smart about failure messages and have something like
self.assertTrue(resp.status in (200, 204), resp.status)
so we can see what the status was when it failed.
Since we don't have to support Python 2.6 any more, we can use
assertIn/assertNotIn and get nice failure messages for free.
Change-Id: I2d46c9969d41207a89e01017b4c2bc533c3d744f
The assert_() method is deprecated and can be safely replaced by assertTrue().
This patch makes sure that running the tests does not generate warnings
all over the screen.
Change-Id: I74705c6498249337bfdf955d62e0ad972035bc1f
Currently, a COPY request for an EC object might go to fail as 499 Client
disconnected because of the difference between destination request content
length and actual transferred bytes.
That is because the conditional response status and content length for
an EC object range GET is handled at calling the response instance on
proxy server. Therefore the calling response instance (resp()) will change
the conditional status from 200 (HTTP_OK) to 206 (PartialContent) and will
change the content length for the range GET.
In EC case, sometimes Swift needs whole stored contents to decode a segment.
It will make 200 HTTP OK response from object-server and proxy-server
will unfortunately set whole content length to the destination content
length and it makes the bug 1467677.
This patch introduces a new method "fix_conditional_response" for
swift.common.swob.Response that calling _response_iter() and cached the
iter in the Response instance. By calling it, Swift can set correct condtional
response any time after setting whole content_length to the response
instance like EC case.
Change-Id: If85826243f955d2f03c6ad395215c73daab509b1
Closes-Bug: #1467677
Since we're dropping Python 2.6 support, we can rely on stdlib's json
and get rid of our dependency on simplejson.
This commit just takes simplejson out of the unit and functional
tests. They still pass.
Change-Id: I96f17df81fa5d265395a938b19213d2638682106
Adds ability to copy objects between different accounts (on server side)
Adds new header to `PUT` request:
`X-Copy-From-Account: <account name>`
Account name corresponds to the last part of storage URL.
Adds new header to `COPY` request:
`Destination-Account: <account name>`
Account name corresponds to the last part of storage URL.
If your storage URL is: http://server:8080/v1/AUTH_test
Then the account name is `AUTH_test`
These headers should be used alongside `X-Copy-From` and `Destination` headers
The legacy headers should specify `<container name>/<object name>` path as usual.
DocImpact
Change-Id: I0285fe6a47df9e699ac20ae4a83b0bf23829e1e6
If a swift cluster configures the proxy server to not accept
/info requests, just do the best we can, skipping tests as
necessary.
To that end, if cors mode is not known we also skip the test.
Change-Id: I34f296c8717e0baf24ae552a7e38f9354ee91974
* additional container tests
* refactor test cross policy copy
* make functional tests cleanup better
In-process functional tests only define a single ring and will skip some of
the multi-storage policy tests, but have been updated to reload_policies with
the patched swift.conf.
DocImpact
Implements: blueprint storage-policies
Change-Id: If17bc7b9737558d3b9a54eeb6ff3e6b51463f002
Use constrainst from the new "swift-constraints" section of test.conf,
fall back to those found in the response to the /info API call,
ultimately falling back to the constraints module's effective
constraints.
Change-Id: Iea01c9c4b5148faa10004a240df411cbe7336a6a
Merge the swift_testing module into the functional test module itself,
so that we can read the configuration once for all unit tests, sharing
the same constraints.
Change-Id: I9fbbfdade9adca329cd79f7d4291ba009327c842
A common pattern that we see clients do is send a HEAD request before a
PUT to see if it exists. This can slow things down quite a bit
especially since 404s on HEAD are currently a bit expensive.
This change will allow a client to include a "If-None-Match: *" header
with a PUT request. In combination with "Expect: 100-Continue" this
allows the server to return that it already has a copy of the object
before any data is sent.
I attempted to also include etag support with the If-None-Match header,
but that turned up having too many hairy edge cases, so was left as a
future excercise.
DocImpact
Change-Id: I94e3754923dbe5faba065719c7a9afa9969652dd
CORS doesn't really work with swift right now. OPTIONS calls for the most part
work but for so called "simple cross-site requests" (i.e. those that don't
require a pre-flight OPTIONS request) Swift always returns the Origin it was
given as the Access-Control-Allow-Origin in the response. This makes CORS
"work" for these requests but if you actually wanted the javascript user agent
to restrict anything for you it wouldn't be able to!
You can duplicate the issue with updated CORS test page:
http://docs.openstack.org/developer/swift/cors.html#test-cors-page
And a public container with an 'X-Container-Meta-Access-Control-Allow-Origin'
that does NOT match the webserver hosting the test-cors-page.
e.g.
with a public container that accepts cross-site requests from "example.com":
`swift post cors-container -m access-control-allow-origin:example.com -r .r:*`
You could point your browser at a copy of the test-cors-page on your
filesystem (the browser will will send 'Origin: null')
Without a token the XMLHttpRequest will not request any custom headers (i.e.
Access-Control-Request-Headers: x-auth-token) and the request will be made
with-out a preflight OPTIONS request (which Swift would have denied anyway
because the origin's don't match)
i.e. fill in "http://saio:8080/v1/AUTH_test/cors-container" for "URL" and
leave "Token" blank.
You would expect that the browser would not complete the request because
"Origin: null" does not match the configured "Access-Control-Allow-Origin:
example.com" on the container metadata, and indeed with this patch - it won't!
Also:
The way cors is set up does not play well with certain applications for swift.
If you are running a CDN on top of swift and you have the
Access-Control-Allow-Origin cors header set to * then you probably want the *
to be cached on the the CDN, not the Origin that happened to result in an
origin request.
Also:
If you were unfortunate enough to allow cors headers to be saved directly
onto objects then this allows them to supersede the headers coming from the
container.
NOTE: There is a change is behavior with this patch. Because its cors, a
spec that was created only to cause annoyance to all, I'll write out
what's being changed and hopefully someone will speak up if it breaks
there stuff.
previous behavior: When a request was made with a Origin header set the
cors_validation decorator would always add that origin as
the Access-Control-Allow-Origin header in the response-
whether the passed origin was a match with the container's
X-Container-Meta-Access-Control-Allow-Origin or not, or even
if the container did not have CORS set up at all.
new behavior: If strict_cors_mode is set to True in the proxy-server.conf
(which is the default) the cors_validation decorator will only
add the Access-Control-Allow-Origin header to the response when
the request's Origin matches the value set in
X-Container-Meta-Access-Control-Allow-Origin. NOTE- if the
container does not have CORS set up it won't just magically start
working. Furthremore, if the Origin doesn't match the
Access-Control-Allow-Origin - a successfully authorized request
(either by token or public ACL) won't be *denied* - it just
won't include the Access-Control-Allow-Origin header (it's up
to the security model in the browser to cancel the request
if the response doesn't include a matching Allow-Origin
header). On the other hand, if you want to restrict requests
with CORS, you can actually do it now.
If you are worried about breaking current functionality you
must set:
strict_cors_mode = False
in the proxy-server.conf. This will continue with returning the
passed in Origin as the Access-Control-Allow-Origin in the
response.
previous: If you had X-Container-Meta-Access-Control-Allow-Origin set to *
and you passed in Origin: http://hey.com you'd get
Access-Control-Allow-Origin: http://hey.com back. This was true for
both OPTIONS and regular reqs.
new: With X-Container-Meta-Access-Control-Allow-Origin set to * you get * back
for both OPTIONS and regular reqs.
previous: cors headers saved directly onto objects (by allowing them to be
saved via the allowed_headers config in the object-server conf)
would be overridden by whatever container cors you have set up.
new: For regular (non-OPTIONS) calls the object headers will be kept. The
container cors will only be applied to objects without the
'Access-Control-Allow-Origin' and 'Access-Control-Expose-Headers' headers.
This behavior doesn't make a whole lot of sense for OPTIONS calls so I
left that as is. I don't think that allowing cors headers to be saved
directly onto objects is a good idea and it should be discouraged.
DocImpact
Change-Id: I9b0219407e77c77a9bb1133cbcb179a4c681c4a8
Move the tests from functionalnosetests under functional, so we no
longer have two seperate trees for functional tests. This also drops
the 'nose' name from the directory, so that it doesn't end up with
confusion if we move to testr. Further, since there are no longer two
test runs in .functests, it nows looks very close to the other two.
Change-Id: I8de025c29d71f05072e257df24899927b82c1382
