Commit Graph

312 Commits

Author SHA1 Message Date
Jenkins
b0142d0cd2 Merge "Retrieve encryption root secret from Barbican" 2017-08-21 21:19:09 +00:00
Jenkins
c9acb0a717 Merge "functests: Only convert headers that should be ints to be ints" 2017-08-18 07:54:17 +00:00
Jenkins
cb1c438006 Merge "Add functests for disallowed COPYs into a versioned container" 2017-08-18 03:04:11 +00:00
Tim Burke
d8c7d21cfc Add functests for disallowed COPYs into a versioned container
While we're at it, have copy and copy_account raise ResponseErrors
on failure, similar to cluster_info, update_metadata, containers, info,
files, delete, initialize, read, sync_metadata, write, and post.

Related-Change: Ia8b92251718d10b1eb44a456f28d3d2569a30003
Change-Id: I9ef42d922a6b7dbf253f2f8f5df83965d8f47e0f
2017-08-08 18:25:14 +00:00
Mathias Bjoerkqvist
77bd74da09 Retrieve encryption root secret from Barbican
This patch adds support for retrieving the encryption root secret from
an external key management system. In practice, this is currently
limited to Barbican.

Change-Id: I1700e997f4ae6fa1a7e68be6b97539a24046e80b
2017-08-02 15:53:09 +03:00
Tim Burke
708fdec831 functests: Only convert headers that should be ints to be ints
Change-Id: I675d05d65b9c3c3ef18684c175d227f662b38f67
Closes-Bug: #1641216
2017-07-20 00:30:40 +00:00
Christian Schwede
2410b616bb Fix swiftdir option and usage of storage policy aliases
If swift-recon/swift-get-nodes/swift-object-info is used with the
swiftdir option they will read rings from the given directory; however
they are still using /etc/swift/swift.conf to find the policies on the
current node.

This makes it impossible to maintain a local swift.conf copy (if you
don't have write access to /etc/swift) or check multiple clusters from
the same node.

Until now swift-recon was also not usable with storage policy aliases,
this patch fixes this as well.

Closes-Bug: 1577582
Closes-Bug: 1604707
Closes-Bug: 1617951
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Change-Id: I13188d42ec19e32e4420739eacd1e5b454af2ae3
2017-07-12 16:23:04 -04:00
Jenkins
c22bab4b34 Merge "Version DLOs, just like every other type of object" 2017-07-03 14:06:07 +00:00
Pete Zaitcev
5dfc3a75fb Open-code eventlet.listen()
Recently out gate started blowing up intermittently with a strange
case of ports mixed up. Sometimes a functional tests tries to
authorize on a port that's clearly an object server port, and
the like. As it turns out, eventlet developers added an unavoidable
SO_REUSEPORT into listen(), which makes listen(("localhost",0)
to reuse ports.

There's an issue about it:
 https://github.com/eventlet/eventlet/issues/411

This patch is working around the problem while eventlet people
consider the issue.

Change-Id: I67522909f96495a6a30e1acdb79835dce2189549
2017-05-11 01:39:14 -06:00
Jenkins
30898435b1 Merge "Stop including Connection header in EC GET responses" 2017-05-05 22:12:05 +00:00
Tim Burke
4c8239503a Add message to account-creation assertion
We've seen some failures in the gate like

    ==============================
    Failed 1 tests - output below:
    ==============================

    setUpModule (test.functional.test_account)
    ------------------------------------------

    Captured traceback:
    ~~~~~~~~~~~~~~~~~~~
        Traceback (most recent call last):
          File "test/functional/test_account.py", line 33, in setUpModule
            tf.setup_package()
          File "test/functional/__init__.py", line 716, in setup_package
            mem_object_server if in_mem_obj else object_server))
          File "test/functional/__init__.py", line 621, in in_process_setup
            create_account(AUTH_test)
          File "test/functional/__init__.py", line 619, in create_account
            assert(resp.status == 201)
        AssertionError

...which aren't terribly useful in figuring out what went wrong.

Change-Id: I3cd31bb480dc8508828fe21416bfae33bc0985b7
2017-05-04 17:48:26 +00:00
Alistair Coles
6c320b2908 Stop including Connection header in EC GET responses
Currently, EC GET responses from proxy to clients, unlike any other
response, include a "Connection: close" header. If the client has sent
a "Connection: keep-alive" header then eventlet.wsgi appends this to
the client response, so clients can receive a response with both
headers:

Connection: close
Connection: keep-alive

This patch fixes the proxy EC GET path to remove any Connection header
from it's response, but does not change the behaviour of eventlet.wsgi
with respect to returning any client provided 'Connection: keep-alive'
header.

Change-Id: I43cd27c978edb4a1a587f031dbbee26e9acdc920
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Closes-Bug:  #1680731
2017-05-01 18:21:18 +01:00
Tim Burke
3ad8773239 Version DLOs, just like every other type of object
Previously, requests involving DLOs would bypass versioned_writes:

 * Any existing DLOs wouldn't get copied to the archive container during
   overwrites (or deletes, with history-mode), so there would be no
   evidence they had ever existed.

 * Any new DLOs wouldn't copy overwritten objects to the archive
   container, potentially leading to data loss.

Now, DLOs will behave like every other type of object under
versioned_writes.

Change-Id: I488e13eead2f33dd272d03f6f898adc52fc7fdad
Related-Change: Ie899290b3312e201979eafefb253d1a60b65b837
Related-Change: Ib5b29a19e1d577026deb50fc9d26064a8da81cd7
Closes-Bug: #1626989
2017-03-27 17:15:13 +00:00
Tim Burke
dfeae59824 Factor out a bunch of common testing setup
Rather than having a comment declaring that the stack/history modes
only diverge after the delete(), move the common code to a separate
function called by tests for both modes.

Change-Id: I657106b11bf0697338a776513d2a51ac6e562c0a
2017-03-23 00:27:39 +00:00
Alistair Coles
5f610c76bd Support EC policy for in process functional tests
Add support for a 2+1 EC policy to be optionally used as default
policy when running in process functional tests.

The EC policy may be selected by setting the env var:

  SWIFT_TEST_IN_PROCESS_CONF_LOADER=ec tox

when running .functests, or by using the new tox test env:

  tox -e func-ec

Change-Id: I02e3553a74a024efdab91dcd609ac1cf4e4f3208
2017-03-09 10:42:34 +00:00
Christopher Bartz
51727c531a ISO 8601 timestamps for tempurl
With this commit, the tempurl middleware accepts (besides
the traditional unix timestamps) also timestamps according
to the format '%Y-%m-%dT%H:%M:%SZ' (one acceptable form of ISO 8601).
The idea is to make the tempurls more user-friendly,
and has been formulated here:

Change-Id: I346a0241060a9559d178b30e60c957792bbeb9f0
Implements: blueprint human-readable-tempurl-timestamp
2017-01-24 17:38:48 +01:00
Jenkins
fab96ed11a Merge "Confirm receipt of SLO PUT with etag" 2017-01-16 20:37:02 +00:00
Jenkins
b90f2d7a23 Merge "Fix download resumption for new SLOs." 2017-01-13 08:30:32 +00:00
Samuel Merritt
0590d20fcc Fix download resumption for new SLOs.
SLOs created after commit 2d25fe6ad3
have an explicit X-Object-Sysmeta-SLO-Size attribute. Fetching one of
those with a range like "bytes=100-" would crash and cause a 500. This
commit fixes the crash.

Closes-Bug: #1656147

Change-Id: I174ed98b4f01a5943af12177d6f8c3586f049778
2017-01-13 01:13:29 +00:00
Jenkins
bce1aa7394 Merge "Support last modified on listing containers" 2017-01-13 00:12:48 +00:00
Kota Tsuyuzaki
652276fea6 Support last modified on listing containers
For now, last modified timestamp is supported only on
object listing. (i.e. GET container)

For example:

GET container with json format results in like as:

[{"hash": "d41d8cd98f00b204e9800998ecf8427e", "last_modified":
"2015-06-10T04:58:23.460230", "bytes": 0, "name": "object",
"content_type": "application/octet-stream"}]

However, container listing (i.e. GET account) shows just a dict
consists of ("name", "bytes", "name") for each container.

For example:

GET accounts with json format result in like as:

[{"count": 0, "bytes": 0, "name": "container"}]

This patch is for supporting last_modified key in the container
listing results as well as object listing like as:

[{"count": 0, "bytes": 0, "name": "container", "last_modified":
"2015-06-10T04:58:23.460230"}]

This patch is changing just output for listing. The original
timestamp to show the last modified is already in container table
of account.db as a "put_timestamp" column.

Note that this patch *DOESN'T* change the put_timestamp semantics.
i.e. the last_modified timestamp will be changed only at both PUT
container and POST container.
(PUT object doesn't affect the timestamp)

Note that the tuple format of returning value from
swift.account.backend.AccountBroker.list_containers is now
(name, object_count, bytes_used, put_timestamp, 0)

* put_timestamp is added *

Original discussion was in working session at Vancouver Summit.
Etherpads are around here:

https://etherpad.openstack.org/p/liberty-swift-contributors-meetup
https://etherpad.openstack.org/p/liberty-container-listing-update

DocImpact

Change-Id: Iba0503916f1481a20c59ae9136436f40183e4c5b
2017-01-12 18:03:10 +00:00
Mahati Chamarthy
fd6edf7bc5 Confirm receipt of SLO PUT with etag
With a multipart-manifest PUT request, if client sends the md5 of the
segments' etags, a 422 Unprocessable Entity response is returned. This
patch fixes that and confirms the etag

Change-Id: I4598a2a3f16ca8727bb07bbb6d8efcfcae777796
Closes-Bug: #1213200
Co-Authored-By: Tim Burke <tim@swiftstack.com>
2016-12-23 01:40:54 +05:30
Cao Xuan Hoang
a67bb2e249 Removes unnecessary utf-8 encoding
The following file(s) added utf-8 encoding but never used. So we can
remove them at all.

test/functional/test_access_control.py
test/unit/common/middleware/crypto/test_keymaster.py
test/unit/obj/test_diskfile.py

Change-Id: I00adc968872ebe9f9c0619a4e393e048c7c1a91e
2016-12-22 10:49:56 +07:00
Jenkins
09fa9dddd7 Merge "Remove duplicate setup code from test_tempurl.py" 2016-12-21 01:56:47 +00:00
Alistair Coles
9f30c5d31e Make tempurl functional tests clean up account keys
Addresses a TODO in test/functional/test_account.py where
an account metadata test was having to clean up tempurl keys
in the account metadata that were left by another test in
a different module. This cleanup is necessary because tests
in test_account.py fail if there is any pre-existing
account metadata.

This patch:

* makes the tempurl tests clean up their keys from account
  metadata.

* makes the test_account.py:TestAccount class remove any
  pre-existing metadata before attempting any tests and
  replacing that metadata when all the tests in that class
  have completed. This is more robust than the existing code
  which only removes any tempurl keys that might be in the
  account - now you could have x-account-meta-foo = bar in
  the test account and test_account.py will still pass.

* consolidates some common setup code currently repeated for
  many of the functional test classes into into a BaseEnv class.

Change-Id: I874a9e23dfcdd1caa934945b46089f11b9f6de65
2016-12-20 10:45:59 -08:00
Alistair Coles
b9f796b8a2 Remove duplicate setup code from test_tempurl.py
TestTempURLPrefix.setUp() almost duplicates the superclass
setUp method, except it set "expires" as an instance variable.
The duplicated setUp code can be removed by simply having the
super class setUp set the "expires" instance variable.

Change-Id: Icf7604460e475523e156daa8c578765336820c8e
2016-12-14 12:22:25 +00:00
Jenkins
c0640f8710 Merge "SLO: Make etag and size_bytes fully optional" 2016-12-13 23:02:27 +00:00
Tim Burke
7fb102dc5d SLO: Make etag and size_bytes fully optional
Previously, we still required that clients send "etag" and "size_bytes"
keys in their segment definitions. This was done as a way to guard
against typos leading to an accidental lack of verification.

However, typos should already be caught when we check for extra keys. As
a result, the only truly required key is "path".

Change-Id: Ie1d8691115f8c68b5a3f3b59317cdab59f9a3fca
2016-12-12 23:14:32 +00:00
Christopher Bartz
8333c4b36a tempurls with a prefix-based scope
The middleware now allows the usage of signatures with a prefix-based
scope. A prefix-based signature grants access to all objects which share
the same prefix. This avoids the creation of a large amount of signatures,
when a whole container or pseudofolder is shared.

Please see spec: https://review.openstack.org/#/c/199607/

Change-Id: I03b68eb74dae6196b5e63e711ef642ff7d2cfdc9
2016-12-12 12:17:12 +01:00
Tim Burke
e8a80e874a Let users know entity size in 416 responses
If a user sends a Range header with no satisfiable ranges, we send back
a 416 Requested Range Not Satisfiable response. Previously however,
there would be no indication of the size of the object they were
requesting, so they wouldn't know how to craft a satisfiable range. We
*do* send a Content-Length, but it is (correctly) the length of the
error message.

The RFC [1] has an answer for this:

>  A server generating a 416 (Range Not Satisfiable) response to a
>  byte-range request SHOULD send a Content-Range header field with an
>  unsatisfied-range value, as in the following example:
>
>    Content-Range: bytes */1234
>
>  The complete-length in a 416 response indicates the current length of
>  the selected representation.

Now, we'll send a Content-Range header for all 416 responses, including
those coming from the object server as well as those generated on a
proxy because of the Range mangling required to support EC policies.

[1] RFC 7233, section 4.2, although similar language was used in RFC
2616, sections 10.4.17 and 14.16

Change-Id: I80c7390fc6f84a10a212b0641bb07a64dfccbd45
2016-11-30 10:52:08 -08:00
Jenkins
18b392b602 Merge "breaking up functional/tests.py a bit further" 2016-11-29 22:24:49 +00:00
Thiago da Silva
aa8bf827e2 breaking up functional/tests.py a bit further
This patch removes the slo, dlo and tempurl tests from tests.py
As before, this patch does not refactor any code it's only a copy
and paste to make it easy to review

Change-Id: I0b49d4f8bc7dd204d06258f4910cfc9a5ba4bdcb
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2016-11-22 21:51:21 -05:00
Thomas Herve
2bdf61fadd Handle deleted files with if-none-match
When an object is deleted, we retrieve its metadata on the next PUT
requests, which make if-none-match requests fail while the object
shouldn't be here. It seems we're only interested in the timestamp of
the deleted object, so get that but forget the metadata.

Change-Id: I4a4a8be9b631598ca1cd52c53885c68c3fbdfc4a
Closes-Bug: #1640448
2016-11-22 21:06:24 -05:00
Thiago da Silva
c3e1d847f4 breaking down tests.py into smaller pieces
tests.py is currently at ~5500 lines of code, it's
time to break it down into smaller files.

I started with an easy middleware set of tests
(i.e., versioned writes, ~600 lines of code ) so I can get
some feedback. There are more complicated tests that cover
multiple middlewares for example, it is not so clear where
those should go.

Change-Id: I2aa6c18ee5b68d0aae73cc6add8cac6fbf7f33da
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2016-11-21 12:16:17 +00:00
Jenkins
15e61c82be Merge "Correctly send 412 Precondition Failed in copy middleware" 2016-11-18 13:14:50 +00:00
Bryan Keller
b94d0db9aa Correctly send 412 Precondition Failed in copy middleware
Previously in copy middleware, if a user entered an invalid destination
path with an invalid `container/object` path the server would return
a 500 Internal Server Error. However, the correct response should be
a 412 Precondition Failed. This patch updates copy so that it catches
the 412 Precondition Failed exception and returns it to the client.

Closes-Bug: #1641980

Change-Id: Ic4677ae033d05b8730c6ad1041bd9c07268e11a9
2016-11-17 17:48:24 +00:00
zhangyanxian
ba1f285fe6 Fix typos in test_container.py
TrivialFix

Change-Id: I6d3422a88b6fd6d2a495d17ff75d292ac96cc251
2016-11-17 02:07:24 +00:00
Ondřej Nový
33c18c579e Remove executable flag from some test modules
Change-Id: I36560c2b54c43d1674b007b8105200869b5f7987
2016-10-31 21:22:10 +00:00
Jenkins
6f0a1102d1 Merge "Add functional test for access control (RBAC INFO) with Keystone" 2016-10-06 00:43:16 +00:00
Jenkins
ba211f4e7d Merge "Add functional test for access control (container ACL) with Keystone" 2016-10-06 00:43:08 +00:00
Jenkins
ea84689598 Merge "Fix failing cross-policy-versioning functional test" 2016-10-05 22:57:19 +00:00
Jenkins
29516db36e Merge "Improve functional tests and test client" 2016-09-30 15:06:49 +00:00
Alistair Coles
1004509cab Fix failing cross-policy-versioning functional test
Since [1] the functional tests for cross-policy
object versioning fails with an error. These tests are
skipped in all CI test jobs because none of them have
more than one policy enabled. However, when running
tests against a system with multiple policies the tests
will not skip and the error is raised.

[1] Related-Change: Ifebc1c3ce558b1df9e576a58a4100f2219dfc7e7

Related-Bug: #1629234

Change-Id: I48028928bc996252a31cbf3abfd2b5e67a74dc95
2016-09-30 10:19:47 +01:00
Kota Tsuyuzaki
03b5e76d09 Improve functional tests and test client
This patch includes a couple of small functional test improvement.

A. Change swift_test_client.File.sync_metadata to follow Swift object
metadata semantics:

swift_test_client.File.sync_metadata is designed to post object user
metadata to an object. However, prior to this patch, the
swift_test_client.File instance keeps the existing object metadata as
its member attribute and if sync_metadata is called, it sends both
existing metadata and incomming metadata from caller. It looks to result
in the odd state as if Swift keeps the existing metadata when POST
object requested.

To tell the correct Swift object metadata semantics, when POST object
requested, the existing metadata in the stored object should be gone
even if no metadata is overwritten.

i.e.
if POST object with 'X-Object-Meta-Key: Val' to a stored object with
'X-Object-Meta-foo: bar', it will result in an object with
'X-Object-Meta-Key' (note that X-Object-Meta-Foo will be deleted)

The prior behavior sometimes make us confused in the reviw [1] so that,
this patch fixes it to send only incomming metadata if it's set.

B. Check the response status code more strictly for ObjectVersioning case

This patch fixes test_versioning_check_acl on both TestObjectVersioning and
TestObjectVersioningHistoryMode to assert the response status code
explisitly instead of asserting just "ResponseError". (e.g. 403 when trying
to delete object from other account)

1: https://review.openstack.org/#/c/360933/1/test/functional/tests.py@4142

Change-Id: Ia3e5b40f17dc0f881b695aa4be39c98b91e2bb06
2016-09-28 21:07:07 -07:00
Jenkins
1e5c5c35bd Merge "Support multi-range GETs for static large objects." 2016-09-28 04:48:34 +00:00
Kota Tsuyuzaki
0c8c764547 Add functional tests for new versioned_write mode
This patch is follow up for [1] and [2] to add new functional
tests for versioned_writes middlware 'history' mode.
(i.e. using X-History-Location header to a container).

The new test class, TestObjectHistoryModeVersioning, will use obvious
setting the mode via new X-History-Location header, since the change [2],
the setting X-Versions-Mode header added since [1] for incomming request has
been deprecated. Hence, since [2], the syntax for stack mode is back to
be same with older Swift than [1] so that the only thing we need now is
just adding a test suite for the new X-History-location.

It means the API has been changing like:
---------------
For stack mode:
---------------
Older than [1]:
X-Versions-Location

[1]~[2]:
X-Vesions-Location (and X-Versions-Mode: 'stack' for obvious)

Newer than [2]:
X-Vesions-Location

-----------------
For history mode:
-----------------
Older than [1]:
(Not supported)

[1]~[2]:
X-Vesions-Location and X-Versions-Mode: 'history'

Newer than [2]:
X-History-Location

Note that this functional tests work on newer swift than [2].

And then, this patch also sets allow_versioned_writes=True
for in-process testing (the container server allow_versions
option was already set, so this is just enabling in the middleware
too). That means that in-process functional tests (such as run by
the tox envs func-in-process-*) because history mode requires the
middleware allow_versioned_writes option to be explicity set to True.

1: https://review.openstack.org/#/c/214922/
2: https://review.openstack.org/#/c/373537/

Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>

Related-Change: I555dc17fefd0aa9ade681aa156da24e018ebe74b
Related-Change: Icfd0f481d4e40dd5375c737190aea7ee8dbc3bf9
Change-Id: Ifebc1c3ce558b1df9e576a58a4100f2219dfc7e7
2016-09-25 19:45:38 -07:00
Jenkins
7b9e2d27ea Merge "Allow DLO manifest to be updated when using post-as-copy" 2016-09-08 15:09:27 +00:00
Alistair Coles
1751db201b Add test for copy using a Range header
Since COPY allows a copy of a ranged GET, add some more test
coverage for that in both unit and functional tests.

Drive-by fix to use better test assertion methods.

Co-Authored-By: Thiago da Silva <thiago@redhat.com>

Change-Id: I5cb202386df0862f953f7388107c4d3466e2e46c
2016-09-06 09:08:32 +00:00
Matthew Oliver
d2fc261457 Authorise versioned write PUTs before copy
Currently a versioned write PUT uses a pre-authed request to move
it into the versioned container before checking whether the
user is authorised. This can lead to some interesting behaviour
whereby a user can select a versioned object path that it does not
have access to, request a put on that versioned object, and this
request will execute the copy part of the request before it fails
due to lack of permissions.

This patch changes the behaviour to be the same as versioned DELETE
where the request is authorised before anything is moved.

Change-Id: Ia8b92251718d10b1eb44a456f28d3d2569a30003
Closes-Bug: #1562175
2016-08-26 11:44:44 -04:00
Janie Richling
06ff865d19 Allow DLO manifest to be updated when using post-as-copy
Currently when using fast-post, the manifest is updated with the given
'x-object-manifest' header on a POST.  If no such header is supplied,
then the manifest will change to a regular object.
This is not currently true when using post-as-copy.

This patch changes the DLO POST using post-as-copy behavior to match
that of using fast-post.  It was also documented that
'x-object-manifest' must be provided on a POST to a manifest file.

Change-Id: Ie1143ab1a2c8f8c21e258a36badbff5d947769d4
Closes-bug: 1612991
2016-08-21 00:09:09 -05:00