testQuotedWWWAuthenticateHeader functional test started failing
due to a change to keystonemiddleware.auth_token, which now adds
its own www-authenticate header in addition to the one that swift
keystoneauth adds.
This patch changes the functional test to check expected
swift generated header value is in the concatenation of
www-authenticate values.
Verified that functional tests still pass using tempauth.
Closes-Bug: 1368048
Change-Id: I913af077df800a559d259c1622f286ad10eae9df
This adds a sanity check on x-delete headers as
part of check_object_creation method
Change-Id: If5069469e433189235b1178ea203b5c8a926f553
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Adds ability to copy objects between different accounts (on server side)
Adds new header to `PUT` request:
`X-Copy-From-Account: <account name>`
Account name corresponds to the last part of storage URL.
Adds new header to `COPY` request:
`Destination-Account: <account name>`
Account name corresponds to the last part of storage URL.
If your storage URL is: http://server:8080/v1/AUTH_test
Then the account name is `AUTH_test`
These headers should be used alongside `X-Copy-From` and `Destination` headers
The legacy headers should specify `<container name>/<object name>` path as usual.
DocImpact
Change-Id: I0285fe6a47df9e699ac20ae4a83b0bf23829e1e6
Replaced throughout code base & tox'd. Functional as well
as probe tests pass with and without policies defined.
POLICY --> 'X-Storage-Policy'
POLICY_INDEX --> 'X-Backend-Storage-Policy-Index'
Change-Id: Iea3d06de80210e9e504e296d4572583d7ffabeac
* additional container tests
* refactor test cross policy copy
* make functional tests cleanup better
In-process functional tests only define a single ring and will skip some of
the multi-storage policy tests, but have been updated to reload_policies with
the patched swift.conf.
DocImpact
Implements: blueprint storage-policies
Change-Id: If17bc7b9737558d3b9a54eeb6ff3e6b51463f002
HTTP header values should be quoted. Since the WWW-Authenticate
header value contains user-supplied strings, it's important to
ensure it's properly quoted to ensure the integrity of the protocol.
Previous to this patch, the URL was unquoted and then the unquoted
value was returned in the header. This patch re-quotes the value
when it is set on the response.
This is filed as CVS-2014-3497
Fixes bug 1327414
Change-Id: If8bd8842f2ce821756e9b4461a18a8ac8d42fb8c
The tempurl functional tests would fail if you put tempurl.methods in
the proxy server's disallowed_sections. The test assumed that the
presence of the key 'tempurl' implied the presence of the subkey
'methods', but since 9cbf8a3 landed, operators can remove arbitrary
things from /info.
Normally I'd just change x['methods'] to x.get('methods', []), but it
turns out this was setting up a class variable that nobody ever looked
out again, so removing the code also works.
Change-Id: Ie899d146bc6fff81a5fae77815897244e8ec6bec
Provide a way to run the functional tests using a p/a/c/o server setup
in the same process running the nosetests infrastructure.
By setting the environment variable, SWIFT_TEST_IN_PROCESS, to a true
value, the functional test framework will construct a set of proxy,
account, container and object servers running in the same process that
is running the functional tests, ignoring any external swift
service. This in-process swift environment is akin to the one used in
test/unit/proxy/test_server.py.
Setting that same environment variable to a false value will ensure the
in-process servers are not used.
When the above environment variable is not present, and the
/etc/swift/test.conf is _not_ present (or present but empty) on the
system where the functional tests are executing, the in-process
environment will be used. Previously, if no /etc/swift/test.conf file
was found, the tests would just be marked as skipped.
Using this in-process method allows one to gather code coverage using
the functional tests to exercise code paths, in addition to the unit
tests, or more easily debug existing functional tests, or even write new
ones.
There are two constraints that are changed for use with the in-process
functional tests: max_file_size is lowered to roughly 8 MB, and
client_timeout is set to 4s.
Change-Id: I5acd65e3068868d6509feae1d1954237d37fad45
Use constrainst from the new "swift-constraints" section of test.conf,
fall back to those found in the response to the /info API call,
ultimately falling back to the constraints module's effective
constraints.
Change-Id: Iea01c9c4b5148faa10004a240df411cbe7336a6a
The only explicit use of Python threading is found in the
testFileSizeLimit test. Using eventlet seems a bit easier to follow,
accomplishing the same goal, and does not constrain us to a
multi-threaded environment.
The chunks() and timeout() module level functions are only used by one
test each, so we just move them to those tests to indicate they are not
used globally.
Change-Id: I50b9fb798fbfd1d552b3c3f90309f6b86da34853
Merge the swift_testing module into the functional test module itself,
so that we can read the configuration once for all unit tests, sharing
the same constraints.
Change-Id: I9fbbfdade9adca329cd79f7d4291ba009327c842
The unconditional module-level change of the locale collation can
affect other tests run under nose. To avoid this, make the change at
the package level, and restore it when finished.
Change-Id: Iacda42aa7155f1cfa7d0dcadbf73b961dfb4b923
We already supported it for object GET requests, but not for
HEAD. This lets clients keep metadata up-to-date without having to
either fetch the whole object when it's changed or do their own date
parsing. They can just treat Last-Modified as opaque and update their
idea of metadata when they get a 200.
Change-Id: Iff25d8989a93d651fd2c327e1e58036e79e1bde1
Looks like we are keeping at least four lists of the default
constraints in various places, and when we added the max_header_size
in initial Static Large Object Support commit (5d73da15) we failed to
add it to the two locations in the test tree (test/sample.conf and
test/functional/tests.py).
With this commit we take a small step to consolidate the list of
constraints to the constraints module by removing the list from the
tests module (note we still have them listed in the sample swift.conf
file and the test module's sample.conf file), document the missing
reference in the test module's sample.conf file, and, while we're at
it, use the SWIFT_CONF_FILE reference where possible.
Change-Id: Ic4fa5f1aedec9bc7a7b43ce1a2ca3cdebce6171f
If a client issues a COPY request for a DLO with the query param
multipart-manifest=get, then the expectation is a new
manifest. However, this wasn't the case; X-Object-Manifest wasn't
being set on the new object, so the result was a normal object with
the same contents as the old manifest (typically 0 bytes).
There was already a mechanism by which middlewares could modify COPY
requests; this commit extends that so they can set headers on the new
object.
Note that this has nothing to do with a "normal" DLO copy, i.e. one
without multipart-manifest=get. That one makes a new object that's the
concatenation of the segments, and it was working just fine.
Change-Id: I1073af9fee6e34ebdfad7b1a89aeb05e4523a151
* Introduce a new privileged account header: X-Account-Access-Control
* Introduce JSON-based version 2 ACL syntax -- see below for discussion
* Implement account ACL authorization in TempAuth
X-Account-Access-Control Header
-------------------------------
Accounts now have a new privileged header to represent ACLs or any other
form of account-level access control. The value of the header is an opaque
string to be interpreted by the auth system, but it must be a JSON-encoded
dictionary. A reference implementation is given in TempAuth, with the
knowledge that historically other auth systems often use TempAuth as a
starting point.
The reference implementation describes three levels of account access:
"admin", "read-write", and "read-only". Adding new access control
features in a future patch (e.g. "write-only" account access) will
automatically be forward- and backward-compatible, due to the JSON
dictionary header format.
The privileged X-Account-Access-Control header may only be read or written
by a user with "swift_owner" status, traditionally the account owner but
now also any user on the "admin" ACL.
Access Levels:
Read-only access is intended to indicate to the auth system that this
list of identities can read everything (except privileged headers) in
the account. Specifically, a user with read-only account access can get
a list of containers in the account, list the contents of any container,
retrieve any object, and see the (non-privileged) headers of the
account, any container, or any object.
Read-write access is intended to indicate to the auth system that this
list of identities can read or write (or create) any container. A user
with read-write account access can create new containers, set any
unprivileged container headers, overwrite objects, delete containers,
etc. A read-write user can NOT set account headers (or perform any
PUT/POST/DELETE requests on the account).
Admin access is intended to indicate to the auth system that this list of
identities has "swift_owner" privileges. A user with admin account access
can do anything the account owner can, including setting account headers
and any privileged headers -- and thus changing the value of
X-Account-Access-Control and thereby granting read-only, read-write, or
admin access to other users.
The auth system is responsible for making decisions based on this header,
if it chooses to support its use. Therefore the above access level
descriptions are necessarily advisory only for other auth systems.
When setting the value of the header, callers are urged to use the new
format_acl() method, described below.
New ACL Format
--------------
The account ACLs introduce a new format for ACLs, rather than reusing the
existing format from X-Container-Read/X-Container-Write. There are several
reasons for this:
* Container ACL format does not support Unicode
* Container ACLs have a different structure than account ACLs
+ account ACLs have no concept of referrers or rlistings
+ accounts have additional "admin" access level
+ account access levels are structured as admin > rw > ro, which seems more
appropriate for how people access accounts, rather than reusing
container ACLs' orthogonal read and write access
In addition, the container ACL syntax is a bit arbitrary and highly custom,
so instead of parsing additional custom syntax, I'd rather propose a next
version and introduce a means for migration. The V2 ACL syntax has the
following benefits:
* JSON is a well-known standard syntax with parsers in all languages
* no artificial value restrictions (you can grant access to a user named
".rlistings" if you want)
* forward and backward compatibility: you may have extraneous keys, but
your attempt to parse the header won't raise an exception
I've introduced hooks in parse_acl and format_acl which currently default
to the old V1 syntax but tolerate the V2 syntax and can easily be flipped
to default to V2. I'm not changing the default or adding code to rewrite
V1 ACLs to V2, because this patch has suffered a lot of scope creep already,
but this seems like a sensible milestone in the migration.
TempAuth Account ACL Implementation
-----------------------------------
As stated above, core Swift is responsible for privileging the
X-Account-Access-Control header (making it only accessible to swift_owners),
for translating it to -sysmeta-* headers to trigger persistence by the
account server, and for including the header in the responses to requests
by privileged users. Core Swift puts no expectation on the *content* of
this header. Auth systems (including TempAuth) are responsible for
defining the content of the header and taking action based on it.
In addition to the changes described above, this patch defines a format
to be used by TempAuth for these headers in the common.middleware.acl
module, in the methods format_v2_acl() and parse_v2_acl(). This patch
also teaches TempAuth to take action based on the header contents. TempAuth
now sets swift_owner=True if the user is on the Admin ACL, authorizes
GET/HEAD/OPTIONS requests if the user is on any ACL, authorizes
PUT/POST/DELETE requests if the user is on the admin or read-write ACL, etc.
Note that the action of setting swift_owner=True triggers core Swift to
add or strip the privileged headers from the responses. Core Swift (not
the auth system) is responsible for that.
DocImpact: Documentation for the new ACL usage and format appears in
summary form in doc/source/overview_auth.rst, and in more detail in
swift/common/middleware/tempauth.py in the TempAuth class docstring.
I leave it to the Swift doc team to determine whether more is needed.
Change-Id: I836a99eaaa6bb0e92dc03e1ca46a474522e6e826
This way, with zero additional effort, SLO will support enhancements
to object storage and retrieval, such as:
* automatic resume of GETs on broken connection (today)
* storage policies (in the near future)
* erasure-coded object segments (in the far future)
This also lets SLOs work with other sorts of hypothetical third-party
middleware, for example object compression or encryption.
Getting COPY to work here is sort of a hack; the proxy's object
controller now checks for "swift.copy_response_hook" in the request's
environment and feeds the GET response (the source of the new object's
data) through it. This lets a COPY of a SLO manifest actually combine
the segments instead of merely copying the manifest document.
Updated ObjectController to expect a response's app_iter to be an
iterable, not just an iterator. (PEP 333 says "When called by the
server, the application object must return an iterable yielding zero
or more strings." ObjectController was just being too strict.) This
way, SLO can re-use the same response-generation logic for GET and
COPY requests.
Added a (sort of hokey) mechanism to allow middlewares to close
incompletely-consumed app iterators without triggering a warning. SLO
does this when it realizes it's performed a ranged GET on a manifest;
it closes the iterable, removes the range, and retries the
request. Without this change, the proxy logs would get 'Client
disconnected on read' in them.
DocImpact
blueprint multi-ring-large-objects
Change-Id: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1
According to HTTP/1.1, servers MUST accept all three formats:
Sun, 06 Nov 1994 08:49:37 GMT # RFC 822, updated by RFC 1123
Sunday, 06-Nov-94 08:49:37 GMT # RFC 850, obsoleted by RFC 1036
Sun Nov 6 08:49:37 1994 # ANSI C's asctime() format
In functional tests, a date value header has 3 kinds of format will be
tested.
Change-Id: I679ed44576208f2a79bffce787cb55bda4b39705
Closes-Bug: #1253207
Also fix some exception-handling stuff in functional tests; at some
point, ResponseError() started needing two more parameters, but the
functional tests (not swift_test_client, just tests.py) still had a
couple spots that were not passing in the new params. Now they're
optional again, and if you omit them, then the stringification of the
ResponseError is just a little less useful than it could be.
Change-Id: I38968c4b590fc04b97b85c5f974c8648291a6689
the Last-Modified header in Response didn't have a suitable
value - an integer part of object's timestamp.
This leads that the the if-[un]modified-since header with the
value from last-modified is always earlier than timestamp
and results the content is always newer than value of these
conditional headers.
Patched code returns math.ceil() of object's timestamp
in Last-Modified header so the later conditional header works
correctly
Closes-Bug: #1248818
Change-Id: I1ece7d008551bf989da74d23f0ed6307c45c5436
The functional tests have some hokey detection of SLO support that
pre-dates the /info API, but we can do better now.
Also moved the SLO check up inside the setUp method so that skipping
the SLO tests should be somewhat faster now.
Change-Id: I645718b459d794a9a97770f7162934558c94f3e8
There's some sort-of-hacky code in there to detect SLO support in
order to skip tests when SLO is off so that the functests won't fail
on older clusters.
Change-Id: I6ad5974a0db7213747b0f4497d08ffc706d3f220
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.
Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
Built-in symbol: file, format, range etc was re-assigned as local vars
in files: test/functional/swift_test_client.py and
test/functional/tests.py, rename these local vars.
Fixes bug #1208078
Change-Id: I627894cd722bd32cad231d534cbe97a0412c7cc9
Add support for functional tests that work with Apache web front end
Change-Id: I72358a12016eeccc842d834461dbebaa188aa117
Implements: blueprint wsgi-application-interface
Unfortunately, SQLite truncates strings with null characters.
Additionally, XML pretty much hates them too.
Change-Id: Id9a8eaa27b841db6350d6959c202d3e3d6462b35
RFC2616 implicitly allows transferring utf-8 in
headers (vals) but not in header names (keys)
Swift functional tests includes testing for utf-8
header names (keys) which is non aligned with the standard.
This seem to be the case when testing file metadata.
File metadata keys are than transferred as part of
the http header keys (header names) as
X-Object-Meta-<metadata_key>=metadata_name.
As a result metadata_keys in swift must be composed from
a subset of ASCII characters - alpha numerics,
some punctuation, not a lot else* and cannot be utf-8.
On branch bp/wsgi-application-interface-4
modified: test/functional/tests.py
Change-Id: I1932e9d4ead83089e8a31a06d28a54f5625efb9c
Fixes: bug 1068026
Implements: blueprint wsgi-application-interface
The directory tests TestContainerPathsEnv creates directorys during SetUp while indicating duplicate application internet media type (MIME) via the Content-Type header.
E.g the output on the line using 'nosetests -v test.functional.tests:TestContainerPaths.testContainerListing' includes:
Host: 127.0.0.1:8080
Accept-Encoding: identity
Content-Type: application/octet-stream
Content-Length: 0
content-type: application/directory
X-Auth-Token: AUTH_tkd40f5550289d4d60af75bb1b6c6067be
On branch bp/wsgi-application-interface-3
modified: test/functional/tests.py
Fixes: Bug #1068499
Implements Blueprint wsgi-application-interface
Change-Id: I49c5d1f73fff3c06d4be18b20c7209eff4e1326c
