The tempurl functional tests would fail if you put tempurl.methods in
the proxy server's disallowed_sections. The test assumed that the
presence of the key 'tempurl' implied the presence of the subkey
'methods', but since 9cbf8a3 landed, operators can remove arbitrary
things from /info.
Normally I'd just change x['methods'] to x.get('methods', []), but it
turns out this was setting up a class variable that nobody ever looked
out again, so removing the code also works.
Change-Id: Ie899d146bc6fff81a5fae77815897244e8ec6bec
Provide a way to run the functional tests using a p/a/c/o server setup
in the same process running the nosetests infrastructure.
By setting the environment variable, SWIFT_TEST_IN_PROCESS, to a true
value, the functional test framework will construct a set of proxy,
account, container and object servers running in the same process that
is running the functional tests, ignoring any external swift
service. This in-process swift environment is akin to the one used in
test/unit/proxy/test_server.py.
Setting that same environment variable to a false value will ensure the
in-process servers are not used.
When the above environment variable is not present, and the
/etc/swift/test.conf is _not_ present (or present but empty) on the
system where the functional tests are executing, the in-process
environment will be used. Previously, if no /etc/swift/test.conf file
was found, the tests would just be marked as skipped.
Using this in-process method allows one to gather code coverage using
the functional tests to exercise code paths, in addition to the unit
tests, or more easily debug existing functional tests, or even write new
ones.
There are two constraints that are changed for use with the in-process
functional tests: max_file_size is lowered to roughly 8 MB, and
client_timeout is set to 4s.
Change-Id: I5acd65e3068868d6509feae1d1954237d37fad45
Use constrainst from the new "swift-constraints" section of test.conf,
fall back to those found in the response to the /info API call,
ultimately falling back to the constraints module's effective
constraints.
Change-Id: Iea01c9c4b5148faa10004a240df411cbe7336a6a
The only explicit use of Python threading is found in the
testFileSizeLimit test. Using eventlet seems a bit easier to follow,
accomplishing the same goal, and does not constrain us to a
multi-threaded environment.
The chunks() and timeout() module level functions are only used by one
test each, so we just move them to those tests to indicate they are not
used globally.
Change-Id: I50b9fb798fbfd1d552b3c3f90309f6b86da34853
Merge the swift_testing module into the functional test module itself,
so that we can read the configuration once for all unit tests, sharing
the same constraints.
Change-Id: I9fbbfdade9adca329cd79f7d4291ba009327c842
The unconditional module-level change of the locale collation can
affect other tests run under nose. To avoid this, make the change at
the package level, and restore it when finished.
Change-Id: Iacda42aa7155f1cfa7d0dcadbf73b961dfb4b923
We already supported it for object GET requests, but not for
HEAD. This lets clients keep metadata up-to-date without having to
either fetch the whole object when it's changed or do their own date
parsing. They can just treat Last-Modified as opaque and update their
idea of metadata when they get a 200.
Change-Id: Iff25d8989a93d651fd2c327e1e58036e79e1bde1
Looks like we are keeping at least four lists of the default
constraints in various places, and when we added the max_header_size
in initial Static Large Object Support commit (5d73da15) we failed to
add it to the two locations in the test tree (test/sample.conf and
test/functional/tests.py).
With this commit we take a small step to consolidate the list of
constraints to the constraints module by removing the list from the
tests module (note we still have them listed in the sample swift.conf
file and the test module's sample.conf file), document the missing
reference in the test module's sample.conf file, and, while we're at
it, use the SWIFT_CONF_FILE reference where possible.
Change-Id: Ic4fa5f1aedec9bc7a7b43ce1a2ca3cdebce6171f
If a client issues a COPY request for a DLO with the query param
multipart-manifest=get, then the expectation is a new
manifest. However, this wasn't the case; X-Object-Manifest wasn't
being set on the new object, so the result was a normal object with
the same contents as the old manifest (typically 0 bytes).
There was already a mechanism by which middlewares could modify COPY
requests; this commit extends that so they can set headers on the new
object.
Note that this has nothing to do with a "normal" DLO copy, i.e. one
without multipart-manifest=get. That one makes a new object that's the
concatenation of the segments, and it was working just fine.
Change-Id: I1073af9fee6e34ebdfad7b1a89aeb05e4523a151
* Introduce a new privileged account header: X-Account-Access-Control
* Introduce JSON-based version 2 ACL syntax -- see below for discussion
* Implement account ACL authorization in TempAuth
X-Account-Access-Control Header
-------------------------------
Accounts now have a new privileged header to represent ACLs or any other
form of account-level access control. The value of the header is an opaque
string to be interpreted by the auth system, but it must be a JSON-encoded
dictionary. A reference implementation is given in TempAuth, with the
knowledge that historically other auth systems often use TempAuth as a
starting point.
The reference implementation describes three levels of account access:
"admin", "read-write", and "read-only". Adding new access control
features in a future patch (e.g. "write-only" account access) will
automatically be forward- and backward-compatible, due to the JSON
dictionary header format.
The privileged X-Account-Access-Control header may only be read or written
by a user with "swift_owner" status, traditionally the account owner but
now also any user on the "admin" ACL.
Access Levels:
Read-only access is intended to indicate to the auth system that this
list of identities can read everything (except privileged headers) in
the account. Specifically, a user with read-only account access can get
a list of containers in the account, list the contents of any container,
retrieve any object, and see the (non-privileged) headers of the
account, any container, or any object.
Read-write access is intended to indicate to the auth system that this
list of identities can read or write (or create) any container. A user
with read-write account access can create new containers, set any
unprivileged container headers, overwrite objects, delete containers,
etc. A read-write user can NOT set account headers (or perform any
PUT/POST/DELETE requests on the account).
Admin access is intended to indicate to the auth system that this list of
identities has "swift_owner" privileges. A user with admin account access
can do anything the account owner can, including setting account headers
and any privileged headers -- and thus changing the value of
X-Account-Access-Control and thereby granting read-only, read-write, or
admin access to other users.
The auth system is responsible for making decisions based on this header,
if it chooses to support its use. Therefore the above access level
descriptions are necessarily advisory only for other auth systems.
When setting the value of the header, callers are urged to use the new
format_acl() method, described below.
New ACL Format
--------------
The account ACLs introduce a new format for ACLs, rather than reusing the
existing format from X-Container-Read/X-Container-Write. There are several
reasons for this:
* Container ACL format does not support Unicode
* Container ACLs have a different structure than account ACLs
+ account ACLs have no concept of referrers or rlistings
+ accounts have additional "admin" access level
+ account access levels are structured as admin > rw > ro, which seems more
appropriate for how people access accounts, rather than reusing
container ACLs' orthogonal read and write access
In addition, the container ACL syntax is a bit arbitrary and highly custom,
so instead of parsing additional custom syntax, I'd rather propose a next
version and introduce a means for migration. The V2 ACL syntax has the
following benefits:
* JSON is a well-known standard syntax with parsers in all languages
* no artificial value restrictions (you can grant access to a user named
".rlistings" if you want)
* forward and backward compatibility: you may have extraneous keys, but
your attempt to parse the header won't raise an exception
I've introduced hooks in parse_acl and format_acl which currently default
to the old V1 syntax but tolerate the V2 syntax and can easily be flipped
to default to V2. I'm not changing the default or adding code to rewrite
V1 ACLs to V2, because this patch has suffered a lot of scope creep already,
but this seems like a sensible milestone in the migration.
TempAuth Account ACL Implementation
-----------------------------------
As stated above, core Swift is responsible for privileging the
X-Account-Access-Control header (making it only accessible to swift_owners),
for translating it to -sysmeta-* headers to trigger persistence by the
account server, and for including the header in the responses to requests
by privileged users. Core Swift puts no expectation on the *content* of
this header. Auth systems (including TempAuth) are responsible for
defining the content of the header and taking action based on it.
In addition to the changes described above, this patch defines a format
to be used by TempAuth for these headers in the common.middleware.acl
module, in the methods format_v2_acl() and parse_v2_acl(). This patch
also teaches TempAuth to take action based on the header contents. TempAuth
now sets swift_owner=True if the user is on the Admin ACL, authorizes
GET/HEAD/OPTIONS requests if the user is on any ACL, authorizes
PUT/POST/DELETE requests if the user is on the admin or read-write ACL, etc.
Note that the action of setting swift_owner=True triggers core Swift to
add or strip the privileged headers from the responses. Core Swift (not
the auth system) is responsible for that.
DocImpact: Documentation for the new ACL usage and format appears in
summary form in doc/source/overview_auth.rst, and in more detail in
swift/common/middleware/tempauth.py in the TempAuth class docstring.
I leave it to the Swift doc team to determine whether more is needed.
Change-Id: I836a99eaaa6bb0e92dc03e1ca46a474522e6e826
This way, with zero additional effort, SLO will support enhancements
to object storage and retrieval, such as:
* automatic resume of GETs on broken connection (today)
* storage policies (in the near future)
* erasure-coded object segments (in the far future)
This also lets SLOs work with other sorts of hypothetical third-party
middleware, for example object compression or encryption.
Getting COPY to work here is sort of a hack; the proxy's object
controller now checks for "swift.copy_response_hook" in the request's
environment and feeds the GET response (the source of the new object's
data) through it. This lets a COPY of a SLO manifest actually combine
the segments instead of merely copying the manifest document.
Updated ObjectController to expect a response's app_iter to be an
iterable, not just an iterator. (PEP 333 says "When called by the
server, the application object must return an iterable yielding zero
or more strings." ObjectController was just being too strict.) This
way, SLO can re-use the same response-generation logic for GET and
COPY requests.
Added a (sort of hokey) mechanism to allow middlewares to close
incompletely-consumed app iterators without triggering a warning. SLO
does this when it realizes it's performed a ranged GET on a manifest;
it closes the iterable, removes the range, and retries the
request. Without this change, the proxy logs would get 'Client
disconnected on read' in them.
DocImpact
blueprint multi-ring-large-objects
Change-Id: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1
According to HTTP/1.1, servers MUST accept all three formats:
Sun, 06 Nov 1994 08:49:37 GMT # RFC 822, updated by RFC 1123
Sunday, 06-Nov-94 08:49:37 GMT # RFC 850, obsoleted by RFC 1036
Sun Nov 6 08:49:37 1994 # ANSI C's asctime() format
In functional tests, a date value header has 3 kinds of format will be
tested.
Change-Id: I679ed44576208f2a79bffce787cb55bda4b39705
Closes-Bug: #1253207
Also fix some exception-handling stuff in functional tests; at some
point, ResponseError() started needing two more parameters, but the
functional tests (not swift_test_client, just tests.py) still had a
couple spots that were not passing in the new params. Now they're
optional again, and if you omit them, then the stringification of the
ResponseError is just a little less useful than it could be.
Change-Id: I38968c4b590fc04b97b85c5f974c8648291a6689
the Last-Modified header in Response didn't have a suitable
value - an integer part of object's timestamp.
This leads that the the if-[un]modified-since header with the
value from last-modified is always earlier than timestamp
and results the content is always newer than value of these
conditional headers.
Patched code returns math.ceil() of object's timestamp
in Last-Modified header so the later conditional header works
correctly
Closes-Bug: #1248818
Change-Id: I1ece7d008551bf989da74d23f0ed6307c45c5436
The functional tests have some hokey detection of SLO support that
pre-dates the /info API, but we can do better now.
Also moved the SLO check up inside the setUp method so that skipping
the SLO tests should be somewhat faster now.
Change-Id: I645718b459d794a9a97770f7162934558c94f3e8
There's some sort-of-hacky code in there to detect SLO support in
order to skip tests when SLO is off so that the functests won't fail
on older clusters.
Change-Id: I6ad5974a0db7213747b0f4497d08ffc706d3f220
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.
Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
Built-in symbol: file, format, range etc was re-assigned as local vars
in files: test/functional/swift_test_client.py and
test/functional/tests.py, rename these local vars.
Fixes bug #1208078
Change-Id: I627894cd722bd32cad231d534cbe97a0412c7cc9
Add support for functional tests that work with Apache web front end
Change-Id: I72358a12016eeccc842d834461dbebaa188aa117
Implements: blueprint wsgi-application-interface
Unfortunately, SQLite truncates strings with null characters.
Additionally, XML pretty much hates them too.
Change-Id: Id9a8eaa27b841db6350d6959c202d3e3d6462b35
RFC2616 implicitly allows transferring utf-8 in
headers (vals) but not in header names (keys)
Swift functional tests includes testing for utf-8
header names (keys) which is non aligned with the standard.
This seem to be the case when testing file metadata.
File metadata keys are than transferred as part of
the http header keys (header names) as
X-Object-Meta-<metadata_key>=metadata_name.
As a result metadata_keys in swift must be composed from
a subset of ASCII characters - alpha numerics,
some punctuation, not a lot else* and cannot be utf-8.
On branch bp/wsgi-application-interface-4
modified: test/functional/tests.py
Change-Id: I1932e9d4ead83089e8a31a06d28a54f5625efb9c
Fixes: bug 1068026
Implements: blueprint wsgi-application-interface
The directory tests TestContainerPathsEnv creates directorys during SetUp while indicating duplicate application internet media type (MIME) via the Content-Type header.
E.g the output on the line using 'nosetests -v test.functional.tests:TestContainerPaths.testContainerListing' includes:
Host: 127.0.0.1:8080
Accept-Encoding: identity
Content-Type: application/octet-stream
Content-Length: 0
content-type: application/directory
X-Auth-Token: AUTH_tkd40f5550289d4d60af75bb1b6c6067be
On branch bp/wsgi-application-interface-3
modified: test/functional/tests.py
Fixes: Bug #1068499
Implements Blueprint wsgi-application-interface
Change-Id: I49c5d1f73fff3c06d4be18b20c7209eff4e1326c
This patch merely fixes a selection of files to the point where
pep8 1.3.3 is happy. Most of the errors are indentation related to
continued lines (E126, E127, E128), bracket positions (E124) and the
use of backslash (E502).
Patch 2 fixes David's comments regarding backslash and an odd comment
- thanks David!
Change-Id: I4fbd77ecf5395743cb96acb95fa946c322c16560
This fixes swob to handle "Range: bytes=-X" where X > len(content); ex.
"Range: bytes=-17" when the object has 10 bytes. Based on the RFC, the
range is satisfiable and all bytes should be returned.
It also fixes "Range: bytes=-0" to be, correctly, not satisfiable. In
addition, this case's response has Content-Length: 0 and has a zero-byte
body.
It also fixes an existing regression in swob for the case
"Range: bytes=100-" for a body of length < 100 (Content-Length was
negative and the body was returned).
The relevant RFC is 2616, section 14.35.1.
Change-Id: Ib3dc672e083173eb970c10801283813623f26e0e
This change replaces WebOb with a mostly compatible local library,
swift.common.swob. Subtle changes to WebOb's API over the years have been a
huge headache. Swift doesn't even run on the current version.
There are a few incompatibilities to simplify the implementation/interface:
* It only implements the header properties we use. More can be easily added.
* Casts header values to str on assignment.
* Response classes ("HTTPNotFound") are no longer subclasses, but partials
on Response, so things like isinstance no longer work on them.
* Unlike newer webob versions, will never return unicode objects.
Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f
Change previously hard-coded constants into config variables. This
allows deployers to tune their cluster more specifically based on
their needs. For example, a deployment that uses direct swift access
for public content may need to set a larger header value constraint to
allow for the full object name to be represented in the Content-
Disposition header (which browsers check to determine the name of a
downloaded object).
The new settings are set in the [swift-constraints] section of
/etc/swift/swift.conf. Comments were also added to this config file.
Cleaned up swift/common/constraints.py to pass pep8 1.3.3
Funtional tests now require constraints to be defined in /etc/test.conf or in
/etc/swift/swift.conf (in the case of running the functional tests against a
local swift cluster). To have any hope of tests passing, the defined
constraints must match the constraints on the tested cluster.
Removed a ton of "magic numbers" in both unit and functional tests.
Change-Id: Ie4588e052fd158314ddca6cd8fca9bc793311465
Based on PatchSet 3 of https://review.openstack.org/#/c/7569/ , make them to pass all funcional tests with both webob 1.x and 1.2.
The additional following compatibility issues were addressed:
- Until patch for range header issue is merged into official webob release, testRangedGetsWithLWSinHeader() should skip test against webob 1.2
(49c175aec2)
- common.constraints.check_utf8() can accept both utf8 str and unicode.
- To convert unicode to utf-8 str if necessary.
- Making proxy_logging can handle invalid utf-8 str
bug 888371
bug 959881
blueprint webob-support
Change-Id: I00e5fd04cd1653259606a4ffdd4926db3c84c496
bug 701248
bug 819303
This change makes syslog optional, by providing a unit test
config with the ability to cause test/unit/__init__.py to
replace SysLogHandler with a fake logging class. The
default behavior is unchanged.
FakeLogger now inherits directly from Handler and mocks out
its API - this was a backward-compatibility issue.
Change-Id: I653dec148c4f6cf81759de03964c6a3290c1a290
