130 Commits

Author SHA1 Message Date
Clay Gerrard
e0307f950b Always use FakeMemcache for in-process tests
Better isolation and consistency for in-process functests to always use
the FakeMemcache.  If you want to test the real memcache you have real
functional tests.

Change-Id: Ic483f794e122130bd7694c9a5f9a2b1cd0b9a653
2014-11-18 10:13:01 +00:00
Alistair Coles
c9f8246378 Make in process functional tests use sample proxy-server.conf
This patch was first motivated by noticing that the proxy
server pipeline used for in process functional tests was
out of date with respect to the pipeline in
/etc/proxy-server.conf.sample. Rather than cut and paste
the current pipeline into the in process setup, it seems
like a better idea would be to have the in process tests
always use the sample config.

A further benefit is that in process functional tests will
pick up changes to the sample config introduced by patches -
previously test/functional/__init__.py would need to be
manually modified to run in process functional tests
on new middleware for example.

Note: because the pipeline is now loaded using entry points,
'python setup.py [develop|install]' will now be needed
before running the tests.

Obvious next steps would be to do the same for the backend
servers, and to allow alternative config files and dir's
to be specified, but this patch is the first step.

Also drive-by fixes some typos in proxy-server.conf.sample

Change-Id: If442bd7c2b1721ec92839c4490924ba33e1545d8
2014-11-14 10:44:41 +00:00
Richard (Rick) Hawkins
5b2c27a587 Fix metadata overall limits bug
Currently metadata limits are checked on a per request basis. If
multiple requests are sent within the per request limits, it is
possible to exceed the overall limits.  This patch adds an overall
metadata check to ensure that multiple requests to add metadata to
an account/container will check overall limits before adding
the additional metadata.

Change-Id: Ib9401a4ee05a9cb737939541bd9b84e8dc239c70
Closes-Bug: 1365350
2014-10-02 22:38:13 +00:00
Jenkins
97c31850db Merge "Add comments to clarify change to www-authenticate test" 2014-09-13 10:56:22 +00:00
Jenkins
034fae630c Merge "Restrict keystone cross-tenant ACLs to IDs" 2014-09-13 00:53:47 +00:00
Alistair Coles
a03732e142 Add comments to clarify change to www-authenticate test
Trivial patch to tidy-up change to the functional test for
www-authenticate header and add a comment to explain
that multiple header values might be returned.

Change-Id: If62cb3fd9e11450a2be0cec71e80ecb74a959d04
Related-bug: 1368048
2014-09-12 10:24:11 +01:00
Jenkins
2a174bfb10 Merge "moving object validation checks to top of PUT method" 2014-09-12 06:22:59 +00:00
Alistair Coles
ab96796dc8 Fix broken www-authenticate functional test
testQuotedWWWAuthenticateHeader functional test started failing
due to a change to keystonemiddleware.auth_token, which now adds
its own www-authenticate header in addition to the one that swift
keystoneauth adds.

This patch changes the functional test to check expected
swift generated header value is in the concatenation of
www-authenticate values.

Verified that functional tests still pass using tempauth.

Closes-Bug: 1368048
Change-Id: I913af077df800a559d259c1622f286ad10eae9df
2014-09-11 10:39:48 +01:00
Jenkins
8329095f11 Merge "account to account copy implementation" 2014-09-11 00:22:51 +00:00
Thiago da Silva
9dcf15f8b5 moving object validation checks to top of PUT method
This adds a sanity check on x-delete headers as
part of check_object_creation method

Change-Id: If5069469e433189235b1178ea203b5c8a926f553
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2014-09-08 10:15:21 +01:00
Jenkins
2c4413a02c Merge "Spelling mistakes corrected in comments." 2014-09-05 01:14:37 +00:00
saranjan
2a8b43e5e7 Spelling mistakes corrected in comments.
Change-Id: Ibbd7511c3a2b08519feb4db18eca6e000603ea32
2014-09-03 10:40:30 -07:00
Clay Gerrard
21adf82cf1 code shuffle post expired headers refactor
Change-Id: I62248d7d3d7e0a3696a30e3d567ac6c2bea3c8eb
2014-08-21 10:45:22 -04:00
Constantine Peresypkin
43ac76373a account to account copy implementation
Adds ability to copy objects between different accounts (on server side)

Adds new header to `PUT` request:
`X-Copy-From-Account: <account name>`
Account name corresponds to the last part of storage URL.

Adds new header to `COPY` request:
`Destination-Account: <account name>`
Account name corresponds to the last part of storage URL.

If your storage URL is: http://server:8080/v1/AUTH_test
Then the account name is `AUTH_test`

These headers should be used alongside `X-Copy-From` and `Destination` headers
The legacy headers should specify `<container name>/<object name>` path as usual.

DocImpact

Change-Id: I0285fe6a47df9e699ac20ae4a83b0bf23829e1e6
2014-08-13 15:43:22 +03:00
anc
a4f634bd89 Restrict keystone cross-tenant ACLs to IDs
The keystoneauth middleware supports cross-tenant access
control using the syntax <tenant>:<user> in container ACLs,
where <tenant> and <user> may currently be either a unique
id or a name. As a result of the keystone v3 API introducing
domains, names are no longer globally unique and are only
unique within a domain. The use of unqualified tenant and
user names in this ACL syntax is therefore not 'safe' in a
keystone v3 environment.

This patch modifies keystoneauth to restrict cross-tenant
ACL matching to use only ids for accounts that are not in
the default domain. For backwards compatibility,
names will still be matched in ACLs when both the requesting
user and tenant are known to be in the default domain AND the
account's tenant is also in the default domain (the default
domain being the domain to which existing tenants are
migrated).

Accounts existing prior to this patch are assumed to be for
tenants in the default domain. New accounts created using a
v2 token scoped on the tenant are also assumed to be in the
default domain. New accounts created using a v3 token scoped
on the tenant will learn their domain membership from the
token info. New accounts created using any unscoped token,
(i.e. with a reselleradmin role) will have unknown domain
membership and therefore be assumed to NOT be in the default
domain.

Despite this provision for backwards compatibility, names
must no longer be used when setting new ACLs in any account,
including new accounts in the default domain.

This change obviously impacts users accustomed to specifying
cross-tenant ACLs in terms of names, and further work will be
necessary to restore those use cases. Some ideas are
discussed under the bug report. With that caveat, this patch
removes the reported vulnerability when using
swift/keystoneauth with a keystone v3 API.

Note: to observe the new 'restricted' behaviour you will need
to setup keystone user(s) and tenant(s) in a non-default domain
and set auth_version = v3.0 in the auth_token middleware config
section of proxy-server.conf. You may also benefit from the
keystone v3 enabled swiftclient patch under review here:
https://review.openstack.org/#/c/91788/

DocImpact

blueprint keystone-v3-support

Closes-Bug:  #1299146

Change-Id: Ib32df093f7450f704127da77ff06b595f57615cb
2014-08-08 15:58:29 +01:00
Peter Portante
3e04606f87 Add a env var to use in-memory obj server in func
Add an environment variable to enable the use of the in-memory object
server during in-process functional test runs.

It might be worth-while to just run under both object servers in-tree,
but this at least enables it, without having to figure out how to make
two test runs in two different environments.

DocImpact

Change-Id: Id76b008e1f273c639ae61550affddc32c5d7c419
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2014-08-04 16:54:50 +01:00
Jenkins
82f4610ff7 Merge "clean up saio env. after runnning func tests" 2014-07-16 19:20:15 +00:00
Thiago da Silva
e5d90a9cc3 clean up saio env. after runnning func tests
Added call to delete container in teardown_package().
Tests are leaving objects and containers in saio environment
after finishing running the tests.

Currently, delete_containers() is called in the setUp of each test
class, which cleans up the containers of the previous test,
but leaves the containers of the last test in the saio environment.

Change-Id: I643d7083a2a310fc3d24eab48c565f3798cff25f
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2014-07-11 14:25:37 -04:00
Peter Portante
0753dd3923 Handle func tests of swift cluster that hide /info
If a swift cluster configures the proxy server to not accept
/info requests, just do the best we can, skipping tests as
necessary.

To that end, if cors mode is not known we also skip the test.

Change-Id: I34f296c8717e0baf24ae552a7e38f9354ee91974
2014-07-11 10:05:53 -07:00
Paul Luse
873c52e608 Replace POLICY and POLICY_INDEX with string literals
Replaced throughout code base &  tox'd. Functional as well
as probe tests pass with and without policies defined.

POLICY --> 'X-Storage-Policy'
POLICY_INDEX --> 'X-Backend-Storage-Policy-Index'

Change-Id: Iea3d06de80210e9e504e296d4572583d7ffabeac
2014-06-23 12:52:50 -07:00
Yuan Zhou
c11ac01252 Add functional tests for Storage Policy
* additional container tests
 * refactor test cross policy copy
 * make functional tests cleanup better

In-process functional tests only define a single ring and will skip some of
the multi-storage policy tests, but have been updated to reload_policies with
the patched swift.conf.

DocImpact
Implements: blueprint storage-policies
Change-Id: If17bc7b9737558d3b9a54eeb6ff3e6b51463f002
2014-06-18 20:57:09 -07:00
John Dickinson
e00da6cabc properly quote www-authenticate header value
HTTP header values should be quoted. Since the WWW-Authenticate
header value contains user-supplied strings, it's important to
ensure it's properly quoted to ensure the integrity of the protocol.

Previous to this patch, the URL was unquoted and then the unquoted
value was returned in the header. This patch re-quotes the value
when it is set on the response.

This is filed as CVS-2014-3497

Fixes bug 1327414

Change-Id: If8bd8842f2ce821756e9b4461a18a8ac8d42fb8c
2014-06-18 11:10:17 -07:00
Samuel Merritt
b5d7c1eaac Remove unused variable
The tempurl functional tests would fail if you put tempurl.methods in
the proxy server's disallowed_sections. The test assumed that the
presence of the key 'tempurl' implied the presence of the subkey
'methods', but since 9cbf8a3 landed, operators can remove arbitrary
things from /info.

Normally I'd just change x['methods'] to x.get('methods', []), but it
turns out this was setting up a class variable that nobody ever looked
out again, so removing the code also works.

Change-Id: Ie899d146bc6fff81a5fae77815897244e8ec6bec
2014-06-02 10:14:22 -06:00
Jenkins
225252b2e6 Merge "Plumb insecure flag to functests" 2014-05-28 01:38:32 +00:00
Alex Gaynor
181e9496fa Fixed a handful of typos
Change-Id: I8cb2de27818332be587df7321ef567f8399910a9
2014-05-26 16:07:41 -05:00
Brian Cline
b4c5a13664 Uses None instead of mutables for function param defaults
As seen on #1174809, changes use of mutable types as default
arguments and defaults them within the method. Otherwise, those
defaults can be unexpectedly persisted with the function between
invocations and erupt into mass hysteria on the streets.

There was indeed a test (TestSimpleClient.test_get_with_retries)
that was erroneously relying on this behavior. Since previous tests
had populated their own instantiations with a token, this test only
passed because the modified headers dict from previous tests was
being overridden. As expected, with the mutable defaults fix in
SimpleClient, this test begain to fail since it never specified any
token, yet it has always passed anyway. This change also now provides
the expected token.

Change-Id: If95f11d259008517dab511e88acfe9731e5a99b5
Related-Bug: #1174809
2014-05-10 11:15:56 +00:00
Pete Zaitcev
14e2a221dc Plumb insecure flag to functests
The problem arises when running functests against pre-existing
clusters configured with self-signed certificates. As far as
security goes, doing that is rather useless, but it happens.
Such testing worked previously, but with the switch to Requests
in the client, we must plumb "insecure" flag. There's no other
way to do it in Requests, no environment variable, anything.

Change-Id: I94a61b60de8ee0529652c8c4a1ab9894c7cdb589
2014-04-30 12:17:25 -06:00
Peter Portante
067b41e854 In-process swift server for functional tests
Provide a way to run the functional tests using a p/a/c/o server setup
in the same process running the nosetests infrastructure.

By setting the environment variable, SWIFT_TEST_IN_PROCESS, to a true
value, the functional test framework will construct a set of proxy,
account, container and object servers running in the same process that
is running the functional tests, ignoring any external swift
service. This in-process swift environment is akin to the one used in
test/unit/proxy/test_server.py.

Setting that same environment variable to a false value will ensure the
in-process servers are not used.

When the above environment variable is not present, and the
/etc/swift/test.conf is _not_ present (or present but empty) on the
system where the functional tests are executing, the in-process
environment will be used. Previously, if no /etc/swift/test.conf file
was found, the tests would just be marked as skipped.

Using this in-process method allows one to gather code coverage using
the functional tests to exercise code paths, in addition to the unit
tests, or more easily debug existing functional tests, or even write new
ones.

There are two constraints that are changed for use with the in-process
functional tests: max_file_size is lowered to roughly 8 MB, and
client_timeout is set to 4s.

Change-Id: I5acd65e3068868d6509feae1d1954237d37fad45
2014-04-29 08:45:11 -04:00
Peter Portante
4dd2670fb0 Load constraints from cluster info dictionary
Use constrainst from the new "swift-constraints" section of test.conf,
fall back to those found in the response to the /info API call,
ultimately falling back to the constraints module's effective
constraints.

Change-Id: Iea01c9c4b5148faa10004a240df411cbe7336a6a
2014-04-14 19:05:25 -04:00
Jenkins
77b8d42dc8 Merge "Support If-[Un]Modified-Since for object HEAD" 2014-04-14 10:55:15 +00:00
Peter Portante
5ff6a4d5d6 Use eventlet instead of threading for timeout
The only explicit use of Python threading is found in the
testFileSizeLimit test. Using eventlet seems a bit easier to follow,
accomplishing the same goal, and does not constrain us to a
multi-threaded environment.

The chunks() and timeout() module level functions are only used by one
test each, so we just move them to those tests to indicate they are not
used globally.

Change-Id: I50b9fb798fbfd1d552b3c3f90309f6b86da34853
2014-04-09 13:25:17 -04:00
Jenkins
62087ab051 Merge "Update callback with proper bytes transferred" 2014-04-09 00:11:15 +00:00
Peter Portante
0e366aa922 Move module level setup under setup_package method
Perform all the package level test setup under the setup_package
method so that we only read the configuration file on disk when
functional tests are run. This let's us "play nice" with other tests.

Change-Id: I699d664bcef6754aac4ec6877830e745b764342f
2014-04-04 07:25:22 -04:00
Peter Portante
44b39417ba Read the configuration once for all func tests
Merge the swift_testing module into the functional test module itself,
so that we can read the configuration once for all unit tests, sharing
the same constraints.

Change-Id: I9fbbfdade9adca329cd79f7d4291ba009327c842
2014-04-02 23:48:01 -04:00
Peter Portante
5ea6c88128 Only change collation for functional tests
The unconditional module-level change of the locale collation can
affect other tests run under nose. To avoid this, make the change at
the package level, and restore it when finished.

Change-Id: Iacda42aa7155f1cfa7d0dcadbf73b961dfb4b923
2014-04-02 23:48:01 -04:00
Peter Portante
07fcf50c3a Rework use of constraints to ease testing
Prior to this patch both mainline code and testing modules imported
and used constraints directly into their own namespace, or relied on
the namespace of other modules that were not the constraints
module. This meant that if a unit test wanted to change a constraint
for its operation, it had to know how that module was using the
constraint, instead of referencing the constraint module itself.

This patch unifies the use of constraints so that all constraints are
referenced via the constraints module. In turn, this allows a test to
leverage the re-loadable nature of the constraints in the constraints
module.

It addition, a number of functional tests where using the default
values for constraints, instead of the configured value discovered in
a test.conf or in an existing swift.conf. This patch removes those
direct references in favor of the load_constraint() method from the
test/functional/tests.py module.

Change-Id: Ia5313d653c667dd9ca800786de59b59334c34eaa
2014-04-02 23:48:01 -04:00
Jenkins
95dcc99a5f Merge "Add "If-None-Match: *" support to PUT" 2014-04-03 01:05:02 +00:00
Peter Portante
f63b37572d Update callback with proper bytes transferred
The first set of bytes transferred was being dropped.

Change-Id: I8e055190d04af1718c5e88bf5e8c44fde6794c74
2014-04-01 22:59:22 -04:00
Hou Ming Wang
87710cc8a2 AssertEquals is deprecated, use assertEqual partI
assertEquals is deprecated in Python 2.7, need drop it.
http://docs.python.org/2/library/unittest.html#deprecated-aliases

Change-Id: Ieec505887f740b99b11ef40878ba3ee7247b78a8
Closes-Bug: #1285040
2014-04-01 16:53:00 -07:00
Chuck Thier
0b893825eb Add "If-None-Match: *" support to PUT
A common pattern that we see clients do is send a HEAD request before a
PUT to see if it exists.  This can slow things down quite a bit
especially since 404s on HEAD are currently a bit expensive.

This change will allow a client to include a "If-None-Match: *" header
with a PUT request.  In combination with "Expect: 100-Continue" this
allows the server to return that it already has a copy of the object
before any data is sent.

I attempted to also include etag support with the If-None-Match header,
but that turned up having too many hairy edge cases, so was left as a
future excercise.

DocImpact

Change-Id: I94e3754923dbe5faba065719c7a9afa9969652dd
2014-04-01 10:42:00 -07:00
Clay Gerrard
45e797b967 additional functional tests for account acls
Change-Id: I7d8a8e48066281707862cd07f6b5a012476e8f24
2014-03-25 19:51:58 -07:00
John Dickinson
08477750a5 fix a skipped account ACLs functional test
closes bug #1292158

Change-Id: Ic74c9d698c312b484a49d759eced6860d46cf399
2014-03-25 15:57:15 -07:00
Samuel Merritt
62a1e7e059 Support If-[Un]Modified-Since for object HEAD
We already supported it for object GET requests, but not for
HEAD. This lets clients keep metadata up-to-date without having to
either fetch the whole object when it's changed or do their own date
parsing. They can just treat Last-Modified as opaque and update their
idea of metadata when they get a 200.

Change-Id: Iff25d8989a93d651fd2c327e1e58036e79e1bde1
2014-03-14 17:55:43 -07:00
Jenkins
fc18d0856d Merge "Make cors work better." 2014-03-14 03:19:32 +00:00
Jenkins
1438baaf96 Merge "DRY out common.constraints a bit more" 2014-03-13 22:44:41 +00:00
Clay Gerrard
7463a46a62 DRY out common.constraints a bit more
Change-Id: If8a71692b79e381036b34245e09697ac1009beca
2014-03-11 23:02:33 -07:00
Jenkins
d054d7ea50 Merge "Functional tests for TempURL and SLO together" 2014-03-12 05:25:07 +00:00
Jenkins
733702627a Merge "Add missing swift constraint for functional tests" 2014-03-12 02:14:16 +00:00
David Goetz
2abb829a5d Make cors work better.
CORS doesn't really work with swift right now. OPTIONS calls for the most part
work but for so called "simple cross-site requests" (i.e. those that don't
require a pre-flight OPTIONS request) Swift always returns the Origin it was
given as the Access-Control-Allow-Origin in the response. This makes CORS
"work" for these requests but if you actually wanted the javascript user agent
to restrict anything for you it wouldn't be able to!

You can duplicate the issue with updated CORS test page:

http://docs.openstack.org/developer/swift/cors.html#test-cors-page

And a public container with an 'X-Container-Meta-Access-Control-Allow-Origin'
that does NOT match the webserver hosting the test-cors-page.

e.g.

with a public container that accepts cross-site requests from "example.com":

`swift post cors-container -m access-control-allow-origin:example.com -r .r:*`

You could point your browser at a copy of the test-cors-page on your
filesystem (the browser will will send 'Origin: null')

Without a token the XMLHttpRequest will not request any custom headers (i.e.
Access-Control-Request-Headers: x-auth-token) and the request will be made
with-out a preflight OPTIONS request (which Swift would have denied anyway
because the origin's don't match)

i.e. fill in "http://saio:8080/v1/AUTH_test/cors-container" for "URL" and
leave "Token" blank.

You would expect that the browser would not complete the request because
"Origin: null" does not match the configured "Access-Control-Allow-Origin:
example.com" on the container metadata, and indeed with this patch - it won't!

Also:

The way cors is set up does not play well with certain applications for swift.
If you are running a CDN on top of swift and you have the
Access-Control-Allow-Origin cors header set to * then you probably want the *
to be cached on the the CDN, not the Origin that happened to result in an
origin request.

Also:

If you were unfortunate enough to allow cors headers to be saved directly
onto objects then this allows them to supersede the headers coming from the
container.

NOTE: There is a change is behavior with this patch. Because its cors, a
      spec that was created only to cause annoyance to all, I'll write out
      what's being changed and hopefully someone will speak up if it breaks
      there stuff.

previous behavior: When a request was made with a Origin header set the
                   cors_validation decorator would always add that origin as
                   the Access-Control-Allow-Origin header in the response-
                   whether the passed origin was a match with the container's
                   X-Container-Meta-Access-Control-Allow-Origin or not, or even
                   if the container did not have CORS set up at all.
new behavior: If strict_cors_mode is set to True in the proxy-server.conf
              (which is the default) the cors_validation decorator will only
              add the Access-Control-Allow-Origin header to the response when
              the request's Origin matches the value set in
              X-Container-Meta-Access-Control-Allow-Origin. NOTE- if the
              container does not have CORS set up it won't just magically start
              working. Furthremore, if the Origin doesn't match the
              Access-Control-Allow-Origin - a successfully authorized request
              (either by token or public ACL) won't be *denied* - it just
              won't include the Access-Control-Allow-Origin header (it's up
              to the security model in the browser to cancel the request
              if the response doesn't include a matching Allow-Origin
              header).  On the other hand, if you want to restrict requests
              with CORS, you can actually do it now.

              If you are worried about breaking current functionality you
              must set:

              strict_cors_mode = False

              in the proxy-server.conf. This will continue with returning the
              passed in Origin as the Access-Control-Allow-Origin in the
              response.

previous: If you had X-Container-Meta-Access-Control-Allow-Origin set to *
          and you passed in Origin: http://hey.com you'd get
          Access-Control-Allow-Origin: http://hey.com back. This was true for
          both OPTIONS and regular reqs.
new: With X-Container-Meta-Access-Control-Allow-Origin set to * you get * back
     for both OPTIONS and regular reqs.

previous: cors headers saved directly onto objects (by allowing them to be
          saved via the allowed_headers config in the object-server conf)
          would be overridden by whatever container cors you have set up.
new: For regular (non-OPTIONS) calls the object headers will be kept. The
     container cors will only be applied to objects without the
     'Access-Control-Allow-Origin' and 'Access-Control-Expose-Headers' headers.
     This behavior doesn't make a whole lot of sense for OPTIONS calls so I
     left that as is. I don't think that allowing cors headers to be saved
     directly onto objects is a good idea and it should be discouraged.

DocImpact

Change-Id: I9b0219407e77c77a9bb1133cbcb179a4c681c4a8
2014-03-11 18:41:46 -07:00
Samuel Merritt
d9d5790d42 Functional tests for TempURL and SLO together
Change-Id: I4893911e50005d5b669b96e7d59b6149c734780e
2014-03-07 15:53:05 -08:00