146 Commits

Author SHA1 Message Date
Jenkins
104a6c8533 Merge "Functional test for SLO PUT overwriting one of its own segments" 2015-05-08 20:40:39 +00:00
Emmanuel Cazenave
1faad248f8 X-Auth-Token should be a bytestring.
Change-Id: I2aa941d74883e17e9548b0144a4a2e2db33aba95
Closes-Bug: 1451773
2015-05-07 14:28:35 +00:00
Samuel Merritt
a2a5b6aa66 Functional test for SLO PUT overwriting one of its own segments
Change-Id: I4855816848f4fdb148d0b82735cf79bc68429617
2015-04-16 11:42:12 -07:00
Richard Hawkins
3c419dfb12 Add additional func tests for TempURLs
Change-Id: I25097c87d0a06c283b8ee5067b840f0a22776369
2015-04-14 13:14:40 -07:00
John Dickinson
e910f7e07d Merge EC feature into master
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I002787f558781bd4d884129b127bc9f108ea9ec4
2015-04-14 08:57:15 -07:00
John Dickinson
dd9d97458e Prevent unauthorized delete in versioned container
An authenticated user can delete the most recent version of any
versioned object who's name is known if the user has listing access
to the x-versions-location container. Only Swift setups with
allow_version setting are affected.

This patch closes this bug, tracked as CVE-2015-1856

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Christian Schwede <info@cschwede.de>
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>

Closes-Bug: 1430645
Change-Id: Ibacc7413afe7cb6f77d92e5941dcfdf4768ffa18
2015-04-13 23:34:10 -07:00
Clay Gerrard
a707829334 Update test infrastructure
* Get FakeConn ready for expect 100 continue
 * Use debug_logger more and with better interfaces
 * Fix patch_policies to be less annoying

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I28c0a3539d994cbb8e6b94d63a23ed4ea6cb956d
2015-04-13 22:57:42 -07:00
Tushar Gohad
ed54066288 Add support for policy types, 'erasure_coding' policy
This patch extends the StoragePolicy class for non-replication storage
policies, the first one being "erasure coding".

Changes:

 - Add 'policy_type' support to BaseStoragePolicy class
 - Disallow direct instantiation of BaseStoragePolicy class

 - Subclass BaseStoragePolicy

   - "StoragePolicy":
     . Replication policy, default
     . policy_type = 'replication'

   - "ECStoragePolicy":
     . Erasure Coding policy
     . policy_type = 'erasure_coding'
     . Private member variables
       ec_type (EC backend),
       ec_num_data_fragments (number of fragments original
         data split into after erasure coding operation),
       ec_num_parity_fragments (number of parity fragments
       generated during erasure coding)
     . Private methods
       EC specific attributes and ring validator methods.

 - Swift will use PyECLib, a Python Erasure Coding library, for
   erasure coding operations. PyECLib is already an approved
   OpenStack core requirement.
   (https://bitbucket.org/kmgreen2/pyeclib/)

 - Add test cases for
   - 'policy_type' StoragePolicy member
   - policy_type == 'erasure_coding'

DocImpact

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386
Implements: blueprint ec-proxy-work
2015-04-13 22:57:42 -07:00
Alistair Coles
7e64c811fe Enable in-process functional test policy to be configured
Currently the in-process tests build a 2 replica, 4 partition,
2 device object ring. This patch allows an alternative policy
and ring to be specified for testing via environment variables
that may optionally be set.

SWIFT_TEST_IN_PROCESS_CONF_DIR - This points the test setup to a
  directory which may have a swift.conf file and ring file. The
  test setup will then prefer these conf files over the samples
  in '/etc'.

SWIFT_TEST_POLICY - This causes the in-process test to
  use the specified policy from the swift.conf file and its
  associated ring for testing (first copying the conf and ring file
  and modifying device parameters to suit in-process testing). If
  not set, the tests will use the default policy.

The in-process tests now start sufficient object servers for the
ring file being tested against.

This should allow in-process functional testing of various policies
and rings (e.g. EC policies) without needing to reconfigure an SAIO
for each test scenario.

The refactoring of the in_process test setup code should also
allow easier addition of other 'hard-coded' test policies/rings
in the future.

Change-Id: I24f5a13de3d296b400da1691dcb53423a9f8a463
2015-03-20 16:13:57 +00:00
Clay Gerrard
555e9894ee Make functests retry auth immediately
I don't think we need to backoff after a 401 - we have a perfectly valid
response telling us exactly what we should do so we can do that right away and
move on.

Assuming you run functional tests at least once a day, according to
http://xkcd.com/1205/ and my benchmarks you can spend roughly a full day
reviewing this change and still come out ahead.

Change-Id: I27c42c4ee3254eb32aad4d3dac08b16b3a43d611
2015-02-26 12:30:30 -08:00
Jenkins
d6467d3385 Merge "Add multiple reseller prefixes and composite tokens" 2015-02-24 16:12:01 +00:00
Donagh McCabe
89397c5b67 Add multiple reseller prefixes and composite tokens
This change is in support of Composite Tokens and Service Accounts
(see http://specs.openstack.org/openstack/swift-specs/specs/in_progress/
service_token.html)

During coding, minor changes were made compared to the original
specification. See https://review.openstack.org/138771 for these changes.

DocImpact

Change-Id: I6072b4efb3a479a8e0cc2d9c11ffda5764b55e30
2015-02-23 15:57:20 +00:00
Richard Hawkins
023e29fa1e Add functional tests for container TempURLs
Change-Id: I7f54594df1522d72dd1d13556be0b9d33e811b30
2015-02-14 10:05:19 -06:00
Alistair Coles
d85371c319 Don't skip account acl functional tests
The functional tests covering account acls are skipped
if keystoneauth is in the pipeline, even if keystone auth
is not being used. Devstack configures the pipeline to have
both tempauth and keystoneauth, so these tests are always
skipped.

This patch changes the condition for skipping account acl
to be based on tempauth being the auth service configured
for the tests.

Change-Id: I378ec6aa0ba52d37a33796057e59a9ebfcab2574
2015-02-03 18:32:33 +00:00
Prashanth Pai
a0977ab8ca dlo: Update doc about manifest containing data
Current behavior:
* If data/body is present in manifest file PUT request, the data/body gets
  saved onto disk, just like for a normal object.
* Generally, this data in manifest file is never served on a GET response.
  However, when the manifest object path itself is part of prefix, GET
  response would contain data present in manifest file as well.
* The query param multipart-manifest=get meant to retrieve SLO manifest
  also works in case of DLO manifest. Hence a COPY request with the
  multipart-manifest=get query param would actually copy DLO manifest.

How things should have been:
* The DLO manifest object is supposed to have no content and only have
  X-Object-Manifest metadata header.
* Query param multipart-manifest=get is SLO specific and shouldn't have
  any role in DLO.

This change intends to only document current behaviour and not change it,
assuming there are users who have previously saved some content in DLO
manifest file and/or have been using multipart-manifest=get to fetch
and/or COPY the DLO manifest file with it's content.

Change-Id: I0f6e175ad7752169ecf94df949336e0665928df7
Signed-off-by: Prashanth Pai <ppai@redhat.com>
2015-01-16 10:24:14 +05:30
Thiago da Silva
24330771af fix dlo manifest file getting versioned
According to documentation dlo manifest files should not
be versioned. This patch fixes this issue and adds
some unit and functional for this scenario.

Change-Id: Ib5b29a19e1d577026deb50fc9d26064a8da81cd7
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2014-12-18 15:14:06 -05:00
Clay Gerrard
e0307f950b Always use FakeMemcache for in-process tests
Better isolation and consistency for in-process functests to always use
the FakeMemcache.  If you want to test the real memcache you have real
functional tests.

Change-Id: Ic483f794e122130bd7694c9a5f9a2b1cd0b9a653
2014-11-18 10:13:01 +00:00
Alistair Coles
c9f8246378 Make in process functional tests use sample proxy-server.conf
This patch was first motivated by noticing that the proxy
server pipeline used for in process functional tests was
out of date with respect to the pipeline in
/etc/proxy-server.conf.sample. Rather than cut and paste
the current pipeline into the in process setup, it seems
like a better idea would be to have the in process tests
always use the sample config.

A further benefit is that in process functional tests will
pick up changes to the sample config introduced by patches -
previously test/functional/__init__.py would need to be
manually modified to run in process functional tests
on new middleware for example.

Note: because the pipeline is now loaded using entry points,
'python setup.py [develop|install]' will now be needed
before running the tests.

Obvious next steps would be to do the same for the backend
servers, and to allow alternative config files and dir's
to be specified, but this patch is the first step.

Also drive-by fixes some typos in proxy-server.conf.sample

Change-Id: If442bd7c2b1721ec92839c4490924ba33e1545d8
2014-11-14 10:44:41 +00:00
Richard (Rick) Hawkins
5b2c27a587 Fix metadata overall limits bug
Currently metadata limits are checked on a per request basis. If
multiple requests are sent within the per request limits, it is
possible to exceed the overall limits.  This patch adds an overall
metadata check to ensure that multiple requests to add metadata to
an account/container will check overall limits before adding
the additional metadata.

Change-Id: Ib9401a4ee05a9cb737939541bd9b84e8dc239c70
Closes-Bug: 1365350
2014-10-02 22:38:13 +00:00
Jenkins
97c31850db Merge "Add comments to clarify change to www-authenticate test" 2014-09-13 10:56:22 +00:00
Jenkins
034fae630c Merge "Restrict keystone cross-tenant ACLs to IDs" 2014-09-13 00:53:47 +00:00
Alistair Coles
a03732e142 Add comments to clarify change to www-authenticate test
Trivial patch to tidy-up change to the functional test for
www-authenticate header and add a comment to explain
that multiple header values might be returned.

Change-Id: If62cb3fd9e11450a2be0cec71e80ecb74a959d04
Related-bug: 1368048
2014-09-12 10:24:11 +01:00
Jenkins
2a174bfb10 Merge "moving object validation checks to top of PUT method" 2014-09-12 06:22:59 +00:00
Alistair Coles
ab96796dc8 Fix broken www-authenticate functional test
testQuotedWWWAuthenticateHeader functional test started failing
due to a change to keystonemiddleware.auth_token, which now adds
its own www-authenticate header in addition to the one that swift
keystoneauth adds.

This patch changes the functional test to check expected
swift generated header value is in the concatenation of
www-authenticate values.

Verified that functional tests still pass using tempauth.

Closes-Bug: 1368048
Change-Id: I913af077df800a559d259c1622f286ad10eae9df
2014-09-11 10:39:48 +01:00
Jenkins
8329095f11 Merge "account to account copy implementation" 2014-09-11 00:22:51 +00:00
Thiago da Silva
9dcf15f8b5 moving object validation checks to top of PUT method
This adds a sanity check on x-delete headers as
part of check_object_creation method

Change-Id: If5069469e433189235b1178ea203b5c8a926f553
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2014-09-08 10:15:21 +01:00
Jenkins
2c4413a02c Merge "Spelling mistakes corrected in comments." 2014-09-05 01:14:37 +00:00
saranjan
2a8b43e5e7 Spelling mistakes corrected in comments.
Change-Id: Ibbd7511c3a2b08519feb4db18eca6e000603ea32
2014-09-03 10:40:30 -07:00
Clay Gerrard
21adf82cf1 code shuffle post expired headers refactor
Change-Id: I62248d7d3d7e0a3696a30e3d567ac6c2bea3c8eb
2014-08-21 10:45:22 -04:00
Constantine Peresypkin
43ac76373a account to account copy implementation
Adds ability to copy objects between different accounts (on server side)

Adds new header to `PUT` request:
`X-Copy-From-Account: <account name>`
Account name corresponds to the last part of storage URL.

Adds new header to `COPY` request:
`Destination-Account: <account name>`
Account name corresponds to the last part of storage URL.

If your storage URL is: http://server:8080/v1/AUTH_test
Then the account name is `AUTH_test`

These headers should be used alongside `X-Copy-From` and `Destination` headers
The legacy headers should specify `<container name>/<object name>` path as usual.

DocImpact

Change-Id: I0285fe6a47df9e699ac20ae4a83b0bf23829e1e6
2014-08-13 15:43:22 +03:00
anc
a4f634bd89 Restrict keystone cross-tenant ACLs to IDs
The keystoneauth middleware supports cross-tenant access
control using the syntax <tenant>:<user> in container ACLs,
where <tenant> and <user> may currently be either a unique
id or a name. As a result of the keystone v3 API introducing
domains, names are no longer globally unique and are only
unique within a domain. The use of unqualified tenant and
user names in this ACL syntax is therefore not 'safe' in a
keystone v3 environment.

This patch modifies keystoneauth to restrict cross-tenant
ACL matching to use only ids for accounts that are not in
the default domain. For backwards compatibility,
names will still be matched in ACLs when both the requesting
user and tenant are known to be in the default domain AND the
account's tenant is also in the default domain (the default
domain being the domain to which existing tenants are
migrated).

Accounts existing prior to this patch are assumed to be for
tenants in the default domain. New accounts created using a
v2 token scoped on the tenant are also assumed to be in the
default domain. New accounts created using a v3 token scoped
on the tenant will learn their domain membership from the
token info. New accounts created using any unscoped token,
(i.e. with a reselleradmin role) will have unknown domain
membership and therefore be assumed to NOT be in the default
domain.

Despite this provision for backwards compatibility, names
must no longer be used when setting new ACLs in any account,
including new accounts in the default domain.

This change obviously impacts users accustomed to specifying
cross-tenant ACLs in terms of names, and further work will be
necessary to restore those use cases. Some ideas are
discussed under the bug report. With that caveat, this patch
removes the reported vulnerability when using
swift/keystoneauth with a keystone v3 API.

Note: to observe the new 'restricted' behaviour you will need
to setup keystone user(s) and tenant(s) in a non-default domain
and set auth_version = v3.0 in the auth_token middleware config
section of proxy-server.conf. You may also benefit from the
keystone v3 enabled swiftclient patch under review here:
https://review.openstack.org/#/c/91788/

DocImpact

blueprint keystone-v3-support

Closes-Bug:  #1299146

Change-Id: Ib32df093f7450f704127da77ff06b595f57615cb
2014-08-08 15:58:29 +01:00
Peter Portante
3e04606f87 Add a env var to use in-memory obj server in func
Add an environment variable to enable the use of the in-memory object
server during in-process functional test runs.

It might be worth-while to just run under both object servers in-tree,
but this at least enables it, without having to figure out how to make
two test runs in two different environments.

DocImpact

Change-Id: Id76b008e1f273c639ae61550affddc32c5d7c419
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2014-08-04 16:54:50 +01:00
Jenkins
82f4610ff7 Merge "clean up saio env. after runnning func tests" 2014-07-16 19:20:15 +00:00
Thiago da Silva
e5d90a9cc3 clean up saio env. after runnning func tests
Added call to delete container in teardown_package().
Tests are leaving objects and containers in saio environment
after finishing running the tests.

Currently, delete_containers() is called in the setUp of each test
class, which cleans up the containers of the previous test,
but leaves the containers of the last test in the saio environment.

Change-Id: I643d7083a2a310fc3d24eab48c565f3798cff25f
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2014-07-11 14:25:37 -04:00
Peter Portante
0753dd3923 Handle func tests of swift cluster that hide /info
If a swift cluster configures the proxy server to not accept
/info requests, just do the best we can, skipping tests as
necessary.

To that end, if cors mode is not known we also skip the test.

Change-Id: I34f296c8717e0baf24ae552a7e38f9354ee91974
2014-07-11 10:05:53 -07:00
Paul Luse
873c52e608 Replace POLICY and POLICY_INDEX with string literals
Replaced throughout code base &  tox'd. Functional as well
as probe tests pass with and without policies defined.

POLICY --> 'X-Storage-Policy'
POLICY_INDEX --> 'X-Backend-Storage-Policy-Index'

Change-Id: Iea3d06de80210e9e504e296d4572583d7ffabeac
2014-06-23 12:52:50 -07:00
Yuan Zhou
c11ac01252 Add functional tests for Storage Policy
* additional container tests
 * refactor test cross policy copy
 * make functional tests cleanup better

In-process functional tests only define a single ring and will skip some of
the multi-storage policy tests, but have been updated to reload_policies with
the patched swift.conf.

DocImpact
Implements: blueprint storage-policies
Change-Id: If17bc7b9737558d3b9a54eeb6ff3e6b51463f002
2014-06-18 20:57:09 -07:00
John Dickinson
e00da6cabc properly quote www-authenticate header value
HTTP header values should be quoted. Since the WWW-Authenticate
header value contains user-supplied strings, it's important to
ensure it's properly quoted to ensure the integrity of the protocol.

Previous to this patch, the URL was unquoted and then the unquoted
value was returned in the header. This patch re-quotes the value
when it is set on the response.

This is filed as CVS-2014-3497

Fixes bug 1327414

Change-Id: If8bd8842f2ce821756e9b4461a18a8ac8d42fb8c
2014-06-18 11:10:17 -07:00
Samuel Merritt
b5d7c1eaac Remove unused variable
The tempurl functional tests would fail if you put tempurl.methods in
the proxy server's disallowed_sections. The test assumed that the
presence of the key 'tempurl' implied the presence of the subkey
'methods', but since 9cbf8a3 landed, operators can remove arbitrary
things from /info.

Normally I'd just change x['methods'] to x.get('methods', []), but it
turns out this was setting up a class variable that nobody ever looked
out again, so removing the code also works.

Change-Id: Ie899d146bc6fff81a5fae77815897244e8ec6bec
2014-06-02 10:14:22 -06:00
Jenkins
225252b2e6 Merge "Plumb insecure flag to functests" 2014-05-28 01:38:32 +00:00
Alex Gaynor
181e9496fa Fixed a handful of typos
Change-Id: I8cb2de27818332be587df7321ef567f8399910a9
2014-05-26 16:07:41 -05:00
Brian Cline
b4c5a13664 Uses None instead of mutables for function param defaults
As seen on #1174809, changes use of mutable types as default
arguments and defaults them within the method. Otherwise, those
defaults can be unexpectedly persisted with the function between
invocations and erupt into mass hysteria on the streets.

There was indeed a test (TestSimpleClient.test_get_with_retries)
that was erroneously relying on this behavior. Since previous tests
had populated their own instantiations with a token, this test only
passed because the modified headers dict from previous tests was
being overridden. As expected, with the mutable defaults fix in
SimpleClient, this test begain to fail since it never specified any
token, yet it has always passed anyway. This change also now provides
the expected token.

Change-Id: If95f11d259008517dab511e88acfe9731e5a99b5
Related-Bug: #1174809
2014-05-10 11:15:56 +00:00
Pete Zaitcev
14e2a221dc Plumb insecure flag to functests
The problem arises when running functests against pre-existing
clusters configured with self-signed certificates. As far as
security goes, doing that is rather useless, but it happens.
Such testing worked previously, but with the switch to Requests
in the client, we must plumb "insecure" flag. There's no other
way to do it in Requests, no environment variable, anything.

Change-Id: I94a61b60de8ee0529652c8c4a1ab9894c7cdb589
2014-04-30 12:17:25 -06:00
Peter Portante
067b41e854 In-process swift server for functional tests
Provide a way to run the functional tests using a p/a/c/o server setup
in the same process running the nosetests infrastructure.

By setting the environment variable, SWIFT_TEST_IN_PROCESS, to a true
value, the functional test framework will construct a set of proxy,
account, container and object servers running in the same process that
is running the functional tests, ignoring any external swift
service. This in-process swift environment is akin to the one used in
test/unit/proxy/test_server.py.

Setting that same environment variable to a false value will ensure the
in-process servers are not used.

When the above environment variable is not present, and the
/etc/swift/test.conf is _not_ present (or present but empty) on the
system where the functional tests are executing, the in-process
environment will be used. Previously, if no /etc/swift/test.conf file
was found, the tests would just be marked as skipped.

Using this in-process method allows one to gather code coverage using
the functional tests to exercise code paths, in addition to the unit
tests, or more easily debug existing functional tests, or even write new
ones.

There are two constraints that are changed for use with the in-process
functional tests: max_file_size is lowered to roughly 8 MB, and
client_timeout is set to 4s.

Change-Id: I5acd65e3068868d6509feae1d1954237d37fad45
2014-04-29 08:45:11 -04:00
Peter Portante
4dd2670fb0 Load constraints from cluster info dictionary
Use constrainst from the new "swift-constraints" section of test.conf,
fall back to those found in the response to the /info API call,
ultimately falling back to the constraints module's effective
constraints.

Change-Id: Iea01c9c4b5148faa10004a240df411cbe7336a6a
2014-04-14 19:05:25 -04:00
Jenkins
77b8d42dc8 Merge "Support If-[Un]Modified-Since for object HEAD" 2014-04-14 10:55:15 +00:00
Peter Portante
5ff6a4d5d6 Use eventlet instead of threading for timeout
The only explicit use of Python threading is found in the
testFileSizeLimit test. Using eventlet seems a bit easier to follow,
accomplishing the same goal, and does not constrain us to a
multi-threaded environment.

The chunks() and timeout() module level functions are only used by one
test each, so we just move them to those tests to indicate they are not
used globally.

Change-Id: I50b9fb798fbfd1d552b3c3f90309f6b86da34853
2014-04-09 13:25:17 -04:00
Jenkins
62087ab051 Merge "Update callback with proper bytes transferred" 2014-04-09 00:11:15 +00:00
Peter Portante
0e366aa922 Move module level setup under setup_package method
Perform all the package level test setup under the setup_package
method so that we only read the configuration file on disk when
functional tests are run. This let's us "play nice" with other tests.

Change-Id: I699d664bcef6754aac4ec6877830e745b764342f
2014-04-04 07:25:22 -04:00
Peter Portante
44b39417ba Read the configuration once for all func tests
Merge the swift_testing module into the functional test module itself,
so that we can read the configuration once for all unit tests, sharing
the same constraints.

Change-Id: I9fbbfdade9adca329cd79f7d4291ba009327c842
2014-04-02 23:48:01 -04:00