An authenticated user can delete the most recent version of any
versioned object who's name is known if the user has listing access
to the x-versions-location container. Only Swift setups with
allow_version setting are affected.
This patch closes this bug, tracked as CVE-2015-1856
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Christian Schwede <info@cschwede.de>
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Closes-Bug: 1430645
Change-Id: Ibacc7413afe7cb6f77d92e5941dcfdf4768ffa18
* Get FakeConn ready for expect 100 continue
* Use debug_logger more and with better interfaces
* Fix patch_policies to be less annoying
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I28c0a3539d994cbb8e6b94d63a23ed4ea6cb956d
This patch extends the StoragePolicy class for non-replication storage
policies, the first one being "erasure coding".
Changes:
- Add 'policy_type' support to BaseStoragePolicy class
- Disallow direct instantiation of BaseStoragePolicy class
- Subclass BaseStoragePolicy
- "StoragePolicy":
. Replication policy, default
. policy_type = 'replication'
- "ECStoragePolicy":
. Erasure Coding policy
. policy_type = 'erasure_coding'
. Private member variables
ec_type (EC backend),
ec_num_data_fragments (number of fragments original
data split into after erasure coding operation),
ec_num_parity_fragments (number of parity fragments
generated during erasure coding)
. Private methods
EC specific attributes and ring validator methods.
- Swift will use PyECLib, a Python Erasure Coding library, for
erasure coding operations. PyECLib is already an approved
OpenStack core requirement.
(https://bitbucket.org/kmgreen2/pyeclib/)
- Add test cases for
- 'policy_type' StoragePolicy member
- policy_type == 'erasure_coding'
DocImpact
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386
Implements: blueprint ec-proxy-work
Currently the in-process tests build a 2 replica, 4 partition,
2 device object ring. This patch allows an alternative policy
and ring to be specified for testing via environment variables
that may optionally be set.
SWIFT_TEST_IN_PROCESS_CONF_DIR - This points the test setup to a
directory which may have a swift.conf file and ring file. The
test setup will then prefer these conf files over the samples
in '/etc'.
SWIFT_TEST_POLICY - This causes the in-process test to
use the specified policy from the swift.conf file and its
associated ring for testing (first copying the conf and ring file
and modifying device parameters to suit in-process testing). If
not set, the tests will use the default policy.
The in-process tests now start sufficient object servers for the
ring file being tested against.
This should allow in-process functional testing of various policies
and rings (e.g. EC policies) without needing to reconfigure an SAIO
for each test scenario.
The refactoring of the in_process test setup code should also
allow easier addition of other 'hard-coded' test policies/rings
in the future.
Change-Id: I24f5a13de3d296b400da1691dcb53423a9f8a463
I don't think we need to backoff after a 401 - we have a perfectly valid
response telling us exactly what we should do so we can do that right away and
move on.
Assuming you run functional tests at least once a day, according to
http://xkcd.com/1205/ and my benchmarks you can spend roughly a full day
reviewing this change and still come out ahead.
Change-Id: I27c42c4ee3254eb32aad4d3dac08b16b3a43d611
The functional tests covering account acls are skipped
if keystoneauth is in the pipeline, even if keystone auth
is not being used. Devstack configures the pipeline to have
both tempauth and keystoneauth, so these tests are always
skipped.
This patch changes the condition for skipping account acl
to be based on tempauth being the auth service configured
for the tests.
Change-Id: I378ec6aa0ba52d37a33796057e59a9ebfcab2574
Current behavior:
* If data/body is present in manifest file PUT request, the data/body gets
saved onto disk, just like for a normal object.
* Generally, this data in manifest file is never served on a GET response.
However, when the manifest object path itself is part of prefix, GET
response would contain data present in manifest file as well.
* The query param multipart-manifest=get meant to retrieve SLO manifest
also works in case of DLO manifest. Hence a COPY request with the
multipart-manifest=get query param would actually copy DLO manifest.
How things should have been:
* The DLO manifest object is supposed to have no content and only have
X-Object-Manifest metadata header.
* Query param multipart-manifest=get is SLO specific and shouldn't have
any role in DLO.
This change intends to only document current behaviour and not change it,
assuming there are users who have previously saved some content in DLO
manifest file and/or have been using multipart-manifest=get to fetch
and/or COPY the DLO manifest file with it's content.
Change-Id: I0f6e175ad7752169ecf94df949336e0665928df7
Signed-off-by: Prashanth Pai <ppai@redhat.com>
According to documentation dlo manifest files should not
be versioned. This patch fixes this issue and adds
some unit and functional for this scenario.
Change-Id: Ib5b29a19e1d577026deb50fc9d26064a8da81cd7
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Better isolation and consistency for in-process functests to always use
the FakeMemcache. If you want to test the real memcache you have real
functional tests.
Change-Id: Ic483f794e122130bd7694c9a5f9a2b1cd0b9a653
This patch was first motivated by noticing that the proxy
server pipeline used for in process functional tests was
out of date with respect to the pipeline in
/etc/proxy-server.conf.sample. Rather than cut and paste
the current pipeline into the in process setup, it seems
like a better idea would be to have the in process tests
always use the sample config.
A further benefit is that in process functional tests will
pick up changes to the sample config introduced by patches -
previously test/functional/__init__.py would need to be
manually modified to run in process functional tests
on new middleware for example.
Note: because the pipeline is now loaded using entry points,
'python setup.py [develop|install]' will now be needed
before running the tests.
Obvious next steps would be to do the same for the backend
servers, and to allow alternative config files and dir's
to be specified, but this patch is the first step.
Also drive-by fixes some typos in proxy-server.conf.sample
Change-Id: If442bd7c2b1721ec92839c4490924ba33e1545d8
Currently metadata limits are checked on a per request basis. If
multiple requests are sent within the per request limits, it is
possible to exceed the overall limits. This patch adds an overall
metadata check to ensure that multiple requests to add metadata to
an account/container will check overall limits before adding
the additional metadata.
Change-Id: Ib9401a4ee05a9cb737939541bd9b84e8dc239c70
Closes-Bug: 1365350
Trivial patch to tidy-up change to the functional test for
www-authenticate header and add a comment to explain
that multiple header values might be returned.
Change-Id: If62cb3fd9e11450a2be0cec71e80ecb74a959d04
Related-bug: 1368048
testQuotedWWWAuthenticateHeader functional test started failing
due to a change to keystonemiddleware.auth_token, which now adds
its own www-authenticate header in addition to the one that swift
keystoneauth adds.
This patch changes the functional test to check expected
swift generated header value is in the concatenation of
www-authenticate values.
Verified that functional tests still pass using tempauth.
Closes-Bug: 1368048
Change-Id: I913af077df800a559d259c1622f286ad10eae9df
This adds a sanity check on x-delete headers as
part of check_object_creation method
Change-Id: If5069469e433189235b1178ea203b5c8a926f553
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Adds ability to copy objects between different accounts (on server side)
Adds new header to `PUT` request:
`X-Copy-From-Account: <account name>`
Account name corresponds to the last part of storage URL.
Adds new header to `COPY` request:
`Destination-Account: <account name>`
Account name corresponds to the last part of storage URL.
If your storage URL is: http://server:8080/v1/AUTH_test
Then the account name is `AUTH_test`
These headers should be used alongside `X-Copy-From` and `Destination` headers
The legacy headers should specify `<container name>/<object name>` path as usual.
DocImpact
Change-Id: I0285fe6a47df9e699ac20ae4a83b0bf23829e1e6
The keystoneauth middleware supports cross-tenant access
control using the syntax <tenant>:<user> in container ACLs,
where <tenant> and <user> may currently be either a unique
id or a name. As a result of the keystone v3 API introducing
domains, names are no longer globally unique and are only
unique within a domain. The use of unqualified tenant and
user names in this ACL syntax is therefore not 'safe' in a
keystone v3 environment.
This patch modifies keystoneauth to restrict cross-tenant
ACL matching to use only ids for accounts that are not in
the default domain. For backwards compatibility,
names will still be matched in ACLs when both the requesting
user and tenant are known to be in the default domain AND the
account's tenant is also in the default domain (the default
domain being the domain to which existing tenants are
migrated).
Accounts existing prior to this patch are assumed to be for
tenants in the default domain. New accounts created using a
v2 token scoped on the tenant are also assumed to be in the
default domain. New accounts created using a v3 token scoped
on the tenant will learn their domain membership from the
token info. New accounts created using any unscoped token,
(i.e. with a reselleradmin role) will have unknown domain
membership and therefore be assumed to NOT be in the default
domain.
Despite this provision for backwards compatibility, names
must no longer be used when setting new ACLs in any account,
including new accounts in the default domain.
This change obviously impacts users accustomed to specifying
cross-tenant ACLs in terms of names, and further work will be
necessary to restore those use cases. Some ideas are
discussed under the bug report. With that caveat, this patch
removes the reported vulnerability when using
swift/keystoneauth with a keystone v3 API.
Note: to observe the new 'restricted' behaviour you will need
to setup keystone user(s) and tenant(s) in a non-default domain
and set auth_version = v3.0 in the auth_token middleware config
section of proxy-server.conf. You may also benefit from the
keystone v3 enabled swiftclient patch under review here:
https://review.openstack.org/#/c/91788/
DocImpact
blueprint keystone-v3-support
Closes-Bug: #1299146
Change-Id: Ib32df093f7450f704127da77ff06b595f57615cb
Add an environment variable to enable the use of the in-memory object
server during in-process functional test runs.
It might be worth-while to just run under both object servers in-tree,
but this at least enables it, without having to figure out how to make
two test runs in two different environments.
DocImpact
Change-Id: Id76b008e1f273c639ae61550affddc32c5d7c419
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Added call to delete container in teardown_package().
Tests are leaving objects and containers in saio environment
after finishing running the tests.
Currently, delete_containers() is called in the setUp of each test
class, which cleans up the containers of the previous test,
but leaves the containers of the last test in the saio environment.
Change-Id: I643d7083a2a310fc3d24eab48c565f3798cff25f
Signed-off-by: Thiago da Silva <thiago@redhat.com>
If a swift cluster configures the proxy server to not accept
/info requests, just do the best we can, skipping tests as
necessary.
To that end, if cors mode is not known we also skip the test.
Change-Id: I34f296c8717e0baf24ae552a7e38f9354ee91974
Replaced throughout code base & tox'd. Functional as well
as probe tests pass with and without policies defined.
POLICY --> 'X-Storage-Policy'
POLICY_INDEX --> 'X-Backend-Storage-Policy-Index'
Change-Id: Iea3d06de80210e9e504e296d4572583d7ffabeac
* additional container tests
* refactor test cross policy copy
* make functional tests cleanup better
In-process functional tests only define a single ring and will skip some of
the multi-storage policy tests, but have been updated to reload_policies with
the patched swift.conf.
DocImpact
Implements: blueprint storage-policies
Change-Id: If17bc7b9737558d3b9a54eeb6ff3e6b51463f002
HTTP header values should be quoted. Since the WWW-Authenticate
header value contains user-supplied strings, it's important to
ensure it's properly quoted to ensure the integrity of the protocol.
Previous to this patch, the URL was unquoted and then the unquoted
value was returned in the header. This patch re-quotes the value
when it is set on the response.
This is filed as CVS-2014-3497
Fixes bug 1327414
Change-Id: If8bd8842f2ce821756e9b4461a18a8ac8d42fb8c
The tempurl functional tests would fail if you put tempurl.methods in
the proxy server's disallowed_sections. The test assumed that the
presence of the key 'tempurl' implied the presence of the subkey
'methods', but since 9cbf8a3 landed, operators can remove arbitrary
things from /info.
Normally I'd just change x['methods'] to x.get('methods', []), but it
turns out this was setting up a class variable that nobody ever looked
out again, so removing the code also works.
Change-Id: Ie899d146bc6fff81a5fae77815897244e8ec6bec
As seen on #1174809, changes use of mutable types as default
arguments and defaults them within the method. Otherwise, those
defaults can be unexpectedly persisted with the function between
invocations and erupt into mass hysteria on the streets.
There was indeed a test (TestSimpleClient.test_get_with_retries)
that was erroneously relying on this behavior. Since previous tests
had populated their own instantiations with a token, this test only
passed because the modified headers dict from previous tests was
being overridden. As expected, with the mutable defaults fix in
SimpleClient, this test begain to fail since it never specified any
token, yet it has always passed anyway. This change also now provides
the expected token.
Change-Id: If95f11d259008517dab511e88acfe9731e5a99b5
Related-Bug: #1174809
The problem arises when running functests against pre-existing
clusters configured with self-signed certificates. As far as
security goes, doing that is rather useless, but it happens.
Such testing worked previously, but with the switch to Requests
in the client, we must plumb "insecure" flag. There's no other
way to do it in Requests, no environment variable, anything.
Change-Id: I94a61b60de8ee0529652c8c4a1ab9894c7cdb589
Provide a way to run the functional tests using a p/a/c/o server setup
in the same process running the nosetests infrastructure.
By setting the environment variable, SWIFT_TEST_IN_PROCESS, to a true
value, the functional test framework will construct a set of proxy,
account, container and object servers running in the same process that
is running the functional tests, ignoring any external swift
service. This in-process swift environment is akin to the one used in
test/unit/proxy/test_server.py.
Setting that same environment variable to a false value will ensure the
in-process servers are not used.
When the above environment variable is not present, and the
/etc/swift/test.conf is _not_ present (or present but empty) on the
system where the functional tests are executing, the in-process
environment will be used. Previously, if no /etc/swift/test.conf file
was found, the tests would just be marked as skipped.
Using this in-process method allows one to gather code coverage using
the functional tests to exercise code paths, in addition to the unit
tests, or more easily debug existing functional tests, or even write new
ones.
There are two constraints that are changed for use with the in-process
functional tests: max_file_size is lowered to roughly 8 MB, and
client_timeout is set to 4s.
Change-Id: I5acd65e3068868d6509feae1d1954237d37fad45
Use constrainst from the new "swift-constraints" section of test.conf,
fall back to those found in the response to the /info API call,
ultimately falling back to the constraints module's effective
constraints.
Change-Id: Iea01c9c4b5148faa10004a240df411cbe7336a6a
The only explicit use of Python threading is found in the
testFileSizeLimit test. Using eventlet seems a bit easier to follow,
accomplishing the same goal, and does not constrain us to a
multi-threaded environment.
The chunks() and timeout() module level functions are only used by one
test each, so we just move them to those tests to indicate they are not
used globally.
Change-Id: I50b9fb798fbfd1d552b3c3f90309f6b86da34853
Perform all the package level test setup under the setup_package
method so that we only read the configuration file on disk when
functional tests are run. This let's us "play nice" with other tests.
Change-Id: I699d664bcef6754aac4ec6877830e745b764342f
Merge the swift_testing module into the functional test module itself,
so that we can read the configuration once for all unit tests, sharing
the same constraints.
Change-Id: I9fbbfdade9adca329cd79f7d4291ba009327c842