test/functional/tests.py:TestAccount.testAccountHead relies
on the account having a known number of containers. The test
setup attempts to delete all existing containers but this fails
a container has versions (and so is not emptied, and cannot be
deleted). The tests then fails because the expected number of
containers does not match the actual.
'bin/resetswift' before running tests will obviously clear all
state but is not always convenient.
This change removes any x-versions-location header before deleting
containers during test setUp.
Steps to recreate the pre-condition for failure on master:
(based on original work by clayg)
swift post target -r '.r:*, .rlistings'
swift post source -H 'x-versions-location: target'
for i in {1..4}; do
echo "junk${i}" > junk
swift upload source junk
done
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I3efb6c20dc1fb3e979087e8a93d04ba7e346b5b6
This patch changes functional test classes to subclass
unittest2.TestCase rather than unittest.TestCase.
This fixes errors when attempting to use
tox -e func -- -n <test_path_including_test_method>
and
tox -e func -- --until-failure
Also migrate from using nose.SkipTest to unittest2.SkipTest
Change-Id: I903033f5e01833550b2f2b945894edca4233c4a2
Closes-Bug: 1526725
Co-Authored-By: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
Updates the functional test to verify the fix applied by
change Iff7274aa631a92cd7332212ed8b4378c27da4a1f
Change-Id: Iae63ac027e4f4acfe46a36dc1325888b1f834ea4
a1c32702, 736cf54a, and 38787d0f remove uses of `simplejson` from
various parts of Swift in favor of the standard libary `json`
module (introduced in Python 2.6). This commit performs the remaining
`simplejson` to `json` replacements, removes two comments highlighting
quirks of simplejson with respect to Unicode, and removes the references
to it in setup documentation and requirements.txt.
There were a lot of places where we were importing json from
swift.common.utils, which is less intuitive than a direct `import json`,
so that replacement is made as well.
(And in two more tiny drive-bys, we add some pretty-indenting to an XML
fragment and use `super` rather than naming a base class explicitly.)
Change-Id: I769e88dda7f76ce15cf7ce930dc1874d24f9498a
The urllib, urllib2 and urlparse modules of Python 2 were reorganized
into a new urllib namespace on Python 3. Replace urllib, urllib2 and
urlparse imports with six.moves.urllib to make the modified code
compatible with Python 2 and Python 3.
The initial patch was generated by the urllib operation of the sixer
tool on: bin/* swift/ test/.
Change-Id: I61a8c7fb7972eabc7da8dad3b3d34bceee5c5d93
Follow up to [1] to add tests for tempurl POSTs not being allowed
to set a DLO manifest header.
[1] I11e68830009d3f6bff44ae4011a41b67139146f6
Change-Id: I7c0ad5a936f71e56c599b8495a586913d3334422
Related-Bug: 1453948
Rewrite object versioning as middleware to simplify the PUT method
in the object controller.
The functionality remains basically the
same with the only major difference being the ability to now
version slo manifest files. dlo manifests are still not
supported as part of this patch.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
DocImpact
Change-Id: Ie899290b3312e201979eafefb253d1a60b65b837
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Get configparser, queue, http_client modules from six.moves.
Patch generated by the six_moves operation of the sixer tool:
https://pypi.python.org/pypi/sixer
Change-Id: I666241ab50101b8cc6f992dd80134ce27327bd7d
* replace "from cStringIO import StringIO"
with "from six.moves import cStringIO as StringIO"
* replace "from StringIO import StringIO"
with "from six import StringIO"
* replace "import cStringIO" and "cStringIO.StringIO()"
with "from six import moves" and "moves.cStringIO()"
* replace "import StringIO" and "StringIO.StringIO()"
with "import six" and "six.StringIO()"
This patch was generated by the stringio operation of the sixer tool:
https://pypi.python.org/pypi/sixer
Change-Id: Iacba77fec3045f96773d1090c0bd48613729a561
Python 2.7.9+ and 3.2.6+ limits the number of maximum headers in httplib to 100
[1,2,3]. This setting is too low for Swift.
By default the maximum number of allowed headers depends on the number of max
allowed metadata settings plus a default value of 32 for regular http headers.
If for some reason this is not enough (custom middleware for example) it can be
increased with the extra_header_count constraint.
[1] https://bugs.python.org/issue16037
[2] https://hg.python.org/cpython/raw-file/15c95b7d81dc/Misc/NEWS
[3] https://hg.python.org/cpython/raw-file/v3.2.6/Misc/NEWS
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Thomas Herve <therve@redhat.com>
Change-Id: I388fd697ec88476024b0e9f1ae75ba35ff765282
The iteritems() of Python 2 dictionaries has been renamed to items() on
Python 3. According to a discussion on the openstack-dev mailing list,
the overhead of creating a temporary list using dict.items() on Python 2
is very low because most dictionaries are small:
http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Patch generated by the following command:
sed -i 's,iteritems,items,g' \
$(find swift -name "*.py") \
$(find test -name "*.py")
Change-Id: I6070bb6c684be76e8e77222a7d280ec6edd43496
Current post as copy routine (i.e. POST object with post_as_copy option
turned on) on Object Controller uses "multipart-manifest" query string
which is feeded to env['copy_hook'] to decide which data (the manifest or
object pointed by the manifest) should be copied.
However, the way using the query string will confuse operators looking at
logging system (or analyzing the log) because whole POST object requests
have 'multipart-manifest=get' like as:
POST /v1/AUTH_test/d4c816b24d38489082f5118599a67920/manifest-abcde%3Fmultipart-manifest%3Dget
We cannot know whether the query string was added by hand
(from user) or not. In addition, the query isn't needed by the
backend conversation between proxy-server and object-server.
(Just needed by "copy_hook" on the proxy controller!)
To remove the confusable query string and to keep the log to be clean,
this patch introduces new environment variable "swift.post_as_copy"
and changes proxy controller and the copy_hook to use the new env.
This item was originally discussed at
https://review.openstack.org/#/c/177132/
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Change-Id: I0cd37520eea1825a10ebd27ccdc7e9162647233e
Add ability to select a storage policy when running
the functional tests by defining the environment variable
SWIFT_TEST_POLICY.
This patch was first introduced by Janie Richling in
feature/ec: https://review.openstack.org/#/c/166097
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Change-Id: Id96354546088df137def0ae7fd434baf64ed0c51
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Current behavior:
* If data/body is present in manifest file PUT request, the data/body gets
saved onto disk, just like for a normal object.
* Generally, this data in manifest file is never served on a GET response.
However, when the manifest object path itself is part of prefix, GET
response would contain data present in manifest file as well.
* The query param multipart-manifest=get meant to retrieve SLO manifest
also works in case of DLO manifest. Hence a COPY request with the
multipart-manifest=get query param would actually copy DLO manifest.
How things should have been:
* The DLO manifest object is supposed to have no content and only have
X-Object-Manifest metadata header.
* Query param multipart-manifest=get is SLO specific and shouldn't have
any role in DLO.
This change intends to only document current behaviour and not change it,
assuming there are users who have previously saved some content in DLO
manifest file and/or have been using multipart-manifest=get to fetch
and/or COPY the DLO manifest file with it's content.
Change-Id: I0f6e175ad7752169ecf94df949336e0665928df7
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Adds ability to copy objects between different accounts (on server side)
Adds new header to `PUT` request:
`X-Copy-From-Account: <account name>`
Account name corresponds to the last part of storage URL.
Adds new header to `COPY` request:
`Destination-Account: <account name>`
Account name corresponds to the last part of storage URL.
If your storage URL is: http://server:8080/v1/AUTH_test
Then the account name is `AUTH_test`
These headers should be used alongside `X-Copy-From` and `Destination` headers
The legacy headers should specify `<container name>/<object name>` path as usual.
DocImpact
Change-Id: I0285fe6a47df9e699ac20ae4a83b0bf23829e1e6
* additional container tests
* refactor test cross policy copy
* make functional tests cleanup better
In-process functional tests only define a single ring and will skip some of
the multi-storage policy tests, but have been updated to reload_policies with
the patched swift.conf.
DocImpact
Implements: blueprint storage-policies
Change-Id: If17bc7b9737558d3b9a54eeb6ff3e6b51463f002
As seen on #1174809, changes use of mutable types as default
arguments and defaults them within the method. Otherwise, those
defaults can be unexpectedly persisted with the function between
invocations and erupt into mass hysteria on the streets.
There was indeed a test (TestSimpleClient.test_get_with_retries)
that was erroneously relying on this behavior. Since previous tests
had populated their own instantiations with a token, this test only
passed because the modified headers dict from previous tests was
being overridden. As expected, with the mutable defaults fix in
SimpleClient, this test begain to fail since it never specified any
token, yet it has always passed anyway. This change also now provides
the expected token.
Change-Id: If95f11d259008517dab511e88acfe9731e5a99b5
Related-Bug: #1174809
Use constrainst from the new "swift-constraints" section of test.conf,
fall back to those found in the response to the /info API call,
ultimately falling back to the constraints module's effective
constraints.
Change-Id: Iea01c9c4b5148faa10004a240df411cbe7336a6a
Also fix some exception-handling stuff in functional tests; at some
point, ResponseError() started needing two more parameters, but the
functional tests (not swift_test_client, just tests.py) still had a
couple spots that were not passing in the new params. Now they're
optional again, and if you omit them, then the stringification of the
ResponseError is just a little less useful than it could be.
Change-Id: I38968c4b590fc04b97b85c5f974c8648291a6689
the Last-Modified header in Response didn't have a suitable
value - an integer part of object's timestamp.
This leads that the the if-[un]modified-since header with the
value from last-modified is always earlier than timestamp
and results the content is always newer than value of these
conditional headers.
Patched code returns math.ceil() of object's timestamp
in Last-Modified header so the later conditional header works
correctly
Closes-Bug: #1248818
Change-Id: I1ece7d008551bf989da74d23f0ed6307c45c5436
The functional tests have some hokey detection of SLO support that
pre-dates the /info API, but we can do better now.
Also moved the SLO check up inside the setUp method so that skipping
the SLO tests should be somewhat faster now.
Change-Id: I645718b459d794a9a97770f7162934558c94f3e8
Remove the useless arg ("start index" = 0) in files, since its default
value is 0, to make code cleaner.
Fixes bug #1259750
Change-Id: I52afac28a3248895bb1c012a5934d39e7c2cc5a9
This way, when something fails in Jenkins, you have some chance of
searching the logs for the relevant transaction.
Change-Id: I3cf606cb4963e32b5c6ac3deda08e73541b3ff7d
When an error occurs during functional tests that use the
swift_test_client module, the reported error message includes the
method and path:
ResponseError: 500: 'Internal Error' ('HEAD' \
'/v1/AUTH_test/d5ce...')
Change-Id: I631cd9e83879fb644778d4ded62625483bf38045
There's some sort-of-hacky code in there to detect SLO support in
order to skip tests when SLO is off so that the functests won't fail
on older clusters.
Change-Id: I6ad5974a0db7213747b0f4497d08ffc706d3f220
Remove extra lines of code that result in
swift_test_client adding params twice to
PUT request urls.
Bug was revealed while developing functional
tests for SLO - SLO manifest PUTs behave incorrectly
because the url constructed by swift_test_client ends
with ?multipart-manifest=put?multipart-manifest=put
Fixes bug 1248121
Change-Id: Ie5a8651a55049bb52ef641edfd6eb29b0ff3c245
* test/__init__.py: Put safe_repr import/implementation here so that it
is available to functional and unit tests.
* test/functional/swift_test_client.py: When a request fails
record why that request failed, how many requests failed, and what the
request was when raising RequestError to aid in debugging. Makes use of
safe_repr from test/__init__.py.
* test/unit/common/test_constraints.py: Remove implementation of
safe_repr and use the implementation in test/__init__.py.
Change-Id: I6c957343fb4b8b95d3875fd5ca87b3cf28a5f47a
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.
Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
Built-in symbol: file, format, range etc was re-assigned as local vars
in files: test/functional/swift_test_client.py and
test/functional/tests.py, rename these local vars.
Fixes bug #1208078
Change-Id: I627894cd722bd32cad231d534cbe97a0412c7cc9
- Make sure storage_url is string and not unicode or the unicode
no_safe_quote test will not get this nicely.
- Fixes bug 1190190.
Change-Id: I945f2b94f2063b836c10131fa4bd57938ed48f3c
