Drive-by: When passing a list or tuple to swob.Response as an app_iter,
check that it's full of byte strings.
Change-Id: Ifc35aacb2e45004f74c871f08ff3c52bc57c1463
Previously, we'd sometimes shove strings into HASH_PATH_PREFIX or
HASH_PATH_SUFFIX, which would blow up on py3. Now, always use bytes.
Change-Id: Icab9981e8920da505c2395eb040f8261f2da6d2e
This way the error is obvious and at initialization time, rather than being
an UnboundLocalError that waits until you actually make requests.
Change-Id: I4c8dafe34fcfea354b254b9137f31184cbd6a762
This boils down to 404, 412, or 416; or 409 when we provided an
X-Timestamp.
This means, among other things, that the expirer won't issue 3 DELETEs
every cycle for every stale work item.
Related-Change: Icd63c80c73f864d2561e745c3154fbfda02bd0cc
Change-Id: Ie5f2d3824e040bbc76d511a54d1316c4c2503732
If we don't drain the body, the proxy logging in the internal client
pipeline will log 499 client disconnect instead of actual error response
code.
For error responses, we try to do the most helpful thing using swob's
closing and caching response body attribute. For non-error responses
which are returned to the client, we endeavour to keep the app_iter
intact and unconsumed, trusting expecting the caller to do the right
thing is the only reasonable interface. We must cleanly close any WSGI
app_iter which we do not return to the client regardless of status code
and allow the logging of the 499 if needed.
Closes-Bug: #1675650
Change-Id: I455b5c38074ad0e72aa5e0b05771e193208905eb
This patch adds 'prefix' argument to iter_containers/iter_objects
method of InternalClient.
This change will be used in general task queue feature [1].
[1]: https://review.openstack.org/#/c/517389/
Change-Id: I8c2067c07fe35681fdc9403da771f451c21136d3
Add a symbolic link ("symlink") object support to Swift. This
object will reference another object. GET and HEAD
requests for a symlink object will operate on the referenced object.
DELETE and PUT requests for a symlink object will operate on the
symlink object, not the referenced object, and will delete or
overwrite it, respectively.
POST requests are *not* forwarded to the referenced object and should
be sent directly. POST requests sent to a symlink object will
result in a 307 Error.
Historical information on symlink design can be found here:
https://github.com/openstack/swift-specs/blob/master/specs/in_progress/symlinks.rst.
https://etherpad.openstack.org/p/swift_symlinks
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Co-Authored-By: Kazuhiro MIYAHARA <miyahara.kazuhiro@lab.ntt.co.jp>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Change-Id: I838ed71bacb3e33916db8dd42c7880d5bb9f8e18
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Follow up for change [1] to add some assertions to check that
marker param is included in sequential GET requests sent during
a full listing.
Extract multiple FakeConn class definitions to single class at
module level and share between all classes.
Also, explicitly unpack the return values from base request calls
made in the full listing section of base_request, and explicitly
return a list to make more consistent with rest of the method.
[1] Change-Id: I6892390d72f70f1bc519b482d4f72603e1570163
Change-Id: Iad038709f46364b8324d25ac79be4317add79df5
The internal_client is used in swift-dispersion-report, and in case one has more
than 10000 containers or objects these are not queried.
This patch adds support to the internal_client to iterate over all
containers/objects if the listing exceeds the default of 10000 entries and the
argument full_listing=True is used.
Closes-Bug: 1314817
Closes-Bug: 1525995
Change-Id: I6892390d72f70f1bc519b482d4f72603e1570163
This change adds a remote HEAD object request before each call to
sync_row.
Currently, container-sync-row attempts to replicate the object
(using PUT) regardless of the existance of the object on the remote side,
thus causing each object to be transferred on the wire several times
(depending on the replication factor)
An alternative to HEAD is to do a conditional PUT (using, 100-continue).
However, this change is more involved and requires upgrade of both the
client and server side clusters to work.
In the Tokyo design summit it was decided to start with the HEAD approach.
Change-Id: I60d982dd2cc79a0f13b0924507cd03d7f9c9d70b
Closes-Bug: #1277223
There was a function in swift.common.utils that was importing
swob.HeaderKeyDict at call time. It couldn't import it at compilation
time since utils can't import from swob or else it blows up with a
circular import error.
This commit just moves HeaderKeyDict into swift.common.header_key_dict
so that we can remove the inline import.
Change-Id: I656fde8cc2e125327c26c589cf1045cb81ffc7e5
As swift no longer supports Python 2.6, replace assertEqual(None, *)
with assertIsNone in tests to have more clear messages in case of
failure.
Change-Id: I94af3e8156ef40465d4f7a2cb79fb99fc7bbda56
Closes-Bug: #1280522
swift/container/sync.py uses swift.common.internal_client.delete_object
and put_object and expected these methods raise ClientException.
But delete_object and put_object never raise the exception so this patch
raises ClientException when urllib2 library raises HTTPError.
Co-Authored-By: Eran Rom <eranr@il.ibm.com>
Closes-Bug: #1419901
Change-Id: I58cbf77988979a07998a46d9d81be84d29b0d9bf
internal_client was not reading response if response status is not 200.
So proxy server treats this as client disconnect and logs 499 in log file.
This patch fixes that by reading response if response status is greater
than or equal to 300 and in acceptable statuses.
Closes-Bug: #1364752
Change-Id: I0512a25895da583956f76031e3c5de5c970bce01
assertEquals is deprecated in py3, replacing it.
Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8
Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
The urllib, urllib2 and urlparse modules of Python 2 were reorganized
into a new urllib namespace on Python 3. Replace urllib, urllib2 and
urlparse imports with six.moves.urllib to make the modified code
compatible with Python 2 and Python 3.
The initial patch was generated by the urllib operation of the sixer
tool on: bin/* swift/ test/.
Change-Id: I61a8c7fb7972eabc7da8dad3b3d34bceee5c5d93
This patch fixes Swift to show message
"No objects to query. Has swift-dispersion-populate been run?"
for "swift-dispersion-report —object-only”
with no container for object dispersion.
Change-Id: I82da56709cfc296a27f5180681709bc56adbc13d
Closes-Bug: #1468120
test.unit.common.test_internal_client.TestSimpleClient.test_proxy
will fail if the environment has http_proxy set and does not have
no_proxy=127.0.0.1 set. This seems to be because urllib overrides
any proxy arg passed to the Request with the env http_proxy var,
unless the Request host is in no_proxy.
We don't need to test urllib, so this patch changes the test to
simply ensure that swift code does pass the correct proxy arg to
urllib2.urlopen. That avoids testers having to make sure their
env vars are compliant to successfully run unit tests. While it is
reasonable to think that 127.0.0.1 is in the no_proxy list, it
shouldn't be a requirement pass swift tests.
Change-Id: Iff91acdb76fabe7056b3e90e2756e27fe5105817
* replace "from cStringIO import StringIO"
with "from six.moves import cStringIO as StringIO"
* replace "from StringIO import StringIO"
with "from six import StringIO"
* replace "import cStringIO" and "cStringIO.StringIO()"
with "from six import moves" and "moves.cStringIO()"
* replace "import StringIO" and "StringIO.StringIO()"
with "import six" and "six.StringIO()"
This patch was generated by the stringio operation of the sixer tool:
https://pypi.python.org/pypi/sixer
Change-Id: Iacba77fec3045f96773d1090c0bd48613729a561
This patch extends the StoragePolicy class for non-replication storage
policies, the first one being "erasure coding".
Changes:
- Add 'policy_type' support to BaseStoragePolicy class
- Disallow direct instantiation of BaseStoragePolicy class
- Subclass BaseStoragePolicy
- "StoragePolicy":
. Replication policy, default
. policy_type = 'replication'
- "ECStoragePolicy":
. Erasure Coding policy
. policy_type = 'erasure_coding'
. Private member variables
ec_type (EC backend),
ec_num_data_fragments (number of fragments original
data split into after erasure coding operation),
ec_num_parity_fragments (number of parity fragments
generated during erasure coding)
. Private methods
EC specific attributes and ring validator methods.
- Swift will use PyECLib, a Python Erasure Coding library, for
erasure coding operations. PyECLib is already an approved
OpenStack core requirement.
(https://bitbucket.org/kmgreen2/pyeclib/)
- Add test cases for
- 'policy_type' StoragePolicy member
- policy_type == 'erasure_coding'
DocImpact
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386
Implements: blueprint ec-proxy-work
Container sync might get stuck without a connection timeout if the remote proxy
is not responding.
This patch sets a default timeout of 5.0 seconds for the connection attempt. The
value is much higher than other connection timeouts inside Swift (0.5); however
there might be a much higher latency to the remote peer, thus playing it safe.
There is also a retry if the attempt timed out.
Note that this setting only applies to the connection request itself. Setting
this timeout does not apply when the remote proxy goes away during a request.
Also added a short test to ensure urlopen is called with the timeout value.
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Change-Id: Ic08a55157fa91fe1316653781adf4d66eead61bc
Partial-Bug: 1419916
Commit 7fcbbebb changed the internal client a little bit and removed the
class variable self.attempts in SimpleClient. This is currently still
needed in swift-dispersion-populate to track the number of retries, thus
re-adding it again.
Closes-Bug: 1351323
Change-Id: I98ba6441607158b34708b2dbb2d1a63788681d63
Previously, this code was attempting to set up http proxying but it
wasn't working. We noticed after a while when we saw traffic going
through an alternate route instead of our set of http proxies with
container sync.
Additional work and testing by clayg; thanks!
Change-Id: I840b8e55a80c13ae85c65bf68de261d735685b27
The SimpleClient.base_request method need only to call urllib2.urlopen
once to get a response from the server. Remove the duplicate call.
* fixup tests that were poorly mocking urllib2
* mock eventlet sleep because we're all busy
Change-Id: I6da93c689200351e9dbd9416b9c4bf9544571f15
Discovered some tests that were coupling the code under test with the
storage policies configured in /etc/swift/swift.conf. There was some
tests that created fake rings in their tempdirs, but didn't reset or
patch the POLICIES global. So if your local config needed more rings
that the fake's were setting up (just 2) the tests would puke when they
loaded up an app that looked for rings. I think this probably started
happening when we added eager object ring loading back into the proxy.
* two TestCases in test_wsgi were missing @patch_policies
* fixed issue with patch_policies that could cause state to bleed
between tests
* patch_policies' legacy and default collections get a FakeRing by
default
* drive-by cleanup for test_loadapp_proxy() ring serialized path
handling
* drive-by cleanup for test_internal_client that was doing basically
the same thing as test_wsgi
Change-Id: Ia706000ba961ed24f2c22b81041e53a0c3f302fc
* add get_object
* allow extra headers passthrough on HEAD/metadata reqeusts
* expose (account|container|get_object)_ring properties
Pipeline propety access to the auto_create_account_prefix also allows us to
bypass the early exit on a container HEAD for auto_create_accounts if the
container-updater hasn't cycled yet.
Allow overriding of storage policy index.
This is something the reconciler will need so that it can GET from one
policy, PUT in another, and then DELETE from the first one again.
DocImpact
Implements: blueprint storage-policies
Change-Id: I9b287d15f2426022d669d1186c9e22dd8ca13fb9
Also fixed a bug where SimpleClient would send ?format=json for object
requests, which is not necessary.
Change-Id: If06a7dcebc9de2d7c8b28a046d60b902dae821c1
As seen on #1174809, changes use of mutable types as default
arguments and defaults them within the method. Otherwise, those
defaults can be unexpectedly persisted with the function between
invocations and erupt into mass hysteria on the streets.
There was indeed a test (TestSimpleClient.test_get_with_retries)
that was erroneously relying on this behavior. Since previous tests
had populated their own instantiations with a token, this test only
passed because the modified headers dict from previous tests was
being overridden. As expected, with the mutable defaults fix in
SimpleClient, this test begain to fail since it never specified any
token, yet it has always passed anyway. This change also now provides
the expected token.
Change-Id: If95f11d259008517dab511e88acfe9731e5a99b5
Related-Bug: #1174809
internal client.
Internal client uses 'chunked' transfer encoding in it's
upload_object method. If content-length is known in advance,
internal_client should allowed to use content length in the
upload_object method and not the chunked method.
Change-Id: I68d2ebde78e01fa16b7187a2f045ea20f4310722
One can argue that it makes sense for the client-facing proxy server
to have certain middlewares like gatekeeper in its pipeline, but that
is not desirable for InternalClient. In particular, it prevents you
from passing in sysmeta headers using InternalClient, and I found
myself wanting to do that earlier today.
Now InternalClient's proxy application gets exactly what's configured;
no more, no less. This will mean that the object expirer can read and
write sysmeta headers, but I think we can trust it to keep our
secrets.
Change-Id: I17b4a89c24e600754701ee1645b40406421fa6f3
Removes the requirement for swiftclient in swift-dispersion-report
and swift-dispersion-populate. To prevent a dependency on
keystoneclient and to avoid reinventing the wheel with an internal
keystoneclient, authentication with keystone is only supported if
swiftclient is available. If not, only auth v1 is supported.
The dependency in swift/container/sync.py has also been removed.
Implements: blueprint remove-swiftclient-dependency
Change-Id: I6ec3b3c85a67b9ab6eb04b90ffc16daf1600e8a7
Hopefuly this extra test case is not too inane and slows us
down for nothing. It is verified to fail with the old code.
Change-Id: I604eca09f7f9ae044a8ad75284cd82a37325f99c
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.
Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
