24 Commits

Author SHA1 Message Date
Christian Schwede
8ad37b2217 Fix swift-dispersion-populate bug when using SimpleClient class
Commit 7fcbbebb changed the internal client a little bit and removed the
class variable self.attempts in SimpleClient. This is currently still
needed in swift-dispersion-populate to track the number of retries, thus
re-adding it again.

Closes-Bug: 1351323
Change-Id: I98ba6441607158b34708b2dbb2d1a63788681d63
2014-08-01 14:18:04 +00:00
Jenkins
3fda874801 Merge "Fix issues with test_wsgi.py and Storage Policies" 2014-07-04 02:08:12 +00:00
gholt
7fcbbebbeb SimpleClient http proxying
Previously, this code was attempting to set up http proxying but it
wasn't working. We noticed after a while when we saw traffic going
through an alternate route instead of our set of http proxies with
container sync.

Additional work and testing by clayg; thanks!

Change-Id: I840b8e55a80c13ae85c65bf68de261d735685b27
2014-07-02 13:05:23 +00:00
Clay Gerrard
0a5b003345 Fix double request in simple client
The SimpleClient.base_request method need only to call urllib2.urlopen
once to get a response from the server.  Remove the duplicate call.

 * fixup tests that were poorly mocking urllib2
 * mock eventlet sleep because we're all busy

Change-Id: I6da93c689200351e9dbd9416b9c4bf9544571f15
2014-06-25 23:27:04 -07:00
Paul Luse
441a171cf5 Fix issues with test_wsgi.py and Storage Policies
Discovered some tests that were coupling the code under test with the
storage policies configured in /etc/swift/swift.conf.  There was some
tests that created fake rings in their tempdirs, but didn't reset or
patch the POLICIES global.  So if your local config needed more rings
that the fake's were setting up (just 2) the tests would puke when they
loaded up an app that looked for rings.  I think this probably started
happening when we added eager object ring loading back into the proxy.

 * two TestCases in test_wsgi were missing @patch_policies
 * fixed issue with patch_policies that could cause state to bleed
   between tests
 * patch_policies' legacy and default collections get a FakeRing by
   default

 * drive-by cleanup for test_loadapp_proxy() ring serialized path
   handling
 * drive-by cleanup for test_internal_client that was doing basically
   the same thing as test_wsgi

Change-Id: Ia706000ba961ed24f2c22b81041e53a0c3f302fc
2014-06-23 23:13:18 -07:00
Clay Gerrard
8bec50838c Extend interface on InternalClient
* add get_object
 * allow extra headers passthrough on HEAD/metadata reqeusts
 * expose (account|container|get_object)_ring properties

Pipeline propety access to the auto_create_account_prefix also allows us to
bypass the early exit on a container HEAD for auto_create_accounts if the
container-updater hasn't cycled yet.

Allow overriding of storage policy index.

This is something the reconciler will need so that it can GET from one
policy, PUT in another, and then DELETE from the first one again.

DocImpact
Implements: blueprint storage-policies
Change-Id: I9b287d15f2426022d669d1186c9e22dd8ca13fb9
2014-06-18 17:31:39 -07:00
gholt
11bbc09bd7 Add debug logging to container sync requests
Also fixed a bug where SimpleClient would send ?format=json for object
requests, which is not necessary.

Change-Id: If06a7dcebc9de2d7c8b28a046d60b902dae821c1
2014-05-28 01:18:28 +00:00
Brian Cline
b4c5a13664 Uses None instead of mutables for function param defaults
As seen on #1174809, changes use of mutable types as default
arguments and defaults them within the method. Otherwise, those
defaults can be unexpectedly persisted with the function between
invocations and erupt into mass hysteria on the streets.

There was indeed a test (TestSimpleClient.test_get_with_retries)
that was erroneously relying on this behavior. Since previous tests
had populated their own instantiations with a token, this test only
passed because the modified headers dict from previous tests was
being overridden. As expected, with the mutable defaults fix in
SimpleClient, this test begain to fail since it never specified any
token, yet it has always passed anyway. This change also now provides
the expected token.

Change-Id: If95f11d259008517dab511e88acfe9731e5a99b5
Related-Bug: #1174809
2014-05-10 11:15:56 +00:00
Gil Vernik
b67a4b9938 Support for content-length in the upload object method for
internal client.

Internal client uses 'chunked' transfer encoding in it's
upload_object method. If content-length is known in advance,
internal_client should allowed to use content length in the
upload_object method and not the chunked method.

Change-Id: I68d2ebde78e01fa16b7187a2f045ea20f4310722
2014-04-07 09:56:03 +03:00
Yuan Zhou
59ad30ce8c Make internal client work with conf.d
Copied from Clay's patch in LP

Fixes bug #1277046

Change-Id: I2052ffaea3a74cdca9596b08273b7120570b1375
2014-03-05 16:23:42 +08:00
Samuel Merritt
3f19041c99 Don't lard up InternalClient with extra middleware
One can argue that it makes sense for the client-facing proxy server
to have certain middlewares like gatekeeper in its pipeline, but that
is not desirable for InternalClient. In particular, it prevents you
from passing in sysmeta headers using InternalClient, and I found
myself wanting to do that earlier today.

Now InternalClient's proxy application gets exactly what's configured;
no more, no less. This will mean that the object expirer can read and
write sysmeta headers, but I think we can trust it to keep our
secrets.

Change-Id: I17b4a89c24e600754701ee1645b40406421fa6f3
2014-02-28 12:55:15 -08:00
Christian Schwede
1f3ae6d8da Remove swiftclient dependency
Removes the requirement for swiftclient in swift-dispersion-report
and swift-dispersion-populate. To prevent a dependency on
keystoneclient and to avoid reinventing the wheel with an internal
keystoneclient, authentication with keystone is only supported if
swiftclient is available. If not, only auth v1 is supported.

The dependency in swift/container/sync.py has also been removed.

Implements: blueprint remove-swiftclient-dependency

Change-Id: I6ec3b3c85a67b9ab6eb04b90ffc16daf1600e8a7
2014-02-06 09:44:58 +00:00
Jenkins
0b594bc3af Merge "Change OpenStack LLC to Foundation" 2013-10-07 16:09:37 +00:00
Greg Lange
285a3a88a1 add seek() to CompressingFileReader
Change-Id: I33d9119684497b53101db3bc380953e86bdf25f0
2013-09-19 19:26:30 +00:00
ZhiQiang Fan
f72704fc82 Change OpenStack LLC to Foundation
Change-Id: I7c3df47c31759dbeb3105f8883e2688ada848d58
Closes-bug: #1214176
2013-09-20 01:02:31 +08:00
Pete Zaitcev
75086a1330 Supply correct arguments to __init__ of a base class
Hopefuly this extra test case is not too inane and slows us
down for nothing. It is verified to fail with the old code.

Change-Id: I604eca09f7f9ae044a8ad75284cd82a37325f99c
2013-09-14 16:09:53 -06:00
Dirk Mueller
3d36a76156 Use Python 3.x compatible except construct
except x,y: was deprected and is removed in Python 3.x.
Use "except x as y:" instead which works in any Python
version >= 2.6.

Change-Id: I7008c74b807340f3457d3a0c8bd0b83f23169d14
2013-09-07 10:50:54 +02:00
Alex Gaynor
ff5a6d0111 Corrected many style violations in the tests.
I focussed primarily on F-category violations, they are all but all fixed with
this patch.

Change-Id: I343f6945c97984ed1093bc347b6def6994297041
2013-07-24 10:18:47 -07:00
Greg Lange
e7b5e9d44f make internal client metadata methods return dicts with lower case keys
Change-Id: I19ced30b6e92133766310f8e41286202607c5b4c
2013-07-09 20:27:52 +00:00
Greg Lange
44f00a23c1 fixed some minor things in tests that pyflakes complained about
Change-Id: Ifeab56a964630bcf941e932fcbe39e6572e62975
2013-03-26 20:42:26 +00:00
Greg Lange
07d85d3ee8 Made internal client handle failed requests better
In two places, the internal client was not doing what it should with failed requests.

Change-Id: I49e566108fe920d85ca60fe76d24debe8ec7ffad
2013-02-04 19:59:38 +00:00
gholt
f46a4d8a2f Fixed bugs with internal client and object expirer
These bug fixes are lumped together because they all caused problems
with the object expirer doing its job.

There was a bug with the internal client doing listings that happened
to run across a Unicode object name for use as a marker.

There was a bug with the object expirer not utf8 encoding object
names it got from json listings, causing deletes to fail.

There was a bug with the object expirer url quoting object names when
calling the internal client's make_request, when make_request already
handles that.

Change-Id: I29fdd351fd60c8e63874b44d604c5fdff35169d4
2012-11-08 00:05:35 +00:00
Greg Lange
52baf4eee4 fixed internal client things
_get_metadata() should only look for metadata on 200 family http status codes

in the path_parts() in the internal client test file should return distinct unicode strings
with spaces for account, container, and obj

Change-Id: I74e335d3b73a59092a0eec06539f4931420061dd
2012-10-11 15:37:56 +00:00
Greg Lange
e7f3a9865e internal client unicode paths
made internal client handle unicode path parts by adding make_path method
fixed pep8 problems in internal client and its test
moved internal client unit test file to correct directory

Change-Id: Id1c81c9cb0db05342e4e8a8393db93552fda4647
2012-10-05 18:57:24 +00:00