Commit Graph

102 Commits (3b21157a844be5b71fba2216486c3ef412e7ae1a)

Author SHA1 Message Date
Timur Alperovich edfeae3723 Add delimiter to get_account().
Exposes the delimiter parameter, which the Swift API supports for
container listings.

Change-Id: Id8dfce01a9b64de9d1222aab9a4a682ce9e0f2b7
2018-11-30 22:58:36 +00:00
Tim Burke 411ef48e5b Stop leaking quite so many connections
While investigating the failures when you move func tests to py3, I
noticed a whole bunch of

   ResourceWarning: unclosed <socket.socket ...>

noise. This should fix it.

While we're at it, make get_capabilities less stupid.

Change-Id: I3913e9334090b04a78143e0b70f621aad30fc642
Related-Change: I86d24104033b490a35178fc504d88c1e4a566628
2018-11-09 09:55:30 -08:00
Tim Burke d1e1f8d8d6 Stop lazy importing keystoneclient
There were two basic problems:

  - We'd try to import on every attempt at getting auth, even when we
    already know keystoneclient isn't available.
  - Sometimes devs would hit some crazy import race involving (some
    combination of?) greenthreads and OS threads.

So let's just try the imports *once*, at import time, and have None
sentinels if it fails. Try both versions separately to decouple
failures; this should let us support a wider range of keystoneclient

Change-Id: I2367310aac74f1b7c5ea0cb1a822a491e4ba8e68
2018-09-07 16:56:13 -07:00
Timur Alperovich f4a2b16c2c Properly handle unicode headers.
Fix unicode handling in Python 3 and Python 2. There are currently two
failure modes. In python 2, swiftclient fails to log in debug mode if
the account name has a non-ASCII character. This is because the account
name will appear in the storage URL, which we attempt to pass to the
logger as a byte string (whereas it should be a unicode string). This
patch changes the behavior to convert the path strings into unicode by
calling the parse_header_string() function.

The second failure mode is with Python 3, where http_lib returns headers
that are latin-1 encoded, but swiftclient expects UTF-8. The patch
automatically converts headers from latin-1 (iso-8859-1) to UTF-8, so
that we can properly handle non-ASCII headers in responses.

Change-Id: Ifa7f3d5af71bde8127129f1f8603772d80d063c1
2018-07-23 14:38:40 -07:00
Zuul 23d29eda8d Merge "Stop mutating header dicts" 2018-07-17 01:05:14 +00:00
Clay Gerrard 1190825054 Make OS_AUTH_URL work in DevStack by default
An earlier change added support for versionless authurls, but the
huristic to detect them didn't work for some configurations I've

Now we use a little bit tighter pattern matching and support auth_url
values with more than one path component.

Change-Id: I5a99c7b4e957ee7c8a5b5470477db49ab2ddba4b
Related-Change-Id: If7ecb67776cb77828f93ad8278cc5040015216b7
2018-06-20 13:25:46 +00:00
Kota Tsuyuzaki e65070964c Add force auth retry mode in swiftclient
This patch attemps to add an option to force get_auth call while retrying
an operation even if it gets errors other than 401 Unauthorized.

Why we need this:
The main reason why we need this is current python-swiftclient requests could
never get succeeded under certion situation using third party proxies/load balancers
between the client and swift-proxy server. I think, it would be general situation
of the use case.

Specifically describing nginx case, the nginx can close the socket from the client
when the response code from swift is not 2xx series. In default, nginx can wait the
buffers from the client for a while (default 30s)[1] but after the time past, nginx
will close the socket immediately. Unfortunately, if python-swiftclient has still been
sending the data into the socket, python-swiftclient will get socket error (EPIPE,
BrokenPipe). From the swiftclient perspective, this is absolutely not an auth error,
so current python-swiftclient will continue to retry without re-auth.
However, if the root cause is sort of 401 (i.e. nginx got 401 unauthorized from the
swift-proxy because of token expiration), swiftclient will loop 401 -> EPIPE -> 401...
until it consume the max retry times.

In particlar, less time to live of the token and multipart object upload with large
segments could not get succeeded as below:

Connection Model:

python-swiftclient -> nginx -> swift-proxy -> swift-backend

Case: Try to create slo with large segments and the auth token expired with 1 hour

1. client create a connection to nginx with successful response from swift-proxy and its auth
2. client continue to put large segment objects
   (e.g. 1~5GB for each and the total would 20~30GB, i.e. 20~30 segments)
3. after some of segments uploaded, 1 hour past but client is still trying to
   send remaining segment objects.
4. nginx got 401 from swift-proxy for a request and wait that the connection is closed
   from the client but timeout past because the python-swiftclient is still sending much data
   into the socket before reading the 401 response.
5. client got socket error because nginx closed the connection during sending the buffer.
6. client retries a new connection to nginx without re-auth...

<loop 4-6>

7. finally python-swiftclient failed with socket error (Broken Pipe)

In operational perspective, setting longer timeout for lingering close would be an option but
it's not complete solution because any other proxy/LB may not support the options.

If we actually do THE RIGHT THING in python-swiftclient, we should send expects: 100-continue
header and handle the first response to re-auth correctly.

HOWEVER, the current python's httplib and requests module used by python-swiftclient doesn't
support expects: 100-continue header [2] and the thread proposed a fix [3] is not super active.
And we know the reason we depends on the library is to fix a security issue that existed
in older python-swiftclient [4] so that we should touch around it super carefully.

In the reality, as the hot fix, this patch try to mitigate the unfortunate situation
described above WITHOUT 100-continue fix, just users can force to re-auth when any errors
occurred during the retries that can be accepted in the upstream.


Change-Id: I3470b56e3f9cf9cdb8c2fc2a94b2c551927a3440
2018-03-13 12:29:48 +09:00
Timur Alperovich a36c3cfda1 Add a query_string option to head_object().
Submitting a path parameter with a HEAD request on an object can be
useful if one is trying to find out information about an SLO/DLO without
retrieving the manifest.

Change-Id: I39efd098e72bd31de271ac51d4d75381929c9638
2018-03-05 17:33:22 -08:00
Jenkins c50823ebf1 Merge "Add support for versionless endpoints" 2017-08-29 02:15:28 +00:00
Tim Burke ae5fd46e87 Stop mutating header dicts
Change-Id: Ia1638c216eff9db6fbe416bc0570c27cfdcfe730
2017-08-25 12:26:03 -07:00
Jenkins 74eb91b176 Merge "Do not set Content-Type to '' with new requests." 2017-06-13 22:07:47 +00:00
Timur Alperovich 32f6b3c642 Do not set Content-Type to '' with new requests.
Previously, python-swiftclient worked around a requests issue where
Content-Type could be set to application/x-www-form-urlencoded when
using python3. This issue has been resolved and a fix released in
requests 2.4 (fixed in subsequent releases as well). The patch makes
the workaround conditional on the requests version, so that with
sufficiently new requests libraries, the Content-Type is not set.

For reference, requests 2.4 was released August 29th, 2014. The
specific issue filed in the requests tracker is:

Related-Change: I035f8b4b9c9ccdc79820b907770a48f86d0343b4
Closes-Bug: #1433767

Change-Id: Ieb2243d2ff5326920a27ce8c3c6f0f5c396701ed
2017-06-13 10:41:01 -07:00
Christian Schwede 2ff3102cf7 Add support for versionless endpoints
Newer deployments are using versionless Keystone endpoints, and most
OpenStack clients already support this.

This patch enables this for Swift: if an auth_url without any path
component is found, it assumes a versionless endpoint will be used.
In this case the v3 suffix will be appended to the path if none
auth_version is set, and v2.0 is appended if auth_version requires v2.

Closes-Bug: 1554885
Related-Bug: 1691106
Change-Id: If7ecb67776cb77828f93ad8278cc5040015216b7
2017-06-13 10:55:50 +02:00
Clay Gerrard dd34af42f8 Fix MockHttpResponse to be more like the Real
This change pulls out that relatively new [1] little string to pull at
in the MockHttpResponse that I think is sorta ugly.  And replaces it
with the correct behavior that's representative of the Real for which
it's standing in (which is sadly our wrapper to make a requests response
feel like a httplib.HTTPResponse).

It's not clear (to me) the history which allowed this difference in the
behavior of the Real and Fake to persist - it seems to have always been
this way [2].

I also reworded a relatively new test [1] to cover more code, and make
assertions on the desired behavior of the client instead of "just" the
http_log method.

FWIW, I don't think there was necessarily anything wrong with the scope
of the new test [1] - and it certainly makes sense to see new tests copy
nearby existing tests.  But I subjectively think this smaller test is
more demonstrative of the desired behavior.

1. Related-Change-Id: I6d7ccbf4ef9b46e890ecec58842c5cdd2804c7a9
2. Related-Change-Id: If07af46cb377f3f3d70f6c4284037241d360a8b7
Change-Id: Ib99a029c1bd1ea1efa8060fe8a11cb01deea41c6
2017-03-08 10:51:43 -08:00
Vitaly Gridnev 028c4824d0 Fix logging of the gzipped body
Change-Id: I6d7ccbf4ef9b46e890ecec58842c5cdd2804c7a9
Closes-bug: 1670620
2017-03-08 00:50:55 +04:00
Jenkins 70c90b2243 Merge "Add additional headers for HEAD/GET/DELETE requests." 2016-11-08 19:30:40 +00:00
Charles Hsu 6cf2bd6626 Add additional headers for HEAD/GET/DELETE requests.
Change-Id: I69276ba711057c122f97deac412e492e313c34dd
Closes-Bug: 1615830
2016-11-07 13:18:29 +08:00
Jenkins e9887703d0 Merge "Adding keystoneauth sessions support" 2016-10-26 11:41:21 +00:00
Jenkins 92544c58c5 Merge "Accept gzip-encoded API responses" 2016-09-01 03:18:18 +00:00
Jenkins 98085c961f Merge "Convert numeric and boolean header values to strings" 2016-08-31 00:58:02 +00:00
Tim Burke f728027bed Accept gzip-encoded API responses
Previously, we would accept gzip-encoded responses, but only because we
were letting requests decode *all* responses (even object data). This
restores the previous capability, but with tighter controls about which
requests will accept gzipped responses and where the decoding happens.

Change-Id: I4fd8b97207b9ab01b1bcf825cc16efd8ad46344a
Related-Bug: 1282861
Related-Bug: 1338464
2016-08-30 15:38:55 -07:00
Jenkins 20e0c515bf Merge "client: renew token on 401 even if retries is 0" 2016-08-26 17:37:41 +00:00
Tim Burke ab60e08e2e Convert numeric and boolean header values to strings
Recently, requests got a bit more picky about what types of data it will
accept as header values [1]. The reasons for this are generally sound;
str()ing arbitrary objects just before pushing them out a socket may not
do what the developer wanted/expected.

However, there are a few standard types that developers may be sending
that we should convert for them as a convenience. Now, we'll convert all
int, float, and bool values to strings before sending them on to

Change-Id: I6c2f451009cb03cb78812f54e4ed8566076de821
Closes-Bug: 1614932
2016-08-25 11:32:08 -07:00
Jamie Lennox fd675b2c5a Use mock patch to handle get_auth_keystone
Setting c.get_auth_keystone = fake_get_auth_keystone is setting a new
method on the module. This information is not reset between test runs
and therefore has the potential to corrupt other tests.

Use mock patch so that the mock is reset after the test is complete.

Change-Id: Ifb9ba72634cf477c72dda080b5aed8f8e3a7ac89
2016-08-25 17:18:20 +10:00
Jenkins b57044a853 Merge "Add copy object method" 2016-08-24 23:59:34 +00:00
Marek Kaleta 4a2465fb12 Add copy object method
Implement copy object method in swiftclient Connection, Service and CLI.

Although COPY functionality can be accomplished with 'X-Copy-From'
header in PUT request, using copy is more convenient especially when
using copy for updating object metadata non-destructively.

Closes-Bug: 1474939
Change-Id: I1338ac411f418f4adb3d06753d044a484a7f32a4
2016-08-23 14:37:11 -07:00
Jenkins 07c960d46f Merge "Query string functionality for containers" 2016-06-14 19:36:45 +00:00
Andrew Welleck 439330cb9c Query string functionality for containers
Added functionality for arbitrary query strings to be passed into container
functions. Additionally a minor typo correction in the README. Added unit
tests for query string functionality.

Closes-Bug: #1542459
Change-Id: Ica2cb3ea439632588388e748d8d2e944e9ed4fa4
2016-06-09 13:51:26 -05:00
Julien Danjou a62b7ee06c client: renew token on 401 even if retries is 0
Gnocchi uses a client with retries=0 to maximize throughtput and not retry N
times on e.g. 404 when checking existence of an object. However, this as the
side effect of never renewing the token since there' no retry on 401 either.

This patches change the behavior so that 401 errors are always retried,
whatever the retries value is.

Closes-Bug: #1589926
Change-Id: Ie06adf4cf17ea4592b5bbd7bbde9828e5e134e3e
2016-06-07 16:58:26 +02:00
Paulo Ewerton 73e4296a38 Adding keystoneauth sessions support
This patch allows authentication in swiftclient with a keystonauth

Co-Authored-By: Tim Burke <>

Change-Id: Ia3fd947ff619c11ff0ce474897533dcf7b49d9b3
Closes-Bug: 1518938
2016-05-19 17:27:31 -07:00
Jenkins f9d0657e70 Merge "Support client certificate/key" 2016-05-19 22:20:17 +00:00
Jenkins 3a6c14981d Merge "Check responses when retrying bodies" 2016-05-11 07:52:47 +00:00
Tim Burke fd5579a154 Check responses when retrying bodies
Previously, if a Range request came back 200 OK (rather than 206 Partial
Content), we would mangle the response body. This could happen if there
was a middleware that would silently drop Range headers, for example.

Now, if the response does not include a Content-Range header, we will
log a warning and seek to our previous position in the stream. If the
Content-Range header has an unexpected value, we will raise an exception.

Change-Id: I94d4536cc1489968d45a2b6ba7edd70c85800275
2016-05-04 15:21:26 -07:00
Cedric Brandily 450f505c35 Support client certificate/key
This change enables to specify a client certificate/key with:
 * usual CLI options (--os-cert/--os-key)
 * usual environment variables ($OS_CERT/$OS_KEY)

Closes-Bug: #1565112
Change-Id: I12e151adcb6084d801c6dfed21d82232a3259aea
2016-04-10 23:20:49 +02:00
Tim Burke 9b8ab67a78 Include response headers in ClientExceptions
Now, client applications can get to things like transaction IDs for
failures without needing to turn on all of logging.

While we're at it, add a from_response factory method for

Co-Authored-By: Alexander Corwin <>
Change-Id: Ib46d5f8fc7f36f651f5908bb9d900316fdaebce3
2016-03-03 17:16:33 +00:00
Jenkins cd3a4dbf0a Merge "Drop testtools from test-requirements.txt" 2016-03-01 17:51:32 +00:00
Jenkins b040ce4e1a Merge "Fix wrong args for get_container with full listing" 2016-02-29 16:26:58 +00:00
Jenkins ba2ff4a6ea Merge "Force header keys/values to bytes/unicode before coercing to unicode" 2016-02-27 01:44:38 +00:00
Tim Burke aa0edd0096 Force header keys/values to bytes/unicode before coercing to unicode
Previously, parse_header_string was only called with data coming out of
requests, which would be either bytes or unicode. Now that we're sending
it request headers as well (see related change), we need to be more

If the value given is neither bytes nor unicode, convert it to a native
string. This will allow developers using the client API to continue
sending header dicts like

  {'X-Delete-After': 2} in Swift's test/probe/

Change-Id: Ie57a93274507b184af5cad4260f244359a585f09
Related-Change: I43dd7254f7281d4db59b286aa2145643c64e1705
2016-02-26 11:25:10 -08:00
Joel Wright 46d8178280 Fix test for redacting sensitive data in client.http_log()
The test should have included utf8 encoded unicode data to
test that encoded unicode data stored in headers was parsed

Also fixes the docstring for swiftclient.safe_value()

Change-Id: Id0def0b3af7a364f1257cc22f67b71c0cc5d8479
2016-02-26 11:36:45 +00:00
Tim Burke c3f0641704 Follow-up to patch 282363
* Improve some formatting
* Be more explicit about how much will be revealed when
* Rename redact_sensitive_tokens to redact_sensitive_headers, as it
  affects more than tokens.

Change-Id: I02b375d914e9f0a210d038ecb31188d09a8ffce3
2016-02-25 10:06:48 -08:00
Joel Wright 4d44dcf360 Do not reveal auth token in swiftclient log messages by default
Currently the swiftclient logs sensitive info in headers when logging
HTTP requests. This patch hides sensitive info in headers such as
'X-Auth-Token' in a similar way to swift itself (we add a
'reveal_sensitive_prefix' configuration to the client).

With this patch, tokens are truncated by removing the specified number
of characters, after which '...' is appended to the logged token to
indicate that it has been redacted.

Co-Authored-By: Li Cheng <>
Co-Authored-By: Zack M. Davis <>
Change-Id: I43dd7254f7281d4db59b286aa2145643c64e1705
Closes-bug: #1516692
2016-02-22 17:55:51 +00:00
Alistair Coles 67f5468ee4 Fix wrong args for get_container with full listing
In client get_container(), when full_listing is true,
the calls back to get_container() pass service_token
as a positional arg which maps its value to the
full_listing arg. It should use a keyword.

Change-Id: Iac2af45df124ff33fcb7fbaf1ba959ef06c96378
Closes-Bug: #1496093
2016-02-22 15:22:44 +00:00
Jenkins da0aa24f28 Merge "_RetryBody doesn't need to take explicit etag/content-length" 2016-02-18 23:40:51 +00:00
Tim Burke bed6bbd5ef Drop testtools from test-requirements.txt
My understanding is that it was mainly being used so we could have sane
testing on py26.  With py26 support being dropped, we no longer need it.

Also drop discover from test-requirements.txt, as we don't seem to
actually use it.

Change-Id: Iee04c42890596d3b483c1473169480a3ae19aac8
Related-Change: I37116731db11449d0c374a6a83a3a43789a19d5f
2016-02-12 09:57:58 -08:00
Jenkins 84d110c63e Merge "Accept token and tenant_id for authenticating against KS" 2016-02-10 19:39:53 +00:00
Pratik Mallya a175689418 Accept token and tenant_id for authenticating against KS
Allow swiftclient to authenticate against keystone using tenant
name/id and token only. Without this patch, the password is
required, which may not always be available. Authentication
against keystone is required to get the service catalog,
which includes the endpoints for swift.

Change-Id: I4477af445474c5fa97ff864c4942f1330b59e5d6
Closes-Bug: #1476002
2016-01-18 10:47:05 -08:00
Tim Burke 7a1e192803 Use bulk-delete middleware when available
When issuing `delete` commands that would require three or more
individual deletes, check whether the cluster supports bulk deletes
and use that if it's available.

Additionally, a new option is added to the `delete` command:

  * --prefix <prefix>

    Delete all objects that start with <prefix>. This is similar to the
    --prefix option for the `list` command.


$ swift delete c --prefix obj_prefix/

    ...will delete from container "c" all objects whose name begins with
    "obj_prefix/", such as "obj_prefix/foo" and "obj_prefix/bar".

Change-Id: I6b9504848d6ef562cf4f570bbcd17db4e3da8264
2016-01-12 15:40:57 -08:00
Jenkins 6ed6c3343f Merge "Retry file uploads via SwiftService" 2016-01-12 13:00:17 +00:00
Tim Burke 5050027610 _RetryBody doesn't need to take explicit etag/content-length
Also, don't try to do int(None) for chunk-encoded responses (like DLOs
that are longer than a single container listing).

Change-Id: Ibacd75d5ee46135d62388786903c895fda8ed3ba
2016-01-11 15:36:37 -08:00