Commit Graph

412 Commits (3b21157a844be5b71fba2216486c3ef412e7ae1a)

Author SHA1 Message Date
Tim Burke 3b21157a84 Clean up warnings from newer flake8
Change-Id: I18a6327b3acdd4db5ae80097080c043f7c20c353
4 years ago
Tim Burke 113eacf3b8 Isolate docs requirements
...since modern sphinx won't install on py27.

While we're at it, clean up some warnings and treat warnings as errors.

Also, fix up how we parse test configs so we can run func tests.

Related-Change: Id3c2ed87230c5918c18e2c01d086df8157f036b1
Change-Id: I3718f69610545b0dbcb0a2ab45b400da3a45682c
4 years ago
ZhijunWei 2ff36fde57 Update hacking version
1. update hacking version to latest
2. fix pep8 failed

Change-Id: Ifc3bfeff4038c93d8c8cf2c9d7814c3003e73504
4 years ago
Timur Alperovich edfeae3723 Add delimiter to get_account().
Exposes the delimiter parameter, which the Swift API supports for
container listings.

Change-Id: Id8dfce01a9b64de9d1222aab9a4a682ce9e0f2b7
5 years ago
Tim Burke 411ef48e5b Stop leaking quite so many connections
While investigating the failures when you move func tests to py3, I
noticed a whole bunch of

   ResourceWarning: unclosed <socket.socket ...>

noise. This should fix it.

While we're at it, make get_capabilities less stupid.

Change-Id: I3913e9334090b04a78143e0b70f621aad30fc642
Related-Change: I86d24104033b490a35178fc504d88c1e4a566628
5 years ago
Tim Burke d1e1f8d8d6 Stop lazy importing keystoneclient
There were two basic problems:

  - We'd try to import on every attempt at getting auth, even when we
    already know keystoneclient isn't available.
  - Sometimes devs would hit some crazy import race involving (some
    combination of?) greenthreads and OS threads.

So let's just try the imports *once*, at import time, and have None
sentinels if it fails. Try both versions separately to decouple
failures; this should let us support a wider range of keystoneclient

Change-Id: I2367310aac74f1b7c5ea0cb1a822a491e4ba8e68
5 years ago
Zuul d80f24f2fd Merge "Back out some version bumps" 5 years ago
Timur Alperovich f4a2b16c2c Properly handle unicode headers.
Fix unicode handling in Python 3 and Python 2. There are currently two
failure modes. In python 2, swiftclient fails to log in debug mode if
the account name has a non-ASCII character. This is because the account
name will appear in the storage URL, which we attempt to pass to the
logger as a byte string (whereas it should be a unicode string). This
patch changes the behavior to convert the path strings into unicode by
calling the parse_header_string() function.

The second failure mode is with Python 3, where http_lib returns headers
that are latin-1 encoded, but swiftclient expects UTF-8. The patch
automatically converts headers from latin-1 (iso-8859-1) to UTF-8, so
that we can properly handle non-ASCII headers in responses.

Change-Id: Ifa7f3d5af71bde8127129f1f8603772d80d063c1
5 years ago
Zuul 23d29eda8d Merge "Stop mutating header dicts" 5 years ago
Tim Burke da362a653e Back out some version bumps
I'm giving up on trying to back out all of the test-requirements
up-revs, but let's try to stay compatibile with old requests/six.

As part of that, only disable some requests warnings on new-enough requests.

Note that we should now be compatible with distro packages back to
Ubuntu 16.04 and CentOS 6. Our six is still too new for Trusty, but
hey, there's less than a year left on that anyway, right?

Change-Id: Iccb23638393616f9ec3da660dd5e39ea4ea94220
Related-Change: I2a8f465c8b08370517cbec857933b08fca94ca38
5 years ago
mmcardle 47fb18c41b Add ability to generate a temporary URL with an
IP range restriction

Change-Id: I4734599886e4f4a563162390d0ff3bb1ef639db4
5 years ago
Zuul c2c5af603f Merge "Add option for user to enter password" 5 years ago
Zuul eb97c69a3c Merge "Make OS_AUTH_URL work in DevStack by default" 5 years ago
Tim Burke 85bb28eab4 Remove some pointless code
Change-Id: I3163834c330c5ea44c1096e83127588c88f0d761
5 years ago
Clay Gerrard 1190825054 Make OS_AUTH_URL work in DevStack by default
An earlier change added support for versionless authurls, but the
huristic to detect them didn't work for some configurations I've

Now we use a little bit tighter pattern matching and support auth_url
values with more than one path component.

Change-Id: I5a99c7b4e957ee7c8a5b5470477db49ab2ddba4b
Related-Change-Id: If7ecb67776cb77828f93ad8278cc5040015216b7
5 years ago
Alistair Coles 33ad9fd4cc Add option for user to enter password
Add the --prompt option for the CLI which will cause the user to be
prompted to enter a password. Any password otherwise specified by
--key, --os-password or an environment variable will be ignored.

The swift client will exit with a warning if the password cannot be
entered without its value being echoed.

Closes-Bug: #1357562
Change-Id: I513647eed460007617f129691069c6fb1bfe62d7
5 years ago
Kota Tsuyuzaki e65070964c Add force auth retry mode in swiftclient
This patch attemps to add an option to force get_auth call while retrying
an operation even if it gets errors other than 401 Unauthorized.

Why we need this:
The main reason why we need this is current python-swiftclient requests could
never get succeeded under certion situation using third party proxies/load balancers
between the client and swift-proxy server. I think, it would be general situation
of the use case.

Specifically describing nginx case, the nginx can close the socket from the client
when the response code from swift is not 2xx series. In default, nginx can wait the
buffers from the client for a while (default 30s)[1] but after the time past, nginx
will close the socket immediately. Unfortunately, if python-swiftclient has still been
sending the data into the socket, python-swiftclient will get socket error (EPIPE,
BrokenPipe). From the swiftclient perspective, this is absolutely not an auth error,
so current python-swiftclient will continue to retry without re-auth.
However, if the root cause is sort of 401 (i.e. nginx got 401 unauthorized from the
swift-proxy because of token expiration), swiftclient will loop 401 -> EPIPE -> 401...
until it consume the max retry times.

In particlar, less time to live of the token and multipart object upload with large
segments could not get succeeded as below:

Connection Model:

python-swiftclient -> nginx -> swift-proxy -> swift-backend

Case: Try to create slo with large segments and the auth token expired with 1 hour

1. client create a connection to nginx with successful response from swift-proxy and its auth
2. client continue to put large segment objects
   (e.g. 1~5GB for each and the total would 20~30GB, i.e. 20~30 segments)
3. after some of segments uploaded, 1 hour past but client is still trying to
   send remaining segment objects.
4. nginx got 401 from swift-proxy for a request and wait that the connection is closed
   from the client but timeout past because the python-swiftclient is still sending much data
   into the socket before reading the 401 response.
5. client got socket error because nginx closed the connection during sending the buffer.
6. client retries a new connection to nginx without re-auth...

<loop 4-6>

7. finally python-swiftclient failed with socket error (Broken Pipe)

In operational perspective, setting longer timeout for lingering close would be an option but
it's not complete solution because any other proxy/LB may not support the options.

If we actually do THE RIGHT THING in python-swiftclient, we should send expects: 100-continue
header and handle the first response to re-auth correctly.

HOWEVER, the current python's httplib and requests module used by python-swiftclient doesn't
support expects: 100-continue header [2] and the thread proposed a fix [3] is not super active.
And we know the reason we depends on the library is to fix a security issue that existed
in older python-swiftclient [4] so that we should touch around it super carefully.

In the reality, as the hot fix, this patch try to mitigate the unfortunate situation
described above WITHOUT 100-continue fix, just users can force to re-auth when any errors
occurred during the retries that can be accepted in the upstream.


Change-Id: I3470b56e3f9cf9cdb8c2fc2a94b2c551927a3440
5 years ago
Timur Alperovich a36c3cfda1 Add a query_string option to head_object().
Submitting a path parameter with a HEAD request on an object can be
useful if one is trying to find out information about an SLO/DLO without
retrieving the manifest.

Change-Id: I39efd098e72bd31de271ac51d4d75381929c9638
5 years ago
Timur Alperovich 2faea93287 Allow for object uploads > 5GB from stdin.
When uploading from standard input, swiftclient should turn the upload
into an SLO in the case of large objects. This patch picks the
threshold as 10MB (and uses that as the default segment size). The
consumers can also supply the --segment-size option to alter that
threshold and the SLO segment size. The patch does buffer one segment
in memory (which is why 10MB default was chosen).

(test is updated)

Change-Id: Ib13e0b687bc85930c29fe9f151cf96bc53b2e594
5 years ago
Zuul cde257de5c Merge "Allow --meta on upload" 6 years ago
Jenkins c50823ebf1 Merge "Add support for versionless endpoints" 6 years ago
Tim Burke ae5fd46e87 Stop mutating header dicts
Change-Id: Ia1638c216eff9db6fbe416bc0570c27cfdcfe730
6 years ago
Timur Alperovich 0982791db2 Allow for uploads from standard input.
If "-" is passed in for the source, python-swiftclient will upload
the object by reading the contents of the standard input. The object
name option must be set, as well, and this cannot be used in
conjunction with other files.

This approach stores the entire contents as one object. A follow on
patch will change this behavior to upload from standard input as SLO,
unless the segment size is larger than the content size.

Change-Id: I1a8be6377de06f702e0f336a5a593408ed49be02
6 years ago
Tim Burke 638d7c789c Buffer reads from disk
Otherwise, Python defaults to 8k reads which seems kinda terrible.

Change-Id: I3160626e947083af487fd1c3cb0aa6a62646527b
Closes-Bug: #1671621
6 years ago
Tim Burke 484d7ee9b2 Allow --meta on upload
Previously, the --meta option was only allowed on post or copy subcommands.

Change-Id: I87bf0338c34b5e89aa946505bee68dbeb37d784c
Closes-Bug: #1616238
6 years ago
Christopher Bartz cde73c196d Option to ignore mtime metadata entry.
Currently, the swiftclient upload command passes a custom metadata
header for each object (called object-meta-mtime), whose value is
the current UNIX timestamp. When downloading such an object with the
swiftclient, the mtime header is parsed and passed as the atime and
mtime for the newly created file.

There are use-cases where this is not desired, for example when using
tmp or scratch directories in which files older than a specific date
are deleted. This commit provides a boolean option for ignoring the
mtime header.

Change-Id: If60b389aa910c6f1969b999b5d3b6d0940375686
6 years ago
Jenkins 1d57403668 Merge "Skip checksum validation on partial downloads" 6 years ago
Jenkins bc3171cfac Merge "Tolerate RFC-compliant ETags" 6 years ago
Jenkins 4515002d78 Merge "Stop sending X-Static-Large-Object headers" 6 years ago
Jenkins 74eb91b176 Merge "Do not set Content-Type to '' with new requests." 6 years ago
Timur Alperovich 32f6b3c642 Do not set Content-Type to '' with new requests.
Previously, python-swiftclient worked around a requests issue where
Content-Type could be set to application/x-www-form-urlencoded when
using python3. This issue has been resolved and a fix released in
requests 2.4 (fixed in subsequent releases as well). The patch makes
the workaround conditional on the requests version, so that with
sufficiently new requests libraries, the Content-Type is not set.

For reference, requests 2.4 was released August 29th, 2014. The
specific issue filed in the requests tracker is:

Related-Change: I035f8b4b9c9ccdc79820b907770a48f86d0343b4
Closes-Bug: #1433767

Change-Id: Ieb2243d2ff5326920a27ce8c3c6f0f5c396701ed
6 years ago
Christian Schwede 2ff3102cf7 Add support for versionless endpoints
Newer deployments are using versionless Keystone endpoints, and most
OpenStack clients already support this.

This patch enables this for Swift: if an auth_url without any path
component is found, it assumes a versionless endpoint will be used.
In this case the v3 suffix will be appended to the path if none
auth_version is set, and v2.0 is appended if auth_version requires v2.

Closes-Bug: 1554885
Related-Bug: 1691106
Change-Id: If7ecb67776cb77828f93ad8278cc5040015216b7
6 years ago
Jenkins f18d070b0b Merge "Fix MockHttpResponse to be more like the Real" 6 years ago
Jenkins 6d5e87a183 Merge "ISO 8601 timestamps for tempurl" 6 years ago
Tim Burke 527f2ff687 Skip checksum validation on partial downloads
If we get back some partial content, we can't validate the MD5.
That's OK.

Change-Id: Ic1d65272190af0d3d982f3cd06833cac5c791a1e
Closes-Bug: 1642021
6 years ago
Tim Burke 64da481ccd Tolerate RFC-compliant ETags
Since time immemorial, Swift has returned unquoted ETags for plain-old
Swift objects -- I hear tell that we once tried to change this, but
quickly backed it out when some clients broke.

However, some proxies (such as nginx) apparently may force the ETag to
adhere to the RFC, which states [1]:

    An entity-tag consists of an opaque *quoted* string

(emphasis mine). See the related bug for an instance of this happening.

Since we can still get the original ETag easily, we should tolerate the
more-compliant format.

[1] or, if you
    prefer the new ones,

Change-Id: I7cfacab3f250a9443af4b67111ef8088d37d9171
Closes-Bug: 1681529
Related-Bug: 1678976
6 years ago
John Dickinson 0cc4d8af18 respect bulk delete page size and fix logic error
Previously, using SwiftService to delete "many" objects would use
bulk delete if available, but it would not respect the bulk delete
page size. If the number of objects to delete exceeded the bulk delete
page size, SwiftService would ignore the error and nothing would be

This patch changes _should_bulk_delete() to be _bulk_delete_page_size();
instead of returning a simple True/False, it returns the page size for
the bulk deleter, or 1 if objects should be deleted one at a time.
Delete SDK calls are then spread across multiple bulk DELETEs if the
requested number of objects to delete exceeds the returned page size.

Fixed the logic in _should_bulk_delete() so that if the object list
is exactly 2x the thread count, it will not bulk delete. This is the
natural conclusion following the logic that existed previously: if
the delete request can be satisfied by every worker thread doing one
or two tasks, don't bulk delete. But if it requires a worker thread
to do three or more tasks, do a bulk delete instead. Previously, the
logic would mean that if every worker thread did exactly two tasks, it
would bulk delete. This patch changes a "<" to a "<=".

Closes-Bug: 1679851
Change-Id: I3c18f89bac1170dc62187114ef06dbe721afcc2e
6 years ago
Tim Burke aaaed55cd4 Stop sending X-Static-Large-Object headers
If we were to include this in a normal PUT, it would 400, but only if
slo is actually in the pipeline. If it's *not*, we'll create a normal
Swift object and the header sticks.

- This is really confusing for users; see the related bug.
- If slo is later enabled in the cluster, Swift starts responding 500
  with a KeyError because the client and on-disk formats don't match!

Change-Id: I1d80c76af02f2ca847123349224ddc36d2a6996b
Related-Change: I986c1656658f874172860469624118cc63bff9bc
Related-Bug: #1680083
6 years ago
Christopher Bartz 8e08931b9f ISO 8601 timestamps for tempurl
Client-side implementation for ISO 8601 timestamp
support of tempurl middleware. Please see

Change-Id: I76da28b48948475ec1bae5258e0b39a316553fb7
6 years ago
Kazufumi Noto 809e4cf98f Close file handle after upload job
The opened file for upload is not closed.
This fix prevents possible file handle leak.

Closes-Bug: #1559079
Change-Id: Ibc58667789e8f54c74ae2bbd32717a45f7b30550
6 years ago
Clay Gerrard dd34af42f8 Fix MockHttpResponse to be more like the Real
This change pulls out that relatively new [1] little string to pull at
in the MockHttpResponse that I think is sorta ugly.  And replaces it
with the correct behavior that's representative of the Real for which
it's standing in (which is sadly our wrapper to make a requests response
feel like a httplib.HTTPResponse).

It's not clear (to me) the history which allowed this difference in the
behavior of the Real and Fake to persist - it seems to have always been
this way [2].

I also reworded a relatively new test [1] to cover more code, and make
assertions on the desired behavior of the client instead of "just" the
http_log method.

FWIW, I don't think there was necessarily anything wrong with the scope
of the new test [1] - and it certainly makes sense to see new tests copy
nearby existing tests.  But I subjectively think this smaller test is
more demonstrative of the desired behavior.

1. Related-Change-Id: I6d7ccbf4ef9b46e890ecec58842c5cdd2804c7a9
2. Related-Change-Id: If07af46cb377f3f3d70f6c4284037241d360a8b7
Change-Id: Ib99a029c1bd1ea1efa8060fe8a11cb01deea41c6
6 years ago
Vitaly Gridnev 028c4824d0 Fix logging of the gzipped body
Change-Id: I6d7ccbf4ef9b46e890ecec58842c5cdd2804c7a9
Closes-bug: 1670620
6 years ago
Jenkins 9531369c23 Merge "Make functests py3-compatible" 6 years ago
Jenkins 5ffd496f1f Merge "Accept more types of input for headers/meta" 6 years ago
Christopher Bartz 3934bd606a prefix-based tempurls support
Implements client-side functionality for
prefix-based tempurls.

Please see:

Change-Id: I8d7701daee888ed1120271a96c0660b01543ca2d
6 years ago
Jenkins 39a0eda486 Merge "modify 'swift <sub_command> —help' display" 7 years ago
Shashirekha Gundur 41666d60c8 modify 'swift <sub_command> —help' display
In python swiftclient:  swift <sub_command> —help will now
display st_<sub_command>_options + st_<sub_command>_help texts

Change-Id: I34e4b2ac29ef395f8ca474ce7a82f59a1fd8c7f4
Closes-Bug: #1621415
7 years ago
Tim Burke a1e2bcde4a Accept more types of input for headers/meta
Previously, we only accepted iterables of strings like 'Header: Value'.
Now, we'll also accept lists of tuples like ('Header', 'Value') as well
as dictionaries like {'Header': 'Value'}.

This should be more intuitive for application developers, who are
already used to being able to pass dicts or lists of tuples to libraries
like requests.

Change-Id: I93ed2f1e8305f0168b7a4bd90c205b04730da836
7 years ago
howardlee 12d42efad2 Replace 'assertEqual(None, ...)' with 'assertIsNone(...)'
[H203] Use assertIs(Not)None to check for None (off by default) Unit
test assertions tend to give better messages for more specific
assertions. As a result, assertIsNone(...) is preferred over
assertEqual(None, ...) and assertIs(None, ...), and assertIsNotNone(...)
is preferred over assertNotEqual(None, ...) and assertIsNot(None,
...). Off by default.

More details, see:

Trivial fix.

Change-Id: Icd268b96dea5e5bb9bd344f597dfcd9cc82253f0
7 years ago
Jenkins 70c90b2243 Merge "Add additional headers for HEAD/GET/DELETE requests." 7 years ago