All classes subclass from object by default in Python 3.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I5a1ad57bcc092861ce969759b06a07c880ad3d35
This mostly affects tests. Nothing too complicated
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Iabc78f651e1d48db35638280722f8019798eccd6
This also upgrades the Requests dependency to 2.4+ (released in 2014)
to avoid having to do version comparisons altogether.
Refs https://bugs.launchpad.net/python-swiftclient/+bug/1904551
Signed-off-by: Aarni Koskela <akx@iki.fi>
Change-Id: I58399f6c526b0b78462f31739c43076314ba9e76
There's a lot of cleanup possible, but this is a start.
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ia1176b7fd5434d52070d482a37abfbb98800cdb3
This reverts commit 1f26c5736949e1c3b57c024a315e33fc419f126e for py2.
Apparently the existence of the __del__ method on Python 2 prevents us
from cleaning up all file descriptors.
Change-Id: Id6cff5dd7b9faf9c4240c0cb26b74d05ed37da5b
Closes-Bug: #1873435
Related-Bug: #1838775
Following the recent v3applicationcredentials patch, if you have your
environment variables set up to work with python-openstackclient using
swiftclient's v1password plugin, swiftclient won't work:
$ env | egrep '^(OS|ST)_'
ST_KEY=testing
ST_USER=test:tester
OS_AUTH_URL=http://saio/auth/v1.0
ST_AUTH=http://saio/auth/v1.0
OS_USERNAME=test:tester
OS_AUTH_TYPE=v1password
OS_PASSWORD=testing
$ openstack object store account show
+------------+----------------------------+
| Field | Value |
+------------+----------------------------+
| Account | AUTH_test |
| Bytes | 0 |
| Containers | 11 |
| Objects | 0 |
+------------+----------------------------+
$ swift stat
Only "v3applicationcredential" is supported for --os-auth-type
We don't really want to allow (and mostly ignore) arbitrary OS_AUTH_TYPE
values, though -- there are a whole bunch of plugins we don't remotely
support. But it seems OK to allow any of the password plugins; while we
won't actually use them (currently), we provide roughly equivalent
functionality.
Handful of other drive-bys:
* Use a None sentinel to determine whether keystoneauth1 is installed
instead of trying to catch a NameError.
* Clarify error state when keystoneauth1 is not installed.
* Fix a typo: "sses" -> "sess".
Change-Id: Id7ea9c3ea8278ae86a04d057a472a8f8a87b8eae
Related-Change: I9190e5e7e24b6a741970fa0d0ac792deccf73d25
It fixes get_capabilities() method to process
correctly endpoints like: 'https://<ip>:<port>/v1',
'https://<ip>:<port>/swift/v1'.
Co-Authored-By: Daniel Cech <dcech@mirantis.com>
Change-Id: Ib4037d0b49da1bce959947100629370805f510d5
Closes-bug: #1712358
* add --versions to list
* add --versions to delete
* add --version-id to stat
* add --version-id to delete
* add --version-id to download
Change-Id: I89802064921778fee7efe57c7d60c976cdde3a27
If an external http connection was not passed into the client, we
create one with a requests.Session() on our own. Once this is used,
it may still have an open socket when the connection is closed. We need
to handle the closing of the requests.Session() ourselves if we created
one. If you do not close it, a ResourceWarning may be reported about the
socket that is left open.
Change-Id: I200ad0cdc8b7999c3f5521b9a822122bd18714bf
Closes-Bug: #1838775
While investigating the failures when you move func tests to py3, I
noticed a whole bunch of
ResourceWarning: unclosed <socket.socket ...>
noise. This should fix it.
While we're at it, make get_capabilities less stupid.
Change-Id: I3913e9334090b04a78143e0b70f621aad30fc642
Related-Change: I86d24104033b490a35178fc504d88c1e4a566628
There were two basic problems:
- We'd try to import on every attempt at getting auth, even when we
already know keystoneclient isn't available.
- Sometimes devs would hit some crazy import race involving (some
combination of?) greenthreads and OS threads.
So let's just try the imports *once*, at import time, and have None
sentinels if it fails. Try both versions separately to decouple
failures; this should let us support a wider range of keystoneclient
versions.
Change-Id: I2367310aac74f1b7c5ea0cb1a822a491e4ba8e68
Since we define the getheader() method on the response from
HTTPConnection, we don't have to call parse_header_string, as the values
will already be converted properly.
Change-Id: Ia81e8674b828b3ff1f014454126b469e41adfc23
Fix unicode handling in Python 3 and Python 2. There are currently two
failure modes. In python 2, swiftclient fails to log in debug mode if
the account name has a non-ASCII character. This is because the account
name will appear in the storage URL, which we attempt to pass to the
logger as a byte string (whereas it should be a unicode string). This
patch changes the behavior to convert the path strings into unicode by
calling the parse_header_string() function.
The second failure mode is with Python 3, where http_lib returns headers
that are latin-1 encoded, but swiftclient expects UTF-8. The patch
automatically converts headers from latin-1 (iso-8859-1) to UTF-8, so
that we can properly handle non-ASCII headers in responses.
Change-Id: Ifa7f3d5af71bde8127129f1f8603772d80d063c1
An earlier change added support for versionless authurls, but the
huristic to detect them didn't work for some configurations I've
encountered.
Now we use a little bit tighter pattern matching and support auth_url
values with more than one path component.
Change-Id: I5a99c7b4e957ee7c8a5b5470477db49ab2ddba4b
Related-Change-Id: If7ecb67776cb77828f93ad8278cc5040015216b7
The valid set of values for auth_version does not include
values starting with the 'v'.
In this particular function, the auth_version variable is
only used for comparisons with v3. So, the code worked
correctly. However, let's clean this up in order to reduce
review confusion and defuse possible future landmine in case
of code changes.
Change-Id: I671016d7992a1922b786b4eb8876b3fbb2532e15
This patch attemps to add an option to force get_auth call while retrying
an operation even if it gets errors other than 401 Unauthorized.
Why we need this:
The main reason why we need this is current python-swiftclient requests could
never get succeeded under certion situation using third party proxies/load balancers
between the client and swift-proxy server. I think, it would be general situation
of the use case.
Specifically describing nginx case, the nginx can close the socket from the client
when the response code from swift is not 2xx series. In default, nginx can wait the
buffers from the client for a while (default 30s)[1] but after the time past, nginx
will close the socket immediately. Unfortunately, if python-swiftclient has still been
sending the data into the socket, python-swiftclient will get socket error (EPIPE,
BrokenPipe). From the swiftclient perspective, this is absolutely not an auth error,
so current python-swiftclient will continue to retry without re-auth.
However, if the root cause is sort of 401 (i.e. nginx got 401 unauthorized from the
swift-proxy because of token expiration), swiftclient will loop 401 -> EPIPE -> 401...
until it consume the max retry times.
In particlar, less time to live of the token and multipart object upload with large
segments could not get succeeded as below:
Connection Model:
python-swiftclient -> nginx -> swift-proxy -> swift-backend
Case: Try to create slo with large segments and the auth token expired with 1 hour
1. client create a connection to nginx with successful response from swift-proxy and its auth
2. client continue to put large segment objects
(e.g. 1~5GB for each and the total would 20~30GB, i.e. 20~30 segments)
3. after some of segments uploaded, 1 hour past but client is still trying to
send remaining segment objects.
4. nginx got 401 from swift-proxy for a request and wait that the connection is closed
from the client but timeout past because the python-swiftclient is still sending much data
into the socket before reading the 401 response.
5. client got socket error because nginx closed the connection during sending the buffer.
6. client retries a new connection to nginx without re-auth...
<loop 4-6>
7. finally python-swiftclient failed with socket error (Broken Pipe)
In operational perspective, setting longer timeout for lingering close would be an option but
it's not complete solution because any other proxy/LB may not support the options.
If we actually do THE RIGHT THING in python-swiftclient, we should send expects: 100-continue
header and handle the first response to re-auth correctly.
HOWEVER, the current python's httplib and requests module used by python-swiftclient doesn't
support expects: 100-continue header [2] and the thread proposed a fix [3] is not super active.
And we know the reason we depends on the library is to fix a security issue that existed
in older python-swiftclient [4] so that we should touch around it super carefully.
In the reality, as the hot fix, this patch try to mitigate the unfortunate situation
described above WITHOUT 100-continue fix, just users can force to re-auth when any errors
occurred during the retries that can be accepted in the upstream.
1: http://nginx.org/en/docs/http/ngx_http_core_module.html#lingering_close
2: https://github.com/requests/requests/issues/713
3: https://bugs.python.org/issue1346874
4: https://review.openstack.org/#/c/69187/
Change-Id: I3470b56e3f9cf9cdb8c2fc2a94b2c551927a3440
Submitting a path parameter with a HEAD request on an object can be
useful if one is trying to find out information about an SLO/DLO without
retrieving the manifest.
Change-Id: I39efd098e72bd31de271ac51d4d75381929c9638
Previously, python-swiftclient worked around a requests issue where
Content-Type could be set to application/x-www-form-urlencoded when
using python3. This issue has been resolved and a fix released in
requests 2.4 (fixed in subsequent releases as well). The patch makes
the workaround conditional on the requests version, so that with
sufficiently new requests libraries, the Content-Type is not set.
For reference, requests 2.4 was released August 29th, 2014. The
specific issue filed in the requests tracker is:
https://github.com/requests/requests/issues/2071.
Related-Change: I035f8b4b9c9ccdc79820b907770a48f86d0343b4
Closes-Bug: #1433767
Change-Id: Ieb2243d2ff5326920a27ce8c3c6f0f5c396701ed
Newer deployments are using versionless Keystone endpoints, and most
OpenStack clients already support this.
This patch enables this for Swift: if an auth_url without any path
component is found, it assumes a versionless endpoint will be used.
In this case the v3 suffix will be appended to the path if none
auth_version is set, and v2.0 is appended if auth_version requires v2.
Closes-Bug: 1554885
Related-Bug: 1691106
Change-Id: If7ecb67776cb77828f93ad8278cc5040015216b7
The only logging we should do is when we've encountered a problem *and
we've dealt with it ourselves*. When we're raising an exception, it
should be up to the caller to decide whether to log anything about it.
Anything else is just rude.
Change-Id: I1c96b76d90a78b7a10ffe63e4a7440c8f579147c
Closes-Bug: 1213179
Related-Bug: 1202229
Previously, we would accept gzip-encoded responses, but only because we
were letting requests decode *all* responses (even object data). This
restores the previous capability, but with tighter controls about which
requests will accept gzipped responses and where the decoding happens.
Change-Id: I4fd8b97207b9ab01b1bcf825cc16efd8ad46344a
Related-Bug: 1282861
Related-Bug: 1338464
Recently, requests got a bit more picky about what types of data it will
accept as header values [1]. The reasons for this are generally sound;
str()ing arbitrary objects just before pushing them out a socket may not
do what the developer wanted/expected.
However, there are a few standard types that developers may be sending
that we should convert for them as a convenience. Now, we'll convert all
int, float, and bool values to strings before sending them on to
requests.
Change-Id: I6c2f451009cb03cb78812f54e4ed8566076de821
Closes-Bug: 1614932
Implement copy object method in swiftclient Connection, Service and CLI.
Although COPY functionality can be accomplished with 'X-Copy-From'
header in PUT request, using copy is more convenient especially when
using copy for updating object metadata non-destructively.
Closes-Bug: 1474939
Change-Id: I1338ac411f418f4adb3d06753d044a484a7f32a4
Added functionality for arbitrary query strings to be passed into container
functions. Additionally a minor typo correction in the README. Added unit
tests for query string functionality.
Closes-Bug: #1542459
Change-Id: Ica2cb3ea439632588388e748d8d2e944e9ed4fa4
