On Python 3, urllib.parse.quote() accepts bytes and str (unicode) types and
always return str (unicode).
Add also more tests with non-ASCII characters.
Change-Id: I8e0f19da7240e874392327d0da074ed4abb7d213
* Replace unicode with six.text_type
* Replace basestring with six.string_types
* The long type doesn't exist in Python 3 anymore: replace 1L with long(1) and
only test this type with Python 2
* Fix quote(): quote the URL if the string is a byte string. Use "bytes" type
instead of "str" to be Python 3 compatible.
Change-Id: I1df5aa85e4e7d07191fb5c654d52fc4bd8b9f440
dict.iteritems() has been removed in Python 3. In Python 2, dict.items()
creates a temporary list, but Samuel Merritt wrote:
"The size of the data is small enough that we don't need to worry about the
memory consumption; it's all just HTTP headers."
Change-Id: Iac427b8bbd032e4a1efb2bdc84a3968217eb6ddf
By default, the HTTPConnection class expose the raw.read urllib3
function which doesn't do any decoding on the http response.
We can fix this by passing decode_content=True to raw.read().
Change-Id: I7d42d31ae516705d1bde2769e642931308116c7a
urlparse netloc contains the TCP port if presents.
There is no need to provide conn.port when building
CLientException.
Change-Id: Id3e4fa8c4f4ab2ad693b0e8702ab69ed7c291830
The requests 'files' parameter adds this 'Content-Type: multipart/form-data'
HTTP header and the whole multipart body data get stored with the object.
This also create a memory hog issue because files are loaded in memory before
being actually sent. This patch removes this behavior and restores what was
done before, ie: direct uploading.
This patches also fixes an issue in requests, when used with glance
CooperativeReader it mis-calculates content-length leading to chunked encoding
for raw upload.
Change-Id: Ie5b0a1078bedd33f09c6157f48b5f88116c589fa
Closes-Bug: #1280072
Closes-Bug: #1280275
Currently, httplib implementation does not support SSL certificate
verification. This patch fixes this. Note that ssl compression parameter
and 100-continue thing is still missing from requests, though those are
lower priority.
Requests now takes care of:
* proxy configuration (get_environ_proxies),
* chunked encoding (with data generator),
* bulk uploading (with files dictionary),
* SSL certificate verification (with 'insecure' and 'cacert' parameter).
This patch have been tested with requests 1.1.0 (CentOS 6) and requests
2.2.1 (current version).
Change-Id: Ib5de962f4102d57c71ad85fd81a615362ef175dc
Closes-Bug: #1199783
DocImpact
SecurityImpact
This patch adds a capabilities option on swiftclient.
This option uses the new /info endpoint to request the
remote capabilities and nicely display it.
Change-Id: Ie34b454511d5527e402e66e1fdb72120f427f2fd
Added a retry_on_ratelimit parameter to the Connection
class so that ratelimited requests can be retried.
DocImpact
Change-Id: I2817a7ea0ed2d69a7659e80111fbd2c91a75d530
As far as proxies usage is concerned, keystone-client API and swift-client API behave differently because the former uses python Request library while the latter uses raw httplib. As a result, Keystone authentication honors environment variables http_proxy, https_proxy and no_proxy while Swift doesn't.
This patch, which code is mainly borrowed from Python Requests, makes Swift
data connections and Swift authentication connections behaving homogeneously.
Change-Id: Ic8a0089c35c458d7ed96e572e22429014298fe4c
Before commit 7d88d14d (http://git.io/hhJWdQ) swifclient used to accept
header value with the type None :
{'Headers': None}
It would just be happy with it with those None headers and not process
them, reinstate the old behavior.
Closes-Bug: 1256441
Change-Id: Ic7f80c38ef2ce9ef1687ed0d6d65521f2e754905
We noticed a difference in download bandwidth while using
authentication details and pre-auth information for download.
Download using authentication details included authentication
time into the download bandwidth at each call of the function.
This time should have been excluded from download bandwidth.
This patch set is adding a timer variable to the client library.
That allows to accurately measure and store the time of each action.
Then while printing the download bandwith in the swift command
authentication time is excluded from bandwidth.
Change-Id: I63df9023e169f637f120ae9e25dac9c90a4e75a0
Fixes: bug #1197443
* log to debug for successes and info for errors
* remove dead code, neither body or raw_body are sent in kwargs
* include resp.reason and response headers in logging
* remove trailing \n in logging. users who want them can set logging.basicConfig(format=FORMAT)
* add --info option to swift cli
Change-Id: If07af46cb377f3f3d70f6c4284037241d360a8b7
Appends the container's name to the URL displayed, to be used to
replay the queries with curl.
Replaying the curl commands displayed by "swift --debug list
<container>" now works properly.
Example:
$ swift --debug list test-container
...
DEBUG:swiftclient:REQ: curl -i http://10.0.48.134:8080/v1/AUTH_6c554b8dd3d74e44878eddb92caf8687/test-container?format=json -X GET -H ...
...
Change-Id: I7ec33d185fedc44a529c016d38f841fde39d20d0
Closes-Bug: #1238612
I was gunna write a test, but mocking ImportErrors is a PITA, so I decided to
just get my tox py26 working instead. Tests seem to fail/pass as expected
without/with the inline NullHanlder.
Closes-Bug: 1212861
Change-Id: I20207374a77ee9969ab201c3a57e4caf40c02a61
This patch extracts the multi-threading code from bin/swift into
swiftclient/multithreading and adds tests. In particular, this new way
of doing it (with context managers) will prevent non-daemonic threads
from wedging the process when unexpected exceptions happen.
I enabled reporting of which lines, specifically, are not covered by
unit tests (added -m option to "coverage report" in .unittests).
This patch includes a drive-by fix for uploading a segmented file with
--use-slo when that object already exists. A key of "name" was used
instead of "path", raising KeyError.
There's also another drive-by fix for uploading segmented objects with
--use-slo. Commit 874e0e4427b80e1b15b74a1557b73ba9d61443ca regressed
this by removing the capturing of thread-worker results in
QueueFunctionThread.run(). This patch restores that functionality and
the feature (uploading SLO objects).
Change-Id: I0b4f677e4a734e83d1a25088d9a74f7d46384e53
Added log statements in swiftclient/client.py:_retry where exceptions
are being raised.
Fixes bug# 1202229
Change-Id: I727537177849d08bb9603aa884152bdebc62fb85
There were two undocumented parameters in there, one introduced
recently in 6411fd3, and the other some time before.
Change-Id: I3a7d59e1c272b382b035b88ba5495bc1c57a3154
See bug #1188896.
Comparing with Curl and Django, they both refuse carriage returns in
header values, so the request() method on the HTTP(S)Connection instance
returned by swiftclient.client.http_connection() will raise an
InvalidHeadersException if any of the headers to be sent contain a
newline.
Drive-by fix for a couple of header values which were integers instead
of strings (Content-Length getting set to zero).
Fixes bug #1188896
Change-Id: Ic6afdb92882284f843aacb06d20f682ddcb47151
The max-backoff concept prevents the backoff used in _retry to double infinitely (which in turn causes wait times of an hour or more between retries).
As per review, changed code to use min instead.
Fixes: bug #1183542
Change-Id: Ic084f54069b7fa7a33604741487045c5e697ff06
When --os-storage-url is specified on the command-line to bin/swift, it
will override the used storage URL regardless of authentication for both
authentication version 1 and version 2. This can be used to bypass a
load-balancer to hit a specific proxy server for testing/debugging
purposes.
Within the client library, this feature is accessed by passing the
desired storage URL into swiftclient.client.Conection.__init__() via the
os_options keyword argument. For example:
conn = Connection(auth_url, user, key, os_options={
'object_storage_url': 'http://overridden.storage.url/AUTH_foo'})
This patch also adds a dependency on mock>=0.8.0, which is the same as
openstack/swift.
Change-Id: Id2a36ed6abffd65e7762b6beea5bbfc6c036e848
