The current code builds a full object listing before performing either a multiple
download or delete operation (and also shuffles this complete list in the case of
a download). This patch removes the creation of the full object list and adds the
ability to turn off shuffle for files when downloading. Also added is a limit on
the number of list results that can be queued by a single call to service.list
without consuming results (reduces memory overhead for large listings).
Some tests added for service.py download and list.
Change-Id: Ie737cbb7f8b1fa8a79bbb88914730b05aa7f2906
This patch adds "--output-dir" and "--remove-prefix" options
to the "download" command and unit tests for it.
Example:
$ swift list example --prefix swift2.2
swift2.2/bin/swift-object-auditor
swift2.2/bin/swift-object-expirer
swift2.2/bin/swift-object-info
swift2.2/bin/swift-object-replicator
swift2.2/bin/swift-object-server
swift2.2/bin/swift-object-updater
When given "--output-dir <directory>", client downloads objects to <directory>.
$ swift download example --prefix swift2.2 \
--output-dir new/swift/dir
The folder structure:
.
└── new
└── swift
└── dir
└── swift2.2
└── bin
├── swift-object-auditor
├── swift-object-expirer
├── swift-object-info
├── swift-object-replicator
├── swift-object-server
└── swift-object-updater
When given "--remove-prefix", client downloads objects without <prefix>.
$ swift download example --prefix swift2.2 \
--remove-prefix \
--output-dir swift
The folder structure:
.
└── swift
└── bin
├── swift-object-auditor
├── swift-object-expirer
├── swift-object-info
├── swift-object-replicator
├── swift-object-server
└── swift-object-updater
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I7463fe2941cc94f9a50a4756a97c2ccdf946294d
Implements: blueprint swiftclient-download-pseudo-folder-to-specific-target
There was always an implicit assumption that they were UTF-8 before, and
by converting them to unicode we close another hole allowing raw bytes
to appear in user-facing messages.
Closes-Bug: #1431866
Change-Id: If2e41d9a592c3ad02818e9c6f0959fd4b73cd0e0
Previously we were trying to test writing bytes in Python 3 using only
native (unicode) string objects. That doesn't test what we thought we
were testing.
Change-Id: I10a0a38143d7f7d850ab9a7005ad87f5d314c375
The gate-swiftclient-dsvm-functional job takes updates from
openstack/requirements and tries to run functional tests against them.
However, the global requirements recently added environment markers
(like "futures>=3.0;python_version=='2.7' or python_version=='2.6'")
which require pbr>=1.2. pbr was getting capped at 1.0 by hacking<0.9, so
bump it to get newer pbr.
Change-Id: I10af36602d23db78c2863d84c0012835b310bbdd
The "unicode" type was renamed to "str" in Python 3.
Use six.text_type to make swiftclient compatible with Python 3.
For more information about changes needed for py3
compatibility, see:
https://wiki.openstack.org/wiki/Python3
Change-Id: Ic65607a69935652a1874340928f626fbcc35c014
Currently SwiftClient populate response dictionary before
removing "./" or "/" at begining of object name. This patch
fixes that by changing that order.
Closes-bug: #1412425
Change-Id: I80222754caba5d42a468f4677ac539e46682dd31
This patch adds a small test to ensure a connection
is released after all chunks have been consumed.
It's a follow up to commit 8756591b and added to ensure
there will be no regression in the future (this test
fails also with that patch not applied).
Change-Id: I6a6fcd26879eb2070f418c8770a395ff6c30aa51
Makes the existing functional tests run using three
auth modes: tempauth (v1), keystone v2 and keystone v3.
The latter uses an account in a non-default domain
(which exists in devstack setup).
This should help avoid regressions in handling different
auth options.
Change-Id: Ifee6a4fa418242892bf73eda5e2cad7b803b1bee
Since tox version 2.0.0 env vars are not passed to the test
env, which means that the SWIFT_TEST_CONFIG_FILE var is not
passed in to tox -e func env. That means that both times
tox -e func runs it is using keystone auth, and never using
tempauth.
Related-Bug: 1455102
Co-Authored-By: Christian Schwede <cschwede@redhat.com>
Change-Id: I23dcdbcde0bf8adc9429eb2d294a2c778005d136
Setting timeout to a default of False in get_auth()
results in a requests timeout of 0.0 in keystoneclient,
which causes a connection failure.
This bug will cause func tests to not run when using
keystoneauth.
Added unit tests to verify correct default timeout is set
in get_auth().
Drive-by: remove what seems like a stale TODO comment
Change-Id: I17c781ce160a682b1768d315422ade0cdd2df198
Adds doc for account_username option to sample test.conf.
This option was added in [1].
[1] change id Ic484e9a0c186c9283c4012c6a2fa77b96b8edf8a
Change-Id: Ic86b274e9d954822da521360981f796d61efaad9
This allows the use of --skip-identical when downloading SLOs and DLOs,
while still only requiring one request for normal objects.
Change-Id: I237ec5a8cc4d1119b6d3f166db81a977a50a4e4e
Related-Bug: #1379252
Previously, we had urllib3 (via requests) automatically decode all
responses with a Content-Encoding of deflate or gzip. This included
object downloads, which would in turn cause etag or content-length
mismatch errors. (See bug 1338464)
This was apparently added in response to a third-party proxy sitting
between the client and server which, having observed that the client
would accept gzip-encoded content while the server sent an unencoded
response, would perform the compression. (See bug 1282861)
Now, we'll no longer let requests send any default headers, nor do any
decoding.
Change-Id: I6cc30a5c12e37de06d7322533a3c36ad15397cc8
Closes-Bug: 1338464
Related-Bug: 1282861
In addition to removing an unnecessary dependency, this closes another
hole that was allowing raw bytes to appear in user-facing messages.
Change-Id: Ia0b76426a38e5a5c368c4c7e7ba2aef286758aca
Extends existing unit test for timeout being passed to get_auth
to cover v2.0 auth when keystone client should get the timeout
kwarg.
Related-Bug: 1447847
Change-Id: Ie9cfc86fa2156b94b45d290ac12e3f71b20d6c4f
A previous change added socket timeouts, but they weren't plumbed
through to the get_auth code path. Make sure we're passing them along
everywhere.
Original-Author: Monty Taylor <mordred@inaugust.com>
Change-Id: I398241898248e66d1f8c436c8ed2ec7a0e9387f6
Closes-bug: 1447847
The underlying requests library supports a timeout input parameter. The
other python-*client libraries support passing it in, so while working
on shade, it seemed like a great idea to be able to pass in timeout to
swiftclient too. For reference, there's a change in shade:
I095c1240693abf024bda2315dd77f4400b24a45b that shows interaction with
the other libs, and a tentative patch that would consume this one.
Change-Id: I699ebb1e092aa010af678de7ba15712da6ed5315
