* Always use testtools.TestCase, since we're relying on testtools
* Always use mock (as opposed to unittest.mock) since we're relying on
mock
* Add note about when a missing logging handler was added
* Stop %-formatting the giant usage string that doesn't actually need
any formatting
* Prefer assertIs, assertIn, assertIsInstance over assertTrue
* Use else-self.fail instead of sentinel values
* Check resp.get('error') is None before checking resp['success'] is
True, so test failures actually tell you something useful
* Tighten some isinstance assertions
* Import MockHttpTest from correct location
* Only populate clean_os_environ once
* Use setUp for setup, not __init__
* Replace assertIn(key, dict) and assertEqual(foo, dict[key]) with
assertEqual(foo, dict.get(key)) when key is a literal and foo is not
None
* Use mock.patch.object instead of manually patching for tests
* Use six.binary_type instead of type(''.encode('utf-8'))
* Stop shadowing builtin bytes
* Reclaim some margin
* Stop checking the return-type of encode_utf8; we already know it's
bytes
Change-Id: I2138ea553378ce88810b7353147c8645a8f8c90e
Some tests rely on the fake connection checking expected request
parameters, but that assumes that the fake ocnnection is even called,
which is not being checked. Add more explicit assertions that requests
are in fact made.
Change-Id: Id1c48235d7d97fd1b0feec6c19ed59a87bebdf89
Headers parameter is required when passing client key for encryption.
It is missing for get_container and head_object.
Change-Id: I35c3b266b3c733f6b1629de4c683ea7d40128032
Attempt to fix the linked bug by initialising the mock instance for
Connection.delete_object before calling the SwiftService upload
method, so that the delete_object mock already exists before the
delete_segments jobs that run in multiple threads call it. Otherwise
there is a risk that the delete_segment job threads could race while
creating either the delete_object mock or the
delete_object.return_value mock, resulting in each thread getting a
different instance. That would explain the intermittent test failures
reported in the bug.
Change-Id: Ia82697c093529076b0bbcc6bccd577afdf0839e1
Partial-Bug: #1480223
The `tempurl` subcommand's second positional argument is called
`seconds` and has heretofore interpreted as the number of seconds for
which the tempURL should be valid, counting from the moment of running
the command. This is indeed a common, if not the most common,
use-case. But some users, occasionally, might want to generate a tempURL
that expires at some particular ("absolute") time, rather than a
particular amount of time relative to the moment of happening to run the
command. (One might make an analogy to the way in which Swift's expiring
object support supports an `X-Delete-At` header in addition to
`X-Delete-After`—and it's the former that must be regarded as
ontologically prior.) Thus, this commit adds an `--absolute` optional
argument to the `tempurl` subcommand; if present, the `seconds` argument
will be interpreted as a Unix timestamp of when the tempURL should be
expire, rather than a duration for which the tempURL should be valid
starting from "now".
Change-Id: If9ded96f2799800958d5063127f3de812f50ef06
Some improvements to the tests for staggered download
that were added in [1].
[1] Ie737cbb7f8b1fa8a79bbb88914730b05aa7f2906
Change-Id: Ib999bc7bd198c1d9c217c57501f751e854d4c6ad
This patch adds exception logging to the swift service API. Each
operation that results in failure of any operation will now log
the exception as well as report a timestamp and full stack trace
in the results returned by the service API calls.
Change-Id: I7336b28354e7740ea7d048bdf355e3c1a1b4436c
When a caller passes an os_options dict to the Connection class
constructor, the constructor may modify the os_options dict,
which can surprise the caller if they re-use the os_options
dict. Specifically the os_options tenant_name and object_storage_url
may be modified, and the changed values would then leak through to a
subsequent Connection constructed using the same os_options dict.
This fix simply constructs a new dict from the supplied os_options.
The patch also adds a test that covers this and also verifies that
a preauth_url passed as a keyword arg to Connection() will take
precedence over any object_storage_url in an os_options parameter.
Closes-Bug: 1488070
Change-Id: Ic6b5cf3ac68c505de155619f2610be9529e15432
Add client changes to allow accessing alternative
reseller_prefixes via a service token.
ie client changes for this server side spec:
https://review.openstack.org/#/c/105228
We assume that the service storage url has been passed in
as a preauthurl.
We rely on get_auth preserving this url.
Change-Id: I1cfda178f0b6c8add46cfebd6bf38440caae2036
fix against H234: assertEquals() logs a DeprecationWarning
in Python3.x. use assertEqual() instead.
Closes-bug: #1480776
Change-Id: Iffda6bb5f2616d4af4567eeea37bb26531e34371
Signed-off-by: Hiroshi Miura <miurahr@nttdata.co.jp>
When a Connection is first __init__ialized (without providing a
preauthurl or preauthtoken), the url and token attributes are None; they
get set (to be reused on future requests) after one of the wrapper
methods internally using _retry (head_account, get_container,
put_object, and similar friends) is called. However, this had not been
the case for get_auth, much to the momentary confusion and
disappointment of programmers using swiftclient who expected to be able
to get the token or storage URL off the Connection object after calling
get_auth (perhaps in order to make an unusual kind of request that
swiftclient doesn't already have a function for). This commit makes
get_auth set the url and token attributes as one might expect.
Change-Id: I0d9593eb5b072c8e3fa84f7d5a4c948c0bc6037a
The current code builds a full object listing before performing either a multiple
download or delete operation (and also shuffles this complete list in the case of
a download). This patch removes the creation of the full object list and adds the
ability to turn off shuffle for files when downloading. Also added is a limit on
the number of list results that can be queued by a single call to service.list
without consuming results (reduces memory overhead for large listings).
Some tests added for service.py download and list.
Change-Id: Ie737cbb7f8b1fa8a79bbb88914730b05aa7f2906
This patch adds "--output-dir" and "--remove-prefix" options
to the "download" command and unit tests for it.
Example:
$ swift list example --prefix swift2.2
swift2.2/bin/swift-object-auditor
swift2.2/bin/swift-object-expirer
swift2.2/bin/swift-object-info
swift2.2/bin/swift-object-replicator
swift2.2/bin/swift-object-server
swift2.2/bin/swift-object-updater
When given "--output-dir <directory>", client downloads objects to <directory>.
$ swift download example --prefix swift2.2 \
--output-dir new/swift/dir
The folder structure:
.
└── new
└── swift
└── dir
└── swift2.2
└── bin
├── swift-object-auditor
├── swift-object-expirer
├── swift-object-info
├── swift-object-replicator
├── swift-object-server
└── swift-object-updater
When given "--remove-prefix", client downloads objects without <prefix>.
$ swift download example --prefix swift2.2 \
--remove-prefix \
--output-dir swift
The folder structure:
.
└── swift
└── bin
├── swift-object-auditor
├── swift-object-expirer
├── swift-object-info
├── swift-object-replicator
├── swift-object-server
└── swift-object-updater
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I7463fe2941cc94f9a50a4756a97c2ccdf946294d
Implements: blueprint swiftclient-download-pseudo-folder-to-specific-target
There was always an implicit assumption that they were UTF-8 before, and
by converting them to unicode we close another hole allowing raw bytes
to appear in user-facing messages.
Closes-Bug: #1431866
Change-Id: If2e41d9a592c3ad02818e9c6f0959fd4b73cd0e0
Previously we were trying to test writing bytes in Python 3 using only
native (unicode) string objects. That doesn't test what we thought we
were testing.
Change-Id: I10a0a38143d7f7d850ab9a7005ad87f5d314c375
Currently SwiftClient populate response dictionary before
removing "./" or "/" at begining of object name. This patch
fixes that by changing that order.
Closes-bug: #1412425
Change-Id: I80222754caba5d42a468f4677ac539e46682dd31
This patch adds a small test to ensure a connection
is released after all chunks have been consumed.
It's a follow up to commit 8756591b and added to ensure
there will be no regression in the future (this test
fails also with that patch not applied).
Change-Id: I6a6fcd26879eb2070f418c8770a395ff6c30aa51
Makes the existing functional tests run using three
auth modes: tempauth (v1), keystone v2 and keystone v3.
The latter uses an account in a non-default domain
(which exists in devstack setup).
This should help avoid regressions in handling different
auth options.
Change-Id: Ifee6a4fa418242892bf73eda5e2cad7b803b1bee
Setting timeout to a default of False in get_auth()
results in a requests timeout of 0.0 in keystoneclient,
which causes a connection failure.
This bug will cause func tests to not run when using
keystoneauth.
Added unit tests to verify correct default timeout is set
in get_auth().
Drive-by: remove what seems like a stale TODO comment
Change-Id: I17c781ce160a682b1768d315422ade0cdd2df198
Adds doc for account_username option to sample test.conf.
This option was added in [1].
[1] change id Ic484e9a0c186c9283c4012c6a2fa77b96b8edf8a
Change-Id: Ic86b274e9d954822da521360981f796d61efaad9
This allows the use of --skip-identical when downloading SLOs and DLOs,
while still only requiring one request for normal objects.
Change-Id: I237ec5a8cc4d1119b6d3f166db81a977a50a4e4e
Related-Bug: #1379252
In addition to removing an unnecessary dependency, this closes another
hole that was allowing raw bytes to appear in user-facing messages.
Change-Id: Ia0b76426a38e5a5c368c4c7e7ba2aef286758aca
