Previously we were trying to test writing bytes in Python 3 using only
native (unicode) string objects. That doesn't test what we thought we
were testing.
Change-Id: I10a0a38143d7f7d850ab9a7005ad87f5d314c375
Extends existing unit test for timeout being passed to get_auth
to cover v2.0 auth when keystone client should get the timeout
kwarg.
Related-Bug: 1447847
Change-Id: Ie9cfc86fa2156b94b45d290ac12e3f71b20d6c4f
The underlying requests library supports a timeout input parameter. The
other python-*client libraries support passing it in, so while working
on shade, it seemed like a great idea to be able to pass in timeout to
swiftclient too. For reference, there's a change in shade:
I095c1240693abf024bda2315dd77f4400b24a45b that shows interaction with
the other libs, and a tentative patch that would consume this one.
Change-Id: I699ebb1e092aa010af678de7ba15712da6ed5315
Currently, get_object returns a generator. This allows access to the
object's data in multiples of 'resp_chunk_size'.
This patch adds a read function to also allow accessing the object data
using read(size).
This allows, for example, the consumer of an object (where no byte range
has been specified) to read up to certain boundaries while streaming to
a new Large Object with segments of a specified size.
Reading and chunking can be safely mixed.
Related-Bug: 1367925
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I1cfb67f45afc7015fd896f1a89bebae048871769
Previously, we compared the ETag from Swift against the MD5 of the
entire large object. However, the ETag for large objects is generally
the MD5 of the concatenation of the ETags for each segment, unless the
object is a DLO whose segments span more than one page of a container
listing. Rather than worry about ETags, just compare each chunk of the
segmented file. This allows the use of --skip-identical when uploading
SLOs and DLOs.
Additionally, there are several test-related improvements:
* The default arguments for OutputManager are now evaluated on
construction, rather than on definition, so that
TestOutputManager.test_instantiation will succeed when using nosetest
as a test runner. (See also: bug 1251507)
* An account_username option is now available in the functional tests
config file for auth systems that do not follow the account:username
format.
* CaptureOutput no longer writes to the captured stream, and
MockHttpTest now captures output. These were polluting test output
unnecessarily. (See also: bug 1201376)
Change-Id: Ic484e9a0c186c9283c4012c6a2fa77b96b8edf8a
Closes-Bug: #1201376
Closes-Bug: #1379252
Related-Bug: #1251507
With MD5Sum checking being added, a concern was brought up that It was
a change with no possibility of reverting to the old behaviour.
This change adds the flag '--ignore-checksum' to the upload subcommand
allowing the checks to be turned off.
Changed occurrences of the magic string for a null md5 to use a descriptive
constant instead.
Updated Error messages generated when validation fails. They should now be more descriptive
and not output a literal newline sequence.
Change-Id: Id1756cbb6700bb7e38f0ee0e75bc535e37f777ed
Changed existing code to calculate the MD5 of the object
during the upload stream. Checks this MD5 against the etag
returned in the response.
An exception is raised if they do not match.
Closes-Bug: 1379263
Change-Id: I6c8bc1366dfb591a26d934a30cd21c9e6b9a04ce
This patch fixes downloading files to stdout and modifies
_SwiftReader to operate as an iterator that performs file
checks at the end of iteration as well as a context manager.
File verification checks have been removed from __exit__
and added to __iter__.
Change-Id: I3250bdeeef8484a9122c4b5b854756a7c8f8731e
Closes-Bug: 1395922
Closes-Bug: 1387376
Removes an account stat from the object upload path.
This stat fails when user is not account admin even
though the user may have container ACL permission to
write objects.
Reduces the severity of the CLI output message when
upload fails to create the given container (this is
not an error since the container may exist - the user
just does not have permission to PUT or POST the
container).
Changes the 'swift upload' exit return code from 1 to
0 if container PUT fails but object PUT succeeds.
For segment uploads, makes the attempt to create the
segment container conditional on it not being the same
as the manifest container. This avoids an unnecessary
container PUT.
Fixes another bug that became apparent: with segmented
upload a container HEAD may be attempted to determine
the policy to be used for the segment container. When
this failed the result dict has headers=None which was
causing an exception in the shell result handler.
Add unit tests for object upload/download and container
list with --os-storage-url option.
Closes-Bug: #1371650
Change-Id: If1f8a02ee7459ea2158ffa6e958f67d299ec529e
If you specify a token and storage url when creating a Connection,
regardless of the auth api version the first request will be made
directly to swift. You can either provide a preauthurl and preauthtoken
or fall back to os_options' object_storage_url and auth_token keys
(exposed as --os-storage-url and --os-auth-token on the command line or
OS_STORAGE_URL and OS_AUTH_TOKEN in the environment).
If a _retry wrapped request on a Connection fails because of invalid
authentication (401) the Connection's cached token and url will be
invalidated. If the Connection's retries attribute is > 0 the
subsequent attempt will call get_auth to refresh the token, but the
pre-configured storage_url will always be re-used. This is consistent
with current auth v2 behavior and less surprising for auth v1.
The pre-existing, but previously undocumented behavior/interface of
get_auth would override the storage_url returned by the auth service if
the 'os_storage_url' option was provided in the os_options dict. To
ensure that this behavior is consistent across auth v1 and v2 from the
command line and when using the Connection class as a library - the
preauthurl is stashed in the os_options dict when provided.
Improved Connection.get_capabilities storage_url handling to better
support the consistent behavior of a preauthurl/object_storage_url on
the connection regardless of auth version.
Fixed up some test infrastructure to enable setting up and testing
multiple requests/responses.
Change-Id: I6950fb73f3e28fdddb62760cae9320e2f4336776
Added the ability to clear the buffers in the
CaptureOutput class so it can be easily used multiple
times in the same context manager.
Also added a option to suppress the SystemExit
associated with printing an error.
Change-Id: Ib59bbbe88256f215eed0a8ebc8282e02181d4377
This change adds some new classes to test.unit.utils which wrap up the
collection and mocking of the std err and std out. This will make the
testing code easier to write and maintain.
Change-Id: I50ad2a736b2bb550ab83f6a43fb5a0fb5393573e
Some client.Connection methods will raise a KeyError
if a response_dict argument is passed and an error
occurs during authentication or making the request.
The fix is straightforward: add a test for existence
of a response_dict before attempting to get it from
kwargs.
The bulk of this patch is adding unit tests for the
response_dict feature.
Closes-Bug: 1381304
Change-Id: Ic7e1b3dfae33909533931c52ac97355867a08a07
Fixes a regression that is preventing swiftclient from
authenticating using keystone v3 options.
swiftclient/shell.py processes the dict of command
line options to set up a child dict ('os_options')
of keystone auth specific options (line 960). The
processing includes stripping --os- prefixes from
the command line options before adding keys to
os_options.
A recent patch https://review.openstack.org/#/c/85453/
introduced a duplication of this option processing
in service.py (line 79) which replaces the os_options
created in shell.py, but omits keystone v3 related
options. Consequently the keystone v3 options are
not being passed to the client get_auth() method.
This patch adds the keystone v3 related options to
the option processing in service.py.
For pragmatic reasons (i.e. fixing the bug quickly)
the option processing code has not been removed from
parse_args in shell.py. It is likely that the code in
parse_args is now redundant, but all code paths between
parse_args and process_options should be inspected and
test coverage added before removing that code.
Unit tests have been added in test_shell.py to verify
that command line options are correctly passed to the
client get_auth method.
The MockHttpTest class is re-used in test_shell.py,
so it is moved from test_swiftclient.py
to tests/unit/utils.py
Closes-bug: #1372465
Change-Id: I4fed013cdb8936509609d06093337cc147ade0d6
Enables swiftclient to authenticate using
the keystone v3 API, allowing user id's, user
domains and tenant/project domains to be
specified.
Since swiftclient imports keystoneclient, the
main changes in swiftclient/client.py are to
selectively import the correct keystoneclient
library version and pass a number of new
options to it via the get_auth() function. In
addition the get_keystoneclient_2_0 method
has been renamed get_auth_keystone to better
reflect its purpose since it now deals with
both v2 and v3 use cases.
In swiftclient/shell.py the new options are
added to the parser. To make the default help
message shorter, help for all the --os-*
options (including the existing v2 options)
is only displayed when explicitly requested
usng a new --os-help option.
A new set of unit tests is added to
test_shell.py to verify the parser. A comment
in tests/sample.conf explains how to
configure the existing functional tests to
run using keystone v3 API.
Note that to use keystone v3
with swift you will need to set
auth_version = v3.0 in the auth_token
middleware config section of
proxy-server.conf.
Change-Id: Ifda0b3263eb919a8c6a1b204ba0a1215ed6f642f
Coverage for swiftclient.client is 71% with these tests.
Unit tests have been moved into another subdirectory
to separate them from functional tests.
Change-Id: Ib8c4d78f7169cee893f82906f6388a5b06c45602
