Currently, if you attempt to stream an upload from an iterator, as in
def data():
yield 'foo'
yield ''
yield 'bar'
conn.put_object('c', 'o', data())
... requests will faithfully emit a zero-length chunk, ending the
transfer. Swift will then close the connection, possibly (if Connection:
keep-alive was set) after attempting to parse the next chunk as a new
request.
Now, Swift will receive all of the bytes from the iterable, and any
zero-byte chunks will be ignored.
This will be fixed in requests [1], but not until an eventual 3.0.0
release.
[1] https://github.com/kennethreitz/requests/pull/2631
Change-Id: I19579ed7a0181ac3f488433e7c1839f7f7a040b8
Add a unit test to test the 'string' like behaviour of
get_object when it is called without resp_chunk_size set.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I496032a76036141d027c30b076c810b34bc6bef0
According to help `swift upload -h` you can add a customized request header 'Content-Type'.
But actually it is ignored (cleared and default is used) if subcommand is upload.
Subcommand post works as expected in help.
Bug fix: Use 'Content-Type' from the customized request headers also if
uploading.
Change-Id: If0d1354b6214b909527341078fe1769aa6587457
* Always use testtools.TestCase, since we're relying on testtools
* Always use mock (as opposed to unittest.mock) since we're relying on
mock
* Add note about when a missing logging handler was added
* Stop %-formatting the giant usage string that doesn't actually need
any formatting
* Prefer assertIs, assertIn, assertIsInstance over assertTrue
* Use else-self.fail instead of sentinel values
* Check resp.get('error') is None before checking resp['success'] is
True, so test failures actually tell you something useful
* Tighten some isinstance assertions
* Import MockHttpTest from correct location
* Only populate clean_os_environ once
* Use setUp for setup, not __init__
* Replace assertIn(key, dict) and assertEqual(foo, dict[key]) with
assertEqual(foo, dict.get(key)) when key is a literal and foo is not
None
* Use mock.patch.object instead of manually patching for tests
* Use six.binary_type instead of type(''.encode('utf-8'))
* Stop shadowing builtin bytes
* Reclaim some margin
* Stop checking the return-type of encode_utf8; we already know it's
bytes
Change-Id: I2138ea553378ce88810b7353147c8645a8f8c90e
Cleanups for change I35c3b266b3c733f6b1629de4c683ea7d40128032
Add missing param definitions to client get_container
and head_object docstrings.
For consistency, add headers parameter to the Connection class
head_object and head_container wrapper methods.
Add tests to verify that the headers parameter of Connection
get_container, head_container and head_object methods is passed to the
module functions.
Change-Id: Ib40d5b626b2793840727c58cffbf725bea55651f
Some tests rely on the fake connection checking expected request
parameters, but that assumes that the fake ocnnection is even called,
which is not being checked. Add more explicit assertions that requests
are in fact made.
Change-Id: Id1c48235d7d97fd1b0feec6c19ed59a87bebdf89
Headers parameter is required when passing client key for encryption.
It is missing for get_container and head_object.
Change-Id: I35c3b266b3c733f6b1629de4c683ea7d40128032
All response headers are now exposed as unicode objects. Any
url-encoding is interpretted as UTF-8; if that causes decoding to fail,
the url-encoded form is returned.
As a result, deleting DLOs with unicode characters will no longer raise
UnicodeEncodeErrors under Python 2.
Related-Bug: #1431866
Change-Id: Idb111c5bf3ac1f5ccfa724b3f4ede8f37d5bfac4
When a caller passes an os_options dict to the Connection class
constructor, the constructor may modify the os_options dict,
which can surprise the caller if they re-use the os_options
dict. Specifically the os_options tenant_name and object_storage_url
may be modified, and the changed values would then leak through to a
subsequent Connection constructed using the same os_options dict.
This fix simply constructs a new dict from the supplied os_options.
The patch also adds a test that covers this and also verifies that
a preauth_url passed as a keyword arg to Connection() will take
precedence over any object_storage_url in an os_options parameter.
Closes-Bug: 1488070
Change-Id: Ic6b5cf3ac68c505de155619f2610be9529e15432
Add client changes to allow accessing alternative
reseller_prefixes via a service token.
ie client changes for this server side spec:
https://review.openstack.org/#/c/105228
We assume that the service storage url has been passed in
as a preauthurl.
We rely on get_auth preserving this url.
Change-Id: I1cfda178f0b6c8add46cfebd6bf38440caae2036
fix against H234: assertEquals() logs a DeprecationWarning
in Python3.x. use assertEqual() instead.
Closes-bug: #1480776
Change-Id: Iffda6bb5f2616d4af4567eeea37bb26531e34371
Signed-off-by: Hiroshi Miura <miurahr@nttdata.co.jp>
When a Connection is first __init__ialized (without providing a
preauthurl or preauthtoken), the url and token attributes are None; they
get set (to be reused on future requests) after one of the wrapper
methods internally using _retry (head_account, get_container,
put_object, and similar friends) is called. However, this had not been
the case for get_auth, much to the momentary confusion and
disappointment of programmers using swiftclient who expected to be able
to get the token or storage URL off the Connection object after calling
get_auth (perhaps in order to make an unusual kind of request that
swiftclient doesn't already have a function for). This commit makes
get_auth set the url and token attributes as one might expect.
Change-Id: I0d9593eb5b072c8e3fa84f7d5a4c948c0bc6037a
This patch adds a small test to ensure a connection
is released after all chunks have been consumed.
It's a follow up to commit 8756591b and added to ensure
there will be no regression in the future (this test
fails also with that patch not applied).
Change-Id: I6a6fcd26879eb2070f418c8770a395ff6c30aa51
Setting timeout to a default of False in get_auth()
results in a requests timeout of 0.0 in keystoneclient,
which causes a connection failure.
This bug will cause func tests to not run when using
keystoneauth.
Added unit tests to verify correct default timeout is set
in get_auth().
Drive-by: remove what seems like a stale TODO comment
Change-Id: I17c781ce160a682b1768d315422ade0cdd2df198
In addition to removing an unnecessary dependency, this closes another
hole that was allowing raw bytes to appear in user-facing messages.
Change-Id: Ia0b76426a38e5a5c368c4c7e7ba2aef286758aca
Extends existing unit test for timeout being passed to get_auth
to cover v2.0 auth when keystone client should get the timeout
kwarg.
Related-Bug: 1447847
Change-Id: Ie9cfc86fa2156b94b45d290ac12e3f71b20d6c4f
A previous change added socket timeouts, but they weren't plumbed
through to the get_auth code path. Make sure we're passing them along
everywhere.
Original-Author: Monty Taylor <mordred@inaugust.com>
Change-Id: I398241898248e66d1f8c436c8ed2ec7a0e9387f6
Closes-bug: 1447847
The underlying requests library supports a timeout input parameter. The
other python-*client libraries support passing it in, so while working
on shade, it seemed like a great idea to be able to pass in timeout to
swiftclient too. For reference, there's a change in shade:
I095c1240693abf024bda2315dd77f4400b24a45b that shows interaction with
the other libs, and a tentative patch that would consume this one.
Change-Id: I699ebb1e092aa010af678de7ba15712da6ed5315
Currently, get_object returns a generator. This allows access to the
object's data in multiples of 'resp_chunk_size'.
This patch adds a read function to also allow accessing the object data
using read(size).
This allows, for example, the consumer of an object (where no byte range
has been specified) to read up to certain boundaries while streaming to
a new Large Object with segments of a specified size.
Reading and chunking can be safely mixed.
Related-Bug: 1367925
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I1cfb67f45afc7015fd896f1a89bebae048871769
With MD5Sum checking being added, a concern was brought up that It was
a change with no possibility of reverting to the old behaviour.
This change adds the flag '--ignore-checksum' to the upload subcommand
allowing the checks to be turned off.
Changed occurrences of the magic string for a null md5 to use a descriptive
constant instead.
Updated Error messages generated when validation fails. They should now be more descriptive
and not output a literal newline sequence.
Change-Id: Id1756cbb6700bb7e38f0ee0e75bc535e37f777ed
Changed existing code to calculate the MD5 of the object
during the upload stream. Checks this MD5 against the etag
returned in the response.
An exception is raised if they do not match.
Closes-Bug: 1379263
Change-Id: I6c8bc1366dfb591a26d934a30cd21c9e6b9a04ce
If you specify a token and storage url when creating a Connection,
regardless of the auth api version the first request will be made
directly to swift. You can either provide a preauthurl and preauthtoken
or fall back to os_options' object_storage_url and auth_token keys
(exposed as --os-storage-url and --os-auth-token on the command line or
OS_STORAGE_URL and OS_AUTH_TOKEN in the environment).
If a _retry wrapped request on a Connection fails because of invalid
authentication (401) the Connection's cached token and url will be
invalidated. If the Connection's retries attribute is > 0 the
subsequent attempt will call get_auth to refresh the token, but the
pre-configured storage_url will always be re-used. This is consistent
with current auth v2 behavior and less surprising for auth v1.
The pre-existing, but previously undocumented behavior/interface of
get_auth would override the storage_url returned by the auth service if
the 'os_storage_url' option was provided in the os_options dict. To
ensure that this behavior is consistent across auth v1 and v2 from the
command line and when using the Connection class as a library - the
preauthurl is stashed in the os_options dict when provided.
Improved Connection.get_capabilities storage_url handling to better
support the consistent behavior of a preauthurl/object_storage_url on
the connection regardless of auth version.
Fixed up some test infrastructure to enable setting up and testing
multiple requests/responses.
Change-Id: I6950fb73f3e28fdddb62760cae9320e2f4336776
Some client.Connection methods will raise a KeyError
if a response_dict argument is passed and an error
occurs during authentication or making the request.
The fix is straightforward: add a test for existence
of a response_dict before attempting to get it from
kwargs.
The bulk of this patch is adding unit tests for the
response_dict feature.
Closes-Bug: 1381304
Change-Id: Ic7e1b3dfae33909533931c52ac97355867a08a07
Fixes a regression that is preventing swiftclient from
authenticating using keystone v3 options.
swiftclient/shell.py processes the dict of command
line options to set up a child dict ('os_options')
of keystone auth specific options (line 960). The
processing includes stripping --os- prefixes from
the command line options before adding keys to
os_options.
A recent patch https://review.openstack.org/#/c/85453/
introduced a duplication of this option processing
in service.py (line 79) which replaces the os_options
created in shell.py, but omits keystone v3 related
options. Consequently the keystone v3 options are
not being passed to the client get_auth() method.
This patch adds the keystone v3 related options to
the option processing in service.py.
For pragmatic reasons (i.e. fixing the bug quickly)
the option processing code has not been removed from
parse_args in shell.py. It is likely that the code in
parse_args is now redundant, but all code paths between
parse_args and process_options should be inspected and
test coverage added before removing that code.
Unit tests have been added in test_shell.py to verify
that command line options are correctly passed to the
client get_auth method.
The MockHttpTest class is re-used in test_shell.py,
so it is moved from test_swiftclient.py
to tests/unit/utils.py
Closes-bug: #1372465
Change-Id: I4fed013cdb8936509609d06093337cc147ade0d6
Enables swiftclient to authenticate using
the keystone v3 API, allowing user id's, user
domains and tenant/project domains to be
specified.
Since swiftclient imports keystoneclient, the
main changes in swiftclient/client.py are to
selectively import the correct keystoneclient
library version and pass a number of new
options to it via the get_auth() function. In
addition the get_keystoneclient_2_0 method
has been renamed get_auth_keystone to better
reflect its purpose since it now deals with
both v2 and v3 use cases.
In swiftclient/shell.py the new options are
added to the parser. To make the default help
message shorter, help for all the --os-*
options (including the existing v2 options)
is only displayed when explicitly requested
usng a new --os-help option.
A new set of unit tests is added to
test_shell.py to verify the parser. A comment
in tests/sample.conf explains how to
configure the existing functional tests to
run using keystone v3 API.
Note that to use keystone v3
with swift you will need to set
auth_version = v3.0 in the auth_token
middleware config section of
proxy-server.conf.
Change-Id: Ifda0b3263eb919a8c6a1b204ba0a1215ed6f642f
get_auth() in client.py raises an exception if
tenant_name is not included in the os_options
dict. This is overly constrained since tenant_id
is equally sufficient. This patch modifies
get_auth to require either tenant_name
or tenant_id.
Change-Id: Ibbcda1704637eb887efa5895579d260a1e072327
A couple of assertTrue in the test_swiftclient.py
unit tests should be assertEqual. Also, the expected
values now need to be bytes literals.
Change-Id: I7cc1bd60d9ba82d1a28fbae2e1243d3c799451bd
This patch fixes three issues that were found running
functional tests with Python 3:
1. python-requests sets 'application/x-www-form-urlencoded' as
content-type if the input is not a string object and no
content-type is given.
2. Encoding of the headers is now only done if required. This
prevents comparisons between unencoded headers and encoded
prefixes and avoids unnecessary forloop-iterations.
One unittest was extended to ensure it works for unencoded
and encoded headers with or without the prefix.
3. Functional tests recently switched to using byte data for
testing, thus the comparison needs to be a byte object as well.
Change-Id: I035f8b4b9c9ccdc79820b907770a48f86d0343b4
There is no testtools.main() at all (this was a
unittest.main() before). Let's remove this unused
and non-working code.
The following code can be used if someone needs
to start tests manually:
python -m unittest <test_file.py>
Change-Id: Id5162ac73825584df6c23dbe68786ff3355fc6ae
Don't encode standard headers keys but just the x-*-meta- ones. That
fixes py3 to not have them casted as bytes and not properly passed to
the requests kwargs.
Other trivial py3 fixes along the way.
Change-Id: I91b95f32fb2aec9b20892a5bb95fd1bc65d002f3
Coverage for swiftclient.client is 71% with these tests.
Unit tests have been moved into another subdirectory
to separate them from functional tests.
Change-Id: Ib8c4d78f7169cee893f82906f6388a5b06c45602
