This allows the use of --skip-identical when downloading SLOs and DLOs,
while still only requiring one request for normal objects.
Change-Id: I237ec5a8cc4d1119b6d3f166db81a977a50a4e4e
Related-Bug: #1379252
Previously, we had urllib3 (via requests) automatically decode all
responses with a Content-Encoding of deflate or gzip. This included
object downloads, which would in turn cause etag or content-length
mismatch errors. (See bug 1338464)
This was apparently added in response to a third-party proxy sitting
between the client and server which, having observed that the client
would accept gzip-encoded content while the server sent an unencoded
response, would perform the compression. (See bug 1282861)
Now, we'll no longer let requests send any default headers, nor do any
decoding.
Change-Id: I6cc30a5c12e37de06d7322533a3c36ad15397cc8
Closes-Bug: 1338464
Related-Bug: 1282861
In addition to removing an unnecessary dependency, this closes another
hole that was allowing raw bytes to appear in user-facing messages.
Change-Id: Ia0b76426a38e5a5c368c4c7e7ba2aef286758aca
Extends existing unit test for timeout being passed to get_auth
to cover v2.0 auth when keystone client should get the timeout
kwarg.
Related-Bug: 1447847
Change-Id: Ie9cfc86fa2156b94b45d290ac12e3f71b20d6c4f
A previous change added socket timeouts, but they weren't plumbed
through to the get_auth code path. Make sure we're passing them along
everywhere.
Original-Author: Monty Taylor <mordred@inaugust.com>
Change-Id: I398241898248e66d1f8c436c8ed2ec7a0e9387f6
Closes-bug: 1447847
The underlying requests library supports a timeout input parameter. The
other python-*client libraries support passing it in, so while working
on shade, it seemed like a great idea to be able to pass in timeout to
swiftclient too. For reference, there's a change in shade:
I095c1240693abf024bda2315dd77f4400b24a45b that shows interaction with
the other libs, and a tentative patch that would consume this one.
Change-Id: I699ebb1e092aa010af678de7ba15712da6ed5315
Previously, we compared the ETag from Swift against the MD5 of the
entire large object. However, the ETag for large objects is generally
the MD5 of the concatenation of the ETags for each segment, unless the
object is a DLO whose segments span more than one page of a container
listing. Rather than worry about ETags, just compare each chunk of the
segmented file. This allows the use of --skip-identical when uploading
SLOs and DLOs.
Additionally, there are several test-related improvements:
* The default arguments for OutputManager are now evaluated on
construction, rather than on definition, so that
TestOutputManager.test_instantiation will succeed when using nosetest
as a test runner. (See also: bug 1251507)
* An account_username option is now available in the functional tests
config file for auth systems that do not follow the account:username
format.
* CaptureOutput no longer writes to the captured stream, and
MockHttpTest now captures output. These were polluting test output
unnecessarily. (See also: bug 1201376)
Change-Id: Ic484e9a0c186c9283c4012c6a2fa77b96b8edf8a
Closes-Bug: #1201376
Closes-Bug: #1379252
Related-Bug: #1251507
Previously, it would only output something like
ClientException: Object PUT failed <url> 413 Request Entity Too Large
Your request is too large.
... which is misleading as it implies that Swift simply can't handle
large objects.
Change-Id: Icec9e76b2671490c3a2249ec08992950107c6828
With MD5Sum checking being added, a concern was brought up that It was
a change with no possibility of reverting to the old behaviour.
This change adds the flag '--ignore-checksum' to the upload subcommand
allowing the checks to be turned off.
Changed occurrences of the magic string for a null md5 to use a descriptive
constant instead.
Updated Error messages generated when validation fails. They should now be more descriptive
and not output a literal newline sequence.
Change-Id: Id1756cbb6700bb7e38f0ee0e75bc535e37f777ed
The chunk looks like nicely containable, but it pushes against the
right edge of the screen annoyingly. This patch _only_ prettifies,
without any observable changes.
Change-Id: I96fcb5848e3e5e0e8a8ebbe398b1d4a0bea5fa24
This patch adds unit tests to verify that
uploads overwriting an existing DLO manifest
will delete any segments pointed to by the
manifest, unless --leave-segments is specified.
Also test the --leave-segments option with an
existing SLO manifest.
Related-Bug: 1418007
Change-Id: I54e8eb138976269bb36392a811089c33c1ace4d3
When using stream=True, we need to make sure we release a connection
back to the connection pool to avoid warnings clogging up the logs. We
can do this by releasing the connection once we've read all the content
from the response. In situations where stream=False, requests already
does this for us.
Related-bug: 1424732
Closes-bug: 1341777
Change-Id: Id1c72ad86135a49d4b985992d788736b65f7dbda
Changed existing code to calculate the MD5 of the object
during the upload stream. Checks this MD5 against the etag
returned in the response.
An exception is raised if they do not match.
Closes-Bug: 1379263
Change-Id: I6c8bc1366dfb591a26d934a30cd21c9e6b9a04ce
Using -l, -d /, or a pseduo folder by itself works fine, but if you
combine all of them, this happens:
[zaitcev@guren s]$ swift list -d / testcont
amazon_lili.jpg
pseudodir/
rin_shibuya_pixiv.jpg
swift-c134855-5a.diff
[zaitcev@guren s]$ swift list -d / -l testcont
151954 2014-06-20 17:13:52 amazon_lili.jpg
Traceback (most recent call last):
File "/q/zaitcev/hail/python-swiftclient-tip/bin/swift", line 24, in <module>
sys.exit(main())
File "/q/zaitcev/hail/python-swiftclient-tip/swiftclient/shell.py", line 1301, in main
globals()['st_%s' % args[0]](parser, argv[1:], output)
File "/q/zaitcev/hail/python-swiftclient-tip/swiftclient/shell.py", line 414, in st_list
total_bytes += item_bytes
TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType'
The patch is a straightforward fix that assigns expected variables.
Change-Id: If24dcc2491e8b8982455107d2aca30c0491a398a
Closes-Bug: 1399678
move the contents of the functional test hook into tox for
consistency.
Change-Id: I730f1bb530215ba7c30bf32a1c3bbc16edd5c217
Required-By: https://review.openstack.org/#/c/156676
This patch fixes a crash and stack trace in stat when an object
name contains non-ascii characters.
Change-Id: Ib6dc686771e593243de56cafc100b17e51d9d9d5
Closes-Bug: 1411665
This patch fixed help message for "<subcommand> --help" in
general help message. Previous help message was only
"this help message". It has not described for subcommand.
Change-Id: I2fe6413baf98ba3881425ee6bed29c618e29fff8
Closes-Bug: 1420253
Missing cacert parameter in get_auth_1_0() and it makes requests_args['verify']
is True in default(likes "swift --os-cacert ssl.crti list").
The requests_args['verify'] should be a ca-certificate file name("ssl.crt").
Change-Id: Id3110f3818f55922ad19eeb1ead80fface824009
Closes-Bug: 1418473
This patch fixes a bug that results in SLO segments being left
behind regardless of the setting of --leave-segments.
Change-Id: I2b3c8ab4817df6facb4b5961dbe570645441a552
Closes-Bug: 1418007
This patch fixes downloading files to stdout and modifies
_SwiftReader to operate as an iterator that performs file
checks at the end of iteration as well as a context manager.
File verification checks have been removed from __exit__
and added to __iter__.
Change-Id: I3250bdeeef8484a9122c4b5b854756a7c8f8731e
Closes-Bug: 1395922
Closes-Bug: 1387376
I'm not sure how test_process_options_defaults ever passed for anyone that had
all the ST_(AUTH|USER|KEY) set in their environment when they ran it. Despite
our efforts to patch os.environ swiftclient.servce was acctually importing
environ from os and then sticking the results into some global state. The
tests seemed to work fine on infra, but for me locally until I `unset ST_AUTH`
the test would always fail. Seems to work now and looks fairly reasonable on
the surface.
Change-Id: Id68590d6af16040a2877e719bc38d1e126603e42
Removes an account stat from the object upload path.
This stat fails when user is not account admin even
though the user may have container ACL permission to
write objects.
Reduces the severity of the CLI output message when
upload fails to create the given container (this is
not an error since the container may exist - the user
just does not have permission to PUT or POST the
container).
Changes the 'swift upload' exit return code from 1 to
0 if container PUT fails but object PUT succeeds.
For segment uploads, makes the attempt to create the
segment container conditional on it not being the same
as the manifest container. This avoids an unnecessary
container PUT.
Fixes another bug that became apparent: with segmented
upload a container HEAD may be attempted to determine
the policy to be used for the segment container. When
this failed the result dict has headers=None which was
causing an exception in the shell result handler.
Add unit tests for object upload/download and container
list with --os-storage-url option.
Closes-Bug: #1371650
Change-Id: If1f8a02ee7459ea2158ffa6e958f67d299ec529e
Modify two tests to use the CaptureOutput class. These
tests were added after the comprehensive transition to using
CaptureOutput made in change [1], so this is just bringing
them in line with that test pattern.
Also deletes an unused mock.
[1] change id Ib59bbbe88256f215eed0a8ebc8282e02181d4377
Change-Id: Ic524311ffb3b0d6566addec0676633ddb8075e25
If you specify a token and storage url when creating a Connection,
regardless of the auth api version the first request will be made
directly to swift. You can either provide a preauthurl and preauthtoken
or fall back to os_options' object_storage_url and auth_token keys
(exposed as --os-storage-url and --os-auth-token on the command line or
OS_STORAGE_URL and OS_AUTH_TOKEN in the environment).
If a _retry wrapped request on a Connection fails because of invalid
authentication (401) the Connection's cached token and url will be
invalidated. If the Connection's retries attribute is > 0 the
subsequent attempt will call get_auth to refresh the token, but the
pre-configured storage_url will always be re-used. This is consistent
with current auth v2 behavior and less surprising for auth v1.
The pre-existing, but previously undocumented behavior/interface of
get_auth would override the storage_url returned by the auth service if
the 'os_storage_url' option was provided in the os_options dict. To
ensure that this behavior is consistent across auth v1 and v2 from the
command line and when using the Connection class as a library - the
preauthurl is stashed in the os_options dict when provided.
Improved Connection.get_capabilities storage_url handling to better
support the consistent behavior of a preauthurl/object_storage_url on
the connection regardless of auth version.
Fixed up some test infrastructure to enable setting up and testing
multiple requests/responses.
Change-Id: I6950fb73f3e28fdddb62760cae9320e2f4336776
Check for None was being done after a method call which
caused an attribute error if the value was None.
The Method call was also a mistake and has been corrected to the
hasattr function like intended.
Added Tests.
Closes-Bug: 1392651
Change-Id: Ifb1c84e26533bccbaddcce5738f434f36feca74e
