When issuing `delete` commands that would require three or more
individual deletes, check whether the cluster supports bulk deletes
and use that if it's available.
Additionally, a new option is added to the `delete` command:
* --prefix <prefix>
Delete all objects that start with <prefix>. This is similar to the
--prefix option for the `list` command.
Example:
$ swift delete c --prefix obj_prefix/
...will delete from container "c" all objects whose name begins with
"obj_prefix/", such as "obj_prefix/foo" and "obj_prefix/bar".
Change-Id: I6b9504848d6ef562cf4f570bbcd17db4e3da8264
This fix makes it possible to upload objects to pseudo-folders by
passing the upload paths via <container> arg regardless of whether the
container or folder path exist or not.
Change-Id: I575e58aa12adcf71cdaa70d025a0ea5c63f46903
Closes-Bug: #1478210
Partial-Bug: #1432734
Related-Bug: #1432734
It was dropping warnings like "UserWarning: Providing attr without
filter_value to get_urls() is deprecated as of the 1.7.0 release and may
be removed in the 2.0.0 release. Either both should be provided or
neither should be provided."
Change-Id: Iead0bcf36b4a46bf465a55a33a21fd7f14f0ac40
* Always use testtools.TestCase, since we're relying on testtools
* Always use mock (as opposed to unittest.mock) since we're relying on
mock
* Add note about when a missing logging handler was added
* Stop %-formatting the giant usage string that doesn't actually need
any formatting
* Prefer assertIs, assertIn, assertIsInstance over assertTrue
* Use else-self.fail instead of sentinel values
* Check resp.get('error') is None before checking resp['success'] is
True, so test failures actually tell you something useful
* Tighten some isinstance assertions
* Import MockHttpTest from correct location
* Only populate clean_os_environ once
* Use setUp for setup, not __init__
* Replace assertIn(key, dict) and assertEqual(foo, dict[key]) with
assertEqual(foo, dict.get(key)) when key is a literal and foo is not
None
* Use mock.patch.object instead of manually patching for tests
* Use six.binary_type instead of type(''.encode('utf-8'))
* Stop shadowing builtin bytes
* Reclaim some margin
* Stop checking the return-type of encode_utf8; we already know it's
bytes
Change-Id: I2138ea553378ce88810b7353147c8645a8f8c90e
Attempt to fix the linked bug by initialising the mock instance for
Connection.delete_object before calling the SwiftService upload
method, so that the delete_object mock already exists before the
delete_segments jobs that run in multiple threads call it. Otherwise
there is a risk that the delete_segment job threads could race while
creating either the delete_object mock or the
delete_object.return_value mock, resulting in each thread getting a
different instance. That would explain the intermittent test failures
reported in the bug.
Change-Id: Ia82697c093529076b0bbcc6bccd577afdf0839e1
Partial-Bug: #1480223
The `tempurl` subcommand's second positional argument is called
`seconds` and has heretofore interpreted as the number of seconds for
which the tempURL should be valid, counting from the moment of running
the command. This is indeed a common, if not the most common,
use-case. But some users, occasionally, might want to generate a tempURL
that expires at some particular ("absolute") time, rather than a
particular amount of time relative to the moment of happening to run the
command. (One might make an analogy to the way in which Swift's expiring
object support supports an `X-Delete-At` header in addition to
`X-Delete-After`—and it's the former that must be regarded as
ontologically prior.) Thus, this commit adds an `--absolute` optional
argument to the `tempurl` subcommand; if present, the `seconds` argument
will be interpreted as a Unix timestamp of when the tempURL should be
expire, rather than a duration for which the tempURL should be valid
starting from "now".
Change-Id: If9ded96f2799800958d5063127f3de812f50ef06
Some improvements to the tests for staggered download
that were added in [1].
[1] Ie737cbb7f8b1fa8a79bbb88914730b05aa7f2906
Change-Id: Ib999bc7bd198c1d9c217c57501f751e854d4c6ad
fix against H234: assertEquals() logs a DeprecationWarning
in Python3.x. use assertEqual() instead.
Closes-bug: #1480776
Change-Id: Iffda6bb5f2616d4af4567eeea37bb26531e34371
Signed-off-by: Hiroshi Miura <miurahr@nttdata.co.jp>
The current code builds a full object listing before performing either a multiple
download or delete operation (and also shuffles this complete list in the case of
a download). This patch removes the creation of the full object list and adds the
ability to turn off shuffle for files when downloading. Also added is a limit on
the number of list results that can be queued by a single call to service.list
without consuming results (reduces memory overhead for large listings).
Some tests added for service.py download and list.
Change-Id: Ie737cbb7f8b1fa8a79bbb88914730b05aa7f2906
There was always an implicit assumption that they were UTF-8 before, and
by converting them to unicode we close another hole allowing raw bytes
to appear in user-facing messages.
Closes-Bug: #1431866
Change-Id: If2e41d9a592c3ad02818e9c6f0959fd4b73cd0e0
Currently SwiftClient populate response dictionary before
removing "./" or "/" at begining of object name. This patch
fixes that by changing that order.
Closes-bug: #1412425
Change-Id: I80222754caba5d42a468f4677ac539e46682dd31
In addition to removing an unnecessary dependency, this closes another
hole that was allowing raw bytes to appear in user-facing messages.
Change-Id: Ia0b76426a38e5a5c368c4c7e7ba2aef286758aca
Extends existing unit test for timeout being passed to get_auth
to cover v2.0 auth when keystone client should get the timeout
kwarg.
Related-Bug: 1447847
Change-Id: Ie9cfc86fa2156b94b45d290ac12e3f71b20d6c4f
Previously, only the first container-listing's worth of segments was
deleted, which would leave behind orphaned segments when the object was
very large with small segments or the server's container_listing_limit
was small.
In addition, process DLO and SLO deletions on the segment thread pool,
rather than the object thread pool.
Change-Id: I1587375261a6237fa55a9cb96bda8dae918cc795
Related-Bug: #1418007
With MD5Sum checking being added, a concern was brought up that It was
a change with no possibility of reverting to the old behaviour.
This change adds the flag '--ignore-checksum' to the upload subcommand
allowing the checks to be turned off.
Changed occurrences of the magic string for a null md5 to use a descriptive
constant instead.
Updated Error messages generated when validation fails. They should now be more descriptive
and not output a literal newline sequence.
Change-Id: Id1756cbb6700bb7e38f0ee0e75bc535e37f777ed
This patch adds unit tests to verify that
uploads overwriting an existing DLO manifest
will delete any segments pointed to by the
manifest, unless --leave-segments is specified.
Also test the --leave-segments option with an
existing SLO manifest.
Related-Bug: 1418007
Change-Id: I54e8eb138976269bb36392a811089c33c1ace4d3
Changed existing code to calculate the MD5 of the object
during the upload stream. Checks this MD5 against the etag
returned in the response.
An exception is raised if they do not match.
Closes-Bug: 1379263
Change-Id: I6c8bc1366dfb591a26d934a30cd21c9e6b9a04ce
This patch fixes a bug that results in SLO segments being left
behind regardless of the setting of --leave-segments.
Change-Id: I2b3c8ab4817df6facb4b5961dbe570645441a552
Closes-Bug: 1418007
This patch fixes downloading files to stdout and modifies
_SwiftReader to operate as an iterator that performs file
checks at the end of iteration as well as a context manager.
File verification checks have been removed from __exit__
and added to __iter__.
Change-Id: I3250bdeeef8484a9122c4b5b854756a7c8f8731e
Closes-Bug: 1395922
Closes-Bug: 1387376
Removes an account stat from the object upload path.
This stat fails when user is not account admin even
though the user may have container ACL permission to
write objects.
Reduces the severity of the CLI output message when
upload fails to create the given container (this is
not an error since the container may exist - the user
just does not have permission to PUT or POST the
container).
Changes the 'swift upload' exit return code from 1 to
0 if container PUT fails but object PUT succeeds.
For segment uploads, makes the attempt to create the
segment container conditional on it not being the same
as the manifest container. This avoids an unnecessary
container PUT.
Fixes another bug that became apparent: with segmented
upload a container HEAD may be attempted to determine
the policy to be used for the segment container. When
this failed the result dict has headers=None which was
causing an exception in the shell result handler.
Add unit tests for object upload/download and container
list with --os-storage-url option.
Closes-Bug: #1371650
Change-Id: If1f8a02ee7459ea2158ffa6e958f67d299ec529e
Modify two tests to use the CaptureOutput class. These
tests were added after the comprehensive transition to using
CaptureOutput made in change [1], so this is just bringing
them in line with that test pattern.
Also deletes an unused mock.
[1] change id Ib59bbbe88256f215eed0a8ebc8282e02181d4377
Change-Id: Ic524311ffb3b0d6566addec0676633ddb8075e25
If you specify a token and storage url when creating a Connection,
regardless of the auth api version the first request will be made
directly to swift. You can either provide a preauthurl and preauthtoken
or fall back to os_options' object_storage_url and auth_token keys
(exposed as --os-storage-url and --os-auth-token on the command line or
OS_STORAGE_URL and OS_AUTH_TOKEN in the environment).
If a _retry wrapped request on a Connection fails because of invalid
authentication (401) the Connection's cached token and url will be
invalidated. If the Connection's retries attribute is > 0 the
subsequent attempt will call get_auth to refresh the token, but the
pre-configured storage_url will always be re-used. This is consistent
with current auth v2 behavior and less surprising for auth v1.
The pre-existing, but previously undocumented behavior/interface of
get_auth would override the storage_url returned by the auth service if
the 'os_storage_url' option was provided in the os_options dict. To
ensure that this behavior is consistent across auth v1 and v2 from the
command line and when using the Connection class as a library - the
preauthurl is stashed in the os_options dict when provided.
Improved Connection.get_capabilities storage_url handling to better
support the consistent behavior of a preauthurl/object_storage_url on
the connection regardless of auth version.
Fixed up some test infrastructure to enable setting up and testing
multiple requests/responses.
Change-Id: I6950fb73f3e28fdddb62760cae9320e2f4336776
The response dictionary in _delete_segment attach's any error
it encounters to the dict key 'exception', all other response
dict's use 'error' to store the exception.
Changed to make it consistent and added tests
Also added a third branch in st_delete as messages
created in _delete_container where being silently dropped.
Change-Id: Ifbc3b1fae78910fbc6acf4a86cfb0f60bb1aa336
Added the ability to clear the buffers in the
CaptureOutput class so it can be easily used multiple
times in the same context manager.
Also added a option to suppress the SystemExit
associated with printing an error.
Change-Id: Ib59bbbe88256f215eed0a8ebc8282e02181d4377
This change adds some new classes to test.unit.utils which wrap up the
collection and mocking of the std err and std out. This will make the
testing code easier to write and maintain.
Change-Id: I50ad2a736b2bb550ab83f6a43fb5a0fb5393573e
When downloading an object, the content_type header is inspected
to determine if a directory needs to be created. This header is
assumed to always be in the response; if it isn't an Exception is
raised. swiftclient should not assume content_type will always be
set.
Change-Id: I156195c02b6e0bc398fa962eb1f78c4dbddd1596
swiftclient.shell.st_post was ignoring the result dict
returned from SwiftService.post and therefore giving no
indication when posts failed e.g. due to invalid auth
credentials.
This patch ensures that SwiftService always returns a
result_dict from post() and then checks the result dict.
On failure the shell now outputs the ClientException
message and exits with error code 1.
Also adds unit tests and cleans up some unnecessary mocked
return values in existing tests.
Closes-Bug: 1383243
Change-Id: I111150eb3d026c8d01c2cac13d3613ca7304e5b9
Changed the message shown when a user doesn't enter project/tenant
id/name to be more informative.
When attempting to stat a container without supplying project/tenant
name or id, an empty response was being returned instead of an error
being raised.
Changed the error raised in swiftclient.client when no tenant or project
is specified to be more specific.
Add tests for basic regression checking.
Closes-Bug: #1372589
Change-Id: I4eb6964d9f1702db119cc0294edc02841b1ecd5f
