Why now?
* argparse was introduced in Python 3.2 and back-ported to Python 2.7.
Until we dropped Python 2.6 support, we were stuck on optparse.
* keystoneauth.loading.cli provides register_argparse_arguments and
load_from_argparse_arguments helper methods. Now that we're moving
toward Keystone Session support, argparse seems required.
Closes-Bug: 1553030
Change-Id: I5139fb64a8631a3010680090fd04345f95c55c7b
Now, client applications can get to things like transaction IDs for
failures without needing to turn on all of logging.
While we're at it, add a from_response factory method for
ClientException.
Co-Authored-By: Alexander Corwin <ancorwin@gmail.com>
Change-Id: Ib46d5f8fc7f36f651f5908bb9d900316fdaebce3
* Improve some formatting
* Be more explicit about how much will be revealed when
* Rename redact_sensitive_tokens to redact_sensitive_headers, as it
affects more than tokens.
Change-Id: I02b375d914e9f0a210d038ecb31188d09a8ffce3
Add checking object-threads, container-threads and segment-threads options validation
The values should be a positive integer
Change-Id: Iaf98e33dae4b9a7f82e33f7cc2e5a0b293a1c76f
Close-Bug: #1546973
Currently the swiftclient logs sensitive info in headers when logging
HTTP requests. This patch hides sensitive info in headers such as
'X-Auth-Token' in a similar way to swift itself (we add a
'reveal_sensitive_prefix' configuration to the client).
With this patch, tokens are truncated by removing the specified number
of characters, after which '...' is appended to the logged token to
indicate that it has been redacted.
Co-Authored-By: Li Cheng <shcli@cn.ibm.com>
Co-Authored-By: Zack M. Davis <zdavis@swiftstack.com>
Change-Id: I43dd7254f7281d4db59b286aa2145643c64e1705
Closes-bug: #1516692
Add support for the auth-version to be specified using
--os-identity-api-version or OS_IDENTITY_API_VERSION
for compatibility with other openstack client command
line options.
The auth version used will be selected as follows:
- if either --auth-version or --os-identity-api-version
is set, use that value
- otherwise use the value of ST_AUTH_VERSION, if set
- otherwise use the value of OS_AUTH_VERSION, if set
- otherwise (new behaviour) use the value of
OS_IDENTITY_API_VERSION, if set
- otherwise default to 1.0
Note that before this change the auth version might have
defaulted to 1.0 despite OS_IDENTITY_API_VERSION being set,
but with this change OS_IDENTITY_API_VERSION is preferred.
Change-Id: Ifba4c4e43560ede3013337b8cdbc77dc2de6e8ff
Closes-Bug: #1541273
When issuing `delete` commands that would require three or more
individual deletes, check whether the cluster supports bulk deletes
and use that if it's available.
Additionally, a new option is added to the `delete` command:
* --prefix <prefix>
Delete all objects that start with <prefix>. This is similar to the
--prefix option for the `list` command.
Example:
$ swift delete c --prefix obj_prefix/
...will delete from container "c" all objects whose name begins with
"obj_prefix/", such as "obj_prefix/foo" and "obj_prefix/bar".
Change-Id: I6b9504848d6ef562cf4f570bbcd17db4e3da8264
The debug and info options need to be set before a subcommand method is called,
otherwise they are simply ignored. This is kind of irritating - other options
(for example -U, -A, -K) are usable after a positional command.
This patch fixes this, and commands like these are no longer ignoring --debug or --info:
swift stat --debug
swift list container --info
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Change-Id: Ib19b05deef7a015881f1eed4a3946025e16bf922
* Always use testtools.TestCase, since we're relying on testtools
* Always use mock (as opposed to unittest.mock) since we're relying on
mock
* Add note about when a missing logging handler was added
* Stop %-formatting the giant usage string that doesn't actually need
any formatting
* Prefer assertIs, assertIn, assertIsInstance over assertTrue
* Use else-self.fail instead of sentinel values
* Check resp.get('error') is None before checking resp['success'] is
True, so test failures actually tell you something useful
* Tighten some isinstance assertions
* Import MockHttpTest from correct location
* Only populate clean_os_environ once
* Use setUp for setup, not __init__
* Replace assertIn(key, dict) and assertEqual(foo, dict[key]) with
assertEqual(foo, dict.get(key)) when key is a literal and foo is not
None
* Use mock.patch.object instead of manually patching for tests
* Use six.binary_type instead of type(''.encode('utf-8'))
* Stop shadowing builtin bytes
* Reclaim some margin
* Stop checking the return-type of encode_utf8; we already know it's
bytes
Change-Id: I2138ea553378ce88810b7353147c8645a8f8c90e
The `tempurl` subcommand's second positional argument is called
`seconds` and has heretofore interpreted as the number of seconds for
which the tempURL should be valid, counting from the moment of running
the command. This is indeed a common, if not the most common,
use-case. But some users, occasionally, might want to generate a tempURL
that expires at some particular ("absolute") time, rather than a
particular amount of time relative to the moment of happening to run the
command. (One might make an analogy to the way in which Swift's expiring
object support supports an `X-Delete-At` header in addition to
`X-Delete-After`—and it's the former that must be regarded as
ontologically prior.) Thus, this commit adds an `--absolute` optional
argument to the `tempurl` subcommand; if present, the `seconds` argument
will be interpreted as a Unix timestamp of when the tempURL should be
expire, rather than a duration for which the tempURL should be valid
starting from "now".
Change-Id: If9ded96f2799800958d5063127f3de812f50ef06
The current code builds a full object listing before performing either a multiple
download or delete operation (and also shuffles this complete list in the case of
a download). This patch removes the creation of the full object list and adds the
ability to turn off shuffle for files when downloading. Also added is a limit on
the number of list results that can be queued by a single call to service.list
without consuming results (reduces memory overhead for large listings).
Some tests added for service.py download and list.
Change-Id: Ie737cbb7f8b1fa8a79bbb88914730b05aa7f2906
- add usage strings for short option such as '-a' for '--all'
This add all short options to usage text.
- add missing --header usage help for download command
- some cometic changes
Closes-bug: #1475511
Change-Id: Ibfecac8764669540fa025787548133a50fa50b10
Signed-off-by: Hiroshi Miura <miurahr@nttdata.co.jp>
This patch adds "--output-dir" and "--remove-prefix" options
to the "download" command and unit tests for it.
Example:
$ swift list example --prefix swift2.2
swift2.2/bin/swift-object-auditor
swift2.2/bin/swift-object-expirer
swift2.2/bin/swift-object-info
swift2.2/bin/swift-object-replicator
swift2.2/bin/swift-object-server
swift2.2/bin/swift-object-updater
When given "--output-dir <directory>", client downloads objects to <directory>.
$ swift download example --prefix swift2.2 \
--output-dir new/swift/dir
The folder structure:
.
└── new
└── swift
└── dir
└── swift2.2
└── bin
├── swift-object-auditor
├── swift-object-expirer
├── swift-object-info
├── swift-object-replicator
├── swift-object-server
└── swift-object-updater
When given "--remove-prefix", client downloads objects without <prefix>.
$ swift download example --prefix swift2.2 \
--remove-prefix \
--output-dir swift
The folder structure:
.
└── swift
└── bin
├── swift-object-auditor
├── swift-object-expirer
├── swift-object-info
├── swift-object-replicator
├── swift-object-server
└── swift-object-updater
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I7463fe2941cc94f9a50a4756a97c2ccdf946294d
Implements: blueprint swiftclient-download-pseudo-folder-to-specific-target
There was always an implicit assumption that they were UTF-8 before, and
by converting them to unicode we close another hole allowing raw bytes
to appear in user-facing messages.
Closes-Bug: #1431866
Change-Id: If2e41d9a592c3ad02818e9c6f0959fd4b73cd0e0
Previously, it would only output something like
ClientException: Object PUT failed <url> 413 Request Entity Too Large
Your request is too large.
... which is misleading as it implies that Swift simply can't handle
large objects.
Change-Id: Icec9e76b2671490c3a2249ec08992950107c6828
With MD5Sum checking being added, a concern was brought up that It was
a change with no possibility of reverting to the old behaviour.
This change adds the flag '--ignore-checksum' to the upload subcommand
allowing the checks to be turned off.
Changed occurrences of the magic string for a null md5 to use a descriptive
constant instead.
Updated Error messages generated when validation fails. They should now be more descriptive
and not output a literal newline sequence.
Change-Id: Id1756cbb6700bb7e38f0ee0e75bc535e37f777ed
The chunk looks like nicely containable, but it pushes against the
right edge of the screen annoyingly. This patch _only_ prettifies,
without any observable changes.
Change-Id: I96fcb5848e3e5e0e8a8ebbe398b1d4a0bea5fa24
Using -l, -d /, or a pseduo folder by itself works fine, but if you
combine all of them, this happens:
[zaitcev@guren s]$ swift list -d / testcont
amazon_lili.jpg
pseudodir/
rin_shibuya_pixiv.jpg
swift-c134855-5a.diff
[zaitcev@guren s]$ swift list -d / -l testcont
151954 2014-06-20 17:13:52 amazon_lili.jpg
Traceback (most recent call last):
File "/q/zaitcev/hail/python-swiftclient-tip/bin/swift", line 24, in <module>
sys.exit(main())
File "/q/zaitcev/hail/python-swiftclient-tip/swiftclient/shell.py", line 1301, in main
globals()['st_%s' % args[0]](parser, argv[1:], output)
File "/q/zaitcev/hail/python-swiftclient-tip/swiftclient/shell.py", line 414, in st_list
total_bytes += item_bytes
TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType'
The patch is a straightforward fix that assigns expected variables.
Change-Id: If24dcc2491e8b8982455107d2aca30c0491a398a
Closes-Bug: 1399678
This patch fixes downloading files to stdout and modifies
_SwiftReader to operate as an iterator that performs file
checks at the end of iteration as well as a context manager.
File verification checks have been removed from __exit__
and added to __iter__.
Change-Id: I3250bdeeef8484a9122c4b5b854756a7c8f8731e
Closes-Bug: 1395922
Closes-Bug: 1387376
Removes an account stat from the object upload path.
This stat fails when user is not account admin even
though the user may have container ACL permission to
write objects.
Reduces the severity of the CLI output message when
upload fails to create the given container (this is
not an error since the container may exist - the user
just does not have permission to PUT or POST the
container).
Changes the 'swift upload' exit return code from 1 to
0 if container PUT fails but object PUT succeeds.
For segment uploads, makes the attempt to create the
segment container conditional on it not being the same
as the manifest container. This avoids an unnecessary
container PUT.
Fixes another bug that became apparent: with segmented
upload a container HEAD may be attempted to determine
the policy to be used for the segment container. When
this failed the result dict has headers=None which was
causing an exception in the shell result handler.
Add unit tests for object upload/download and container
list with --os-storage-url option.
Closes-Bug: #1371650
Change-Id: If1f8a02ee7459ea2158ffa6e958f67d299ec529e
