Currently, a user with read/write access to a container (but without
access to creat new containers) recieves a warning every time they
upload. Now, allow them to avoid the extra request and warning by
specifying --skip-container-put on the command line.
This is also useful when testing: developers can HEAD a container to
ensure it's in memcache, shut down all container servers, then upload
and creaate a bunch of async pendings. Previously, the 503 on container
PUT would prevent the object upload from even being attempted.
Closes-Bug: 1317956
Related-Bug: 1204558
Change-Id: I3d9129a0b6b65c6c6187ae6af003b221afceef47
Related-Change: If1f8a02ee7459ea2158ffa6e958f67d299ec529e
Previously, we would write out totals for every page of listings, like
$ swift list sync --prefix=09-21 --total -l
80000000000
80000000000
80000000000
58096000000
Now, roll those all into a single total:
$ swift list sync --prefix=09-21 --total -l
298096000000
Change-Id: Icc265636815220e33e8c9eec0a3ab80e9f899038
It's fairly annoying getting a traceback in swift's probe tests then
only having a URL and status code to go searching for in logs.
Leave the shell.py output untouched, though, since we output the
transaction ID on a new line anyway.
Change-Id: Idb849848ec08b6c04812b088467c9a687c2a7e27
Following the recent v3applicationcredentials patch, if you have your
environment variables set up to work with python-openstackclient using
swiftclient's v1password plugin, swiftclient won't work:
$ env | egrep '^(OS|ST)_'
ST_KEY=testing
ST_USER=test:tester
OS_AUTH_URL=http://saio/auth/v1.0
ST_AUTH=http://saio/auth/v1.0
OS_USERNAME=test:tester
OS_AUTH_TYPE=v1password
OS_PASSWORD=testing
$ openstack object store account show
+------------+----------------------------+
| Field | Value |
+------------+----------------------------+
| Account | AUTH_test |
| Bytes | 0 |
| Containers | 11 |
| Objects | 0 |
+------------+----------------------------+
$ swift stat
Only "v3applicationcredential" is supported for --os-auth-type
We don't really want to allow (and mostly ignore) arbitrary OS_AUTH_TYPE
values, though -- there are a whole bunch of plugins we don't remotely
support. But it seems OK to allow any of the password plugins; while we
won't actually use them (currently), we provide roughly equivalent
functionality.
Handful of other drive-bys:
* Use a None sentinel to determine whether keystoneauth1 is installed
instead of trying to catch a NameError.
* Clarify error state when keystoneauth1 is not installed.
* Fix a typo: "sses" -> "sess".
Change-Id: Id7ea9c3ea8278ae86a04d057a472a8f8a87b8eae
Related-Change: I9190e5e7e24b6a741970fa0d0ac792deccf73d25
Have `--versions` imply `--long` and add a new column for version_id.
Also, have version-aware listings show all versions as "null" on old
Swifts that don't support object versioning (or when object versioning
is not enabled).
Change-Id: I0e009bce2471d1c140ac9b83700591cb355fee3f
* add --versions to list
* add --versions to delete
* add --version-id to stat
* add --version-id to delete
* add --version-id to download
Change-Id: I89802064921778fee7efe57c7d60c976cdde3a27
Replace the 1 always concatenated to printed object names for each
successfully deleted object in bulk-delete with an optional [after x
attempts] if x > 1
Change-Id: If4af9141fe4f3436a4e9e0e2dfc24c6ec7292996
Closes-Bug: 1852808
Symlinks have recently added some new keys to container listings. It's
very convenient to be able to see and reason about the extra information
in container listings.
Allowing raw json output is similar with what the client already does
for the info command, and it's forward compatible with any listing
enhancements added by future middleware development.
Change-Id: I88fb38529342ac4e4198aeccd2f10c69c7396704
This patch basically follows the bash completion
model that other OpenStack clients use. It creates
a new command to swiftclient called `bash_completion`.
The `bash_completion` command by default will print
all base flags and exsiting commands. If you pass
it a command, it'll print out all base flags and
any flags that command accepts. So as you type out
your swift command and auto-complete, only the current
available flags are offered to you.
This is used by the swift.bash_completion script to
allow swift commands to be bash completed.
To make it work, place the swift.bash_completion file
into /etc/bash_completion.d and source it:
cp tools/swift.bash_completion /etc/bash_completion.d/swift
source /etc/bash_completion.d/swift
Because swiftclient itself is creating this flag/command output
it should automatically add anything we add to the swiftclient
CLI.
Change-Id: I5609a19018269762b4640403daae5827bb9ad724
I'm giving up on trying to back out all of the test-requirements
up-revs, but let's try to stay compatibile with old requests/six.
As part of that, only disable some requests warnings on new-enough requests.
Note that we should now be compatible with distro packages back to
Ubuntu 16.04 and CentOS 6. Our six is still too new for Trusty, but
hey, there's less than a year left on that anyway, right?
Change-Id: Iccb23638393616f9ec3da660dd5e39ea4ea94220
Related-Change: I2a8f465c8b08370517cbec857933b08fca94ca38
Add the --prompt option for the CLI which will cause the user to be
prompted to enter a password. Any password otherwise specified by
--key, --os-password or an environment variable will be ignored.
The swift client will exit with a warning if the password cannot be
entered without its value being echoed.
Closes-Bug: #1357562
Change-Id: I513647eed460007617f129691069c6fb1bfe62d7
ading multiple options on the same line makes
it easy to miss when quickly scanning the options.
Change-Id: I8e324fca48cd05d9e381d5106135542274c2ff7f
Signed-off-by: Thiago da Silva <thiago@redhat.com>
This patch attemps to add an option to force get_auth call while retrying
an operation even if it gets errors other than 401 Unauthorized.
Why we need this:
The main reason why we need this is current python-swiftclient requests could
never get succeeded under certion situation using third party proxies/load balancers
between the client and swift-proxy server. I think, it would be general situation
of the use case.
Specifically describing nginx case, the nginx can close the socket from the client
when the response code from swift is not 2xx series. In default, nginx can wait the
buffers from the client for a while (default 30s)[1] but after the time past, nginx
will close the socket immediately. Unfortunately, if python-swiftclient has still been
sending the data into the socket, python-swiftclient will get socket error (EPIPE,
BrokenPipe). From the swiftclient perspective, this is absolutely not an auth error,
so current python-swiftclient will continue to retry without re-auth.
However, if the root cause is sort of 401 (i.e. nginx got 401 unauthorized from the
swift-proxy because of token expiration), swiftclient will loop 401 -> EPIPE -> 401...
until it consume the max retry times.
In particlar, less time to live of the token and multipart object upload with large
segments could not get succeeded as below:
Connection Model:
python-swiftclient -> nginx -> swift-proxy -> swift-backend
Case: Try to create slo with large segments and the auth token expired with 1 hour
1. client create a connection to nginx with successful response from swift-proxy and its auth
2. client continue to put large segment objects
(e.g. 1~5GB for each and the total would 20~30GB, i.e. 20~30 segments)
3. after some of segments uploaded, 1 hour past but client is still trying to
send remaining segment objects.
4. nginx got 401 from swift-proxy for a request and wait that the connection is closed
from the client but timeout past because the python-swiftclient is still sending much data
into the socket before reading the 401 response.
5. client got socket error because nginx closed the connection during sending the buffer.
6. client retries a new connection to nginx without re-auth...
<loop 4-6>
7. finally python-swiftclient failed with socket error (Broken Pipe)
In operational perspective, setting longer timeout for lingering close would be an option but
it's not complete solution because any other proxy/LB may not support the options.
If we actually do THE RIGHT THING in python-swiftclient, we should send expects: 100-continue
header and handle the first response to re-auth correctly.
HOWEVER, the current python's httplib and requests module used by python-swiftclient doesn't
support expects: 100-continue header [2] and the thread proposed a fix [3] is not super active.
And we know the reason we depends on the library is to fix a security issue that existed
in older python-swiftclient [4] so that we should touch around it super carefully.
In the reality, as the hot fix, this patch try to mitigate the unfortunate situation
described above WITHOUT 100-continue fix, just users can force to re-auth when any errors
occurred during the retries that can be accepted in the upstream.
1: http://nginx.org/en/docs/http/ngx_http_core_module.html#lingering_close
2: https://github.com/requests/requests/issues/713
3: https://bugs.python.org/issue1346874
4: https://review.openstack.org/#/c/69187/
Change-Id: I3470b56e3f9cf9cdb8c2fc2a94b2c551927a3440
When uploading from standard input, swiftclient should turn the upload
into an SLO in the case of large objects. This patch picks the
threshold as 10MB (and uses that as the default segment size). The
consumers can also supply the --segment-size option to alter that
threshold and the SLO segment size. The patch does buffer one segment
in memory (which is why 10MB default was chosen).
(test is updated)
Change-Id: Ib13e0b687bc85930c29fe9f151cf96bc53b2e594
If "-" is passed in for the source, python-swiftclient will upload
the object by reading the contents of the standard input. The object
name option must be set, as well, and this cannot be used in
conjunction with other files.
This approach stores the entire contents as one object. A follow on
patch will change this behavior to upload from standard input as SLO,
unless the segment size is larger than the content size.
Change-Id: I1a8be6377de06f702e0f336a5a593408ed49be02
Currently, the swiftclient upload command passes a custom metadata
header for each object (called object-meta-mtime), whose value is
the current UNIX timestamp. When downloading such an object with the
swiftclient, the mtime header is parsed and passed as the atime and
mtime for the newly created file.
There are use-cases where this is not desired, for example when using
tmp or scratch directories in which files older than a specific date
are deleted. This commit provides a boolean option for ignoring the
mtime header.
Change-Id: If60b389aa910c6f1969b999b5d3b6d0940375686
Client-side implementation for ISO 8601 timestamp
support of tempurl middleware. Please see
https://review.openstack.org/#/c/422679/
Change-Id: I76da28b48948475ec1bae5258e0b39a316553fb7
The SwiftService and shell support the ability to limit
deletions to only those objects that match a specified
prefix, so let's expose that (really useful) behaviour
in the command line help as well :)
Change-Id: I9ef177aa96e4829196b5200dd8e9d0d2f7f89b63
In python swiftclient: swift <sub_command> —help will now
display st_<sub_command>_options + st_<sub_command>_help texts
e.g. http://paste.openstack.org/show/589752/
Change-Id: I34e4b2ac29ef395f8ca474ce7a82f59a1fd8c7f4
Closes-Bug: #1621415
The patch add the information about the syntax of provding read and
write access to the users within a project.
The current status is:
account:* does not work with tempauth
account does not work with keystoneauth
Furthermore, the concept of "account" is fuzzy. It could be a
"project", but even that doesn't distinguish between a project
id and a project name.
Change-Id: I4462230fe05aa4a742e825f5b6312deb93bc1170
Closes-Bug: #1587677
If the supplied path is not of the form /v1/a/c/o then
swift tempurl subcommand will now return an error message.
Also removes redundant check for seconds parameter being an int
from shell.py because the same check is made when calling
utils.generate_temp_url.
Drive-by fix for missing param definition for generate_temp_url.
Change-Id: I41f4389948b01fadaa5fc4939ea12e0ed2167345
Related-Change: I0fb2ce125fe12d660e4deb778265016bdd5ff31b
In usage messages, it only makes sense to have an
object option if the container option has been specified.
Fixes the parentheses to indicate this.
Change-Id: I550ba99879ea6ce5489617a94cec36b66be270c8
Related-Change: Ie84a90eb0278dd2cdf5775948572bb140f8d2c4b
When printing the help menu for Swift commands, it is confusing to see
both positional arguments and options not distinguished. The
positional arguments are supposed to be enclosed in angular braces <>,
options in square braces [] and optional positional arguments in [<>].
Change-Id: Ie84a90eb0278dd2cdf5775948572bb140f8d2c4b
