Previously, using SwiftService to delete "many" objects would use
bulk delete if available, but it would not respect the bulk delete
page size. If the number of objects to delete exceeded the bulk delete
page size, SwiftService would ignore the error and nothing would be
deleted.
This patch changes _should_bulk_delete() to be _bulk_delete_page_size();
instead of returning a simple True/False, it returns the page size for
the bulk deleter, or 1 if objects should be deleted one at a time.
Delete SDK calls are then spread across multiple bulk DELETEs if the
requested number of objects to delete exceeds the returned page size.
Fixed the logic in _should_bulk_delete() so that if the object list
is exactly 2x the thread count, it will not bulk delete. This is the
natural conclusion following the logic that existed previously: if
the delete request can be satisfied by every worker thread doing one
or two tasks, don't bulk delete. But if it requires a worker thread
to do three or more tasks, do a bulk delete instead. Previously, the
logic would mean that if every worker thread did exactly two tasks, it
would bulk delete. This patch changes a "<" to a "<=".
Closes-Bug: 1679851
Change-Id: I3c18f89bac1170dc62187114ef06dbe721afcc2e
If we were to include this in a normal PUT, it would 400, but only if
slo is actually in the pipeline. If it's *not*, we'll create a normal
Swift object and the header sticks.
- This is really confusing for users; see the related bug.
- If slo is later enabled in the cluster, Swift starts responding 500
with a KeyError because the client and on-disk formats don't match!
Change-Id: I1d80c76af02f2ca847123349224ddc36d2a6996b
Related-Change: I986c1656658f874172860469624118cc63bff9bc
Related-Bug: #1680083
Client-side implementation for ISO 8601 timestamp
support of tempurl middleware. Please see
https://review.openstack.org/#/c/422679/
Change-Id: I76da28b48948475ec1bae5258e0b39a316553fb7
Probably the most common format for documenting arguments is
reST field lists [1]. This change updates some docstrings to
comply with the field lists syntax.
[1] http://sphinx-doc.org/domains.html#info-field-lists
Change-Id: Ic011fd3e3a8c5bafa24a3438a6ed5bb126b50e95
The opened file for upload is not closed.
This fix prevents possible file handle leak.
Closes-Bug: #1559079
Change-Id: Ibc58667789e8f54c74ae2bbd32717a45f7b30550
The SwiftService and shell support the ability to limit
deletions to only those objects that match a specified
prefix, so let's expose that (really useful) behaviour
in the command line help as well :)
Change-Id: I9ef177aa96e4829196b5200dd8e9d0d2f7f89b63
In python swiftclient: swift <sub_command> —help will now
display st_<sub_command>_options + st_<sub_command>_help texts
e.g. http://paste.openstack.org/show/589752/
Change-Id: I34e4b2ac29ef395f8ca474ce7a82f59a1fd8c7f4
Closes-Bug: #1621415
Previously, we only accepted iterables of strings like 'Header: Value'.
Now, we'll also accept lists of tuples like ('Header', 'Value') as well
as dictionaries like {'Header': 'Value'}.
This should be more intuitive for application developers, who are
already used to being able to pass dicts or lists of tuples to libraries
like requests.
Change-Id: I93ed2f1e8305f0168b7a4bd90c205b04730da836
The only logging we should do is when we've encountered a problem *and
we've dealt with it ourselves*. When we're raising an exception, it
should be up to the caller to decide whether to log anything about it.
Anything else is just rude.
Change-Id: I1c96b76d90a78b7a10ffe63e4a7440c8f579147c
Closes-Bug: 1213179
Related-Bug: 1202229
This lets us use Keystone sessions against endpoints like swauth and
tempauth with code like:
import keystoneauth1.loading
import keystoneauth1.session
import swiftclient
loader = keystoneauth1.loading.get_plugin_loader('v1password')
auth_plugin = loader.load_from_options(
auth_url='http://saio:8080/auth/v1.0',
username='test:tester',
password='testing')
keystone_session = keystoneauth1.session.Session(auth_plugin)
conn = swiftclient.Connection(session=keystone_session)
The plugin includes an optional project_name option, which may be used
to override the swift account from the storage url that was returned.
Additionally, it includes enough infrastructure to support some commands
in python-openstackclient>=3.0:
export OS_AUTH_TYPE=v1password
export OS_AUTH_URL=http://saio:8080/auth/v1.0
export OS_PROJECT_NAME=AUTH_test2
export OS_USERNAME=test:tester
export OS_PASSWORD=testing
openstack token issue
openstack catalog list
openstack catalog show object-store
openstack object store account show
openstack container list
openstack container create <container>
openstack container save <container>
openstack container show <container>
openstack container delete <container>
openstack object list <container>
openstack object create <container> <file>
openstack object save <container> <object>
opsentack object show <container> <object>
openstack object delete <container> <object>
Change-Id: Ia963dc44415f72a6518227e86d9528a987e07491
The patch add the information about the syntax of provding read and
write access to the users within a project.
The current status is:
account:* does not work with tempauth
account does not work with keystoneauth
Furthermore, the concept of "account" is fuzzy. It could be a
"project", but even that doesn't distinguish between a project
id and a project name.
Change-Id: I4462230fe05aa4a742e825f5b6312deb93bc1170
Closes-Bug: #1587677
Previously the tempurl subcommand would dump a traceback
due to a TypeError if the seconds option was not an int
value. With this patch it will now return the same
error message as if the seconds option were negative or not
a number.
Also changes the error message to state that the seconds
option should be a "whole number" rather than a "positive
integer", since 0 is a valid value.
Change-Id: Ie940d470f2be8006aa8eb7fe242f092457aeae21
Closes-Bug: #1621817
If the supplied path is not of the form /v1/a/c/o then
swift tempurl subcommand will now return an error message.
Also removes redundant check for seconds parameter being an int
from shell.py because the same check is made when calling
utils.generate_temp_url.
Drive-by fix for missing param definition for generate_temp_url.
Change-Id: I41f4389948b01fadaa5fc4939ea12e0ed2167345
Related-Change: I0fb2ce125fe12d660e4deb778265016bdd5ff31b
In usage messages, it only makes sense to have an
object option if the container option has been specified.
Fixes the parentheses to indicate this.
Change-Id: I550ba99879ea6ce5489617a94cec36b66be270c8
Related-Change: Ie84a90eb0278dd2cdf5775948572bb140f8d2c4b
When printing the help menu for Swift commands, it is confusing to see
both positional arguments and options not distinguished. The
positional arguments are supposed to be enclosed in angular braces <>,
options in square braces [] and optional positional arguments in [<>].
Change-Id: Ie84a90eb0278dd2cdf5775948572bb140f8d2c4b
Currently the options help, info and debug must exist in the options
dictionary; however it might be that this is not the case if a method is
imported externally. This patch fixes this.
Also support using a storage URL or path as an argument for tempurl, and
adding a test to verify correct output in both cases.
Related-Bug: 1607519
Related-Bug: 1607523
Closes-Bug: 1607521
Change-Id: I0fb2ce125fe12d660e4deb778265016bdd5ff31b
Previously, we would accept gzip-encoded responses, but only because we
were letting requests decode *all* responses (even object data). This
restores the previous capability, but with tighter controls about which
requests will accept gzipped responses and where the decoding happens.
Change-Id: I4fd8b97207b9ab01b1bcf825cc16efd8ad46344a
Related-Bug: 1282861
Related-Bug: 1338464
Recently, requests got a bit more picky about what types of data it will
accept as header values [1]. The reasons for this are generally sound;
str()ing arbitrary objects just before pushing them out a socket may not
do what the developer wanted/expected.
However, there are a few standard types that developers may be sending
that we should convert for them as a convenience. Now, we'll convert all
int, float, and bool values to strings before sending them on to
requests.
Change-Id: I6c2f451009cb03cb78812f54e4ed8566076de821
Closes-Bug: 1614932
Implement copy object method in swiftclient Connection, Service and CLI.
Although COPY functionality can be accomplished with 'X-Copy-From'
header in PUT request, using copy is more convenient especially when
using copy for updating object metadata non-destructively.
Closes-Bug: 1474939
Change-Id: I1338ac411f418f4adb3d06753d044a484a7f32a4
New versions of requests will raise an InvalidHeader error otherwise.
Change-Id: Idf3bcd8ac359bdda9a847bf99a78988943374134
Closes-Bug: #1614280
Closes-Bug: #1613814
This lets us do things like:
$ swift info --json | jq '[.swift.policies[].name]'
[
"Standard-Replica",
"EC"
]
Also, escape more dashes in the man page, so they won't be
misinterpreted as hyphens.
Change-Id: Ic7690bdbcfc55f55e5dde9bc11bb0644085973ce
Previously, we weren't encoding paths and keys as UTF-8, which would
trigger a UnicodeEncodeError on py27.
Change-Id: I2fad428369406c2ae32343a5e943ffb2cd1ca6ef
Added functionality for arbitrary query strings to be passed into container
functions. Additionally a minor typo correction in the README. Added unit
tests for query string functionality.
Closes-Bug: #1542459
Change-Id: Ica2cb3ea439632588388e748d8d2e944e9ed4fa4
Gnocchi uses a client with retries=0 to maximize throughtput and not retry N
times on e.g. 404 when checking existence of an object. However, this as the
side effect of never renewing the token since there' no retry on 401 either.
This patches change the behavior so that 401 errors are always retried,
whatever the retries value is.
Closes-Bug: #1589926
Change-Id: Ie06adf4cf17ea4592b5bbd7bbde9828e5e134e3e
