Currently, if you attempt to stream an upload from an iterator, as in
def data():
yield 'foo'
yield ''
yield 'bar'
conn.put_object('c', 'o', data())
... requests will faithfully emit a zero-length chunk, ending the
transfer. Swift will then close the connection, possibly (if Connection:
keep-alive was set) after attempting to parse the next chunk as a new
request.
Now, Swift will receive all of the bytes from the iterable, and any
zero-byte chunks will be ignored.
This will be fixed in requests [1], but not until an eventual 3.0.0
release.
[1] https://github.com/kennethreitz/requests/pull/2631
Change-Id: I19579ed7a0181ac3f488433e7c1839f7f7a040b8
Add a unit test to test the 'string' like behaviour of
get_object when it is called without resp_chunk_size set.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I496032a76036141d027c30b076c810b34bc6bef0
It was dropping warnings like "UserWarning: Providing attr without
filter_value to get_urls() is deprecated as of the 1.7.0 release and may
be removed in the 2.0.0 release. Either both should be provided or
neither should be provided."
Change-Id: Iead0bcf36b4a46bf465a55a33a21fd7f14f0ac40
New documentation for python-swiftclient that introduces
the APIs available and gives some opinionated advice about
when to use the shell, the client API and the service API.
Change-Id: I19020f041fab2e72469979f712ffe3951c431d24
According to help `swift upload -h` you can add a customized request header 'Content-Type'.
But actually it is ignored (cleared and default is used) if subcommand is upload.
Subcommand post works as expected in help.
Bug fix: Use 'Content-Type' from the customized request headers also if
uploading.
Change-Id: If0d1354b6214b909527341078fe1769aa6587457
* Always use testtools.TestCase, since we're relying on testtools
* Always use mock (as opposed to unittest.mock) since we're relying on
mock
* Add note about when a missing logging handler was added
* Stop %-formatting the giant usage string that doesn't actually need
any formatting
* Prefer assertIs, assertIn, assertIsInstance over assertTrue
* Use else-self.fail instead of sentinel values
* Check resp.get('error') is None before checking resp['success'] is
True, so test failures actually tell you something useful
* Tighten some isinstance assertions
* Import MockHttpTest from correct location
* Only populate clean_os_environ once
* Use setUp for setup, not __init__
* Replace assertIn(key, dict) and assertEqual(foo, dict[key]) with
assertEqual(foo, dict.get(key)) when key is a literal and foo is not
None
* Use mock.patch.object instead of manually patching for tests
* Use six.binary_type instead of type(''.encode('utf-8'))
* Stop shadowing builtin bytes
* Reclaim some margin
* Stop checking the return-type of encode_utf8; we already know it's
bytes
Change-Id: I2138ea553378ce88810b7353147c8645a8f8c90e
Cleanups for change I35c3b266b3c733f6b1629de4c683ea7d40128032
Add missing param definitions to client get_container
and head_object docstrings.
For consistency, add headers parameter to the Connection class
head_object and head_container wrapper methods.
Add tests to verify that the headers parameter of Connection
get_container, head_container and head_object methods is passed to the
module functions.
Change-Id: Ib40d5b626b2793840727c58cffbf725bea55651f
Some tests rely on the fake connection checking expected request
parameters, but that assumes that the fake ocnnection is even called,
which is not being checked. Add more explicit assertions that requests
are in fact made.
Change-Id: Id1c48235d7d97fd1b0feec6c19ed59a87bebdf89
Headers parameter is required when passing client key for encryption.
It is missing for get_container and head_object.
Change-Id: I35c3b266b3c733f6b1629de4c683ea7d40128032
Attempt to fix the linked bug by initialising the mock instance for
Connection.delete_object before calling the SwiftService upload
method, so that the delete_object mock already exists before the
delete_segments jobs that run in multiple threads call it. Otherwise
there is a risk that the delete_segment job threads could race while
creating either the delete_object mock or the
delete_object.return_value mock, resulting in each thread getting a
different instance. That would explain the intermittent test failures
reported in the bug.
Change-Id: Ia82697c093529076b0bbcc6bccd577afdf0839e1
Partial-Bug: #1480223
The `tempurl` subcommand's second positional argument is called
`seconds` and has heretofore interpreted as the number of seconds for
which the tempURL should be valid, counting from the moment of running
the command. This is indeed a common, if not the most common,
use-case. But some users, occasionally, might want to generate a tempURL
that expires at some particular ("absolute") time, rather than a
particular amount of time relative to the moment of happening to run the
command. (One might make an analogy to the way in which Swift's expiring
object support supports an `X-Delete-At` header in addition to
`X-Delete-After`—and it's the former that must be regarded as
ontologically prior.) Thus, this commit adds an `--absolute` optional
argument to the `tempurl` subcommand; if present, the `seconds` argument
will be interpreted as a Unix timestamp of when the tempURL should be
expire, rather than a duration for which the tempURL should be valid
starting from "now".
Change-Id: If9ded96f2799800958d5063127f3de812f50ef06
All response headers are now exposed as unicode objects. Any
url-encoding is interpretted as UTF-8; if that causes decoding to fail,
the url-encoded form is returned.
As a result, deleting DLOs with unicode characters will no longer raise
UnicodeEncodeErrors under Python 2.
Related-Bug: #1431866
Change-Id: Idb111c5bf3ac1f5ccfa724b3f4ede8f37d5bfac4
Some improvements to the tests for staggered download
that were added in [1].
[1] Ie737cbb7f8b1fa8a79bbb88914730b05aa7f2906
Change-Id: Ib999bc7bd198c1d9c217c57501f751e854d4c6ad
This patch adds exception logging to the swift service API. Each
operation that results in failure of any operation will now log
the exception as well as report a timestamp and full stack trace
in the results returned by the service API calls.
Change-Id: I7336b28354e7740ea7d048bdf355e3c1a1b4436c
When a caller passes an os_options dict to the Connection class
constructor, the constructor may modify the os_options dict,
which can surprise the caller if they re-use the os_options
dict. Specifically the os_options tenant_name and object_storage_url
may be modified, and the changed values would then leak through to a
subsequent Connection constructed using the same os_options dict.
This fix simply constructs a new dict from the supplied os_options.
The patch also adds a test that covers this and also verifies that
a preauth_url passed as a keyword arg to Connection() will take
precedence over any object_storage_url in an os_options parameter.
Closes-Bug: 1488070
Change-Id: Ic6b5cf3ac68c505de155619f2610be9529e15432
Add client changes to allow accessing alternative
reseller_prefixes via a service token.
ie client changes for this server side spec:
https://review.openstack.org/#/c/105228
We assume that the service storage url has been passed in
as a preauthurl.
We rely on get_auth preserving this url.
Change-Id: I1cfda178f0b6c8add46cfebd6bf38440caae2036
fix against H234: assertEquals() logs a DeprecationWarning
in Python3.x. use assertEqual() instead.
Closes-bug: #1480776
Change-Id: Iffda6bb5f2616d4af4567eeea37bb26531e34371
Signed-off-by: Hiroshi Miura <miurahr@nttdata.co.jp>
When a Connection is first __init__ialized (without providing a
preauthurl or preauthtoken), the url and token attributes are None; they
get set (to be reused on future requests) after one of the wrapper
methods internally using _retry (head_account, get_container,
put_object, and similar friends) is called. However, this had not been
the case for get_auth, much to the momentary confusion and
disappointment of programmers using swiftclient who expected to be able
to get the token or storage URL off the Connection object after calling
get_auth (perhaps in order to make an unusual kind of request that
swiftclient doesn't already have a function for). This commit makes
get_auth set the url and token attributes as one might expect.
Change-Id: I0d9593eb5b072c8e3fa84f7d5a4c948c0bc6037a
The current code builds a full object listing before performing either a multiple
download or delete operation (and also shuffles this complete list in the case of
a download). This patch removes the creation of the full object list and adds the
ability to turn off shuffle for files when downloading. Also added is a limit on
the number of list results that can be queued by a single call to service.list
without consuming results (reduces memory overhead for large listings).
Some tests added for service.py download and list.
Change-Id: Ie737cbb7f8b1fa8a79bbb88914730b05aa7f2906
This patch adds "--output-dir" and "--remove-prefix" options
to the "download" command and unit tests for it.
Example:
$ swift list example --prefix swift2.2
swift2.2/bin/swift-object-auditor
swift2.2/bin/swift-object-expirer
swift2.2/bin/swift-object-info
swift2.2/bin/swift-object-replicator
swift2.2/bin/swift-object-server
swift2.2/bin/swift-object-updater
When given "--output-dir <directory>", client downloads objects to <directory>.
$ swift download example --prefix swift2.2 \
--output-dir new/swift/dir
The folder structure:
.
└── new
└── swift
└── dir
└── swift2.2
└── bin
├── swift-object-auditor
├── swift-object-expirer
├── swift-object-info
├── swift-object-replicator
├── swift-object-server
└── swift-object-updater
When given "--remove-prefix", client downloads objects without <prefix>.
$ swift download example --prefix swift2.2 \
--remove-prefix \
--output-dir swift
The folder structure:
.
└── swift
└── bin
├── swift-object-auditor
├── swift-object-expirer
├── swift-object-info
├── swift-object-replicator
├── swift-object-server
└── swift-object-updater
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I7463fe2941cc94f9a50a4756a97c2ccdf946294d
Implements: blueprint swiftclient-download-pseudo-folder-to-specific-target
There was always an implicit assumption that they were UTF-8 before, and
by converting them to unicode we close another hole allowing raw bytes
to appear in user-facing messages.
Closes-Bug: #1431866
Change-Id: If2e41d9a592c3ad02818e9c6f0959fd4b73cd0e0
Previously we were trying to test writing bytes in Python 3 using only
native (unicode) string objects. That doesn't test what we thought we
were testing.
Change-Id: I10a0a38143d7f7d850ab9a7005ad87f5d314c375
Currently SwiftClient populate response dictionary before
removing "./" or "/" at begining of object name. This patch
fixes that by changing that order.
Closes-bug: #1412425
Change-Id: I80222754caba5d42a468f4677ac539e46682dd31
