The documentation of "swift download" hints that "marker" option
is supported, but in reality we forgot to patch it through, so
all downloads were always done with the default, empty marker.
Closes-Bug: #1565393
Change-Id: I38bd29d2baa9188b61397dec75ce1d864041653c
Previously, we were using a content-type of text/directory, but that is
already defined in RFC 2425 and doesn't reflect our usage:
The text/directory Content-Type is defined for holding a variety
of directory information, for example, name, or email address,
or logo.
(From there it goes on to describe a superset of the vCard format
defined in RFC 2426.)
application/directory, on the other hand, is used by Static Web [1] and
is used by cloudfuse [2]. Seems like as sane a choice as any to
standardize on.
[1] https://github.com/openstack/swift/blob/2.5.0/swift/common/middleware/staticweb.py#L71-L75
[2] https://github.com/redbo/cloudfuse/blob/1.0/README#L105-L106
Change-Id: I19e30484270886292d83f50e7ee997b6e1623ec7
...using a new "application/swiftclient-segment" content-type.
Segments uploaded by swiftclient are expected to have a many-to-one
relationship to large objects, rather than the more-general many-to-many
relationship that SLO and DLO generally allow. Later, we may use this
information to make more intelligent decisions, such as when to
automatically clean up segments.
Change-Id: Ie56a3aa10065db754ac572cc37d93f2c901aac60
SwiftService().post(cont, [SwiftPostObject(obj, options]) currently
ignores options['header'], raises exception when options['headers']
is set and make malformed metadata when options['meta'] is set.
Fix tipos in code, add unittest for SwiftService().post
Closes-Bug: #1560052
Change-Id: Ie460f753492e9b73836b4adfc7c9c0f2130a8a91
in python-swiftclient/swiftclient/service.py,
there is a method
def download(self, container=None, objects=None, options=None):
if container is specified but objects not, it download all
objects in specified container.
if both container and objects are specified, it download all
specified objects in the container.
when it comes to the case that, objects argument is specified,
but it turned out to be empty array [ ], the download method
download all the objects under specified container.
this may be not reasonable.
for example,
the objects was not empty when it came from command line,
but it's filtered, maybe by --prefix argument.
at last, it turned out to be empty array.
when calling download method with objects arguments
being empty array, we should download nothing instead of
all the objects under the specified container.
Change-Id: I81aab935533a50b40679c8b3575f298c285233a8
Closes-bug: #1549881
My understanding is that it was mainly being used so we could have sane
testing on py26. With py26 support being dropped, we no longer need it.
Also drop discover from test-requirements.txt, as we don't seem to
actually use it.
Change-Id: Iee04c42890596d3b483c1473169480a3ae19aac8
Related-Change: I37116731db11449d0c374a6a83a3a43789a19d5f
This fix ensures creation and use of the correct default segment
container when pseudo-folder paths are passed via <container> arg.
Change-Id: I90356b041dc9dfbd55eb341271975621759476b9
Closes-Bug: 1532981
Related-Bug: 1478210
New documentation for python-swiftclient that introduces
the APIs available and gives some opinionated advice about
when to use the shell, the client API and the service API.
Change-Id: I19020f041fab2e72469979f712ffe3951c431d24
* Always use testtools.TestCase, since we're relying on testtools
* Always use mock (as opposed to unittest.mock) since we're relying on
mock
* Add note about when a missing logging handler was added
* Stop %-formatting the giant usage string that doesn't actually need
any formatting
* Prefer assertIs, assertIn, assertIsInstance over assertTrue
* Use else-self.fail instead of sentinel values
* Check resp.get('error') is None before checking resp['success'] is
True, so test failures actually tell you something useful
* Tighten some isinstance assertions
* Import MockHttpTest from correct location
* Only populate clean_os_environ once
* Use setUp for setup, not __init__
* Replace assertIn(key, dict) and assertEqual(foo, dict[key]) with
assertEqual(foo, dict.get(key)) when key is a literal and foo is not
None
* Use mock.patch.object instead of manually patching for tests
* Use six.binary_type instead of type(''.encode('utf-8'))
* Stop shadowing builtin bytes
* Reclaim some margin
* Stop checking the return-type of encode_utf8; we already know it's
bytes
Change-Id: I2138ea553378ce88810b7353147c8645a8f8c90e
Some improvements to the tests for staggered download
that were added in [1].
[1] Ie737cbb7f8b1fa8a79bbb88914730b05aa7f2906
Change-Id: Ib999bc7bd198c1d9c217c57501f751e854d4c6ad
This patch adds exception logging to the swift service API. Each
operation that results in failure of any operation will now log
the exception as well as report a timestamp and full stack trace
in the results returned by the service API calls.
Change-Id: I7336b28354e7740ea7d048bdf355e3c1a1b4436c
The current code builds a full object listing before performing either a multiple
download or delete operation (and also shuffles this complete list in the case of
a download). This patch removes the creation of the full object list and adds the
ability to turn off shuffle for files when downloading. Also added is a limit on
the number of list results that can be queued by a single call to service.list
without consuming results (reduces memory overhead for large listings).
Some tests added for service.py download and list.
Change-Id: Ie737cbb7f8b1fa8a79bbb88914730b05aa7f2906
This patch adds "--output-dir" and "--remove-prefix" options
to the "download" command and unit tests for it.
Example:
$ swift list example --prefix swift2.2
swift2.2/bin/swift-object-auditor
swift2.2/bin/swift-object-expirer
swift2.2/bin/swift-object-info
swift2.2/bin/swift-object-replicator
swift2.2/bin/swift-object-server
swift2.2/bin/swift-object-updater
When given "--output-dir <directory>", client downloads objects to <directory>.
$ swift download example --prefix swift2.2 \
--output-dir new/swift/dir
The folder structure:
.
└── new
└── swift
└── dir
└── swift2.2
└── bin
├── swift-object-auditor
├── swift-object-expirer
├── swift-object-info
├── swift-object-replicator
├── swift-object-server
└── swift-object-updater
When given "--remove-prefix", client downloads objects without <prefix>.
$ swift download example --prefix swift2.2 \
--remove-prefix \
--output-dir swift
The folder structure:
.
└── swift
└── bin
├── swift-object-auditor
├── swift-object-expirer
├── swift-object-info
├── swift-object-replicator
├── swift-object-server
└── swift-object-updater
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I7463fe2941cc94f9a50a4756a97c2ccdf946294d
Implements: blueprint swiftclient-download-pseudo-folder-to-specific-target
Currently SwiftClient populate response dictionary before
removing "./" or "/" at begining of object name. This patch
fixes that by changing that order.
Closes-bug: #1412425
Change-Id: I80222754caba5d42a468f4677ac539e46682dd31
This allows the use of --skip-identical when downloading SLOs and DLOs,
while still only requiring one request for normal objects.
Change-Id: I237ec5a8cc4d1119b6d3f166db81a977a50a4e4e
Related-Bug: #1379252
In addition to removing an unnecessary dependency, this closes another
hole that was allowing raw bytes to appear in user-facing messages.
Change-Id: Ia0b76426a38e5a5c368c4c7e7ba2aef286758aca
Previously, we compared the ETag from Swift against the MD5 of the
entire large object. However, the ETag for large objects is generally
the MD5 of the concatenation of the ETags for each segment, unless the
object is a DLO whose segments span more than one page of a container
listing. Rather than worry about ETags, just compare each chunk of the
segmented file. This allows the use of --skip-identical when uploading
SLOs and DLOs.
Additionally, there are several test-related improvements:
* The default arguments for OutputManager are now evaluated on
construction, rather than on definition, so that
TestOutputManager.test_instantiation will succeed when using nosetest
as a test runner. (See also: bug 1251507)
* An account_username option is now available in the functional tests
config file for auth systems that do not follow the account:username
format.
* CaptureOutput no longer writes to the captured stream, and
MockHttpTest now captures output. These were polluting test output
unnecessarily. (See also: bug 1201376)
Change-Id: Ic484e9a0c186c9283c4012c6a2fa77b96b8edf8a
Closes-Bug: #1201376
Closes-Bug: #1379252
Related-Bug: #1251507
With MD5Sum checking being added, a concern was brought up that It was
a change with no possibility of reverting to the old behaviour.
This change adds the flag '--ignore-checksum' to the upload subcommand
allowing the checks to be turned off.
Changed occurrences of the magic string for a null md5 to use a descriptive
constant instead.
Updated Error messages generated when validation fails. They should now be more descriptive
and not output a literal newline sequence.
Change-Id: Id1756cbb6700bb7e38f0ee0e75bc535e37f777ed
Changed existing code to calculate the MD5 of the object
during the upload stream. Checks this MD5 against the etag
returned in the response.
An exception is raised if they do not match.
Closes-Bug: 1379263
Change-Id: I6c8bc1366dfb591a26d934a30cd21c9e6b9a04ce
This patch fixes downloading files to stdout and modifies
_SwiftReader to operate as an iterator that performs file
checks at the end of iteration as well as a context manager.
File verification checks have been removed from __exit__
and added to __iter__.
Change-Id: I3250bdeeef8484a9122c4b5b854756a7c8f8731e
Closes-Bug: 1395922
Closes-Bug: 1387376
I'm not sure how test_process_options_defaults ever passed for anyone that had
all the ST_(AUTH|USER|KEY) set in their environment when they ran it. Despite
our efforts to patch os.environ swiftclient.servce was acctually importing
environ from os and then sticking the results into some global state. The
tests seemed to work fine on infra, but for me locally until I `unset ST_AUTH`
the test would always fail. Seems to work now and looks fairly reasonable on
the surface.
Change-Id: Id68590d6af16040a2877e719bc38d1e126603e42
Check for None was being done after a method call which
caused an attribute error if the value was None.
The Method call was also a mistake and has been corrected to the
hasattr function like intended.
Added Tests.
Closes-Bug: 1392651
Change-Id: Ifb1c84e26533bccbaddcce5738f434f36feca74e
The response dictionary in _delete_segment attach's any error
it encounters to the dict key 'exception', all other response
dict's use 'error' to store the exception.
Changed to make it consistent and added tests
Also added a third branch in st_delete as messages
created in _delete_container where being silently dropped.
Change-Id: Ifbc3b1fae78910fbc6acf4a86cfb0f60bb1aa336
actual_md5 was being used instead of _actual_md5 only when assigning
the md5 object. This resulted in all checks and updates of the md5
to be skipped.
When fixed it revealed another bug that the md5 of the read content
is always checked even if there is no _expected_etag.
Added tests
Change-Id: Iaf1e21324ce592049d02cd5be123604b99833e86
