If "-" is passed in for the source, python-swiftclient will upload
the object by reading the contents of the standard input. The object
name option must be set, as well, and this cannot be used in
conjunction with other files.
This approach stores the entire contents as one object. A follow on
patch will change this behavior to upload from standard input as SLO,
unless the segment size is larger than the content size.
Change-Id: I1a8be6377de06f702e0f336a5a593408ed49be02
Currently, the swiftclient upload command passes a custom metadata
header for each object (called object-meta-mtime), whose value is
the current UNIX timestamp. When downloading such an object with the
swiftclient, the mtime header is parsed and passed as the atime and
mtime for the newly created file.
There are use-cases where this is not desired, for example when using
tmp or scratch directories in which files older than a specific date
are deleted. This commit provides a boolean option for ignoring the
mtime header.
Change-Id: If60b389aa910c6f1969b999b5d3b6d0940375686
Since time immemorial, Swift has returned unquoted ETags for plain-old
Swift objects -- I hear tell that we once tried to change this, but
quickly backed it out when some clients broke.
However, some proxies (such as nginx) apparently may force the ETag to
adhere to the RFC, which states [1]:
An entity-tag consists of an opaque *quoted* string
(emphasis mine). See the related bug for an instance of this happening.
Since we can still get the original ETag easily, we should tolerate the
more-compliant format.
[1] https://tools.ietf.org/html/rfc2616.html#section-3.11 or, if you
prefer the new ones, https://tools.ietf.org/html/rfc7232#section-2.3
Change-Id: I7cfacab3f250a9443af4b67111ef8088d37d9171
Closes-Bug: 1681529
Related-Bug: 1678976
Previously, using SwiftService to delete "many" objects would use
bulk delete if available, but it would not respect the bulk delete
page size. If the number of objects to delete exceeded the bulk delete
page size, SwiftService would ignore the error and nothing would be
deleted.
This patch changes _should_bulk_delete() to be _bulk_delete_page_size();
instead of returning a simple True/False, it returns the page size for
the bulk deleter, or 1 if objects should be deleted one at a time.
Delete SDK calls are then spread across multiple bulk DELETEs if the
requested number of objects to delete exceeds the returned page size.
Fixed the logic in _should_bulk_delete() so that if the object list
is exactly 2x the thread count, it will not bulk delete. This is the
natural conclusion following the logic that existed previously: if
the delete request can be satisfied by every worker thread doing one
or two tasks, don't bulk delete. But if it requires a worker thread
to do three or more tasks, do a bulk delete instead. Previously, the
logic would mean that if every worker thread did exactly two tasks, it
would bulk delete. This patch changes a "<" to a "<=".
Closes-Bug: 1679851
Change-Id: I3c18f89bac1170dc62187114ef06dbe721afcc2e
If we were to include this in a normal PUT, it would 400, but only if
slo is actually in the pipeline. If it's *not*, we'll create a normal
Swift object and the header sticks.
- This is really confusing for users; see the related bug.
- If slo is later enabled in the cluster, Swift starts responding 500
with a KeyError because the client and on-disk formats don't match!
Change-Id: I1d80c76af02f2ca847123349224ddc36d2a6996b
Related-Change: I986c1656658f874172860469624118cc63bff9bc
Related-Bug: #1680083
Probably the most common format for documenting arguments is
reST field lists [1]. This change updates some docstrings to
comply with the field lists syntax.
[1] http://sphinx-doc.org/domains.html#info-field-lists
Change-Id: Ic011fd3e3a8c5bafa24a3438a6ed5bb126b50e95
The opened file for upload is not closed.
This fix prevents possible file handle leak.
Closes-Bug: #1559079
Change-Id: Ibc58667789e8f54c74ae2bbd32717a45f7b30550
Previously, we only accepted iterables of strings like 'Header: Value'.
Now, we'll also accept lists of tuples like ('Header', 'Value') as well
as dictionaries like {'Header': 'Value'}.
This should be more intuitive for application developers, who are
already used to being able to pass dicts or lists of tuples to libraries
like requests.
Change-Id: I93ed2f1e8305f0168b7a4bd90c205b04730da836
Implement copy object method in swiftclient Connection, Service and CLI.
Although COPY functionality can be accomplished with 'X-Copy-From'
header in PUT request, using copy is more convenient especially when
using copy for updating object metadata non-destructively.
Closes-Bug: 1474939
Change-Id: I1338ac411f418f4adb3d06753d044a484a7f32a4
New versions of requests will raise an InvalidHeader error otherwise.
Change-Id: Idf3bcd8ac359bdda9a847bf99a78988943374134
Closes-Bug: #1614280
Closes-Bug: #1613814
The documentation of "swift download" hints that "marker" option
is supported, but in reality we forgot to patch it through, so
all downloads were always done with the default, empty marker.
Closes-Bug: #1565393
Change-Id: I38bd29d2baa9188b61397dec75ce1d864041653c
Previously, we were using a content-type of text/directory, but that is
already defined in RFC 2425 and doesn't reflect our usage:
The text/directory Content-Type is defined for holding a variety
of directory information, for example, name, or email address,
or logo.
(From there it goes on to describe a superset of the vCard format
defined in RFC 2426.)
application/directory, on the other hand, is used by Static Web [1] and
is used by cloudfuse [2]. Seems like as sane a choice as any to
standardize on.
[1] https://github.com/openstack/swift/blob/2.5.0/swift/common/middleware/staticweb.py#L71-L75
[2] https://github.com/redbo/cloudfuse/blob/1.0/README#L105-L106
Change-Id: I19e30484270886292d83f50e7ee997b6e1623ec7
...using a new "application/swiftclient-segment" content-type.
Segments uploaded by swiftclient are expected to have a many-to-one
relationship to large objects, rather than the more-general many-to-many
relationship that SLO and DLO generally allow. Later, we may use this
information to make more intelligent decisions, such as when to
automatically clean up segments.
Change-Id: Ie56a3aa10065db754ac572cc37d93f2c901aac60
SwiftService().post(cont, [SwiftPostObject(obj, options]) currently
ignores options['header'], raises exception when options['headers']
is set and make malformed metadata when options['meta'] is set.
Fix tipos in code, add unittest for SwiftService().post
Closes-Bug: #1560052
Change-Id: Ie460f753492e9b73836b4adfc7c9c0f2130a8a91
in python-swiftclient/swiftclient/service.py,
there is a method
def download(self, container=None, objects=None, options=None):
if container is specified but objects not, it download all
objects in specified container.
if both container and objects are specified, it download all
specified objects in the container.
when it comes to the case that, objects argument is specified,
but it turned out to be empty array [ ], the download method
download all the objects under specified container.
this may be not reasonable.
for example,
the objects was not empty when it came from command line,
but it's filtered, maybe by --prefix argument.
at last, it turned out to be empty array.
when calling download method with objects arguments
being empty array, we should download nothing instead of
all the objects under the specified container.
Change-Id: I81aab935533a50b40679c8b3575f298c285233a8
Closes-bug: #1549881
This fix ensures creation and use of the correct default segment
container when pseudo-folder paths are passed via <container> arg.
Change-Id: I90356b041dc9dfbd55eb341271975621759476b9
Closes-Bug: 1532981
Related-Bug: 1478210
When issuing `delete` commands that would require three or more
individual deletes, check whether the cluster supports bulk deletes
and use that if it's available.
Additionally, a new option is added to the `delete` command:
* --prefix <prefix>
Delete all objects that start with <prefix>. This is similar to the
--prefix option for the `list` command.
Example:
$ swift delete c --prefix obj_prefix/
...will delete from container "c" all objects whose name begins with
"obj_prefix/", such as "obj_prefix/foo" and "obj_prefix/bar".
Change-Id: I6b9504848d6ef562cf4f570bbcd17db4e3da8264
This fix makes it possible to upload objects to pseudo-folders by
passing the upload paths via <container> arg regardless of whether the
container or folder path exist or not.
Change-Id: I575e58aa12adcf71cdaa70d025a0ea5c63f46903
Closes-Bug: #1478210
Partial-Bug: #1432734
Related-Bug: #1432734
New documentation for python-swiftclient that introduces
the APIs available and gives some opinionated advice about
when to use the shell, the client API and the service API.
Change-Id: I19020f041fab2e72469979f712ffe3951c431d24
This commit adds the utils module to those for which Sphinx
automatically generates documentation from docstrings. (Many of the
functions here may be of little interest to users, but
`generate_temp_url`, at least, definitely deserves to be in the
documentation; in this way, this commit can be seen as a spiritual
companion to ca70dd9e.)
Also, a few markup errors and perceived infelicities in existing
docstrings are amended.
Change-Id: I8c66a23cb359d7dd9302a16459fad9825fedb690
All response headers are now exposed as unicode objects. Any
url-encoding is interpretted as UTF-8; if that causes decoding to fail,
the url-encoded form is returned.
As a result, deleting DLOs with unicode characters will no longer raise
UnicodeEncodeErrors under Python 2.
Related-Bug: #1431866
Change-Id: Idb111c5bf3ac1f5ccfa724b3f4ede8f37d5bfac4
This patch adds exception logging to the swift service API. Each
operation that results in failure of any operation will now log
the exception as well as report a timestamp and full stack trace
in the results returned by the service API calls.
Change-Id: I7336b28354e7740ea7d048bdf355e3c1a1b4436c
The current code builds a full object listing before performing either a multiple
download or delete operation (and also shuffles this complete list in the case of
a download). This patch removes the creation of the full object list and adds the
ability to turn off shuffle for files when downloading. Also added is a limit on
the number of list results that can be queued by a single call to service.list
without consuming results (reduces memory overhead for large listings).
Some tests added for service.py download and list.
Change-Id: Ie737cbb7f8b1fa8a79bbb88914730b05aa7f2906
This patch adds "--output-dir" and "--remove-prefix" options
to the "download" command and unit tests for it.
Example:
$ swift list example --prefix swift2.2
swift2.2/bin/swift-object-auditor
swift2.2/bin/swift-object-expirer
swift2.2/bin/swift-object-info
swift2.2/bin/swift-object-replicator
swift2.2/bin/swift-object-server
swift2.2/bin/swift-object-updater
When given "--output-dir <directory>", client downloads objects to <directory>.
$ swift download example --prefix swift2.2 \
--output-dir new/swift/dir
The folder structure:
.
└── new
└── swift
└── dir
└── swift2.2
└── bin
├── swift-object-auditor
├── swift-object-expirer
├── swift-object-info
├── swift-object-replicator
├── swift-object-server
└── swift-object-updater
When given "--remove-prefix", client downloads objects without <prefix>.
$ swift download example --prefix swift2.2 \
--remove-prefix \
--output-dir swift
The folder structure:
.
└── swift
└── bin
├── swift-object-auditor
├── swift-object-expirer
├── swift-object-info
├── swift-object-replicator
├── swift-object-server
└── swift-object-updater
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I7463fe2941cc94f9a50a4756a97c2ccdf946294d
Implements: blueprint swiftclient-download-pseudo-folder-to-specific-target
Currently SwiftClient populate response dictionary before
removing "./" or "/" at begining of object name. This patch
fixes that by changing that order.
Closes-bug: #1412425
Change-Id: I80222754caba5d42a468f4677ac539e46682dd31
