This patch extracts the multi-threading code from bin/swift into
swiftclient/multithreading and adds tests. In particular, this new way
of doing it (with context managers) will prevent non-daemonic threads
from wedging the process when unexpected exceptions happen.
I enabled reporting of which lines, specifically, are not covered by
unit tests (added -m option to "coverage report" in .unittests).
This patch includes a drive-by fix for uploading a segmented file with
--use-slo when that object already exists. A key of "name" was used
instead of "path", raising KeyError.
There's also another drive-by fix for uploading segmented objects with
--use-slo. Commit 874e0e4427b80e1b15b74a1557b73ba9d61443ca regressed
this by removing the capturing of thread-worker results in
QueueFunctionThread.run(). This patch restores that functionality and
the feature (uploading SLO objects).
Change-Id: I0b4f677e4a734e83d1a25088d9a74f7d46384e53
When --os-storage-url is specified on the command-line to bin/swift, it
will override the used storage URL regardless of authentication for both
authentication version 1 and version 2. This can be used to bypass a
load-balancer to hit a specific proxy server for testing/debugging
purposes.
Within the client library, this feature is accessed by passing the
desired storage URL into swiftclient.client.Conection.__init__() via the
os_options keyword argument. For example:
conn = Connection(auth_url, user, key, os_options={
'object_storage_url': 'http://overridden.storage.url/AUTH_foo'})
This patch also adds a dependency on mock>=0.8.0, which is the same as
openstack/swift.
Change-Id: Id2a36ed6abffd65e7762b6beea5bbfc6c036e848
Allow the ability to download a subset of containers (--all with -p) or
a subset of objects within a container (container name with -p).
This patch also includes a drive-by fix for "download --all" which would
not actually download any objects (for me, at least) because the object
queue got filled with "stop" messages before the container workers had
run long enough to put work in the object queue. Doh!
I also closed up a few holes where an (unexpected, obviously) Exception
could cause the process to hang because non-daemon threads still
existed.
Change-Id: I71c6935c60282b5353badc2dfce8a935d47e3bb7
Allows optionally disabling SSL compression. This can significantly
improve HTTPS upload/download performance in some cases -- in particular
when the object is not compressible and you have very high network
bandwidth.
Implements blueprint ssl-compression.
Change-Id: I1260055f9c2e83cdabfeb51aed11b3899bed4d55
The current implementation uses busy-wait and checking flags to
communicate between threads. It wastes a lot of CPU powers. With
python 2.6 is required for Swift, the communication between
threads should now be using queue and signal mechanisms. This
patch removed the busy-wait loops and use queue and queue join
for threads coordination which will not consume CPU cycles if
a thread is blocked.
Change-Id: I648cd637a92a159d5c13baa83f357cee2dfe7937
Some mechancical replacement of the deprecated except x,y:
construct with except x as y, which works with any Python
version >= 2.6
Change-Id: Ic245049dc7b408a5c89b9e27dfd2bd7c08acc5b5
Add to the help strings to clarify the need for --os-auth-token to
also have --os-storage-url supplied.
Change-Id: I6a7259c38ce01813ae96f307efaf46e7863bd223
Also fixed bug with current large objects with segment listing prefixes.
Allow retry for object PUTs when possible.
Change-Id: I0edff127fd5d5c53da33aa7cb76a4f4dc85bf6e6
Currently swift client does not allow content type to be specified
when upload an object. It does not allow a user to change content
type neither. This patch will allow a user to specify content
type by using upload command or modify the content type by using
post command.
For example:
1. To upload an object with specific content type:
swift upload -H content-type:applicaiton/xml con1 file1
2. To modify an object to a specific content type:
swift post -H content-type:application/josn con1 file1
This patch also allow a client to specify other request headers.
Change-Id: I12db83a1d465285e0906889cc67dfaa44e059568
Fixes: bug #1154621
Fix bug 1110924
Add segment container option as part of 'swift upload -S'. If this
option is omitted, the segments will go to <container>_segments.
Example:
$ swift upload c1 obj1 -S 1 -C user_segments
Please check swift upload --help for more details
Change-Id: Ib71aae322485d8d3ac89916d37ebcac053f49e3b
Add env[SWIFTCLIENT_INSECURE] as default of --insecure option.
If set to 'true', allow to access insecure keystone server.
The name follows 'NOVACLIENT_INSECURE' in novaclient.
Change-Id: I322674eba9c07e6def97bce339815fa15191a92d
- Remove PYTHON_SWIFTCLIENT and use --debug making it more consistents
with other openstack clients libraries.
- Fix printing the curl command.
- Don't show the body of the GET objects it could get bad on large
objects and it's an iterator anyway.
- Use -I for showing HEADs (and not -X HEAD).
Change-Id: I954e7d3d795401fc3679725440dec36cdc80af87
Add support to specify a ca certificate bundle to verify
keystone TLS (https) certificates. This only verifies
certificates on the keystone connection, swift https
connections are unchanged.
Change-Id: I14351b405af4fd3d1970ba6656c1282a5d0a1082
If enable this option , swift CLI is allowed to access a keystone
server with self signed certificate.
Change-Id: I5e219fe875b246b68ac51a077e7ff15e95463adf
Adding a trailing slash to the authentication URL may interfere with
auth service implementations not using /FOO/ but /FOO as authentication
URL.
OpenStack Object Storage API 1.0 doesn't specify that the auth URL must
end in a slash and swiftclient.client doesn't force it either.
Change-Id: I4e6b6d758d5ccc884e253880513e3d8763ffc2ff
Fixes: bug #1085827
str(float) isn't a good way of converting a float to a string with all the precision
Eg
>>> f = os.path.getmtime("z")
>>> f
1347717491.343554
>>> str(f)
'1347717491.34'
>>> "%f" % f
'1347717491.343554'
Change-Id: I6eb02f7f6730eff814c348d1039eae4606642b52
Add --os-region-name (and OS_REGION_NAME env) to bin/swift
Add region_name to the os_options in Connection class.
bug 1019832
Change-Id: Id8515d97e5351638dce10581c7368f61518e1fa7
This patch adds a "--no-download" option to the "download" command.
When given, all writing to disk is bypassed, while still actually
downloading the data and validating etags.
This can be handy when you're using the swift command-line client to
test out a swift cluster and don't want client-side disk writing to be a
bottle-neck (but you still want to know about any etag validation
failures).
Change-Id: I0a511f473a64820161d1eb529b995900742794f2
This changes every command-line option with a '_' in its name
and changes them to '-'. The old option names are maintained
for backward compatibility but are no longer in the help text.
BP command-options
Change-Id: I79b3c03f59ce8e253aa0dcbf0c2ed5a56d71cd0c
Description: The swift command tool will set the auth version
to 2 if OS_AUTH_URL is set even use -V 1 option to set Version
to 1.So when use nova/glance client and swift client in the
same shell, and export environment OS_AUTH_URL, it will lead
swift client to raise 400 error if swift not use keystone
for auth.
Fixes bug 1034158
Change-Id: I8003ff2ad4ac25fd710f87c4dab1507f6040ed3d
When downloading the same containers or objects with multiple
invocations of the swift command-line client, you'll get better
throughput and avoid "hot spots" if each client randomizes its download
order.
Note that the marker must be picked *before* shuffling the containers or
objects.
Change-Id: I7240eda57a80e1708c2483827c6329fd57d5fc51
When using the swift command-line tool to evaluate a Swift cluster, it
can be very handy to get some insight into the download timing. This
patch adds timing data to verbose output for the download command. For
each downloaded file, the printed line will also contain:
- The time it took to send the request and receive the header
- The total time the request took (including writing the file out
locally)
- The average throughput of the download
Change-Id: Ib4a995623af973bb1eed4fb52c8c0e5da935964d
