When downloading the same containers or objects with multiple
invocations of the swift command-line client, you'll get better
throughput and avoid "hot spots" if each client randomizes its download
order.
Note that the marker must be picked *before* shuffling the containers or
objects.
Change-Id: I7240eda57a80e1708c2483827c6329fd57d5fc51
When using the swift command-line tool to evaluate a Swift cluster, it
can be very handy to get some insight into the download timing. This
patch adds timing data to verbose output for the download command. For
each downloaded file, the printed line will also contain:
- The time it took to send the request and receive the header
- The total time the request took (including writing the file out
locally)
- The average throughput of the download
Change-Id: Ib4a995623af973bb1eed4fb52c8c0e5da935964d
Fix race condition in _delete_container() where all elements of
object_queue have been removed, but the last one (per thread) may not
have actually been deleted yet when the container deletion thread calls
conn.delete_container(container). Fixes bug 1032879.
Improves container deletion throughput by immediately deleting
containers with no objects instead of waiting for all pending object
deletes to complete. Fixes bug 1032878.
Change-Id: I404229a4c608995294e0ada77724ac8afe8d6f3c
- This allows us to delegate all 2.0 authentication directly to the
library without reimplementing ourselves.
- Support reusing a token / storage-url without re-authenticating every
time via the switch os_storage_url os_auth_token.
- Allow auth via tenant_id instead of just tenant_name via the switch
os_tenant_id.
- Refactor a bit to make it easier in the future to add new OS features
(i.e: region).
- Implements blueprint use-keystoneclient-for-swiftclient.
- Fixes bug 1016641.
Change-Id: I532f38a68af884de25326aaac05a2050f5ffa1c7
The 'delete', 'download', and 'upload' commands use multiple threads
for concurrency. However, the number of threads was hardcoded at
10. This patch simply makes those configurable.
For example, if I'm downloading a lot of files but I don't want to
saturate the downstream on my Internet connection, I might choose to
use only 1 or 2 threads for object downloads. Conversely, if I'm
uploading a lot of small files across a fast network, I would want
lots of threads to speed things along.
The default number of threads is 10, so the default behavior is
unchanged.
Change-Id: I64c06741b24ca97fef5ded206d7e898bf5cab3b8
The first time the user presses ctrl-c, all QueueFunctionThreads will
have abort=True set on them. This will cause them to finish the work
they're doing (e.g. finish uploading the current file) but then ignore
any further work and let the process exit. Also, a notification of
this is written to stderr so the user understands why the process
didn't exit immediately.
The second time the user presses ctrl-c, the process will exit
immediately. Any in-progress operations are abandoned.
Change-Id: Ie6927f78726ac1c7998e920cb608682ead10f25b
Before, if a QueueFunctionThread's function raised an exception, then
its thread would log the exception and exit, leaving the rest of the
jobs in the queue and ensuring that the swift client would hang.
Now, the exception is logged and processing continues, so all the
messages get handled eventually and the client exits.
Change-Id: I43d4df212847a2a85732b304de319ea2cce82ddd
