Commit Graph

99 Commits (3b21157a844be5b71fba2216486c3ef412e7ae1a)

Author SHA1 Message Date
Christian Schwede 0a5a88b9e6 Make bin/swift testable part 2
Follow up for patch

Change-Id: I6801b75cb78a3da0631ca8a60b7c2756b8930e72
2014-03-25 11:36:39 +00:00
Christian Schwede 100419c0b3 Make bin/swift testable part 1
To make bin/swift importable for testing it needs to be renamed/moved.
The final idea is to move it to swiftclient/ and add only a stub
in bin/swift to import swiftclient/

However, doing this in a single step makes reviewing very hard, because
now there are > 1400 lines deleted from bin/swift and added to
swiftclient/ - Gerrit doesn't detect the moved file.

For example:

This patch first moves the code to swiftclient/ and uses to create the stub file in bin/swift. A follow up
( will then add the stub itself
in bin/swift (and remove the entry in

The change to tox.ini is related to bug 1290562 and can be removed in
the future.

Change-Id: Id86392ccaf09e1d80a5d808fb49827d6d18c2588
2014-03-25 11:32:51 +00:00
Andreas Jaeger 3dd9eddeb4 Improve help strings
Make help strings more consistent:
* Fix wording and grammar, remove duplicated words
* Add "." consistently everywhere.

Change-Id: I37f94600df2a58532633f6bab2ab82fc5e91a75b
2014-03-10 18:17:21 +01:00
Clay Gerrard 07dd668082 add "info" as an alias to "capabilities"
Change-Id: Ia07554250a6aae8f3a6be6a70f83690ec3fc108c
2014-03-06 13:01:49 -08:00
Luis de Bethencourt 25a0e63619 Help string format persistent
Keep the format of these strings constant through all options and commands.

Change-Id: I67a47bdd5681b4b221baf55933a175568aa8e16d
2014-02-26 15:52:35 -05:00
Luis de Bethencourt f2cd140ccd Make the help strings constant
Have all the option help strinsg match the usage text for that command.

Change-Id: I575a71769600f95c33cc4bbc9904b313f890e997
2014-02-26 15:52:35 -05:00
Jenkins 21cec82ff1 Merge "Add missing backslash." 2014-02-18 06:24:59 +00:00
Jenkins 0aa0f4f6cc Merge "Add option to skip downloading/uploading identical files" 2014-02-18 01:45:54 +00:00
Jenkins 19d7e1812a Merge "Port to python-requests" 2014-02-14 00:41:39 +00:00
tanlin c3107cd665 Rename Openstack to OpenStack
Change-Id: I091b1d2de41ff59b21ce136738e2f69b759d682b
2014-02-13 16:45:38 +08:00
Tristan Cacqueray b182112719 Port to python-requests
Currently, httplib implementation does not support SSL certificate
verification. This patch fixes this. Note that ssl compression parameter
and 100-continue thing is still missing from requests, though those are
lower priority.

Requests now takes care of:
* proxy configuration (get_environ_proxies),
* chunked encoding (with data generator),
* bulk uploading (with files dictionary),
* SSL certificate verification (with 'insecure' and 'cacert' parameter).

This patch have been tested with requests 1.1.0 (CentOS 6) and requests
2.2.1 (current version).

Change-Id: Ib5de962f4102d57c71ad85fd81a615362ef175dc
Closes-Bug: #1199783
2014-02-12 13:21:26 +01:00
Christian Schwede 9b3ec3705f Add option to skip downloading/uploading identical files
swift has already the upload option "--changed" to upload only changed
files. This patch adds a similar feature by comparing the md5 of the
local file to the remote object etag.

When used in combination with download the MD5 hexdigest of each file
is sent with an "If-None-Match" header to skip downloading identical files.

When used in combination with upload the MD5 is compared to the remote
etag by using the already existing HEAD request.

Change-Id: I727b0456558c6a7742b2428c6d1c45c4bfaf66e9
2014-02-11 09:13:26 +00:00
Christian Schwede 545ea2e032 Fix swiftclient help
Command help was wrong in some cases. While we're at it, add some
defaults to help and change some brackets to be consistent with

Change-Id: I5d7b2a703294d97fc5e3de43b9ec375c3ea55a6e
2014-02-07 13:48:51 +00:00
Shane Wang 0d3f4a0aa2 Fix misspellings in python swiftclient
Fix misspellings detected by:
* pip install misspellings
* git ls-files | grep -v locale | misspellings -f -

Change-Id: I2fb042b3d9eb59cd404c52c775e5cc870e6157e1
Closes-Bug: #1257295
2014-02-07 13:58:17 +08:00
Chmouel Boudjnah 504e871967 Add missing backslash.
The one character commit of the day that needed two iterations.

Change-Id: I3bfe91c7e164d45d03348b18e0a457314b8baec7
2014-02-05 12:15:15 +01:00
Fabien Boucher 533c9c5ba1 Add capabilities option
This patch adds a capabilities option on swiftclient.
This option uses the new /info endpoint to request the
remote capabilities and nicely display it.

Change-Id: Ie34b454511d5527e402e66e1fdb72120f427f2fd
2014-01-17 10:26:34 +01:00
Jenkins 20cd3402b2 Merge "Add --object-name" 2013-12-25 11:20:10 +00:00
Taurus Cheung 26ecec0f6d Add --object-name
Add parameter --object-name, which:
1) Sets target object name when upload single file
2) Sets object prefix when upload a directory

Change-Id: Idc4357c3355e66d31c100540b901e70db20b03c3
Closes-Bug: 1012979
2013-12-24 10:27:17 +08:00
Sahid Orentino Ferdjaoui 222f283b22 Fix help of some optional arguments
Some positional arguments are optional but the help indicate
they are mendatory and can be confuse for an user.

  swift delete <container> [object]
  swift post <container> [object]

Change-Id: I7be525905b51d3ef160b0f47194e817b5cba0f9b
2013-12-18 10:12:22 +00:00
jola-mirecka 9b7e24c449 Fix download bandwidth for swift command.
We noticed a difference in download bandwidth while using
authentication details and pre-auth information for download.
Download using authentication details included authentication
time into the download bandwidth at each call of the function.
This time should have been excluded from download bandwidth.

This patch set is adding a timer variable to the client library.
That allows to accurately measure and store the time of each action.
Then while printing the download bandwith in the swift command
authentication time is excluded from bandwidth.

Change-Id: I63df9023e169f637f120ae9e25dac9c90a4e75a0
Fixes: bug #1197443
2013-11-28 13:02:54 -05:00
Jenkins a253b5cecd Merge "Allow custom headers when using swift download (CLI)" 2013-11-27 10:16:55 +00:00
Jenkins 04e0cb2783 Merge "Add verbose output to all stat commands" 2013-11-21 08:50:33 +00:00
Matthieu Huin 3cabda8c48 Allow custom headers when using swift download (CLI)
This fixes bug #1051046

A repeatable option, --header or -H, is added so a user can specify
custom headers such as Range or If-Modified-Since when downloading
an object with the swift CLI.

Change-Id: I1f7dcf64cf625f2e5a4488c210894cfe6e0d5974
2013-11-04 17:41:55 +01:00
Romain Hardouin 4b78b4d136 Fix a typo in help text: "downlad"
Change-Id: Icd6cf75f3c5758c13118060c19c9c000e28ef3ae
Closes-Bug: #1245082
2013-10-27 01:41:04 +02:00
Jenkins 775a24b1bc Merge "enhance swiftclient logging" 2013-10-18 21:43:05 +00:00
Leah Klearman b682935055 enhance swiftclient logging
* log to debug for successes and info for errors
* remove dead code, neither body or raw_body are sent in kwargs
* include resp.reason and response headers in logging
* remove trailing \n in logging. users who want them can set logging.basicConfig(format=FORMAT)
* add --info option to swift cli

Change-Id: If07af46cb377f3f3d70f6c4284037241d360a8b7
2013-10-17 16:05:36 -07:00
Jenkins 1f2717e334 Merge "Extend usage message for `swift download`" 2013-10-17 17:42:42 +00:00
Jenkins efde2378c2 Merge "user defined headers added to swift post queries" 2013-10-17 16:59:03 +00:00
Pete Zaitcev e08ec3ab8a Clarify main help for post subcommand
Fixes bug 1231321

Change-Id: I7014aa160ebe5ba0bc5a91de05e6858fc58f69e0
2013-10-14 15:04:40 -06:00
Samuel Merritt 7d61c54399 Make pbr only a build-time dependency.
This lets you build python-swiftclient packages that don't require pbr
to be installed at all. You would need pbr on the machine running
rpmbuild / debuild, but not on the machines that install the packages.

Unfortunately, this does not make python-swiftclient able to be
installed via pip 0.3.1 on Lucid; you'll need to uninstall the system
python-pip package and install a new pip some other way. Given that
pip < 1.3 doesn't perform SSL certificate validation for pypi (trivial
MITM attack, anyone?), you'd probably want to get a new pip anyway.

Change-Id: I85d4d77aacf094e48d39e48e750594b95dbc7af0
2013-10-10 11:57:47 -07:00
Clay Gerrard d687060a44 Add verbose output to all stat commands
When you stat a container or object with the verbose flag the full path of the
reousrce will be displayed with the token similarlly to how an account stat
displays the auth url and token.

 * move some logic out of bin/swift.st_stat to test it
 * new module swiftclient.commnad_helpers for code you want to test
 * moved prt_bytes into swiftclient.utils to test it
 * fixed IndexError with prt_bytes on sizes >= 1024Y

Change-Id: Iaaa96e0308b08c554205b0055b8a04de581fefa4
2013-10-09 14:31:47 -07:00
Matthieu Huin afa1f9f76d user defined headers added to swift post queries
This commit fixes a problem where headers passed through the option
--header were not taken into account when doing "post" queries.

Change-Id: Ia34808c53a1816d9956a98e0803d8c4d20d2ad44
Closes-Bug: #1226256
2013-09-27 11:40:34 +02:00
David Goetz 266217e32c fix bug with replace old *LOs
Change-Id: I9e6278b62608cf36fa7e5e228f234cc8199c17fc
2013-09-10 13:12:29 -07:00
Alexis Lee 47c33cc4a3 Extend usage message for `swift download`
Make it clear that you do not have to provide either the container or
object positional parameters, and what swiftclient will do if you omit

Change-Id: I3766fc1eabd51f197e2175771af579577a96c192
2013-09-06 14:57:34 +01:00
Stanislaw Pitucha a47d3d6468 Remove redundant unit suffix
Transfer speed already has a "MB/s" unit. The "s" suffix was probably a
copy/paste mistake.

Change-Id: Iccd7894d5ca11aac8a9f98365834c5d3deef0720
2013-08-28 16:26:28 +01:00
Jakub Krajcovic b0eb857b69 Reformat help outputs
Reformatted the help outputs in the swift client to bring them more
in line with the rest of the openstack clients. Changes were added to
all of the st_command_help variables and a new variable
st_command_options was added to help with correct formatting of
help outputs.

Change-Id: I0c549cf96292384ddb36b43ecec41a883d8b2e13
Implements: changes to help outputs
2013-08-19 20:45:00 +10:00
Darrell Bishop 9198e95468 Move multi-threading code to a library.
This patch extracts the multi-threading code from bin/swift into
swiftclient/multithreading and adds tests.  In particular, this new way
of doing it (with context managers) will prevent non-daemonic threads
from wedging the process when unexpected exceptions happen.

I enabled reporting of which lines, specifically, are not covered by
unit tests (added -m option to "coverage report" in .unittests).

This patch includes a drive-by fix for uploading a segmented file with
--use-slo when that object already exists.  A key of "name" was used
instead of "path", raising KeyError.

There's also another drive-by fix for uploading segmented objects with
--use-slo.  Commit 874e0e4427 regressed
this by removing the capturing of thread-worker results in  This patch restores that functionality and
the feature (uploading SLO objects).

Change-Id: I0b4f677e4a734e83d1a25088d9a74f7d46384e53
2013-07-28 22:08:17 -07:00
Jenkins 65e5ef7b81 Merge "Allow setting # of retries in the binary" 2013-07-12 15:17:03 +00:00
Jenkins c460ebff0a Merge "Allow storage url override for both auth vers." 2013-07-10 07:06:08 +00:00
Ben McCann 188e835ead Allow setting # of retries in the binary
Change-Id: I7ef4f1f86cc6ea4937043ea7bf7d9526e23bd2c7
2013-07-08 17:09:44 -07:00
Kun Huang 94d7c23181 Note '-V 2' is necessary for auth 2.0
Here are some questions in Launchpad, just because they miss '-V 2' for
auth 2.0.
So I add somethins for noteing people add the '-V 2'

Change-Id: I2e0fef969be4eca316975de03fdbc516a1069acf
2013-07-07 16:22:01 +08:00
Darrell Bishop 1d4d51b218 Allow storage url override for both auth vers.
When --os-storage-url is specified on the command-line to bin/swift, it
will override the used storage URL regardless of authentication for both
authentication version 1 and version 2.  This can be used to bypass a
load-balancer to hit a specific proxy server for testing/debugging

Within the client library, this feature is accessed by passing the
desired storage URL into swiftclient.client.Conection.__init__() via the
os_options keyword argument.  For example:

  conn = Connection(auth_url, user, key, os_options={
    'object_storage_url': ''})

This patch also adds a dependency on mock>=0.8.0, which is the same as

Change-Id: Id2a36ed6abffd65e7762b6beea5bbfc6c036e848
2013-07-05 14:12:59 -05:00
Jenkins 6f7458a290 Merge "Add -p option to download command." 2013-07-05 17:16:34 +00:00
Darrell Bishop f022aac0cf Add -p option to download command.
Allow the ability to download a subset of containers (--all with -p) or
a subset of objects within a container (container name with -p).

This patch also includes a drive-by fix for "download --all" which would
not actually download any objects (for me, at least) because the object
queue got filled with "stop" messages before the container workers had
run long enough to put work in the object queue.  Doh!

I also closed up a few holes where an (unexpected, obviously) Exception
could cause the process to hang because non-daemon threads still

Change-Id: I71c6935c60282b5353badc2dfce8a935d47e3bb7
2013-06-26 23:00:46 -07:00
Mark Seger 9f5b334f7a add -t for totals to list command and --lh to stat
Change-Id: Ieaba5e7762032bf9a2652ead39870d55e85b4672
2013-06-26 15:48:21 +00:00
Jenkins 473167ff31 Merge "Add -l and --lh switches to swift 'list' command" 2013-06-11 22:03:34 +00:00
Mark Seger 17bb3fa779 Add -l and --lh switches to swift 'list' command
Also updated man page and added an example in 'list'
command help message

Change-Id: Icf38de9bab6d387a9a1deed444a91f68b4389cfe
2013-06-10 16:49:44 +00:00
Stuart McLaren 790f087a67 Add option to disable SSL compression
Allows optionally disabling SSL compression. This can significantly
improve HTTPS upload/download performance in some cases -- in particular
when the object is not compressible and you have very high network

Implements blueprint ssl-compression.

Change-Id: I1260055f9c2e83cdabfeb51aed11b3899bed4d55
2013-06-04 16:33:18 +00:00
Tong Li 874e0e4427 remove busy-wait so that swift client won't use up all CPU cycles
The current implementation uses busy-wait and checking flags to
communicate between threads. It wastes a lot of CPU powers. With
python 2.6 is required for Swift, the communication between
threads should now be using queue and signal mechanisms. This
patch removed the busy-wait loops and use queue and queue join
for threads coordination which will not consume CPU cycles if
a thread is blocked.

Change-Id: I648cd637a92a159d5c13baa83f357cee2dfe7937
2013-05-29 19:09:49 +00:00
David Goetz adebab739d do not warn about etag for slo
Change-Id: I17fa1643f28aa1cd160c2d770243875a3c4a80ee
2013-05-13 07:37:15 -07:00