Replace URLs for workflow documentation to appropriate parts of the
OpenStack Project Infrastructure Manual.
Change-Id: I7513a141c95f3126ad348ed679b2baa335bdf45f
This change adds some new classes to test.unit.utils which wrap up the
collection and mocking of the std err and std out. This will make the
testing code easier to write and maintain.
Change-Id: I50ad2a736b2bb550ab83f6a43fb5a0fb5393573e
actual_md5 was being used instead of _actual_md5 only when assigning
the md5 object. This resulted in all checks and updates of the md5
to be skipped.
When fixed it revealed another bug that the md5 of the read content
is always checked even if there is no _expected_etag.
Added tests
Change-Id: Iaf1e21324ce592049d02cd5be123604b99833e86
swiftclient.shell.st_post was ignoring the result dict
returned from SwiftService.post and therefore giving no
indication when posts failed e.g. due to invalid auth
credentials.
This patch ensures that SwiftService always returns a
result_dict from post() and then checks the result dict.
On failure the shell now outputs the ClientException
message and exits with error code 1.
Also adds unit tests and cleans up some unnecessary mocked
return values in existing tests.
Closes-Bug: 1383243
Change-Id: I111150eb3d026c8d01c2cac13d3613ca7304e5b9
Changed the message shown when a user doesn't enter project/tenant
id/name to be more informative.
When attempting to stat a container without supplying project/tenant
name or id, an empty response was being returned instead of an error
being raised.
Changed the error raised in swiftclient.client when no tenant or project
is specified to be more specific.
Add tests for basic regression checking.
Closes-Bug: #1372589
Change-Id: I4eb6964d9f1702db119cc0294edc02841b1ecd5f
Some client.Connection methods will raise a KeyError
if a response_dict argument is passed and an error
occurs during authentication or making the request.
The fix is straightforward: add a test for existence
of a response_dict before attempting to get it from
kwargs.
The bulk of this patch is adding unit tests for the
response_dict feature.
Closes-Bug: 1381304
Change-Id: Ic7e1b3dfae33909533931c52ac97355867a08a07
SysExit is raised if the OutputManager has non-zero
error_count. However, error_count is incremented in
a print thread, but tested on the main thread.
Currently error_count is tested before the OutputManager
print threads have exited, which results in a race. This race
means that SysExit is not always raised even when an error
has occured.
This change moves the error_count test after the
OutputManager context manager has exited, which ensures that
the OutputManager threads have exited.
Change-Id: I5ef7d9f10057fe2b41f48ab95066a7265b86a3ac
Closes-Bug: #1379229
During a segmented object upload, attempts are made
to create the container for the manifest and the
segment container. These jobs are currently placed
on separate threads and can occur out of order which
causes the unit test failure seen in the bug report.
The container jobs should be ordered because the
segment container job attempts to HEAD the manifest
container to determine correct storage policy.
This patch enforces ordering of the two jobs and
modifies the unit test to assert that both container
PUTs occur in the expected order.
Change-Id: If90bec126867a4be2df34067ccefee660202f59f
Closes-Bug: #1376878
We forgot to propagate options into http_connection() for the info
aka capabilities command. Fix is to use the proper helper.
Fixes bug: 1381866
Change-Id: I9515203c969a1bba38dd909412355080383905f9
Instead of always specifying segment size in bytes the user
can now use B,K,M or G as suffixes for the corresponding
size.
Conversion is done with Binary units (1024) rather than SI
units (1000).
e.g
swift upload test_container -S 1073741824 large_file
can now be written
swift upload test_container -S 1G large_file
The change is backwards compatible as it ignores arguments to -S
that don't have a valid suffix.
Updated unit tests and help message.
Change-Id: I6314b4e45cf2fbffde2fe57a02df77a25e911e84
SkipTest is raised if there is no func_test section in the Swift test
config file. However, the currently raised Exception will result in a
failed test, not in a test marked as skipped.
Since we already use testtools we can use the included skipTest easily.
Change-Id: I1bbb1f9dbe31fb0698d774550708d1196b266625
If 'x-timestamp' header is not present in 'swift list' result,
date string cannot be correctly generated. Check for correct
exception to catch this case and return a special string value.
This is just a workaround for faulty RadosGW behavior.
Change-Id: I172843a2668e33efac00924906ff0dca3e790050
Closes-Bug: 1349528
This patch removes a debugging print statement that slipped into
service.py in patch https://review.openstack.org/#/c/85453/
Change-Id: I80eee49eaa74b52e9c36f00669206aa7b2cdec9b
tests/unit/test_shell.py:TestParsing tests can fail if
there are OS_* variables set in the environment. There
is already code in the test setUp to remove ST_* variables,
so we should do the same for OS_* variables.
This patch also changes the mechanism used to remove and
then restore any unwanted variables found in os.environ.
The existing setUp() takes a copy of os.environ and then
deletes any ST_* variables in the original. In tearDown() it
sets os.environ as the copy. However, the environ imported
into shell.py remains pointing to the original os.environ
object. So after the first call to tearDown, subsequent
mocking of os.environ has no effect on shell.environ.
This renders some of the tests ineffective e.g.
test_insufficient_env_vars_v3 is not actually setting any
vars in shell.environ.
The issue can be provoked by repeating a test:
nosetests -w tests/unit/ test_shell.py:TestParsing.test_args_v3 \
test_shell.py:TestParsing.test_args_v3
The test will pass first time and fail second time.
Change-Id: I5d100f81115e74878d510326acb5777e6a3626c8
Fixes a regression that is preventing swiftclient from
authenticating using keystone v3 options.
swiftclient/shell.py processes the dict of command
line options to set up a child dict ('os_options')
of keystone auth specific options (line 960). The
processing includes stripping --os- prefixes from
the command line options before adding keys to
os_options.
A recent patch https://review.openstack.org/#/c/85453/
introduced a duplication of this option processing
in service.py (line 79) which replaces the os_options
created in shell.py, but omits keystone v3 related
options. Consequently the keystone v3 options are
not being passed to the client get_auth() method.
This patch adds the keystone v3 related options to
the option processing in service.py.
For pragmatic reasons (i.e. fixing the bug quickly)
the option processing code has not been removed from
parse_args in shell.py. It is likely that the code in
parse_args is now redundant, but all code paths between
parse_args and process_options should be inspected and
test coverage added before removing that code.
Unit tests have been added in test_shell.py to verify
that command line options are correctly passed to the
client get_auth method.
The MockHttpTest class is re-used in test_shell.py,
so it is moved from test_swiftclient.py
to tests/unit/utils.py
Closes-bug: #1372465
Change-Id: I4fed013cdb8936509609d06093337cc147ade0d6
When the account response includes 'X-Storage-Policy-X-Container-Count' the
command helpers will stick a helpful extra line in the output.
If the swift response doesn't include the information the behavior is
unchanged.
Change-Id: Iaaa55d207ebe6a9c16e52adb56eb858c57cfb26a
Cannot help streamlining the code a bit, sorry. But the meat is that
we should not try to make directories if output path is explicit.
Previously we created directories using the URL path, which is
obviously wrong if explicit output file is supplied... unless
a crafty user supplied the same path with -o that is contained
in the URL path. If anyone was doing such tricks, it's not going
to work anymore (we are forcing a regression for the sake of
theoretical correctness here).
Fixes bug: 1369546
Change-Id: Ifce31f2ba233eb55550f3810348bf16bf2447d62
This patch adds a SwiftService class that incorporates the high
level logic from swiftclient/shell.py. It also ports shell.py to
use the new class, and updates the code in swiftclient/multithreading.py
to allow the SwiftService to be used for multiple operations whilst
using only one thread pool.
Currently, code that imports swiftclient has to have its own logic for
things like creating large objects, parallel uploads, and parallel
downloads. This patch adds a SwiftService class that makes that
functionality available in Python code as well as through the shell.
Change-Id: I08c5796b4c01001d79fd571651c3017c16462ffd
Implements: blueprint bin-swift-logic-as-importable-library
A Swift client executable needs to be provided on Windows since Python scripts
cannot be executed directly.
This approach is also consistent with other OpenStack Python clients (e.g.
Nova, Neutron, Cinder, etc.).
Change-Id: I7f03b7c305dceed4fa6f4fe1c1281659820606e4
Closes-Bug: #1359360
This is a fix-up for the commit 3d56b65c. The code change was good,
but the additional test had a side effect of creating an empty
directory "pseudo/". Fix this by mocking-out mkdirs().
Change-Id: Iead55e72d8a75d96bde0a7491aca12bbdcc269cf
If a user creates an object with name ending with a slash, then
downloading such container ends in a traceback like this:
..............
test5g.file [auth 1.516s, headers 1.560s, total 244.565s, 22.089 MB/s]
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/swiftclient/multithreading.py", lin
result = self.func(item, *self.args, **self.kwargs)
File "/usr/lib/python2.6/site-packages/swiftclient/shell.py", line 403, in
fp = open(path, 'wb')
IOError: [Errno 21] Is a directory: 'first-pseudo-folder/'
The proposed fix is not to save this object. Note that the contents of
the object are available with --output option, as before. Only the
crash is fixed.
Even though we do not use the contents, we download the object and
check its Etag, in case. We also create a corresponding directory,
in case the pseudo-directory contains no objects.
The format of printout is changed, so user realizes easier when
pseudo-directory convention is in effect. Note that this is not a
compatibility issue because previously there was crash in such case.
Change-Id: I3352f7a4eaf9970961af0cc84c4706fc1eab281d
Storage policy stats was not well parsed in account stat. This
patch parses the stats and print out the stats in a format like below:
$swift -A http://swift_cluster/auth/v1.0 -U test:tester -K testing stat
Account: AUTH_test
Containers: 5
Objects: 1
Bytes: 2097152
Objects in policy "golden": 1
Bytess in policy "golden": 2097152
Objects in policy "silver": 0
Bytes in policy "silver": 0
X-Timestamp: 1404697760.88809
X-Trans-Id: txec519e24b44a413abb705-0053da2dcb
Content-Type: text/plain; charset=utf-8
Accept-Ranges: bytes
Change-Id: I7ad0ee6d88f8393e3a93e90cd52b9b592da7072d
The current developer docs theme used is out of sync with the other
openstack projects. This patch will update the docs to provide a more
consistent look and feel when using developer docs
Change-Id: I2c66f60f6cfdd7b7b50a457c931f11851f668fa6
Add a tox job to make it easier for developers to generate their
own docs before submitting a patch.
Change-Id: I86dbfc64a9d174697048c254c6d3d042a62350b4
Enables swiftclient to authenticate using
the keystone v3 API, allowing user id's, user
domains and tenant/project domains to be
specified.
Since swiftclient imports keystoneclient, the
main changes in swiftclient/client.py are to
selectively import the correct keystoneclient
library version and pass a number of new
options to it via the get_auth() function. In
addition the get_keystoneclient_2_0 method
has been renamed get_auth_keystone to better
reflect its purpose since it now deals with
both v2 and v3 use cases.
In swiftclient/shell.py the new options are
added to the parser. To make the default help
message shorter, help for all the --os-*
options (including the existing v2 options)
is only displayed when explicitly requested
usng a new --os-help option.
A new set of unit tests is added to
test_shell.py to verify the parser. A comment
in tests/sample.conf explains how to
configure the existing functional tests to
run using keystone v3 API.
Note that to use keystone v3
with swift you will need to set
auth_version = v3.0 in the auth_token
middleware config section of
proxy-server.conf.
Change-Id: Ifda0b3263eb919a8c6a1b204ba0a1215ed6f642f
Make it so that swift <cmd> --help will print the info
subcommand help for info and tempurl just like all the
other subcommands.
Also add unit tests to verify subcommand help.
Change-Id: Id3666dcf72a9727fbfda2f74c23293ada1c53aa0
Client already supports -H/--header option when creating container
or uploading objects. This patch extends this option to support
Storage Policy.
e.g.,
swift post con -H 'X-Storage-Policy:p1'
This creates one container 'con' with storage policy 'p1'.
swift upload con obj -H 'X-Storage-Policy:p2'
This creates container 'con' with storage policy 'p2' and uploads
object 'obj' into it.
Also fixes segmented uploading to non-default storage policy container
When uploading large objects with segmentation to container with
non-default storage policy, there will be another 'xxx_segments'
container created, but with the default storage policy. This
results all the segments to be stored with the wrong policy.
This patch is for the Storage Policy feature, and also
compatible with old versions w/o Storage Policy support.
Change-Id: I5c19e90604a0bcf2c85e1732b8a0b97ae6801994
