349 Commits

Author SHA1 Message Date
Jenkins
cacb3a3a15 Merge "Make tempurl command check for valid object path" 2016-09-14 20:45:43 +00:00
Alistair Coles
4c955751d3 Make tempurl command check for valid object path
If the supplied path is not of the form /v1/a/c/o then
swift tempurl subcommand will now return an error message.

Also removes redundant check for seconds parameter being an int
from shell.py because the same check is made when calling
utils.generate_temp_url.

Drive-by fix for missing param definition for generate_temp_url.

Change-Id: I41f4389948b01fadaa5fc4939ea12e0ed2167345
Related-Change: I0fb2ce125fe12d660e4deb778265016bdd5ff31b
2016-09-14 16:24:22 +01:00
Ji-Wei
5eb0f3debc Fix order of arguments in assertIs
Some tests used incorrect order assertIs(observed, expected).
The correct order expected by testtools is

assertIs(expected, observed, message='')
Assert that expected is observed.
Parameters:
  expected The expected value.
  observed The observed value.
  message An optional message describing the error.
https://docs.python.org/2/library/unittest.html

Change-Id: I5f406336b7804c13ca042dd0c6b30645a711c5f2
2016-09-09 17:30:46 +08:00
Christian Schwede
89ff29e788 Make options arguments truely optional
Currently the options help, info and debug must exist in the options
dictionary; however it might be that this is not the case if a method is
imported externally. This patch fixes this.

Also support using a storage URL or path as an argument for tempurl, and
adding a test to verify correct output in both cases.

Related-Bug: 1607519
Related-Bug: 1607523
Closes-Bug: 1607521
Change-Id: I0fb2ce125fe12d660e4deb778265016bdd5ff31b
2016-09-06 07:38:08 +00:00
Jenkins
92544c58c5 Merge "Accept gzip-encoded API responses" 2016-09-01 03:18:18 +00:00
Jenkins
98085c961f Merge "Convert numeric and boolean header values to strings" 2016-08-31 00:58:02 +00:00
Tim Burke
f728027bed Accept gzip-encoded API responses
Previously, we would accept gzip-encoded responses, but only because we
were letting requests decode *all* responses (even object data). This
restores the previous capability, but with tighter controls about which
requests will accept gzipped responses and where the decoding happens.

Change-Id: I4fd8b97207b9ab01b1bcf825cc16efd8ad46344a
Related-Bug: 1282861
Related-Bug: 1338464
2016-08-30 15:38:55 -07:00
Jenkins
20e0c515bf Merge "client: renew token on 401 even if retries is 0" 2016-08-26 17:37:41 +00:00
Tim Burke
ab60e08e2e Convert numeric and boolean header values to strings
Recently, requests got a bit more picky about what types of data it will
accept as header values [1]. The reasons for this are generally sound;
str()ing arbitrary objects just before pushing them out a socket may not
do what the developer wanted/expected.

However, there are a few standard types that developers may be sending
that we should convert for them as a convenience. Now, we'll convert all
int, float, and bool values to strings before sending them on to
requests.

Change-Id: I6c2f451009cb03cb78812f54e4ed8566076de821
Closes-Bug: 1614932
2016-08-25 11:32:08 -07:00
Jamie Lennox
fd675b2c5a Use mock patch to handle get_auth_keystone
Setting c.get_auth_keystone = fake_get_auth_keystone is setting a new
method on the module. This information is not reset between test runs
and therefore has the potential to corrupt other tests.

Use mock patch so that the mock is reset after the test is complete.

Change-Id: Ifb9ba72634cf477c72dda080b5aed8f8e3a7ac89
2016-08-25 17:18:20 +10:00
Jenkins
5acefd27e4 Merge "Strip leading/trailing whitespace from headers" 2016-08-25 04:24:16 +00:00
Jenkins
b57044a853 Merge "Add copy object method" 2016-08-24 23:59:34 +00:00
Marek Kaleta
4a2465fb12 Add copy object method
Implement copy object method in swiftclient Connection, Service and CLI.

Although COPY functionality can be accomplished with 'X-Copy-From'
header in PUT request, using copy is more convenient especially when
using copy for updating object metadata non-destructively.

Closes-Bug: 1474939
Change-Id: I1338ac411f418f4adb3d06753d044a484a7f32a4
2016-08-23 14:37:11 -07:00
Tim Burke
8fbe118cea Strip leading/trailing whitespace from headers
New versions of requests will raise an InvalidHeader error otherwise.

Change-Id: Idf3bcd8ac359bdda9a847bf99a78988943374134
Closes-Bug: #1614280
Closes-Bug: #1613814
2016-08-19 19:53:11 +00:00
Jenkins
2766744ba1 Merge "Fix unicode issues in tempurl command" 2016-08-07 23:43:13 +00:00
Jenkins
92178ac17c Merge "Add --json option to swift capabilities / swift info" 2016-08-06 00:10:32 +00:00
Tim Burke
335570e511 Add --json option to swift capabilities / swift info
This lets us do things like:

    $ swift info --json | jq '[.swift.policies[].name]'
    [
      "Standard-Replica",
      "EC"
    ]

Also, escape more dashes in the man page, so they won't be
misinterpreted as hyphens.

Change-Id: Ic7690bdbcfc55f55e5dde9bc11bb0644085973ce
2016-08-02 16:18:44 -07:00
zheng yin
d460db84ed Modify assert
For example: self.assertEqual(a,None) is equal to self.assertIsNone(a)

Change-Id: I063145d034979cf3d36645a00a30cc27defac7c4
2016-07-26 22:23:43 +08:00
Tim Burke
dca4ea5fd6 Fix unicode issues in tempurl command
Previously, we weren't encoding paths and keys as UTF-8, which would
trigger a UnicodeEncodeError on py27.

Change-Id: I2fad428369406c2ae32343a5e943ffb2cd1ca6ef
2016-07-19 12:52:51 -07:00
yuyafei
f4f2e6c261 Add __ne__ built-in function
In Python 3 __ne__ by default delegates to __eq__ and inverts the
result, but in Python 2 they urge you to define __ne__ when you
define __eq__ for it to work properly [1].There are no implied
relationships among the comparison operators. The truth of x==y
does not imply that x!=y is false. Accordingly, when defining
__eq__(), one should also define __ne__() so that the operators
will behave as expected.
[1]https://docs.python.org/2/reference/datamodel.html#object.__ne__

Change-Id: I5485022f010cdcb88c0d4ebe5c44a39a1bf242b0
2016-07-05 14:58:42 +08:00
Jenkins
07c960d46f Merge "Query string functionality for containers" 2016-06-14 19:36:45 +00:00
Andrew Welleck
439330cb9c Query string functionality for containers
Added functionality for arbitrary query strings to be passed into container
functions. Additionally a minor typo correction in the README. Added unit
tests for query string functionality.

Closes-Bug: #1542459
Change-Id: Ica2cb3ea439632588388e748d8d2e944e9ed4fa4
2016-06-09 13:51:26 -05:00
Julien Danjou
a62b7ee06c client: renew token on 401 even if retries is 0
Gnocchi uses a client with retries=0 to maximize throughtput and not retry N
times on e.g. 404 when checking existence of an object. However, this as the
side effect of never renewing the token since there' no retry on 401 either.

This patches change the behavior so that 401 errors are always retried,
whatever the retries value is.

Closes-Bug: #1589926
Change-Id: Ie06adf4cf17ea4592b5bbd7bbde9828e5e134e3e
2016-06-07 16:58:26 +02:00
Jenkins
c91c8d575e Merge "Add an option: disable etag check on downloads" 2016-06-06 10:33:25 +00:00
Cheng Li
69bf4634b9 Add an option: disable etag check on downloads
This patch is to add an option of disable etag
check on downloads.

Change-Id: I9ad389dd691942dea6db470ca3f0543eb6e9703e
Closes-bug: #1581147
2016-06-02 22:53:18 +08:00
Tim Burke
ff505d72cd Suppress InsecureRequestWarning when using --insecure
The user already knows this is insecure, there's no point in bringing it
up again and again.

See also: https://github.com/kennethreitz/requests/issues/2214

Change-Id: I7991b2e568407269f84138bc03711147ed080c9c
2016-06-01 18:57:38 -07:00
Jenkins
f9d0657e70 Merge "Support client certificate/key" 2016-05-19 22:20:17 +00:00
Jenkins
8ffc5c11ae Merge "Use application/directory content-type for dir markers" 2016-05-19 08:53:50 +00:00
Jenkins
3a6c14981d Merge "Check responses when retrying bodies" 2016-05-11 07:52:47 +00:00
Jenkins
3faa99bac1 Merge "Tighten up testing for sloppy auth version" 2016-05-08 22:21:10 +00:00
Alistair Coles
a0d65e4ccd Tighten up testing for sloppy auth version
The existing test that os-identity-api-version=2 would
result in the correct auth_version=2.0 (note 2 -> 2.0)
does not verify the sloppy version handling code path.
When auth, key and user options are all missing the
auth version is always set to 2.0 so the test will always
pass. Adding auth, key and user options to the test command
will cause the test to verify that the os-identity-api-version
option was used and mapped to 2.0.

Change-Id: Ifa10cd2b8bf81c082d5e3fa846f993871194fea0
2016-05-05 15:25:16 +01:00
Tim Burke
fd5579a154 Check responses when retrying bodies
Previously, if a Range request came back 200 OK (rather than 206 Partial
Content), we would mangle the response body. This could happen if there
was a middleware that would silently drop Range headers, for example.

Now, if the response does not include a Content-Range header, we will
log a warning and seek to our previous position in the stream. If the
Content-Range header has an unexpected value, we will raise an exception.

Change-Id: I94d4536cc1489968d45a2b6ba7edd70c85800275
2016-05-04 15:21:26 -07:00
Tim Burke
67f629cdeb Default to v3 auth if we find a (user|project)-domain-(name|id) option
Change-Id: I4616492752b620de0bf90672142f1071ec9bac83
2016-05-03 14:41:07 -07:00
Tim Burke
c3766319b9 Pull option processing out to service.py
...because it seems silly that we do nearly the same thing in two
different places

Change-Id: Iafafe3c553d00652adb91ceefd0de1479cbcb5da
2016-05-03 14:18:34 -07:00
Tim Burke
5b714f104d Parse options to dict
This is a first step toward unifying the options parsing magic between
shell.py and service.py

Change-Id: If001e07978c0bae729ac0cd9b2c2934092c98447
2016-05-03 14:03:15 -07:00
Jenkins
c26dec66f6 Merge "Add tests for thread option validation" 2016-05-02 21:11:57 +00:00
Jenkins
6b065dd4a6 Merge "Identify segments uploaded via swiftclient" 2016-05-02 10:35:20 +00:00
Jenkins
ab636c1ee3 Merge "Fix downloading from "marker" item" 2016-04-14 02:02:34 +00:00
Cedric Brandily
450f505c35 Support client certificate/key
This change enables to specify a client certificate/key with:
 * usual CLI options (--os-cert/--os-key)
 * usual environment variables ($OS_CERT/$OS_KEY)

Closes-Bug: #1565112
Change-Id: I12e151adcb6084d801c6dfed21d82232a3259aea
2016-04-10 23:20:49 +02:00
Sergey Gotliv
909bdf8954 Fix downloading from "marker" item
The documentation of "swift download" hints that "marker" option
is supported, but in reality we forgot to patch it through, so
all downloads were always done with the default, empty marker.

Closes-Bug: #1565393
Change-Id: I38bd29d2baa9188b61397dec75ce1d864041653c
2016-04-08 20:55:18 -06:00
Tim Burke
9fd537a082 Use application/directory content-type for dir markers
Previously, we were using a content-type of text/directory, but that is
already defined in RFC 2425 and doesn't reflect our usage:

   The text/directory Content-Type is defined for holding a variety
   of directory information, for example, name, or email address,
   or logo.

(From there it goes on to describe a superset of the vCard format
defined in RFC 2426.)

application/directory, on the other hand, is used by Static Web [1] and
is used by cloudfuse [2]. Seems like as sane a choice as any to
standardize on.

[1] https://github.com/openstack/swift/blob/2.5.0/swift/common/middleware/staticweb.py#L71-L75
[2] https://github.com/redbo/cloudfuse/blob/1.0/README#L105-L106

Change-Id: I19e30484270886292d83f50e7ee997b6e1623ec7
2016-04-08 16:57:51 -07:00
Tim Burke
4a6fa02c28 Identify segments uploaded via swiftclient
...using a new "application/swiftclient-segment" content-type.

Segments uploaded by swiftclient are expected to have a many-to-one
relationship to large objects, rather than the more-general many-to-many
relationship that SLO and DLO generally allow. Later, we may use this
information to make more intelligent decisions, such as when to
automatically clean up segments.

Change-Id: Ie56a3aa10065db754ac572cc37d93f2c901aac60
2016-04-08 13:41:29 -07:00
Jenkins
015903e383 Merge "Port from optparse to argparse" 2016-04-08 09:36:31 +00:00
Tim Burke
17aa6c789e Port from optparse to argparse
Why now?

* argparse was introduced in Python 3.2 and back-ported to Python 2.7.
  Until we dropped Python 2.6 support, we were stuck on optparse.

* keystoneauth.loading.cli provides register_argparse_arguments and
  load_from_argparse_arguments helper methods. Now that we're moving
  toward Keystone Session support, argparse seems required.

Closes-Bug: 1553030
Change-Id: I5139fb64a8631a3010680090fd04345f95c55c7b
2016-04-06 14:20:27 -07:00
Tim Burke
f86b2d8138 Clean up some unnecessary variables
Change-Id: Iac93ced6344d4a6fee7e6390e891fde765814c03
2016-03-24 13:06:42 -07:00
Marek Kaleta
51a8a5a7ae Fix SwiftPostObject options usage in SwiftService
SwiftService().post(cont, [SwiftPostObject(obj, options]) currently
ignores options['header'], raises exception when options['headers']
is set and make malformed metadata when options['meta'] is set.

Fix tipos in code, add unittest for SwiftService().post
Closes-Bug: #1560052

Change-Id: Ie460f753492e9b73836b4adfc7c9c0f2130a8a91
2016-03-23 15:38:10 +01:00
Jenkins
39e017d807 Merge "Initialize delete_object mock *before* creating all the threads" 2016-03-18 10:43:09 +00:00
Tim Burke
d2bd2f0859 Initialize delete_object mock *before* creating all the threads
Previously, we'd occasionally get spurious failures like

    FAIL: test_delete_account (tests.unit.test_shell.TestShell)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File ".../mock/mock.py", line 1721, in _inner
        return f(*args, **kw)
      File ".../mock/mock.py", line 1305, in patched
        return func(*args, **keywargs)
      File ".../tests/unit/test_shell.py", line 788, in test_delete_account
        response_dict={})], any_order=True)
      File ".../mock/mock.py", line 983, in assert_has_calls
        ), cause)
      File ".../six.py", line 718, in raise_from
        raise value
    AssertionError: (call(u'container', u'object', query_string=None,
                          response_dict={}),)
    not all found in call list

Related-Bug: #1539536
Related-Bug: #1480223
Change-Id: I810894545ca74d3b2f2dbde2d0388eb69c2ba710
2016-03-17 15:43:45 -07:00
Jenkins
c4ce65f2d2 Merge "Include response headers in ClientExceptions" 2016-03-17 22:20:35 +00:00
Tim Burke
f5224a696e Add tests for thread option validation
Change-Id: If84714c7ea6be1c95c5898a82db2d4b6c9637242
2016-03-17 12:39:44 -07:00