70 Commits

Author SHA1 Message Date
Clay Gerrard
ed6fd60915 Fix swiftclient output regression
Fix swiftclient output regression introduced by the related change:
  - output for SLO object download: fix incorrect error from
    SwiftReader about SLO object ETag header not matching MD5 checksum
  - output for object stat: fix duplicated ETag
  - output for account/container stat: fix duplicated byte/object counts

Co-Authored-By: Yan Xiao <yanxiao@nvidia.com>
Related-Change: Ice9cc9fe68684563f18ee527996e5a4292230a96
Change-Id: I5b2d79f89d1b6016de69d6b58879e5c2ef31e107
2024-05-02 16:02:59 -07:00
Zuul
fa9718deee Merge "Add transaction id to errors" 2024-03-01 20:38:52 +00:00
Yan Xiao
ac2a60dc0d Add transaction id to errors
Add transaction id to errors to help troubleshoot, including error
when downloading object with truncted/missing segment, stat'ing or
listing non-existent container.

Change response headers returned by client.py Connection APIs
such as get_object() from lowercase dict to CaseInsensitiveDict,
as some error paths rely on this to retrieve the transaction id,
while there are existing code paths using response.headers which
is CaseInsensitiveDict from requests lib.

Change-Id: Ice9cc9fe68684563f18ee527996e5a4292230a96
2024-03-01 09:59:33 -05:00
Zuul
9366d86220 Merge "shell: Print friendly account byte quotas" 2024-02-27 07:32:53 +00:00
Matthew Oliver
1e0ab242ca lint: Up-rev hacking
Without up-reving hacking python 3.12 can't be used for pep8/flake8
checks. Also address a few new pyflakes issues that seem reasonable
to enforce:

   E275 missing whitespace after keyword
   H214 Use assertIn/NotIn(A, B) rather than
        assertTrue/False(A in/not in B)
   H211 do not compare types, use assert{Is,IsNot}Instance

Add F811 (redefinition of unused 'function'), as this check makes
client.py fail.

And noqa some longer lines that we're happy to carry.

Change-Id: Ic97698a310ffe7114b668b8c4bc0195f6997bb45
2024-02-21 17:14:27 +00:00
Zuul
7e65d10199 Merge "tests: Fix call assertion" 2024-01-02 20:17:06 +00:00
Tim Burke
6f01f84ce2 tests: Fix call assertion
Under py312, MagicMock.has_calls raises an AttributeError. This is
actually a good thing, as previously we weren't actually making the
assertions we thought we were making.

Fix the one place that we forgot the leading `assert_`, and switch it
to `any_order=True`; this is required for the test to pass, as there
are additional calls between the two expected calls.

See https://github.com/python/cpython/commit/1d4d677d

Change-Id: Ia2ff2f198b4c1312a3e7dce32f11b87050683d45
2023-12-15 12:26:21 -08:00
Zuul
c62357c32f Merge "tests: Skip keystoneauth tests if not available" 2023-10-17 20:50:21 +00:00
Tim Burke
4698f1f642 shell: Print friendly account byte quotas
This gives us an opportunity to give it more human-friendly units
when doing a `stat --lh`.

Change-Id: I01c02ffcba94d89ae03f87c1c64afca4d491a895
2023-10-06 10:00:39 -07:00
Tim Burke
9331880e73 tests: Fix timeout assertion for py36, py37
The kwargs property was introduced in py38.

Change-Id: Id380fee6a2131bf88baa27145c8158277eef10c1
2023-08-25 09:43:43 -07:00
Tim Burke
c52e404262 tests: Skip keystoneauth tests if not available
Change-Id: I80af28b49415c5bea6399683922b6649f04ed011
2023-07-07 14:32:05 -07:00
Tim Burke
ee952c1bf2 Ensure v*password auth_type correctly implies auth version
Change-Id: I787b0988b7518850f1e1f93ca269f39f8cdec15c
2023-03-23 20:32:23 -07:00
Zuul
31c279ff0e Merge "Use SLO by default for segmented uploads if the cluster supports it" 2023-03-22 16:00:44 +00:00
Tim Burke
6a8675e897 Use SLO by default for segmented uploads if the cluster supports it
We've had SLO for nearly ten years, and it is preferable for most
use-cases. It's time to default to SLO rather than DLO.

Add a new --use-dlo option to give users a way to return to old
behavior; ensure there is still a --use-slo option so we don't break
existing scripts that may use it.

UpgradeImpact:
==============
The default segmented-upload behavior has changed; Static Large Objects
are now used by default rather than Dynamic Large Objects. To revert to
the old behavior:

* CLI users may use the new `--use-dlo` option
* Service API users may explicitly set `use_slo` to False in their
  options dicts

Change-Id: Iebcd64ced8eab6efebb9f02a0c4be6bf6338cb3b
2023-03-21 12:46:44 -07:00
Zuul
62641a2f49 Merge "Retry on SSLError" 2023-03-21 19:37:06 +00:00
Tim Burke
9e703b9345 shell: Allow timeouts to have units
Related-Change: Ibbe7e5aa8aa8e54935da76109c2ea13fb83bc7ab
Change-Id: Ifeaaea790d1dadc84b157a7cf2be7590949c70f0
2023-03-20 17:23:38 -07:00
Tim Burke
099dd66265 Retry on SSLError
...provided it isn't something to do with certification validation.

Closes-Bug: #1641363
Closes-Bug: #2011731
Change-Id: Ic3acbf431e444fcd8bc0fe79571fe2db4140cf22
2023-03-20 16:57:15 -07:00
Tim Burke
e343298240 Include transaction ID on content-check failures
Change-Id: I6b667db26ffc5dccdcadfc8c73f7accb81f03dac
2023-03-15 12:37:02 -07:00
Tim Burke
6adb8bb17f service: Check content-length before etag
If the received content-length does not match expectations, of course
the etag won't match!

Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I1a0c066c11b94718fffbb11e13b82d0b16e01626
2023-03-14 11:56:03 -07:00
Tim Burke
ff6b359d06 Retry with fresh socket on 499
Change-Id: I0c22eefb587375997672724c03744c9cda473708
2023-02-15 21:01:07 -08:00
Tim Burke
f4e62191bc Allow tempurl to be used to sign /info requests
Change-Id: I1f9d26541e9c8f5aec7a6790c87df397d178efe6
2022-08-29 13:13:34 -07:00
Zuul
662e530d8d Merge "Allow tempurl times to have units" 2022-08-25 07:38:36 +00:00
Zuul
02509acc71 Merge "Add option to skip container PUT during upload" 2022-08-23 00:37:52 +00:00
Tim Burke
defbb4a8f3 Allow tempurl times to have units
Specifically, let users add a suffix for seconds, minutes, hours, or
days.

Change-Id: Ibbe7e5aa8aa8e54935da76109c2ea13fb83bc7ab
2022-08-19 17:40:10 -07:00
Zuul
249201f7ea Merge "Fix misuse of assertTrue" 2022-08-19 01:50:40 +00:00
Zuul
c1f5e8b7c1 Merge "Expand retry handling on ratelimit response" 2022-08-19 00:43:25 +00:00
Zuul
3b30f155a7 Merge "Enable retry_on_ratelimit by default" 2022-08-19 00:42:56 +00:00
Clay Gerrard
653cbcb686 Expand retry handling on ratelimit response
We have seen middlewares that return ratelimit responses as 498 or 429,
so tolerate either.

Closes-Bug: #1879572
Change-Id: I027222157f6c2ad7882a0508302c9de097baae4c
2022-08-18 12:14:04 -07:00
Tim Burke
a1d2f31131 Enable retry_on_ratelimit by default
UpgradeImpact
=============
The Connection class now enables retry_on_ratelimit by default.
If you need to return to the old behavior, explicitly pass
retry_on_ratelimit=False as a keyword arg.

The SwiftService class will now enables the retry_on_ratelimit option by
default. If you need to return to the old behavior, explicitly set it to
false in your options dict.

Change-Id: I3221fda84f0b8031c50128aa600e2c19deb5b102
2022-08-18 12:12:35 -07:00
Takashi Natsume
0bd2ab5cb0 Fix misuse of assertTrue
Replace assertTrue with assertEqual.

Change-Id: Ia3524bc5b3b01c0039bede6bb172535eb85bac08
Closes-Bug: 1986948
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
2022-08-18 22:39:35 +09:00
Zuul
7f2649bfb7 Merge "More cleanup following py2 removal" 2022-08-18 01:49:01 +00:00
Tim Burke
9eee29d2e4 tempurl: Support sha256 and sha512 signatures
Up the default to sha256 since

* the proxy has supported (and defaulted to allowing) it for four years
  now, and
* Rackspace has supported it for even longer.

Include a note in the --help about older clusters likely requiring sha1.

Change-Id: Ibac2bb7e2e4c9946c7384f0aab8e43d0d79ba645
Related-Change: Ia9dd1a91cc3c9c946f5f029cdefc9e66bcf01046
Related-Bug: #1733634
Closes-Bug: #1977867
2022-06-08 09:30:21 -07:00
Tim Burke
5d451fb920 More cleanup following py2 removal
* Drop py2-only hacking pin from test-requirements.

* Remove quote() helper; urllib.parse.quote() works fine.

* Remove some useless code.

Change-Id: I9ffc923f58f1d11538f83ff26f7beb53cdf134c3
2022-05-24 08:33:02 -07:00
Steve Kowalik
20c97e83d3 Remove use of mock
Since Python 3.4, the unittest module has provided mock, negating the
need for the external dependancy. Switch to using unittest.mock.

Change-Id: Idec3aaed2fddd1ece3ed86ee0bcc48f7616d56fa
2022-05-24 11:56:35 +10:00
Stephen Finucane
20d837a276 Remove unnecessary unicode prefixes
All strings are unicode by default in Python 3. No need to mark them as
such.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I68fb60ef271abfddebcc9d2137424f5db2a17e92
2022-03-21 18:32:30 +00:00
Stephen Finucane
61ce5ac824 Remove unnecessary object subclassing
All classes subclass from object by default in Python 3.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I5a1ad57bcc092861ce969759b06a07c880ad3d35
2022-03-21 18:32:00 +00:00
Stephen Finucane
fa137a5bf1 Remove six
This mostly affects tests. Nothing too complicated

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Iabc78f651e1d48db35638280722f8019798eccd6
2022-03-21 18:31:10 +00:00
Stephen Finucane
4983b90983 Remove coding comments
Everything is unicode in Python 3.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I6a076dc67c461f265ed99878e3959e1992a88189
2022-03-21 18:29:25 +00:00
Stephen Finucane
3d10744c55 Remove __future__ imports
These aren't needed in modern Python 3 versions.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I5e81d6fb2e2cb8e4bfae4ed746da002f44e871c4
2022-03-21 18:29:25 +00:00
Aarni Koskela
c09621eb42 Don't patch Requests globally on import
This also upgrades the Requests dependency to 2.4+ (released in 2014)
to avoid having to do version comparisons altogether.

Refs https://bugs.launchpad.net/python-swiftclient/+bug/1904551

Signed-off-by: Aarni Koskela <akx@iki.fi>
Change-Id: I58399f6c526b0b78462f31739c43076314ba9e76
2022-03-21 08:05:54 -07:00
Stephen Finucane
2636965f38 Drop support for Python 2
There's a lot of cleanup possible, but this is a start.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ia1176b7fd5434d52070d482a37abfbb98800cdb3
2022-03-16 15:06:19 -07:00
Tim Burke
f1858d89e0 Add option to skip container PUT during upload
Currently, a user with read/write access to a container (but without
access to creat new containers) recieves a warning every time they
upload. Now, allow them to avoid the extra request and warning by
specifying --skip-container-put on the command line.

This is also useful when testing: developers can HEAD a container to
ensure it's in memcache, shut down all container servers, then upload
and creaate a bunch of async pendings. Previously, the 503 on container
PUT would prevent the object upload from even being attempted.

Closes-Bug: 1317956
Related-Bug: 1204558
Change-Id: I3d9129a0b6b65c6c6187ae6af003b221afceef47
Related-Change: If1f8a02ee7459ea2158ffa6e958f67d299ec529e
2022-01-11 16:39:38 -08:00
Zuul
3f5d5b0252 Merge "Include storage policy when listing account with --long" 2021-10-13 01:11:03 +00:00
Zuul
ec6a346afb Merge "Allow functional tests to pass with etag_quoter enabled by default" 2021-09-24 02:35:55 +00:00
Zuul
e03a11c649 Merge "Allow unit tests to be run via pytest" 2021-09-23 23:56:23 +00:00
Tim Burke
373fa26ce1 Correctly aggregate totals for >10k items
Previously, we would write out totals for every page of listings, like

    $ swift list sync --prefix=09-21 --total -l
     80000000000
     80000000000
     80000000000
     58096000000

Now, roll those all into a single total:

    $ swift list sync --prefix=09-21 --total -l
    298096000000

Change-Id: Icc265636815220e33e8c9eec0a3ab80e9f899038
2021-09-21 16:06:56 -07:00
Tim Burke
ad3e8e49d0 Include storage policy when listing account with --long
Change-Id: Ibc2f9445b5a8e80cfb73d0706e20a7e4c62eec4a
2021-09-21 08:40:45 -07:00
Tim Burke
553e34ebfe Improve formatting for billions of objects
Change-Id: If8aa08c4c8c8ad6ca2c861602baf1eefa8642a8a
2021-09-20 15:49:59 -07:00
Tim Burke
6d8138ebcf Allow functional tests to pass with etag_quoter enabled by default
Change-Id: I861b5e0a172f0ea0a5b1fe8389cd70da8d4b5d5d
2021-05-11 13:42:47 -07:00
Tim Burke
99b5b81217 Allow unit tests to be run via pytest
You can run all tests by just running

   $ pytest

Or just unit tests with

   $ pytest test/unit/

Or one specific test with

   $ pytest test/unit/test_swiftclient.py::TestConnection::test_reauth

Change-Id: I1dfa239f9ee9ea85663b5c1f22631a97f87b4dfc
2021-05-11 13:34:35 -07:00