65 Commits

Author SHA1 Message Date
Matthew Oliver
1e0ab242ca lint: Up-rev hacking
Without up-reving hacking python 3.12 can't be used for pep8/flake8
checks. Also address a few new pyflakes issues that seem reasonable
to enforce:

   E275 missing whitespace after keyword
   H214 Use assertIn/NotIn(A, B) rather than
        assertTrue/False(A in/not in B)
   H211 do not compare types, use assert{Is,IsNot}Instance

Add F811 (redefinition of unused 'function'), as this check makes
client.py fail.

And noqa some longer lines that we're happy to carry.

Change-Id: Ic97698a310ffe7114b668b8c4bc0195f6997bb45
2024-02-21 17:14:27 +00:00
Zuul
7e65d10199 Merge "tests: Fix call assertion" 2024-01-02 20:17:06 +00:00
Tim Burke
6f01f84ce2 tests: Fix call assertion
Under py312, MagicMock.has_calls raises an AttributeError. This is
actually a good thing, as previously we weren't actually making the
assertions we thought we were making.

Fix the one place that we forgot the leading `assert_`, and switch it
to `any_order=True`; this is required for the test to pass, as there
are additional calls between the two expected calls.

See https://github.com/python/cpython/commit/1d4d677d

Change-Id: Ia2ff2f198b4c1312a3e7dce32f11b87050683d45
2023-12-15 12:26:21 -08:00
Zuul
c62357c32f Merge "tests: Skip keystoneauth tests if not available" 2023-10-17 20:50:21 +00:00
Tim Burke
9331880e73 tests: Fix timeout assertion for py36, py37
The kwargs property was introduced in py38.

Change-Id: Id380fee6a2131bf88baa27145c8158277eef10c1
2023-08-25 09:43:43 -07:00
Tim Burke
c52e404262 tests: Skip keystoneauth tests if not available
Change-Id: I80af28b49415c5bea6399683922b6649f04ed011
2023-07-07 14:32:05 -07:00
Tim Burke
ee952c1bf2 Ensure v*password auth_type correctly implies auth version
Change-Id: I787b0988b7518850f1e1f93ca269f39f8cdec15c
2023-03-23 20:32:23 -07:00
Zuul
31c279ff0e Merge "Use SLO by default for segmented uploads if the cluster supports it" 2023-03-22 16:00:44 +00:00
Tim Burke
6a8675e897 Use SLO by default for segmented uploads if the cluster supports it
We've had SLO for nearly ten years, and it is preferable for most
use-cases. It's time to default to SLO rather than DLO.

Add a new --use-dlo option to give users a way to return to old
behavior; ensure there is still a --use-slo option so we don't break
existing scripts that may use it.

UpgradeImpact:
==============
The default segmented-upload behavior has changed; Static Large Objects
are now used by default rather than Dynamic Large Objects. To revert to
the old behavior:

* CLI users may use the new `--use-dlo` option
* Service API users may explicitly set `use_slo` to False in their
  options dicts

Change-Id: Iebcd64ced8eab6efebb9f02a0c4be6bf6338cb3b
2023-03-21 12:46:44 -07:00
Zuul
62641a2f49 Merge "Retry on SSLError" 2023-03-21 19:37:06 +00:00
Tim Burke
9e703b9345 shell: Allow timeouts to have units
Related-Change: Ibbe7e5aa8aa8e54935da76109c2ea13fb83bc7ab
Change-Id: Ifeaaea790d1dadc84b157a7cf2be7590949c70f0
2023-03-20 17:23:38 -07:00
Tim Burke
099dd66265 Retry on SSLError
...provided it isn't something to do with certification validation.

Closes-Bug: #1641363
Closes-Bug: #2011731
Change-Id: Ic3acbf431e444fcd8bc0fe79571fe2db4140cf22
2023-03-20 16:57:15 -07:00
Tim Burke
e343298240 Include transaction ID on content-check failures
Change-Id: I6b667db26ffc5dccdcadfc8c73f7accb81f03dac
2023-03-15 12:37:02 -07:00
Tim Burke
6adb8bb17f service: Check content-length before etag
If the received content-length does not match expectations, of course
the etag won't match!

Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I1a0c066c11b94718fffbb11e13b82d0b16e01626
2023-03-14 11:56:03 -07:00
Tim Burke
ff6b359d06 Retry with fresh socket on 499
Change-Id: I0c22eefb587375997672724c03744c9cda473708
2023-02-15 21:01:07 -08:00
Tim Burke
f4e62191bc Allow tempurl to be used to sign /info requests
Change-Id: I1f9d26541e9c8f5aec7a6790c87df397d178efe6
2022-08-29 13:13:34 -07:00
Zuul
662e530d8d Merge "Allow tempurl times to have units" 2022-08-25 07:38:36 +00:00
Zuul
02509acc71 Merge "Add option to skip container PUT during upload" 2022-08-23 00:37:52 +00:00
Tim Burke
defbb4a8f3 Allow tempurl times to have units
Specifically, let users add a suffix for seconds, minutes, hours, or
days.

Change-Id: Ibbe7e5aa8aa8e54935da76109c2ea13fb83bc7ab
2022-08-19 17:40:10 -07:00
Zuul
249201f7ea Merge "Fix misuse of assertTrue" 2022-08-19 01:50:40 +00:00
Zuul
c1f5e8b7c1 Merge "Expand retry handling on ratelimit response" 2022-08-19 00:43:25 +00:00
Zuul
3b30f155a7 Merge "Enable retry_on_ratelimit by default" 2022-08-19 00:42:56 +00:00
Clay Gerrard
653cbcb686 Expand retry handling on ratelimit response
We have seen middlewares that return ratelimit responses as 498 or 429,
so tolerate either.

Closes-Bug: #1879572
Change-Id: I027222157f6c2ad7882a0508302c9de097baae4c
2022-08-18 12:14:04 -07:00
Tim Burke
a1d2f31131 Enable retry_on_ratelimit by default
UpgradeImpact
=============
The Connection class now enables retry_on_ratelimit by default.
If you need to return to the old behavior, explicitly pass
retry_on_ratelimit=False as a keyword arg.

The SwiftService class will now enables the retry_on_ratelimit option by
default. If you need to return to the old behavior, explicitly set it to
false in your options dict.

Change-Id: I3221fda84f0b8031c50128aa600e2c19deb5b102
2022-08-18 12:12:35 -07:00
Takashi Natsume
0bd2ab5cb0 Fix misuse of assertTrue
Replace assertTrue with assertEqual.

Change-Id: Ia3524bc5b3b01c0039bede6bb172535eb85bac08
Closes-Bug: 1986948
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
2022-08-18 22:39:35 +09:00
Zuul
7f2649bfb7 Merge "More cleanup following py2 removal" 2022-08-18 01:49:01 +00:00
Tim Burke
9eee29d2e4 tempurl: Support sha256 and sha512 signatures
Up the default to sha256 since

* the proxy has supported (and defaulted to allowing) it for four years
  now, and
* Rackspace has supported it for even longer.

Include a note in the --help about older clusters likely requiring sha1.

Change-Id: Ibac2bb7e2e4c9946c7384f0aab8e43d0d79ba645
Related-Change: Ia9dd1a91cc3c9c946f5f029cdefc9e66bcf01046
Related-Bug: #1733634
Closes-Bug: #1977867
2022-06-08 09:30:21 -07:00
Tim Burke
5d451fb920 More cleanup following py2 removal
* Drop py2-only hacking pin from test-requirements.

* Remove quote() helper; urllib.parse.quote() works fine.

* Remove some useless code.

Change-Id: I9ffc923f58f1d11538f83ff26f7beb53cdf134c3
2022-05-24 08:33:02 -07:00
Steve Kowalik
20c97e83d3 Remove use of mock
Since Python 3.4, the unittest module has provided mock, negating the
need for the external dependancy. Switch to using unittest.mock.

Change-Id: Idec3aaed2fddd1ece3ed86ee0bcc48f7616d56fa
2022-05-24 11:56:35 +10:00
Stephen Finucane
20d837a276 Remove unnecessary unicode prefixes
All strings are unicode by default in Python 3. No need to mark them as
such.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I68fb60ef271abfddebcc9d2137424f5db2a17e92
2022-03-21 18:32:30 +00:00
Stephen Finucane
61ce5ac824 Remove unnecessary object subclassing
All classes subclass from object by default in Python 3.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I5a1ad57bcc092861ce969759b06a07c880ad3d35
2022-03-21 18:32:00 +00:00
Stephen Finucane
fa137a5bf1 Remove six
This mostly affects tests. Nothing too complicated

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Iabc78f651e1d48db35638280722f8019798eccd6
2022-03-21 18:31:10 +00:00
Stephen Finucane
4983b90983 Remove coding comments
Everything is unicode in Python 3.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I6a076dc67c461f265ed99878e3959e1992a88189
2022-03-21 18:29:25 +00:00
Stephen Finucane
3d10744c55 Remove __future__ imports
These aren't needed in modern Python 3 versions.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I5e81d6fb2e2cb8e4bfae4ed746da002f44e871c4
2022-03-21 18:29:25 +00:00
Aarni Koskela
c09621eb42 Don't patch Requests globally on import
This also upgrades the Requests dependency to 2.4+ (released in 2014)
to avoid having to do version comparisons altogether.

Refs https://bugs.launchpad.net/python-swiftclient/+bug/1904551

Signed-off-by: Aarni Koskela <akx@iki.fi>
Change-Id: I58399f6c526b0b78462f31739c43076314ba9e76
2022-03-21 08:05:54 -07:00
Stephen Finucane
2636965f38 Drop support for Python 2
There's a lot of cleanup possible, but this is a start.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ia1176b7fd5434d52070d482a37abfbb98800cdb3
2022-03-16 15:06:19 -07:00
Tim Burke
f1858d89e0 Add option to skip container PUT during upload
Currently, a user with read/write access to a container (but without
access to creat new containers) recieves a warning every time they
upload. Now, allow them to avoid the extra request and warning by
specifying --skip-container-put on the command line.

This is also useful when testing: developers can HEAD a container to
ensure it's in memcache, shut down all container servers, then upload
and creaate a bunch of async pendings. Previously, the 503 on container
PUT would prevent the object upload from even being attempted.

Closes-Bug: 1317956
Related-Bug: 1204558
Change-Id: I3d9129a0b6b65c6c6187ae6af003b221afceef47
Related-Change: If1f8a02ee7459ea2158ffa6e958f67d299ec529e
2022-01-11 16:39:38 -08:00
Zuul
3f5d5b0252 Merge "Include storage policy when listing account with --long" 2021-10-13 01:11:03 +00:00
Zuul
ec6a346afb Merge "Allow functional tests to pass with etag_quoter enabled by default" 2021-09-24 02:35:55 +00:00
Zuul
e03a11c649 Merge "Allow unit tests to be run via pytest" 2021-09-23 23:56:23 +00:00
Tim Burke
373fa26ce1 Correctly aggregate totals for >10k items
Previously, we would write out totals for every page of listings, like

    $ swift list sync --prefix=09-21 --total -l
     80000000000
     80000000000
     80000000000
     58096000000

Now, roll those all into a single total:

    $ swift list sync --prefix=09-21 --total -l
    298096000000

Change-Id: Icc265636815220e33e8c9eec0a3ab80e9f899038
2021-09-21 16:06:56 -07:00
Tim Burke
ad3e8e49d0 Include storage policy when listing account with --long
Change-Id: Ibc2f9445b5a8e80cfb73d0706e20a7e4c62eec4a
2021-09-21 08:40:45 -07:00
Tim Burke
553e34ebfe Improve formatting for billions of objects
Change-Id: If8aa08c4c8c8ad6ca2c861602baf1eefa8642a8a
2021-09-20 15:49:59 -07:00
Tim Burke
6d8138ebcf Allow functional tests to pass with etag_quoter enabled by default
Change-Id: I861b5e0a172f0ea0a5b1fe8389cd70da8d4b5d5d
2021-05-11 13:42:47 -07:00
Tim Burke
99b5b81217 Allow unit tests to be run via pytest
You can run all tests by just running

   $ pytest

Or just unit tests with

   $ pytest test/unit/

Or one specific test with

   $ pytest test/unit/test_swiftclient.py::TestConnection::test_reauth

Change-Id: I1dfa239f9ee9ea85663b5c1f22631a97f87b4dfc
2021-05-11 13:34:35 -07:00
Tim Burke
74c50dee2d Have delete --all imply --versions for the CLI
Change-Id: Id5a6d4cef3d4ed76c897a099a62a4ba3ed8f8dab
2020-10-16 13:57:02 -07:00
Zuul
bf862981a6 Merge "Close connections created when calling module-level functions" 2020-10-15 22:41:10 +00:00
Zuul
90c10fab43 Merge "tests: Make test_delete_container_versions less flakey" 2020-10-14 03:56:46 +00:00
Tim Burke
97aa3e6541 Close connections created when calling module-level functions
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Id62e63afc6f2ffa32eb8640787c78559481050f9
Related-Change: I200ad0cdc8b7999c3f5521b9a822122bd18714bf
Related-Bug: #1873435
Closes-Bug: #1838775
2020-10-13 10:45:07 -07:00
Zuul
89994a7ad0 Merge "Add cacert test config option" 2020-09-23 22:14:55 +00:00