8847 Commits

Author SHA1 Message Date
Zuul
ba01f93ad9 Merge "func tests: work with etag-quoter on by default" 2020-06-05 20:52:30 +00:00
Zuul
56278100a1 Merge "relinker: Improve performance by limiting I/O" 2020-06-05 06:09:24 +00:00
Zuul
9aced4db27 Merge "Switch to newer openstackdocstheme and reno versions" 2020-06-03 18:32:59 +00:00
Zuul
99e6a49208 Merge "Remove <py3.5 dependencies from requirements.txt" 2020-06-03 18:32:57 +00:00
Andreas Jaeger
3d105b623d Switch to newer openstackdocstheme and reno versions
Switch to openstackdocstheme 2.2.1 and reno 3.1.0 versions. Using
these versions will allow especially:
* Linking from HTML to PDF document
* Allow parallel building of documents
* Fix some rendering problems

Update Sphinx version as well.

Disable openstackdocs_auto_name to use 'project' variable as name.

Set openstackdocs_pdf_link to link to PDF file. Note that
the link to the published document only works on docs.openstack.org
where the PDF file is placed in the top-level html directory. The
site-preview places the PDF in a pdf directory.

Change pygments_style to 'native' since old theme version always used
'native' and the theme now respects the setting and using 'sphinx' can
lead to some strange rendering.

Remove docs requirements from lower-constraints, they are not needed
during install or test but only for docs building.

openstackdocstheme renames some variables, so follow the renames
before the next release removes them. A couple of variables are also
not needed anymore, remove them.

See also
http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html

Change-Id: I131850d2a5c6164dfd48c9c95885d4754b5236c6
2020-06-03 08:31:04 +02:00
Andreas Jaeger
019bade19c Remove <py3.5 dependencies from requirements.txt
The requirements repo is support python 3.5 as oldest python version
while swift still supports py27.

thus, requirements-check will fail on a couple of lines in swift.  The
check is only run when these files are touched.

The py2.7 packagers we know about aren't depending on upstream
requirements.txt for correctness and aside from all the production
deployments running on py2.7 we only realistically support >=py3.7

There's no good reason for our requirements.txt to be "unspported" by
the openstack requirements check job.  Since they only support >=py3.5
we can change our requirements.txt inline with that.  This should be
fine for everything we could hope to get out of both our
requirements.txt and the check!

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ibf8000498528c401707be8b0b91b8355cd993786
2020-06-03 08:30:43 +02:00
Tim Burke
738514c164 docs: s3api doesn't support tagging
Change-Id: I227e8c2defe96fb5f33054167677ae551830d9e8
2020-06-02 21:51:00 -07:00
Zuul
45a9ca39a4 Merge "Enable s3api and staticweb tests across all func tests" 2020-06-02 22:25:15 +00:00
Zuul
b26d208b61 Merge "Simplify wsgify()" 2020-06-02 19:53:58 +00:00
Thiago da Silva
20c6bdb71c Enable s3api and staticweb tests across all func tests
This patch removed the separate s3api, staticweb functional tests
gate jobs and added them across all other functional test jobs.

Change-Id: Ie1c606132a054defc2b3cc14a66031090e7b8449
2020-06-01 19:16:48 -07:00
Tim Burke
99947150dd func tests: work with etag-quoter on by default
Also, run the in-process encryption func tests like that.

Change-Id: I984ab8d1304d23b89589973950b10dda4aea0db3
2020-06-01 18:38:23 -05:00
Zuul
6c1bc3949d Merge "ratelimit: Allow multiple placements" 2020-06-01 21:25:41 +00:00
Clay Gerrard
ede9dad9f6 Better functest quarantine cleanup
Change-Id: I9218aaeb5fcd21f1bc2a5d655e3216059a209aeb
2020-06-01 14:43:05 +00:00
Zuul
4851c11600 Merge "Ussuri contrib docs community goal" 2020-05-30 06:55:23 +00:00
Zuul
9b1ca0eb67 Merge "dsvm: Run service-user tests under keystoneauth" 2020-05-30 01:07:34 +00:00
Zuul
4f440ad44d Merge "dsvm: Run service-user tests under tempauth" 2020-05-28 00:27:40 +00:00
Zuul
e545a9291f Merge "dsvm: Run domain_remap tests with tempauth" 2020-05-28 00:27:38 +00:00
Zuul
8f1a40eaae Merge "swift-dsvm: Enable s3api" 2020-05-28 00:27:36 +00:00
Tim Burke
60f052f69a dsvm: Run service-user tests under keystoneauth
Change-Id: I5b0ae0b78d37a31928e0b34a1b6118802a3a8236
2020-05-27 15:07:52 -07:00
Tim Burke
3aa4e3ec85 Remove swift-dsvm-functional-py3 job
Following https://github.com/openstack/devstack/commit/6b6bdc711,
the swift-dsvm-functional job has been running everything under
Python 3 anyway.

Change-Id: Ie285f513e1ed71dbaf4e12fe747ea6087664f843
2020-05-27 13:16:55 -07:00
OpenStack Proposal Bot
650272eacb Imported Translations from Zanata
For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

Change-Id: I1bb2aa262730d45a5852eaa3b3351f455a6ab337
2020-05-27 10:03:59 +00:00
Zuul
ee59909b65 Merge "Skip tests when only changing README.rst, CONTRIBUTING.rst, etc." 2020-05-27 04:30:54 +00:00
Tim Burke
fa768b4342 Simplify wsgify()
Change-Id: Iec399aa8b58e72152a17265f2af1131f02667131
2020-05-27 03:19:13 +00:00
Zuul
5d9c373618 Merge "versioning: Have versioning symlinks make pre-auth requests to reserved container" 2020-05-27 00:15:58 +00:00
Tim Burke
5034916c18 Skip tests when only changing README.rst, CONTRIBUTING.rst, etc.
Change-Id: Ib78398111ee3d150b6f56aa4e59a65750aa115dd
2020-05-26 16:42:52 -07:00
Matthew Oliver
78cce72f8a Ussuri contrib docs community goal
This patch standardizes the CONTRIBUTING.rst file and adds the
required doc/source/contributor/contributing.rst

Swift already had a detailed CONTIRBUTING.rst and an informative
REVIEW_GUIDELINES.rst in the root of the repo. So we are also pulling
them into the contributor documentation so they can not only be easily
found in the checked repo but in the online documentation.

Change-Id: I4c84efbe50eb25ab922c9d6b69198dae341af48b
2020-05-26 15:06:02 -07:00
Zuul
7b6546b076 Merge "fix s3api functional tests" 2020-05-26 20:36:06 +00:00
Zuul
22719f0d0d Merge "Breakup reclaim into batches" 2020-05-26 20:36:03 +00:00
Tim Burke
73f0b143d1 dsvm: Run service-user tests under tempauth
Change-Id: I0bdd3a1d044f8f99873f6270ca821862bb994d72
2020-05-26 12:54:17 -07:00
Tim Burke
63b16a3683 dsvm: Run domain_remap tests with tempauth
Change-Id: Idcc33c71869ccf7ad5a3134767d36ae94aab4598
2020-05-26 12:41:03 -07:00
Zuul
c3e6575a5a Merge "Test for Versioned Object If-Match" 2020-05-26 18:04:29 +00:00
Tim Burke
a8e03f42e0 versioning: Have versioning symlinks make pre-auth requests to reserved container
Previously, the lack of container ACLs on the reserved container would
mean that attempting to grant access to the user-visible container would
not work; the user could not access the backing object.

Now, have symlinks with the allow-reserved-names sysmeta set be
pre-authed. Note that the user still has to be authorized to read the
symlink, and if the backing object was *itself* a symlink, that will be
authed separately.

Change-Id: Ifd744044421ef2ca917ce9502b155a6514ce8ecf
Closes-Bug: #1880013
2020-05-26 10:09:56 -05:00
Clay Gerrard
63e02fa9fa Test for Versioned Object If-Match
Regular objects can do conditional requests, versioned objects should be
able to as well.

Change-Id: If69dbf2a4c876fa4b830f6d8652f22f39ceaf549
2020-05-26 13:49:15 +00:00
Tim Burke
bb9b0326fd swift-dsvm: Enable s3api
Depends-On: https://review.opendev.org/#/c/571021/
Change-Id: I3ac3288cd61b745ce7dbf2bded8eade026d0418f
2020-05-22 16:04:52 -07:00
Zuul
c5191a040a Merge "Add py38 package metadata" 2020-05-22 22:15:58 +00:00
Clay Gerrard
aab45880f8 Breakup reclaim into batches
We want to do the table scan without locking and group the locking
deletes into small indexed operations to minimize the impact of
background processes calling reclaim each cycle.

Change-Id: I3ccd145c14a9b68ff8a9da61f79034549c9bc127
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Closes-Bug: #1877651
2020-05-22 21:15:57 +00:00
zhangboye
f25705ee13 Add py38 package metadata
Change-Id: Icd5312dec647f69a5df111c3a6766093ecfcb6ef
2020-05-22 15:31:08 +08:00
Tim Burke
67598b3a4a Bump up timeouts for DSVM and probe test jobs
DSVM recently got a bunch more middlewares enabled, so it's running more
tests than it used to.

I can't think of much that's changed for probe tests, but I feel like
I've seen it popping timeouts more often lately.

Note that the new timeouts are still lower than the typical run-time of
our longest-running jobs (currently grenade / tempest-ipv6-only).

Change-Id: Iffbb567124096df02b04981550faec8204d5d1ec
Related-Change: I3cbbcd2ea9ced0923bee4a6b0783e4cf5e82e95b
2020-05-20 15:22:12 -07:00
Tim Burke
1db11df4f2 ratelimit: Allow multiple placements
We usually want to have ratelimit fairly far left in the pipeline -- the
assumption is that something like an auth check will be fairly expensive
and we should try to shield the auth system so it doesn't melt under the
load of a misbehaved swift client.

But with S3 requests, we can't know the account/container that a request
is destined for until *after* auth. Fortunately, we've already got some
code to make s3api play well with ratelimit.

So, let's have our cake and eat it, too: allow operators to place
ratelimit once, before auth, for swift requests and again, after auth,
for s3api. They'll both use the same memcached keys (so users can't
switch APIs to effectively double their limit), but still only have each
S3 request counted against the limit once.

Change-Id: If003bb43f39427fe47a0f5a01dbcc19e1b3b67ef
2020-05-19 11:10:22 -07:00
Zuul
70dede1ae9 Merge "versioning: Use CONTAINER_LISTING_LIMIT when doing container listings" 2020-05-17 08:21:39 +00:00
Zuul
b8afd36fd8 Merge "object-updater: Ignore ENOENT when trying to unlink stale pending files" 2020-05-17 05:08:59 +00:00
Zuul
31645d5bd4 Merge "updater: Shuffle suffixes so we don't keep hitting the same failures" 2020-05-16 06:39:50 +00:00
Thiago da Silva
f4bc951508 fix s3api functional tests
Connection was hard coded to use saio config

Change-Id: I9c11162de89fa3aa2a78aea093b187d0309860f5
Signed-off-by: Thiago da Silva <thiago@redhat.com>
2020-05-15 22:38:04 -07:00
Tim Burke
1358d37328 versioning: Use CONTAINER_LISTING_LIMIT when doing container listings
Change-Id: If274a01a6660776cf01b71efbdd102159c520794
2020-05-15 16:03:02 -07:00
Zuul
08db36a295 Merge "py27: Suppress UnicodeWarnings in ShardRange setters" 2020-05-13 17:44:56 +00:00
Tim Burke
0fd23ee546 Fix pep8 job
New flake8 came out with new & improved rules. Ignore E741; it would be
too much churn. Fix the rest.

Change-Id: I9125c8c53423232309a75cbcc5b695b378864c1b
2020-05-13 00:24:13 -07:00
Tim Burke
dee98a74d4 updater: Shuffle suffixes so we don't keep hitting the same failures
When tuning your updater, you often want to try a new config, see how it
changes your metrics, then adjust concurrency up or down depending on
how your container layer is responding.

If your containers haven't been doing well, though, and you've got a
giant backlog of async pendings to work through, updater restarts to
change concurrency previously posed a problem: the updater would walk
the suffix directories in the same order every start-up. So, if you
found a config that was making decent progress for a while but still had
*some* failures, and you wanted to try tweaking settings to see if you
could *reduce* those failures -- you'd likely start getting *all*
failures as it went to retry the failed ones first and all at once. If
you continued trying to tweak configs to get your failures to a
reasonable rate, you'd almost certainly over-correct for these handful 
of overwhelmed DBs and not the overall cluster.

Now, shuffle the suffixes before we walk them.

Change-Id: I3ef34119f0cb563ab405a6517335a24dbaf2b4c3
Closes-Bug: #1878056
2020-05-11 17:33:31 +00:00
Tim Burke
f57d4cfa71 object-updater: Ignore ENOENT when trying to unlink stale pending files
Change-Id: Iaac1fb891d70707af38c567d9cca5913b8355b7d
Closes-Bug: #1877924
2020-05-11 00:23:34 -07:00
Zuul
d050ef82f7 Merge "Use local py2 upper-constraints" 2020-05-07 09:54:44 +00:00
Zuul
a792255e79 Merge "swift-dsvm: Enable more middlewares" 2020-05-06 19:37:13 +00:00