The proxy_logging middleware needs an X-Backend-Storage-Policy-Index
header to populate the storage policy field in logs, and will look in
both request and response headers to find it.
Previously, the s3api middleware would indiscriminately copy the
X-Backend-Storage-Policy-Index from swift backend requests into the
S3Request headers [1]. This works for logging but causes the header
to leak between backend requests [2] and break mixed policy
multipart uploads. This patch sets the X-Backend-Storage-Policy-Index
header on s3api responses rather than requests.
Additionally, the middleware now looks for the
X-Backend-Storage-Policy-Index header in the swift backend request
*and* response headers, in the same way that proxy_logging would
(preferring a response header over a request header). This means that
a policy index is now logged for bucket requests, which only have
X-Backend-Storage-Policy-Index header in their response headers.
The s3api adds the value from the *final* backend request/response
pair to its response headers. Returning the policy index from the
final backend request/response is consistent with swift.backend_path
being set to that backend request's path i.e. proxy_logging will log
the correct policy index for the logged path.
The FakeSwift helper no longer looks in registered object responses
for an X-Backend-Storage-Policy-Index header to update an object
request. Real Swift object responses do not have an
X-Backend-Storage-Policy-Index header. By default, FakeSwift will now
update *all* object requests with an X-Backend-Storage-Policy-Index as
follows:
- If a matching container HEAD response has been registered then
any X-Backend-Storage-Policy-Index found with that is used.
- Otherwise the default policy index is used.
Furthermore, FakeSwift now adds the X-Backend-Storage-Policy-Index
header to the request *after* the request has been captured. Tests
using FakeSwift.calls_wth_headers() to make assertions about captured
headers no longer need to make allowance for the header that FakeSwift
added.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Closes-Bug: #2038459
[1] Related-Change: I5fe5ab31d6b2d9f7b6ecb3bfa246433a78e54808
[2] Related-Change: I40b252446b3a1294a5ca8b531f224ce9c16f9aba
Change-Id: I2793e335a08ad373c49cbbe6759d4e97cc420867
For tox 3.x and earlier, passenv was a space-separated list; as of tox
4.0.0, it's comma-separated. For a while, our spaces would be silently
included in the now-one-and-only passenv value parsed (which wasn't
great, but mostly just caused confusion) -- as of tox 4.0.6, however, it
became a hard error, and all tests would fail like
pass_env values cannot contain whitespace, use comma to have multiple
values in a single line, invalid values found 'SWIFT_* *_proxy'
Unfortunately, we don't really know what versions of tox all our various
stakeholders might want/need to use (though we previously set a
minversion of 2.3.2). We might be able to spread values over multiple
lines to make it compatible with both tox 3 *and* tox 4, but I'm fairly
certain *_proxy was only included for some variables that are recent
versions of tox include by default anyway, so just increase our
minversion (which was too low, anyway -- allowlist_externals which we
already configure was added in 3.18.0) and get rid of *_proxy.
FWIW, python-swiftclient was already specifying 3.18.0 as a minversion,
so I expect the new minversion to not be a problem.
Also, add ./.functests to a bunch of allowlist_externals, as newer tox
is more strict about that sort of thing.
Drop skipsdist in a bunch of places so we can import swift from func
tests and docs. (Still not sure why I don't see us hitting a similar
problem for unit tests...)
Change-Id: I4be1e86e3291ad1619c695fb93d7cadf053b556d
nose has not seen active development for many years now. With py310, we
can no longer use it due to import errors.
Also update lower contraints
Closes-Bug: #1993531
Change-Id: I215ba0d4654c9c637c3b97953d8659ac80892db8
PasteDeploy version 3.0 (2022-10-16) dropped support for python2 (as
well as <3.7), which causes our py2 tests to fail. So cap the version
here at 2.1.1, the last which is compatible with our tests.
Even doing this doesn't stop pip install swift pulling in a newer
PasteDeploy in the
tools/playbooks/saio_single_node_setup/make_rings.yaml playbook
(causing the probes test on CentOS-7 to fail); so handle CentOS 7
explicitly.
Change-Id: If69ae0f8eac8fe8ff7d5e4f4f1bff6d0ea9e7a8b
Signed-off-by: Matthew Vernon <mvernon@wikimedia.org>
Now that OpenStack writ large has dropped support for py36,
we're going to have to manage our constraints ourselves
more, just like we did for py27.
Change-Id: I0973be6692e3a9b871eb61cbf759e13644a25107
As part of that, the ceph test runner needed up-rev'ing to run under
py3. As a result, the known-failures shifted.
Trim the on-demand rolling upgrade jobs list -- now that it's running
py3, we only expect it to pass for train and beyond.
Also, pin smmap version on py2 -- otherwise, the remaining experimental
jobs running on centos-7 fail.
Change-Id: Ibe46aecf0f4461be59eb206bfe9063cc1bfff706
We've recently started seeing some failures in the gate related to these
projects, and they have final py2-supporting versions.
Change-Id: If81fc352c8b2b1f03f3fa7b79c56dfcf981ced70
Some selenium webdrivers (e.g. Chrome, Firefox) support a headless
option so we can expose that as an option for the test runner too.
Use this in an attempt to fix "Error: cannot open display: :99" errors
seen in the gate.
Related-Bug: #1918864
Change-Id: I2a549ce829eb0bc38406575582202e1d8dd1a0e2
Drive-bys:
* Only copy outputs when there are outputs to copy
* Print tracebacks when there's an issue running a selenium driver
Change-Id: I0807af4525a13a30baf27ada40eeabe311b44296
This adds support for presigned GET URLs, at least.
Note that there is no support yet for preflight requests, so a whole
bunch of other CORS stuff *doesn't* work (yet). This was just an easy
first step.
Change-Id: I43150a630a2a7620099e6bfecaed3bbe958ba423
If you've got selenium installed (and working), the whole thing can be
automated pretty well; run main.py, wait while some windows pop up (or
use xvfb-run to run things on a virtual display), then check out what
tests were run on which browsers and whether any of them failed. Exit
code is the number of failed tests.
Includes tests against:
- Account
- Containers, with various ACLs/CORS settings
- Objects
- /info
- SLOs
- DLOs
- Symlinks
Include a gate job that runs the tests in firefox.
Areas for future work:
- Install chromium and chromedriver in the gate; tests should
automatically pick up on the fact that it's available
- Capture the web browser's console logs, too, so we can get
more info when things go wrong
Change-Id: Ic1d3a062419f1133c6e2f00a598867d567358c9f
On every sharder cycle up update in progress recon stats for each sharding
container. However, we tend to not run it one final time once sharding
is complete because the DB state is changed to SHARDED and therefore the
in_progress stats never get their final update.
For those collecting this data to monitor, this makes sharding/cleaving shards
never complete.
This patch, adds a new option `recon_shared_timeout` which will now
allow sharded containers to be processed by `_record_sharding_progress()`
after they've finished sharding for an amount of time.
Change-Id: I5fa39d41f9cd3b211e45d2012fd709f4135f595e
We've seen failures with probe tests lately where dnspython 2.0.0 is
getting installed even though it doesn't support py2 anymore. I think
using latest pip should be better about noticing that and installing the
last 1.x release intead?
Change-Id: I6eda54ccd2792effadb334ce9324887132b62b6f
Note that existing SAIOs with 60xx ports should still work fine.
Change-Id: If5dd79f926fa51a58b3a732b212b484a7e9f00db
Related-Change: Ie1c778b159792c8e259e2a54cb86051686ac9d18
Hopefully this will fix the currently-broken probe test gate?
Depends-On: https://review.opendev.org/#/c/736070/
Change-Id: Ib652534b35236fdb6bcab131c7dc08a079bf72f6
crudini seems to have trouble on py3 -- still not sure *why* it's using
py3 for the losf job, though...
Change-Id: Id98055994c8d59e561372417c9eb4aec969afc6a
* Stop specifying logbufs=8; that's the default
* Stop including nodiratime with noatime; the latter implies the former
Nothing wrong with being explicit, I suppose, but may as well keep the
mount options to what we can easily explain: we want noatime because
Swift does not use atime, so we don't want to lose any performance to
tracking atime.
Change-Id: I1e52b4368ad7eb375964eee5132bc50297536355
XFS no longer supports nobarrier mount option.
It has been deprecated for a long time[1] and removed in
recent kernel versions resulting in an error when trying to
mount: "kernel: XFS (loop0): unknown mount option [nobarrier]."
[1] - https://patchwork.kernel.org/patch/9486549/
Change-Id: Iaa9208fb20545ae9ac990f0e180899108d983123
Give storage nodes more time to complete requests for multi-node upgrade
and probetests.
Also slightly decouple probetests from default configs.
Change-Id: I334ef517d833916a3b7be3151a812d4f9c66a6e1
Previously, we'd install development versions of Swift as root, causing
later tox runs as zuul to fail on a permissions error because the
generated egg-info (at least) was locked down.
Change-Id: Ia688790f8b23ed1cf76947b5809c208df5dee8bb
Note that eventlet 0.22.0+ closes connections between requests when
it stops accepting connections.
Partial-Bug: #1792615
Change-Id: Ia8d9ab95e2aad40e8d797acc3423a917e809ffdb
