We recently started trying to use O_TMPFILE without the kernel version
check, we gracefully degrade when it isn't available, and we don't give
operators any way to avoid the O_TMPFILE attempt -- logging at warning
seems way too noisy.
Change-Id: I8f42127fd6eb930282161930c65e8614f3e3175f
Related-Change: I3599e2ab257bcd99467aee83b747939afac639d8
Internal clients used to retry operations even on client errors, where
we had no expectation that we would get a different response. Among
other things, this would cause container-sync to needlessly retry
deletes that would 404, even though we had code specifically in place to
say that 404s are OK.
While we're at it, also flag 409s as being OK (since the remote has
newer data than us).
Change-Id: I1833a8ff2ac2f4126b6274d6bba47e2b58a41745
Closes-Bug: #1849841
This works fine; we continue processing the other rows in the DB. But it
*does* take longer than it really ought to require. See the related bug;
we ought to be able to shave some 17s off the test time by not retrying
on the 404.
Change-Id: I9ca2511651e9b2bc0045894baa4062d20bc15369
Related-Bug: #1849841
Whitelisted headers include X-Delete-At/X-Delete-After and all Object
metadata headers (X-Object-Meta-*)
Closes-Bug: 1857546
Change-Id: If5fb164693e395f89d57899fb8ab355f1e3f817c
Prior to the related change, clients may have written down X-Delete-At headers
that are outside of the Timestamp range, for example.
Change-Id: Ib8ae7ebcbdb32e0aa58446bd1ef949e5e2f63e74
Related-Change: I23666ec8a067d829eaf9bfe54bd086c320b3429e
Related-Bug: 1821204
Partial-Bug: 1860149
Previously, we took the native string filename attribute and put it
directly in the (WSGI string) PATH_INFO field. Now, we convert it to
a WSGI string first.
Change-Id: I30e3beb8707b88c36bd3cdc7a0887d069e943ba6
Closes-Bug: #1858259
unittest2 was needed for Python version <= 2.6, so it hasn't been needed
for quite some time. See unittest2 note one:
https://docs.python.org/2.7/library/unittest.html
This drops unittest2 in favor of the standard unittest module.
Change-Id: I2e787cfbf1709b7f9c889230a10c03689e032957
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Apparently FakeSwift has always been persnickety about the order of
query params of registered responses and the app making the calls.
Since query params can often be converted to dictionaries the order of
the encoded params should be able to change without effecting the apps
dispatching of registered responses.
Change-Id: Ied68c9334201a7663e9c85f3bdaa5b0643d4b6db
If we move it to constraints it's more globally accessible in our code,
but more importantly it's more obvious to ops that everything breaks if
you try to mis-configure different values per-service.
Change-Id: Ib8f7d08bc48da12be5671abe91a17ae2b49ecfee
Previously, if you were on Python 2.7.10+ [0], such a newline would cause the
sharder to fail, complaining about invalid header values when trying to create
the shard containers. On older versions of Python, it would most likely cause a
parsing error in the container-server that was trying to handle the PUT.
Now, quote all places that we pass around container paths. This includes:
* The X-Container-Sysmeta-Shard-(Quoted-)Root sent when creating the (empty)
remote shards
* The X-Container-Sysmeta-Shard-(Quoted-)Root included when initializing the
local handoff for cleaving
* The X-Backend-(Quoted-)Container-Path the proxy sends to the object-server
for container updates
* The Location header the container-server sends to the object-updater
Note that a new header was required in requests so that servers would
know whether the value should be unquoted or not. We can get away with
reusing Location in responses by having clients opt-in to quoting with
a new X-Backend-Accept-Quoted-Location header.
During a rolling upgrade,
* old object-servers servicing requests from new proxy-servers will
not know about the container path override and so will try to update
the root container,
* in general, object updates are more likely to land in the root
container; the sharder will deal with them as misplaced objects, and
* shard containers created by new code on servers running old code
will think they are root containers until the server is running new
code, too; during this time they'll fail the sharder audit and report
stats to their account, but both of these should get cleared up upon
upgrade.
Drive-by: fix a "conainer_name" typo that prevented us from testing that
we can shard a container with unicode in its name. Also, add more UTF8
probe tests.
[0] See https://bugs.python.org/issue22928
Change-Id: Ie08f36e31a448a547468dd85911c3a3bc30e89f1
Closes-Bug: 1856894
Since we don't use 404s from handoffs anymore, we need to not let errors
on handoffs overwhelm primary responses either
Change-Id: I2624e113c9d945542f787e5f18f487bd7be3d32e
Closes-Bug: #1857909
Otherwise, we waste a request on some 416/206 response that won't be
helpful.
To do this, add a new X-Backend-Ignore-Range-If-Metadata-Present header
whose value is a comma-separated list of header names. Middlewares may
include this header to tell object-servers to send the whole object
(rather than a 206 or 416) if *any* of the metadata are present.
Have dlo and symlink use it, too; it won't save us any round-trips, but
it should clean up some object-server logging.
Change-Id: I4ff2a178d0456e7e37d561109ef57dd0d92cbd4e
... just like they always have and server per port strategy still does.
Related-Change-Id: I3e5229d2fb04be67e53533ff65b0870038accbb7
Change-Id: I14e3ed201ceaceef0f8dbc44685395f350a0e7fc
Starting with Python 3.4, newly-created file descriptors are non-inheritable
[0], which causes trouble when we try to use a pipe for IPC. Fortunately, the
same PEP that implemented this change *also* provided a new API to mark file
descriptors as being inheritable -- so just do that.
While we're at it,
* Fix up the probe tests to work on py3
* Fix up the probe tests to work when policy-0 is erasure-coded
* Decode the bytes read so py3 doesn't log a b'pid'
* Log a warning if the read() is empty; something surely went wrong
in the re-exec
[0] https://www.python.org/dev/peps/pep-0446/
Change-Id: I2a8a9f3dc78abb99bf9cbcf6b44c32ca644bb07b
Related-Change: I3e5229d2fb04be67e53533ff65b0870038accbb7
Reserve the namespace starting with the NULL byte for internal
use-cases. Backend services will allow path names to include the NULL
byte in urls and validate names in the reserved namespace. Database
services will filter all names starting with the NULL byte from
responses unless the request includes the header:
X-Backend-Allow-Reserved-Names: true
The proxy server will not allow path names to include the NULL byte in
urls unless a middlware has set the X-Backend-Allow-Reserved-Names
header. Middlewares can use the reserved namespace to create objects
and containers that can not be directly manipulated by clients. Any
objects and bytes created in the reserved namespace will be aggregated
to the user's account totals.
When deploying internal proxys developers and operators may configure
the gatekeeper middleware to translate the X-Allow-Reserved-Names header
to the Backend header so they can manipulate the reserved namespace
directly through the normal API.
UpgradeImpact: it's not safe to rollback from this change
Change-Id: If912f71d8b0d03369680374e8233da85d8d38f85
Swift servers can now be seamlessly reloaded by sending them a SIGUSR1
(instead of a SIGHUP). The server forks off a synchronized child to
wait to close the old listen socket(s) until the new server has started
up and bound its listen socket(s). The new server is exec'ed from the
old one so its PID doesn't change. This makes Systemd happier, so a
ReloadExec= stanza can now be used.
The seamless part means that incoming connections will alwyas get
accepted either by the old server or the new one. This eliminates
client-perceived "downtime" during server reloads, while allowing the
server to fully reload, re-reading configuration, becoming a fresh
Python interpreter instance, etc. The SO_REUSEPORT socket option has
already been getting used, so nothing had to change there.
This patch also includes a non-invasive fix for a current eventlet bug;
see https://github.com/eventlet/eventlet/pull/590
That bug prevents a SIGHUP "reload" from properly servicing existing
requests before old worker processes close sockets and exit. The
existing probtests missed this, but the new ones, in this patch, caught
it.
New probe tests cover both old SIGHUP "reload" behavior as well as the
new SIGUSR1 seamless reload behavior.
Change-Id: I3e5229d2fb04be67e53533ff65b0870038accbb7
Currently, The internal client module is missing a delete_account method.
I added a delete_account in InternalClient
Change-Id: I97cb090820b4099944ba49eaeb5b02f9bbcf1d58
Closes-Bug: #1680377
It was parctically the same between both, only the table name was
different -- and we already have a way to accomodate that.
Change-Id: I112bef1cf44c7d6eeaacd30cea27cb71fb2e2406
Internal Client's doest not read body for 404.
So, When internal deleing object that not existed, proxy-server logs
its response code to 499.
In this patch, add a code that drain the reponse body to prevent
unexpected disconnect in proxy-server
Change-Id: I6d170fead798d0d539c69d27a6fa8a2d0123ca99
Closes-Bug: #1835324
These are known to not work until https://bugs.python.org/issue37093
is addressed in CPython upstream.
Change-Id: I4a6877907d14b632a9a477c887913488427b62b7
