This patch adds a new object versioning mode. This new mode provides
a new set of APIs for users to interact with older versions of an
object. It also changes the naming scheme of older versions and adds
a version-id to each object.
This new mode is not backwards compatible or interchangeable with the
other two modes (i.e., stack and history), especially due to the changes
in the namimg scheme of older versions. This new mode will also serve
as a foundation for adding S3 versioning compatibility in the s3api
middleware.
Note that this does not (yet) support using a versioned container as
a source in container-sync. Container sync should be enhanced to sync
previous versions of objects.
Change-Id: Ic7d39ba425ca324eeb4543a2ce8d03428e2225a1
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Co-Authored-By: Thiago da Silva <thiagodasilva@gmail.com>
This allows static symlinks to be synced before their target. Dynamic
symlinks could already be synced even if target object has not been
synced, but static links previously required that target object existed
before it can be PUT. Now, have container_sync middleware plumb in an
override like it does for SLO.
Change-Id: I3bfc62b77b247003adcee6bd4d374168bfd6707d
Whitelisted headers include X-Delete-At/X-Delete-After and all Object
metadata headers (X-Object-Meta-*)
Closes-Bug: 1857546
Change-Id: If5fb164693e395f89d57899fb8ab355f1e3f817c
Prior to the related change, clients may have written down X-Delete-At headers
that are outside of the Timestamp range, for example.
Change-Id: Ib8ae7ebcbdb32e0aa58446bd1ef949e5e2f63e74
Related-Change: I23666ec8a067d829eaf9bfe54bd086c320b3429e
Related-Bug: 1821204
Partial-Bug: 1860149
Previously, we took the native string filename attribute and put it
directly in the (WSGI string) PATH_INFO field. Now, we convert it to
a WSGI string first.
Change-Id: I30e3beb8707b88c36bd3cdc7a0887d069e943ba6
Closes-Bug: #1858259
unittest2 was needed for Python version <= 2.6, so it hasn't been needed
for quite some time. See unittest2 note one:
https://docs.python.org/2.7/library/unittest.html
This drops unittest2 in favor of the standard unittest module.
Change-Id: I2e787cfbf1709b7f9c889230a10c03689e032957
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Apparently FakeSwift has always been persnickety about the order of
query params of registered responses and the app making the calls.
Since query params can often be converted to dictionaries the order of
the encoded params should be able to change without effecting the apps
dispatching of registered responses.
Change-Id: Ied68c9334201a7663e9c85f3bdaa5b0643d4b6db
This is to fix the duplicated words issue, like:
* "both of which which must be stripped off"
* "In addition the the values set"
and so on
Change-Id: Id3d84281f15815b4185c76874575e91a3589981b
If we move it to constraints it's more globally accessible in our code,
but more importantly it's more obvious to ops that everything breaks if
you try to mis-configure different values per-service.
Change-Id: Ib8f7d08bc48da12be5671abe91a17ae2b49ecfee
An odd thing happened: when my code did only
"from swift.common.ring import Ring", and nothing else,
the interpreter looped on the CPU.
The import from the top of the ring module is our standard
protocol for Ring. It causes no problem in places like
container updater or list_endpoints.py. It is a big
mystery why it causes Python 3.7.3 to loop, and only
in specific circumstances.
But we noticed that the recursive import is unnecesasry
in this case, so an obviously harmless fix exists.
Change-Id: I7373bbb0a50d090d6becf141e5832f8ae98381a4
Previously, if you were on Python 2.7.10+ [0], such a newline would cause the
sharder to fail, complaining about invalid header values when trying to create
the shard containers. On older versions of Python, it would most likely cause a
parsing error in the container-server that was trying to handle the PUT.
Now, quote all places that we pass around container paths. This includes:
* The X-Container-Sysmeta-Shard-(Quoted-)Root sent when creating the (empty)
remote shards
* The X-Container-Sysmeta-Shard-(Quoted-)Root included when initializing the
local handoff for cleaving
* The X-Backend-(Quoted-)Container-Path the proxy sends to the object-server
for container updates
* The Location header the container-server sends to the object-updater
Note that a new header was required in requests so that servers would
know whether the value should be unquoted or not. We can get away with
reusing Location in responses by having clients opt-in to quoting with
a new X-Backend-Accept-Quoted-Location header.
During a rolling upgrade,
* old object-servers servicing requests from new proxy-servers will
not know about the container path override and so will try to update
the root container,
* in general, object updates are more likely to land in the root
container; the sharder will deal with them as misplaced objects, and
* shard containers created by new code on servers running old code
will think they are root containers until the server is running new
code, too; during this time they'll fail the sharder audit and report
stats to their account, but both of these should get cleared up upon
upgrade.
Drive-by: fix a "conainer_name" typo that prevented us from testing that
we can shard a container with unicode in its name. Also, add more UTF8
probe tests.
[0] See https://bugs.python.org/issue22928
Change-Id: Ie08f36e31a448a547468dd85911c3a3bc30e89f1
Closes-Bug: 1856894
Since we don't use 404s from handoffs anymore, we need to not let errors
on handoffs overwhelm primary responses either
Change-Id: I2624e113c9d945542f787e5f18f487bd7be3d32e
Closes-Bug: #1857909
Otherwise, we waste a request on some 416/206 response that won't be
helpful.
To do this, add a new X-Backend-Ignore-Range-If-Metadata-Present header
whose value is a comma-separated list of header names. Middlewares may
include this header to tell object-servers to send the whole object
(rather than a 206 or 416) if *any* of the metadata are present.
Have dlo and symlink use it, too; it won't save us any round-trips, but
it should clean up some object-server logging.
Change-Id: I4ff2a178d0456e7e37d561109ef57dd0d92cbd4e
Previously, passing a relative path would confuse the ContainerBroker about
which DB files are available, leading to an IndexError when none were found.
Just call realpath() on whatever the user provided so we don't have to muck
with any of the broker code.
Change-Id: Icdf100dfcd006d975b49d151b99aa9272452d013
This shouldn't have any practical effect, since *any* False-y value for
chunk will give us the correct behavior, but let's be consistent in the
type used with that name.
Change-Id: I1928883792e7a9945f3731ddeea992d14882d679
