Often, we want the current timestamp. May as well improve the ergonomics
a bit and provide a class method for it.
Change-Id: I3581c635c094a8c4339e9b770331a03eab704074
In particular, in TestObjectController.test_object_delete_at_async_update
Rarely (<0.1% of the time?), it would fail with:
======================================================================
FAIL: test_object_delete_at_async_update
(test.unit.obj.test_server.TestObjectController)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/vagrant/swift/test/unit/obj/test_server.py", line 4826, in
test_object_delete_at_async_update
resp = req.get_response(self.object_controller)
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/vagrant/swift/test/unit/__init__.py", line 1075, in
mocked_http_conn
raise AssertionError('left over status %r' % left_over_status)
AssertionError: left over status [500, 500]
-------------------- >> begin captured stdout << ---------------------
test INFO: None - - [26/Apr/2017:22:32:13 +0000] "PUT /sda1/p/a/c/o" 400
19 "-" "-" "-" 0.0003 "-" 23801 0
--------------------- >> end captured stdout << ----------------------
>> raise AssertionError('left over status %r' % [500, 500])
----------------------------------------------------------------------
Related-Bug: 1514111
Change-Id: I1af4a291fb67cf4b1829f167998a08644117a800
EC object metadata can currently have a mixture of bytestrings and
unicode. The ssync_sender.send_put() method raises an
UnicodeDecodeError when it attempts to concatenate the metadata
values, if any bytestring has non-ascii characters.
The root cause of this issue is that the object server uses unicode
for the keys of some object metadata items that are received in the
footer of an EC PUT request, whereas all other object metadata keys
and values are persisted as bytestrings.
This patch fixes the bug by changing diskfile write_metadata()
function to encode all unicode metadata keys and values as utf8
encoded bytes before writing to disk. To cope with existing objects
that have a mixture of unicode and bytestring metadata, the diskfile
read_metadata() function is also changed so that all returned unicode
metadata keys and values are utf8 encoded. This ensures that
ssync_sender.send_put() (and any other caller of diskfile
read_metadata) only reads bytestrings from object metadata.
Closes-Bug: #1678018
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: Ic23c55754ee142f6f5388dcda592a3afc9845c39
DiskFileMixin and DiskFileManagerMixin has almost
identical setUp() and tearDown() methods, and both
inherit BaseDiskFileTestMixin, so this moves the common
code into the abstract superclass.
Also remove repeated declaration of vars in
test_diskfile.py:run_quarantine_invalids
and a duplicated qualified import in obj/test_server.py
Change-Id: Id99ba151c7802c3b61e483a7e766bf6f2b2fe3df
I tried to move res.fix_conditional_response() into the get_log_line
method in utils but couldn't find a container request that was
*actually* conditional. We return 412 from the container on a couple
of bad requests where we really probably shouldn't - but regardless those
responses aren't really conditional.
Plus I'd have to make a synthetic test in utils somewhere.
So this seems better - leave the fix closer to the problem.
Change-Id: I76237d5d5025a653e879141b03df863205c59f3a
Related-Change-Id: I66c871f1f15fb43f19d59331b05cc7567ce10063
This only affects chunked transfers to replicated policies.
EC policies already have comparable error handling in
swift.common.utils._MultipartMimeFileLikeObject.read
See also: https://github.com/eventlet/eventlet/commit/c3ce3ee
but note that eventlet.wsgi.ChunkReadError and
swift.common.exceptions.ChunkReadError are entirely unrelated.
Change-Id: Ic9bcb73ea518c672a58a71d2b92152892f1ce2e4
The assumption that we don't need to write an entry in the invalidations
file when the hashes.pkl does not exist turned out to be a premature
optimization and also wrong.
Primarily we should recognize the creation of hashes.pkl is the first
thing that happens in a part when it lands on a new primary. The code
should be optimized toward the assumption of the most common disk state.
Also, in this case the extra stat calls to check if the hashes.pkl exists
were not only un-optimized - but introducing a race.
Consider the common case:
proc 1 | proc 2
-------------------------------|---------------------------
a) read then truncate journal |
b) do work | c) append to journal
d) apply "a" to index |
The index written at "d" may not (yet) reflect the entry writen by proc
2 at "c"; however, it's clearly in the journal so it's easy to see we're
safe.
Adding in the extra stat call for the index existence check increases
the state which can effect correctness.
proc 1 | proc 2
------------------------------|---------------------------
a) no index, truncate journal |
b) do work | b) iff index exists
| c) append to journal
d) apply (or create) index |
If step "c" doesn't happen because the index does not yet exist - the
update is clearly lost.
In our case we'd skip marking a suffix as invalid when the hashes.pkl
does not exist because we know "the next time we rehash" we'll have to
os.listdir suffixes anyway. But if another process is *currently*
rehashing (and has already done it's os.listdir) instead we've just
dropped an invalidation on the floor.
Don't do that.
Instead - write down the invalidation. The running rehash is welcome to
proceed on outdated information - as long as the next pass will grab and
hash the new suffix.
Known-Issue(s):
If the suffix already exists there's an even chance the running rehash
will hash in the very update for which we want to invalidate the suffix,
but that's ok it's idempotent.
Co-Author: Pavel Kvasnička <pavel.kvasnicka@firma.seznam.cz>
Co-Author: Alistair Coles <alistair.coles@hpe.com>
Co-Author: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Related-Change-Id: I64cadb1a3feb4d819d545137eecfc295389794f0
Change-Id: I2b48238d9d684e831d9777a7b18f91a3cef57cd1
Closes-Bug: #1651530
In [1] it was necessary to override the monkey patching
of mimetools so that mimetools would apply a default
Content-Type=text/plain header when missing from object server
requests. Otherwise some tests fail due to Content-Type=None.
This patch fixes the fragile tests by adding a Content-Type header,
and thus removes the need to override the mimetool monkey patching.
This is closer to the real world in which mimetools is patched in
the object server and PUT requests should have a Content-Type.
Drive-by fix some typos and apply config assertions to all policies.
[1] Related-Change: I5b5f90bb898a335e6336f043710a05a44e3b810f
Change-Id: I4c5bca42f753be4165cb731b8e3957eb4cdd28d5
The reclaim_age is a DiskFile option, it doesn't make sense for two
different object services or nodes to use different values.
I also driveby cleanup the reclaim_age plumbing from get_hashes to
cleanup_ondisk_files since it's a method on the Manager and has access
to the configured reclaim_age. This fixes a bug where finalize_put
wouldn't use the [DEFAULT]/object-server configured reclaim_age - which
is normally benign but leads to weird behavior on DELETE requests with
really small reclaim_age.
There's a couple of places in the replicator and reconstructor that
reach into their manager to borrow the reclaim_age when emptying out
the aborted PUTs that failed to cleanup their files in tmp - but that
timeout doesn't really need to be coupled with reclaim_age and that
method could have just as reasonably been implemented on the Manager.
UpgradeImpact: Previously the reclaim_age was documented to be
configurable in various object-* services config sections, but that did
not work correctly unless you also configured the option for the
object-server because of REPLICATE request rehash cleanup. All object
services must use the same reclaim_age. If you require a non-default
reclaim age it should be set in the [DEFAULT] section. If there are
different non-default values, the greater should be used for all object
services and configured only in the [DEFAULT] section.
If you specify a reclaim_age value in any object related config you
should move it to *only* the [DEFAULT] section before you upgrade. If
you configure a reclaim_age less that your consistency window you are
likely to be eaten by a Grue.
Closes-Bug: #1626296
Change-Id: I2b9189941ac29f6e3be69f76ff1c416315270916
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
This is just tests to demonstrate the current behavior - and provide
context of behavioral differences/consistency in the related change.
See the related change for context.
The related change slightly modifies the outcome of the
reclaim/REPLICATE test.
The related change has no effect on the config handling.
The related change greatly simplfies the reclaim_age option
cleanup_ondisk_files plumbing tests.
Related-Change-Id: I2b9189941ac29f6e3be69f76ff1c416315270916
Co-Author: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I5b5f90bb898a335e6336f043710a05a44e3b810f
This patch just adds some unit tests for the patch 395582
Change-Id: I78cda3f9a64b162926dc0d2b3bb8e68095bea71a
Signed-off-by: Thiago da Silva <thiago@redhat.com>
This patch makes the ECDiskFileReader check the validity of EC
fragment metadata as it reads chunks from disk and quarantine a
diskfile with bad metadata. This in turn means that both the object
auditor and a proxy GET request will cause bad EC fragments to be
quarantined.
This change is motivated by bug 1631144 which may result in corrupt EC
fragments being written to disk but appear valid to the object auditor
md5 hash and content-length checks.
NotImplemented:
* perform metadata check when a read starts on any frag_size
boundary, not just at zero
Related-Bug: #1631144
Closes-Bug: #1633647
Change-Id: Ifa6a7f8aaca94c7d39f4aeb9d4fa3f59c4f6ee13
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
When Object-Server saves async-update in pickle files,
'User-Agent: object-server %(pid)' (the key is title style) is included
in the pickles. However, Object-Updater will add
'user-agent: object-updater %(pid)' (the key is lower case style) to the
pickled headers and use it to make http connection. According to RFC
7230, each header field consists of a case-insensitive field name,
therefore either of 'User-Agent' and 'user-agent' should not be included
in the headers.
This patch removes old 'User-Agent' header from object-updater's
requests.
Change-Id: Ia624558395584457c718b311fe80e1a8406e22ad
Closes-Bug:#1635114
Instead of using a separate .durable file to indicate
the durable status of a .data file, rename the .data
to include a durable marker in the filename. This saves
one inode for every EC fragment archive.
An EC policy PUT will, as before, first rename a temp
file to:
<timestamp>#<frag_index>.data
but now, when the object is committed, that file will be
renamed:
<timestamp>#<frag_index>#d.data
with the '#d' suffix marking the data file as durable.
Diskfile suffix hashing returns the same result when the
new durable-data filename or the legacy durable file is
found in an object directory. A fragment archive that has
been created on an upgraded object server will therefore
appear to be in the same state, as far as the consistency
engine is concerned, as the same fragment archive created
on an older object server.
Since legacy .durable files will still exist in deployed
clusters, many of the unit tests scenarios have been
duplicated for both new durable-data filenames and legacy
durable files.
Change-Id: I6f1f62d47be0b0ac7919888c77480a636f11f607
It has valid_timestamp() to check timestamps in /POST/PUT/DELETE methods.
It has test_POST_bad_timestamps in Post, but it has no test_PUT/DELETE_bad_timestamps
int PUT/DELETE. Therefore,I add them
Change-Id: Ib531f3b7819c1c8dc69a7b51d990581bf1e24dab
test_GET_HEAD_with_fragment_preferences seemed to fail on ocassion on
my machine; the test seemed to be assuming the order of dictionary
keys in json output sent in headers; the fix was convert back to a
dictionary when making the assertion on the context of the json.
Change-Id: I1ea1b734c2a9fb12b8f59262bb1229421803e48e
Closes-Bug: #1625956
This is *only* sysmeta, to ensure we neither change the semantics of the
response (which could happen if we included allowed_headers) nor reveal
previously-written metadata to write-only clients (which could happen if
we ever get updateable object metadata and sent back user meta).
We could conceivably send back transient sysmeta, though it seems
better to wait for a use-case that demands it.
Change-Id: I04fe0b36b85e546b5379bed12089ffd1bce01fcf
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
This patch improves EC GET response handling:
- The proxy no longer requires all object servers to have a
durable file for the fragment archive that they return in
response to a GET. The proxy will now be satisfied if just
one object server has a durable file at the same timestamp
as fragments from other object servers.
This means that the proxy can now successfully GET an
object that had missing durable files when it was PUT.
- The proxy will now ensure that it has a quorum of *unique*
fragment indexes from object servers before considering a
GET to be successful.
- The proxy is now able to fetch multiple fragment archives
having different indexes from the same node. This enables
the proxy to successfully GET an object that has some
fragments that have landed on the same node, for example
after a rebalance.
This new behavior is facilitated by an exchange of new
headers on a GET request and response between the proxy and
object servers.
An object server now includes with a GET (or HEAD) response:
- X-Backend-Fragments: the value of this describes all
fragment archive indexes that the server has for the
object by encoding a map of the form: timestamp -> <list
of fragment indexes>
- X-Backend-Durable-Timestamp: the value of this is the
internal form of the timestamp of the newest durable file
that was found, if any.
- X-Backend-Data-Timestamp: the value of this is the
internal form of the timestamp of the data file that was
used to construct the diskfile.
A proxy server now includes with a GET request:
- X-Backend-Fragment-Preferences: the value of this
describes the proxy's current preference with respect to
those fragments that it would have object servers
return. It encodes a list of timestamp, and for each
timestamp a list of fragment indexes that the proxy does
NOT require (because it already has them).
The presence of a X-Backend-Fragment-Preferences header
(even one with an empty list as its value) will cause the
object server to search for the most appropriate fragment
to return, disregarding the existence or not of any
durable file. The object server assumes that the proxy
knows best.
Closes-Bug: 1469094
Closes-Bug: 1484598
Change-Id: I2310981fd1c4622ff5d1a739cbcc59637ffe3fc3
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
This change improves the readability of the object-server's unit
tests by breaking down some long assertTrue statements into smaller
and much easier to read and more relevant assert statements.
For example:
assertTrue(a in resp.headers and
b in resp.headers and
c not in resp.headers)
Is equal to:
assertIn(a, resp.headers)
assertIn(b, resp.headers)
assertNotIn(c, resp.headers)
Change-Id: Iba746ecfb1a1dc541856b7a4c9d2f00d08e4ad51
Adds one more scenario to the tests for preference of
container etag override values in sysmeta or backend
headers and footers.
Change-Id: Iacdaec8c98c7001029163a9d50321a13dc8d5a19
Related-Change: I074fbecb6440fb1d04279cd892d38d2acc44b47d
A make_timestamp_iter was added into setUp in the Related-Change.
There was a couple of different tests that were using timestamp iters
in different ways, consistently we apply the use of next(self.ts)
which I believe is becoming more common/standardized in unitests.
Related-Change: I074fbecb6440fb1d04279cd892d38d2acc44b47d
Change-Id: Ib6b883afec242355ae08c50c1e685a20e5efadc7
The change in preference to the Related-Change is simply to prefer
sysmeta overrides in the headers to backend overrides in the footers:
Before:
sysmeta footers > backend footers > sysmeta headers > backend headers
After:
sysmeta footers > sysmeta headers > backend footers > backend headers
This change just breaks up the tests to try to make it more obvious
what already worked and what has changed. The justification seems to
be that overrides in sysmeta headers only work on policies that don't
send backend footers, but sysmeta overrides should always have a
higher preference than backend overrides.
Related-Change: Idb40361ac72da51e1390dff690723dbc2c653a13
Change-Id: I074fbecb6440fb1d04279cd892d38d2acc44b47d
Previously, all footer overrides (whether from the X-Backend-* or
X-Object-Sysmeta-* namespace) would take priority over any header
override.
However, middleware should be able to set a Sysmeta override without
needing to worry about whether it's working with a replicated policy
(where setting it in headers will suffice) or an EC policy (where it
would need to install a footers callback). This could be mitigated by
*always* installing a footer callback, but doing so would incur
additional overhead that would otherwise be unnecessary.
Change-Id: Idb40361ac72da51e1390dff690723dbc2c653a13
Adds a new form of system metadata for objects.
Sysmeta cannot be updated by an object POST because
that would cause all existing sysmeta to be deleted.
Crypto middleware will want to add 'system' metadata
to object metadata on PUTs and POSTs, but it is ok
for this metadata to be replaced en-masse on every
POST.
This patch introduces x-object-transient-sysmeta-*
that is persisted by object servers and returned
in GET and HEAD responses, just like user metadata,
without polluting the x-object-meta-* namespace.
All headers in this namespace will be filtered
inbound and outbound by the gatekeeper, so cannot
be set or read by clients.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Change-Id: I5075493329935ba6790543fc82ea6e039704811d
Whatever container update override etag is sent to the object server
with a PUT must be used in container updates for subsequent
POSTs. Unfortunately the current container update override headers
(x-backend-container-update-override-*) are not persisted with the
object metadata so are not available when handling a POST.
For EC there is an ugly hack in the object server to use the
x-object-sysmeta-ec-[etag,size] values when doing a container update
for a POST.
With crypto, the encryption middleware needs to override the etag
(possibly overriding the already overridden EC etag value) with an
encrypted etag value. We therefore have a similar problem that this
override value is not persisted at the object server.
This patch introduces a new namespace for container override headers,
x-object-sysmeta-container-update-override-*, which uses object
sysmeta so that override values are persisted. This allows a general
mechanism in the object server to apply the override values (if any
have been set) from object sysmeta when constructing a container
update for a PUT or a POST. Middleware should use the
x-object-sysmeta-container-update-override-* namespace when setting
container update overrides. Middleware should be aware that other
middleware may have already set container override headers, in which
case consideration should be given to whether any existing value should
take precedence.
For backwards compatibility the existing
x-backend-container-update-override-* style headers are still
supported in the object server for EC override values, and the ugly
hack for EC etag/size override in POST updates remains in the object
server. That allows an older proxy server to be used with an upgraded
object server. The proxy server continues to use the
x-backend-container-update-override-* style headers for EC values so
that an older object server will continue to work with an upgraded
proxy server.
x-object-sysmeta-container-update-override-* headers take precedence
over x-backend-container-update-override-* headers and the use of
x-backend-container-update-override-* headers by middleware is
deprecated. Existing third party middleware that is using
x-backend-container-update-override-* headers should be modified to
use x-object-sysmeta-container-update-override-* headers in order to
be compatible with other middleware such as encryption and to ensure
that container updates during POST requests carry correct values. If
targeting multiple versions of Swift object servers it may be
necessary to send headers from both namespaces. However, in general it
is recommended to upgrade all backend servers, then upgrade proxy
servers before finally upgrading third party middleware.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
UpgradeImpact
Change-Id: Ib80b4db57dfc2d37ea8ed3745084a3981d082784
Before this patch, the proxy ObjectController supported sending
metadata from the proxy server to object servers in "footers" that
trail the body of HTTP PUT requests, but this support was for EC
policies only. The encryption feature requires that footers are sent
with both EC and replicated policy requests in order to persist
encryption specific sysmeta, and to override container update headers
with an encrypted Etag value.
This patch:
- Moves most of the functionality of ECPutter into a generic Putter
class that is used for replicated object PUTs without footers.
- Creates a MIMEPutter subclass to support multipart and multiphase
behaviour required for any replicated object PUT with footers and
all EC PUTs.
- Modifies ReplicatedObjectController to use Putter objects in place
of raw connection objects.
- Refactors the _get_put_connections method and _put_connect_node methods
so that more code is in the BaseObjectController class and therefore
shared by [EC|Replicated]ObjectController classes.
- Adds support to call a callback that middleware may have placed
in the environ, so the callback can set footers. The
x-object-sysmeta-ec- namespace is reserved and any footer values
set by middleware in that namespace will not be forwarded to
object servers.
In addition this patch enables more than one value to be added to the
X-Backend-Etag-Is-At header. This header is used to point to an
(optional) alternative sysmeta header whose value should be used when
evaluating conditional requests with If-[None-]Match headers. This is
already used with EC policies when the ECObjectController has
calculated the actual body Etag and sent it using a footer
(X-Object-Sysmeta-EC-Etag). X-Backend-Etag-Is-At is in that case set
to X-Object-Sysmeta-Ec-Etag so as to point to the actual body Etag
value rather than the EC fragment Etag.
Encryption will also need to add a pointer to an encrypted Etag value.
However, the referenced sysmeta may not exist, for example if the
object was created before encryption was enabled. The
X-Backend-Etag-Is-At value is therefore changed to support a list of
possible locations for alternate Etag values. Encryption will place
its expected alternative Etag location on this list, as will the
ECObjectController, and the object server will look for the first
object metadata to match an entry on the list when matching
conditional requests. That way, if the object was not encrypted then
the object server will fall through to using the EC Etag value, or in
the case of a replicated policy will fall through to using the normal
Etag metadata.
If your proxy has a third-party middleware that uses X-Backend-Etag-Is-At
and it upgrades before an object server it's talking to then conditional
requests may be broken.
UpgradeImpact
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Closes-Bug: #1594739
Change-Id: I12a6e41150f90de746ce03623032b83ed1987ee1
When a PUT request is made to an EC object the resulting container
update must include the override values for the actual object
etag and size, as opposed to the fragment etag and size. When a POST
request is made the same override values should be included in the
container update, but currently the update includes the incorrect EC
fragment size (but the correct body etag).
This is ok so long as the update for the object PUT request arrives at
the container server first (whether by direct update or replication)
because the etag and size values in an update due to an object POST
will not have a newer timestamp that the PUT and will therefore be
ignored at the container server.
However, if the update due to the object PUT request has not arrived
at the container server when the update due to the object POST
arrives, then the etag and incorrect size sent with the POST update
will be recorded in the container server. If the update due to the PUT
subsequently arrives it will not fix this error because the timestamp
of its etag and size values is not greater than that of the already
recorded values.
Fortunately the correct object body size is persisted with the object
as X-Backend-Container-Update-Override-Size sysmeta so this patch
fixes the container update due to a POST to use that value instead of
the Content-Length metadata.
Closes-Bug: #1582723
Change-Id: Ide7c9c59eb41aa09eaced2acfd0700f882c6eab1
hash_cleanup_listdir was removed in [1], this patch
renames all references to it in test_diskfile to refer to
the cleanup_ondisk_files method that is now tested directly.
Also remove the final references to the now non-existent
function in a few comments.
[1] I0b96dfde32b4c666eebda6e88228516dd693ef92
Change-Id: I1e151799fc2774de9a1af092afff875af24a630c
Related-Bug: #1550569
Changing the recommended ports for Swift services
from ports 6000-6002 to unused ports 6200-6202;
so they do not conflict with X-Windows or other services.
Updated SAIO docs.
DocImpact
Closes-Bug: #1521339
Change-Id: Ie1c778b159792c8e259e2a54cb86051686ac9d18
The fake_spawn() context manager wraps spawn and waits
for greenthreads to complete (such as the async_update threads).
The wait needs to be done before http_connect is un-mocked, so
the fake_spawn context manager should exit *before* any context
manager that mocks the http_connect method.
Also add fake_spawn to _test_PUT_then_POST_async_pendings()
Related-Bug: #1536376
Related-Bug: #1514111
Closes-Bug: #1555739
Change-Id: I15f36e191cfe3ee6c82b4be56e8618ec0230e328
Without the correct request header, the test that should be
using an EC policy was in fact using a replication policy.
Change-Id: Id44d0d615a4fd09aadfb286425939aea6abdf7b0
Each object update to a container server is saved in a
pending file if the initial update attempt fails. Pending
file names were derived from the update request's x-timestamp,
which is equal to the object's data file timestamp. This meant
that updates due to an object POST used the same async pending
file as updates due to the object's PUT.
This is not so bad because the object POST update has a superset
of the metadata included in the PUT update. But there is a risk
of a race condition causing an update to be lost: the updater may
open an update file due to a PUT whuile the object server is
writing an update due to a POST to the same file name. The
updater could then unlink the file before the more recent update
for the POST is sent.
This patch changes the POST update pending file name to be derived
from the object's metadata timestamp, thus making it distinct from
the PUT update pending file name. There is no upgrade impact since
existing pending files will continue to be processed.
Change-Id: I1b093c837efe8c2a64e92075ebd5e1b93e30efb9
There was a function in swift.common.utils that was importing
swob.HeaderKeyDict at call time. It couldn't import it at compilation
time since utils can't import from swob or else it blows up with a
circular import error.
This commit just moves HeaderKeyDict into swift.common.header_key_dict
so that we can remove the inline import.
Change-Id: I656fde8cc2e125327c26c589cf1045cb81ffc7e5
This patch makes a number of changes to enable content-type
metadata to be updated when using the fast-POST mode of
operation, as proposed in the associated spec [1].
* the object server and diskfile are modified to allow
content-type to be updated by a POST and the updated value
to be stored in .meta files.
* the object server accepts PUTs and DELETEs with older
timestamps than existing .meta files. This is to be
consistent with replication that will leave a later .meta
file in place when replicating a .data file.
* the diskfile interface is modified to provide accessor
methods for the content-type and its timestamp.
* the naming of .meta files is modified to encode two
timestamps when the .meta file contains a content-type value
that was set prior to the latest metadata update; this
enables consistency to be achieved when rsync is used for
replication.
* ssync is modified to sync meta files when content-type
differs between local and remote copies of objects.
* the object server issues container updates when handling
POST requests, notifying the container server of the current
immutable metadata (etag, size, hash, swift_bytes),
content-type with their respective timestamps, and the
mutable metadata timestamp.
* the container server maintains the most recently reported
values for immutable metadata, content-type and mutable
metadata, each with their respective timestamps, in a single
db row.
* new probe tests verify that replication achieves eventual
consistency of containers and objects after discrete updates
to content-type and mutable metadata, and that container-sync
sync's objects after fast-post updates.
[1] spec change-id: I60688efc3df692d3a39557114dca8c5490f7837e
Change-Id: Ia597cd460bb5fd40aa92e886e3e18a7542603d01
This renames test_object_delete_at_aysnc to
test_object_delete_at_async to match with the function
async_update which it tests.
Change-Id: I726afcbf08a3449c7af2834e573b97be378a86cd
Closes-Bug: 1550067
Swift object-server will return 409 Conflict when the incomming
request X-Timestamp is older than the local disk file to prevent
wasted data transfer and disk space consumption. However, IFF,
the local disk file is a tombstone, current object-server will
write the data in the device and cleanup it according to timestamp
comparison result in hash_cleanup_list_dir, and then return 201
Created. That's wasted and far from the semantics in the normal case.
This patch fixes object-server to catch the DiskFileDeleted exception
at PUT request, and then, set the timestamp from tombstone as
original_timestamp to be able to compare with request timestamp.
Co-Authored-By: Kazuhiro Miyahara <miyahara.kazuhiro@lab.ntt.co.jp>
Change-Id: I078c9cb90707a3c320708e76ea42cbfa73e1ea4b
Closes-Bug: #1546865
Previously, attempting to PUT a new object with an X-Timestamp header
less than or equal to zero (ie, for a timestamp on or before 1970-01-01
00:00:00) would cause the object-server to 500.
While we're at it, cap X-Timestamp at 9999999999 (2286-11-20 17:46:40)
so we don't get an eleventh digit before the decimal point.
Closes-Bug: 1532471
Change-Id: I23666ec8a067d829eaf9bfe54bd086c320b3429e
