Rewrite server side copy and 'object post as copy' feature as middleware to
simplify the PUT method in the object controller code. COPY is no longer
a verb implemented as public method in Proxy application.
The server side copy middleware is inserted to the left of dlo, slo and
versioned_writes middlewares in the proxy server pipeline. As a result,
dlo and slo copy_hooks are no longer required. SLO manifests are now
validated when copied so when copying a manifest to another account the
referenced segments must be readable in that account for the manifest
copy to succeed (previously this validation was not made, meaning the
manifest was copied but could be unusable if the segments were not
readable).
With this change, there should be no change in functionality or existing
behavior. This is asserted with (almost) no changes required to existing
functional tests.
Some notes (for operators):
* Middleware required to be auto-inserted before slo and dlo and
versioned_writes
* Turning off server side copy is not configurable.
* object_post_as_copy is no longer a configurable option of proxy server
but of this middleware. However, for smooth upgrade, config option set
in proxy server app is also read.
DocImpact: Introducing server side copy as middleware
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Change-Id: Ic96a92e938589a2f6add35a40741fd062f1c29eb
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Signed-off-by: Thiago da Silva <thiago@redhat.com>
In addition to the container sync stat. report, keeping per container
statistics allows administrator with more control over bytes
transfered over a specific time per user account: The per container stats
are crucial for billing purposes and provides the operator a 'progress
bar' equivalent on the container's replication status.
Change-Id: Ia8abcdaf53e466e8d60a957c76e32c2b2c5dc3fa
DiskFile already fills in the _ondisk_info attribute when it tries to open
a diskfile - even if the DiskFile's fileset is not valid or deleted.
During this process the rsync tempfiles would be discovered and logged,
but no-one would attempt to clean them up - even if they were really old.
Instead of logging and ignoring unexpected files when validate a DiskFile
fileset we'll add unexpected files to the unexpected key in the
_ondisk_info attribute.
With a little bit of re-organization in the auditor's object_audit method
to get things into a single return path we can add an unconditional check
for unexpected files and remove those that are "old enough".
Since the replicator will kill any rsync processes that are running longer
than the configured rsync_timeout we know that any rsync tempfiles older
than this can be deleted.
Split unlink_older_than in common.utils into two functions to allow an
explicit list of previously discovered paths to be passed in to avoid an
extra listdir. Since the getmtime handling already ignores OSError
there's less concern of race condition where a previous discovered
unexpected file is reaped by rsync while we're attempting to clean it up.
Update some doc on the new config option.
Closes-Bug: #1554005
Change-Id: Id67681cb77f605e3491b8afcb9c69d769e154283
Updates docs to remove warnings that container sync only
works with object_post_as_copy=True. Since commit e91de49
container sync will also sync POST updates when using
object_post_as_copy=False.
Change-Id: I5cc3cc6e8f9ba2fef6f896f2b11d2a4e06825f7f
Bring overview_auth.rst and proxy server man page
up to date with changes made in [1]
[1] Change-Id: I373734933189c87c4094203b0752dd3762689034
Change-Id: Ia16f0c391e7c357ccb9c13945839dc5647e49a13
Swift now uses SSYNC verb instead of old REPLICATION verb for ssync
protocol. This patch replaces all docs written as REPLICATION into
SSYNC and fix a few words for explanation.
Change-Id: I1253210d4f49749e7d425d6252dd262b650d9548
This change adds 2 new parameters to enable and control concurrent GETs
in swift, these are 'concurrent_gets' and 'concurrency_timeout'.
'concurrent_gets' allows you to turn on or off concurrent GETs, when
on it will set the GET/HEAD concurrency to replica count. And in the
case of EC HEADs it will set it to ndata.
The proxy will then serve only the first valid source to respond.
This applies to all account, container and object GETs except
for EC. For EC only HEAD requests are effected.
It achieves this by changing the request sending mechanism to using
GreenAsyncPile and green threads with a time out between each
request.
'concurrency_timeout' is related to concurrent_gets. And is the
amount of time to wait before firing the next thread. A value of 0
will fire at the same time (fully concurrent), setting another value
will stagger the firing allowing you the ability to give a node a
shorter chance to respond before firing the next. This value is a float
and should be somewhere between 0 and node_timeout. The default is
conn_timeout. Meaning by default it will stagger the firing.
DocImpact
Implements: blueprint concurrent-reads
Change-Id: I789d39472ec48b22415ff9d9821b1eefab7da867
This patch cleans up some rough edges that were left (due to
time constraints) in the original commit.
Change-Id: Id4480be8dc1b5c920c19988cb89ca8b60ace91b4
Co-Authored-By: Gerry Drudy gerry.drudy@hpe.com
This removes notes stating support for Erasure coding as beta. Questions
regarding the stability of EC are coming up regularly, and are often referring
to the docs that state EC as still in beta.
Besides this, a note marking statsd support as beta has been removed as well.
Change-Id: If4fb6a5c4cb741d42953db3cee8cb17a1d774e15
This is the operational procedures guide that HPE used
to operate and monitor their public Swift systems.
It has been made publicly available.
Change-Id: Iefb484893056d28beb69265d99ba30c3c84add2b
There's no RST file that uses ".. math" and thus
the pngmath Sphinx extension is not used and can
get removed.
Change-Id: I3ea4b529025c8ac3c4092f3720124647861e6668
Some doc files and swift.conf-sample were added execute permissions in past changes.
This patch remove execute permissions from them.
Change-Id: Id8844989a8321578e9207566ebd6660f5b9523f0
Using unprintable characters for swift_hash_path_prefix/suffix might lead to
hard problems when parsing of these values changes, for example due to newer
Python versions or changes in the parsers itself. Let's avoid this and add a
note that deployers should use printable strings for these values.
Change-Id: I976982b753b6af831ab91d7190f50f8f15bf73bf
Added a link to the API reference (where headers and query
strings are documented; click the "detail" button to see).
Also added a reference to Swift section of the OpenStack end
user guide. This contains some additional details about the API.
No attempt was made to reconcile duplicate information. Instead
this patch links documents that might otherwise be overlooked.
However, I fixed text, originally in a table, that had become
garbled in a prior patch.
Change-Id: I0910cbeb0c8bffc00e510f35585603e7b7a67790
The variable max_large_object_get_time is no longer used and was
removed to reflect the change.
Change-Id: I43051181dcb38245de6d13fab63876e83f46fc39
Closes-Bug: #1538834
Multiple files and documents showed that log_statsd_host had
a default value, usually localhost. This was incorrect, instead
setting a value for log_statsd_host enables statsd logging.
Removed any reference of log_statsd_host having a default value.
Also changed descriptions to show setting a value enables logging.
Change-Id: I3ca5c0e8b8e4981de3aa6db0c476072b5a59723d
Closes-Bug: #1542227
