Swift returns `X-Openstack-Request-Id` for container and
object requests as well as account. A couple api-ref docs are
missing this value in the response examples.
Change-Id: Ifcd67a620e04be5e92b43c7749ee4acb50bb171d
Continuing the clean up in account and container
info, removed duplicated validation from account_info
and container_info methods, since the same validations
were recently added to get_account_info and get_container_info.
Change-Id: I1ad745fe809367d22649c83f38c4de7a74cac44e
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Changes the file parameters on command line to use the convention of
<file> for a mandatory option rather than [file] which suggests an
optional flag. Also drop to lower case as is convention in other man
pages.
Change-Id: Icfb4b4786343eb493a73b092cad1fdadc935d066
swift-account-info, swift-container-info, swift-object-info
and swift-get-node do not contain any information about the available
options. This patch adds OPTIONS in these manpages.
Change-Id: I832c621460ee6bf19adac8e0aadeab32be4b8da0
The warning message for rsync tempfiles was removed in the related
change. However, because our regex match might result in a false
positive maybe it's still useful to log a debug message. Instead of
silently ignoring rsync tempfiles, when running in debug we note the
file and how we classified it - but still no warning will occur.
I also consolidate our use of the regex for rsync tempfiles into the
diskfile module and move the negative test for the warning logger
message a little next to the positive test.
Change-Id: Idb2a1a76aa275c9c2e9bad8aceea913b8f5b1c71
Related-Change: I5a5d6e24710e4880776b32edcbc07021acf77676
This should give the necessary tooling to set up the right
links in the right places so Swift has release notes listed
alongside other projects.
Change-Id: I4e5f1ce1fcfbb2943036c821a24a0b4a3a2d9fc8
Decode header to latin1 on python3
encode header to utf-8 on python2.
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Change-Id: I10f205a05bb3a566e52a597d9315b3a8b8c14664
Despite a check to prevent zero values in the denominator python
integer division could result in ZeroDivisionError in the compute_eta
helper function. Make sure we always have a non-zero value even if it
is small.
NotImplemented:
* stats calculation is still not great, see lp bug #1488608
Closes-Bug: #1549110
Change-Id: I54f2081c92c2a0b8f02c31e82f44f4250043d837
The intentional use of "bare except" handling in catch_errors and some
daemons to prevent propagation on unexpected errors that do not
inherit from Exception (like eventlet.Timeout) or even BaseException
(like old-style classes) has the side effect of spuriously "handling"
*expected* errors like when a signal handler raises SystemExit.
The signal handler installed in our Daemon is intended to ensure first
that the entire process group and any forked processes (like rsync's)
receive the SIGTERM signal and also that the process itself
terminates.
The use of sys.exit was not a concious grandiose plans for graceful
shutdown (like the running[0] = False trick that wsgi server parent
process do) - the desired behavior for SIGTERM is to stop - hard.
This change ensures the original goals and intentions of our signal
handler are fulfilled without the undesirable side effect that can
cause our daemons to confusingly log an expected message to stop as an
unexpected error, and start ignoring additional SIGTERM messages;
forcing our kind operators to resort to brutal process murder.
Closes-Bug: #1489209
Change-Id: I9d2886611f6db2498cd6a8f81a58f2a611f40905
When an SSYNC connection fails after shipping a fragment, but before
cleaning itself up - it can lead to multiple replicas of the same
fragment index serviceable in the node_iter at the same time. Or
perhaps more likely if a partner nodes win a race to rebuild waiting
on a handoff. In this case, the proxy need not fail to service a GET
just because of extra information. A similar guard was added to the
reconstructor in a related change [1].
This underlying bug is acctually fixed by the related change [2], this
probetest is just encoding the failure to help with future maintenance.
1. Related-Change: I91f3d4af52cbc66c9f7ce00726f247b5462e66f9
2. Related-Change: I2310981fd1c4622ff5d1a739cbcc59637ffe3fc3
Closes-Bug: 1624176
Change-Id: I06fc381a25613585dd18916d50e7ca2c68d406b6
Previously the reconstructor would only reconstruct a missing fragment
when a set of ec_ndata other fragments was available, *all* of which
were durable. Since change [1] it has been possible to retrieve
non-durable fragments from object servers. This patch changes the
reconstructor to take advantage of [1] and use non-durable fragments.
A new probe test is added to test scenarios with a mix of failed and
non-durable nodes. The existing probe tests in
test_reconstructor_rebuild.py and test_reconstructor_durable.py were
broken. These were intended to simulate cases where combinations of
nodes were either failed or had non-durable fragments, but the test
scenarios defined were not actually created - every test scenario
broke only one node instead of the intent of breaking multiple
nodes. The existing tests have been refactored to re-use most of their
setup and assertion code, and merged with the new test into a single
class in test_reconstructor_rebuild.py.
test_reconstructor_durable.py is removed.
[1] Related-Change: I2310981fd1c4622ff5d1a739cbcc59637ffe3fc3
Change-Id: Ic0cdbc7cee657cea0330c2eb1edabe8eb52c0567
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Closes-Bug: #1624088
Since the related change [1] the auditor checks deleted objects in
case their tombstone is ready to be reclaimed. However, the auditor
mistakes an expired object for a deleted object because
DiskFileExpired is a subclass of DiskFileDeleted. This causes a
KeyError to be raised and logged because the expired object has no
tombstone info in the ondisk_info data structure.
This patch makes the auditor catch and ignore DiskFileExpired
exceptions before handling DiskFileDeleted exceptions.
[1] Related-Change: I3e99dc702d55a7424c6482969e03cb4afac854a4
Change-Id: I9872b6997d09dcfd8a868c4b91b9379407283b8e
Closes-Bug: #1638016
eventlet.wsgi.server is adaptive to dealing with the named log
argument being either a file like or a log like [1], but message
handling is a more direct pass through if it's a log like.
Since all we want to do is make eventlet's logging it a noop - it's
easy to provide either interface!
1. https://github.com/eventlet/eventlet/blob/4d2cdc/eventlet/wsgi.py#L246
Change-Id: I2d792176c96931eafb3f140e6653ba8b31eda429
I changed asserts with more specific assert methods.
e.g.: from assertTrue(sth == None) to assertIsNone(*) or
assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type) or
assertTrue(not sth) to assertFalse(sth).
The code gets more readable, and a better description will be shown on fail.
Change-Id: I3768faa568e3964e726ecc48ac8cb133cb088284
This patch makes the ECDiskFileReader check the validity of EC
fragment metadata as it reads chunks from disk and quarantine a
diskfile with bad metadata. This in turn means that both the object
auditor and a proxy GET request will cause bad EC fragments to be
quarantined.
This change is motivated by bug 1631144 which may result in corrupt EC
fragments being written to disk but appear valid to the object auditor
md5 hash and content-length checks.
NotImplemented:
* perform metadata check when a read starts on any frag_size
boundary, not just at zero
Related-Bug: #1631144
Closes-Bug: #1633647
Change-Id: Ifa6a7f8aaca94c7d39f4aeb9d4fa3f59c4f6ee13
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
When Object-Server saves async-update in pickle files,
'User-Agent: object-server %(pid)' (the key is title style) is included
in the pickles. However, Object-Updater will add
'user-agent: object-updater %(pid)' (the key is lower case style) to the
pickled headers and use it to make http connection. According to RFC
7230, each header field consists of a case-insensitive field name,
therefore either of 'User-Agent' and 'user-agent' should not be included
in the headers.
This patch removes old 'User-Agent' header from object-updater's
requests.
Change-Id: Ia624558395584457c718b311fe80e1a8406e22ad
Closes-Bug:#1635114
Do not log unexpected file warning for rsync temp files when the parsing
of the timespamp fails. If the file passes a regex test suppress the
logger waring, but still return it as an unexpected file from
get_ondisk_files.
Closes-Bug: 1616504
Change-Id: I5a5d6e24710e4880776b32edcbc07021acf77676
Many other OpenStack services use a `[X-]OpenStack-Request-Id` header
to return a unique identifier for the request. Swift will now return
`X-Trans-Id` as well as `X-Openstack-Request-Id`.
Change-Id: I56cd4738808b99c0a08463f83c100be51a62db05
Closes-Bug: #1572786
This is consistent with what we already do for other
semantically-invalid values like "bytes=--1". Previously, we would
return a 416 Requested Range Not Satisfiable response like we do for the
semantically-valid-but-not-really-meaningful "bytes=-0"
Change-Id: I932b42406c9a5ee7eaa6978a655e61022a957415
Previously, we didn't make any assertions about the backend requests
but rather just verified the user-visible behavior.
Change-Id: Iddd4b705ee9b724a4a8a88c6fbaff36cca9612cf
Fixies this problem:
* swift-drive-audit needs to be run by root, because only root have
"umount" permission
* swift-object servers typically runs as user swift
* if swift-drive-audit is run by root, /var/cache/swift/drive.recon is
owned by root, with 0o600
* recon middleware (inside swift-object-server) can't read this cache
file: swift-object: Error reading recon cache file
This patch adds "user" option to drive-audit config file. Recon cache
is chowned to this user.
Change-Id: Ibf20543ee690b7c5a37fabd1540fd5c0c7b638c9
