The object server runs certain IO-intensive methods outside the main
pthread for performance. If one of those methods tries to log, this can
cause a crash that eventually leads to an object server with hundreds
or thousands of greenthreads, all deadlocked.
The short version of the story is that logging.SysLogHandler has a
mutex which Eventlet monkey-patches. However, the monkey-patched mutex
sometimes breaks if used across different pthreads, and it breaks in
such a way that it is still considered held. After that happens, any
attempt to emit a log message blocks the calling greenthread forever.
The fix is to use a mutex that works across different greenlets and
across different pthreads. This patch introduces such a lock based on
an anonymous pipe.
Change-Id: I57decefaf5bbed57b97a62d0df8518b112917480
Closes-Bug: 1710328
Fixes test_GET_with_duplicate_but_sufficient_frag_indexes
failing intermittently.
The original non-duplicated-frags version of the test in
TestECObjController verified that the controller would search all
primaries, ignoring repeated fragment indexes and finally find the
last necessary unique fragment on the first handoff. The version of
the test for a duplicated fragments policy in
TestECDuplicationObjController had the final necessary fragment on the
first of the 'duplicate' primaries (index ec_num_unique_fragments + 1)
so there was a chance that it would be found before all of the first
ec_num_unique_fragments had been tried, causing the test to fail as
reported in the linked bug report.
This patch moves the final necessary unique fragment to the first
handoff (index 2 * index ec_num_unique_fragments + 1) and aligns the
test with the intent of the equivalent test in TestECObjController,
so that the two tests can be moved to a single implementation in an
inherited mixin.
test_GET_with_duplicate_but_insufficient_frag is also moved to the
new mixin since the two implementations of that test were also very
similar.
Change-Id: I4d99dea8e941bccb6fa610bab39e38a6976a3721
Closes-Bug: #1710951
Avoid ever having two objects in same test with same etag.
Refactor some common test helper methods to a common superclass.
Change-Id: Ib956c997f8805df1859ebf3582f7ce59c8f65129
Closes-Bug: #1710924
...by using a random ring, looking at *all* partitions, and making
assertions about the distribution of how many times we have to call
next().
Change-Id: Ia5feb9396d4bf6fd35f16bbc5280e63022ed2c47
Unit tests use the random module in places to randomise
test inputs, but once the tests in test_builder.py or
test_ring_builder_analyzer.py have been run the random
seed is left in a repeatable state because calls are made
to RingBuilder.balance with a seed value. Consequently,
subsequent calls to random in other tests get repeatable
results from one test run to another.
This patch resets the state of the random module before
returning from RingBuilder.rebalance.
Closes-Bug: #1639755
Change-Id: I4b74030afc654e60452e65b3e0f1b45a189c16e3
While we're at it, have copy and copy_account raise ResponseErrors
on failure, similar to cluster_info, update_metadata, containers, info,
files, delete, initialize, read, sync_metadata, write, and post.
Related-Change: Ia8b92251718d10b1eb44a456f28d3d2569a30003
Change-Id: I9ef42d922a6b7dbf253f2f8f5df83965d8f47e0f
When a fragment of an expired object was missing, the reconstructor
ssync job would send a DELETE sub-request. This leads to situation
where, for the same object and timestamp, some nodes have a data file,
while others can have a tombstone file.
This patch forces the reconstructor to reconstruct a data file, even
for expired objects. DELETE requests are only sent for tombstoned
objects.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Closes-Bug: #1652323
Change-Id: I7f90b732c3268cb852b64f17555c631d668044a8
We have a test for get_local_devices, but let's make some broader
assertions as well.
Related-Bug: #1707595
Change-Id: Ifa696207ffdb3b39650dfeaa3e7c6cfda94050db
Since the related change, object-reconstructor gathers the local devices
for ec policy via get_local_devices method but the method causes TypeError when
attempting *reduce* for empty set list. the list can be empty when no EC config
found in swift.conf.
This patch fixes the get_local_devices to return empty set even when no ec config
in swift.conf without errors.
Co-Authored-By: Kirill Zaitsev <k.zaitsev@me.com>
Change-Id: Ic121fb547966787a43f9eae83c91bb2bf640c4be
Related-Change: 701a172afac37229b85ea762f20428f6f422d29b
Closes-Bug: #1707595
* Fixes warnings in RST file
* Suppress warning log from pyeclib during the doc build.
pyeclib emits a warning message on an older liberasurecode [1]
and sphinx treats this as error (when warning-is-error is set).
There is no need to check warnings during the doc build,
so we can safely suppress the warning.
This is a part of the doc migration community-wide effort.
http://specs.openstack.org/openstack/docs-specs/specs/pike/os-manuals-migration.html
[1] https://github.com/openstack/pyeclib/commit/d163972b
Change-Id: I9adaee29185a2990cc3985bbe0dd366e22f4f1a2
Add a test that verifies that get_all_devices does
fetch devices from the ring.
Related-Change: I28925a37f3985c9082b5a06e76af4dc3ec813abe
Change-Id: Ie2f83694f14f9a614b5276bbb859b9a3c0ec5dcb
This change adds a new Strategy concept to the daemon module similar to
how we manage WSGI workers. We need to leverage multiple python
processes to get the concurrency properties we need. More workers will
rebalance much faster on dense chassis with many devices.
Currently the default is still only one process, and no workers. Set
reconstructor_workers in the [object-reconstructor] section to some
whole number <= the number of devices on a node to get that many
reconstructor workers.
Each worker will operate on a different subset of disks.
Once mode works as before, but tends to want to update recon drops a
little bit more.
If you change the rings, the strategy will shutdown workers and spawn
new ones.
You can kill the worker pids and the daemon strategy will respawn them.
New per-disk reconstructor stats are dumped to recon under the
object_reconstruction_per_disk key. To maintain legacy compatibility
and replication monitoring based on cycle times they are aggregated
every stats_interval (default 5 mins).
Change-Id: I28925a37f3985c9082b5a06e76af4dc3ec813abe
This is follow up for the related change [1].
Just adding small docs and removing unnecessary arg in the test.
[1] Related-Change: I1b30cb3d776be441346a4131007d2487a5440a81
Change-Id: I48c21b6028bd2ef454e084032a2bd8c0d4239e4b
Clarify in usage statement and man pages that CLI override options for
swift-object-reconstructor and swift-object-replicator only have
effect when --once is used.
Also add a link to object reconstructor source code docs to the doc
index page for consistency with the other object services.
Change-Id: If348b340d59a672d3a19d4df231ebdb74f4aed51
