The reconstructor handoffs_only needs to aggressively avoid erroneous
I/O related to rehash of primary suffixes.
While in handoffs_only mode the reconstructor won't even look at primary
partitions.
This has a *huge* impact on cycle time once the node has completed
processing handoffs; which results in a much faster and stronger signal
that that it's either time to rebalance again or turn off handoffs_only.
Related-Change-Id: Idde4b6cf92fab6c45f2c0c2733277701eb436898
Change-Id: If4bbb778d511efe13713590639c8b91615556f22
Source code should be licensed under the Apache 2.0 license.
Add Apache License in swift/probe/__init__.py file.
Change-Id: I3b6bc2ec5fe5caac87ee23f637dbcc7a5d8fc331
When deleting objects in multi-region swift delpoyment with write
affinity configured, users always get 404 when deleting object before
it's replcated to approriate nodes.
This patch adds a config item 'write_affinity_handoff_delete_count' so
that operator could define how many local handoff nodes should swift
send request to get more candidates for the final response, or by
default just leave it to swift to calculate the appropriate number.
Change-Id: Ic4ef82e4fc1a91c85bdbc6bf41705a76f16d1341
Closes-Bug: #1503161
Use assertIsNone() instead of assertEqual(), because assertEqual()
still fails on false values when compared to None
Change-Id: Ic52c319e3e55135df834fdf857982e1721bc44bb
If you're running servers_per_port > 0 and threads_per_disk = 0 (as it
should be with servers_per_port on), each object-server process will
have 20 IO threads waiting around to service eventlet.tpool
calls. This is far too many; with servers_per_port, there's no real
benefit to having so many IO threads.
This commit makes it so that, when servers_per_port > 0, each object
server defaults to having one main thread and one IO thread.
Also, eventlet's tpool size is now configurable via the object-server
config file. If a tpool size is set, that's what we'll use regardless
of servers_per_port. This allows operators with an excess of threads
to remove some regardless of servers_per_port.
Change-Id: I8f8914b7e70f2510393eb7c5e6be9708631ac027
Closes-Bug: 1554233
After the change to reuse device id's [1], the order of devices in the
output of swift-ring-builder is confusing. This patch list the devices
in order of (region, zone, ip, device).
The effect of this patch is as illustrated in [2].
This patch also partially fix Bug 1545016.
1. https://review.openstack.org/#/c/265461/
2. https://github.com/MicrowiseOnGitHub/tempfiles/blob/master/reorder_ring_output
Change-Id: I564ed1b8d0cd4a6250649689b1bce7ad3574fe57
Partial-Bug: 1545016
Closes-Bug: 1536743
In some unit tests instead of self.fail(msg) statements
self.assertTrue(False, msg) were used, which might be ambiguous.
Using assertTrue(False, msg) gives the following message on fail:
File "C:\Python361\lib\unittest\case.py", line 678, in assertTrue
raise self.failureException(msg)
AssertionError: False is not true : msg
'False is not true' message implies that unit test failed (as the
result is False while we asserted True).
Replaced with self.fail(msg) is less ambiguous and more readable.
File "C:\Python361\lib\unittest\case.py", line 666, in fail
raise self.failureException(msg)
AssertionError: msg
TrivialFix
Change-Id: Ib56a0ed8549fd7af2724eb59222106888781e9c8
container and object updaters sleeps "slowdown" (default 0.01) seconds
after every processed container/object. Because time.sleep call adds overhead,
use ratelimit_sleep from common.utils instead. Same as in auditor.
Change-Id: I362aa0f13c78ad03ce1f76ee0257b0646f981212
We said we were going to do it, we've had two releases saying we'd do
it, we've even backported our saying it to Newton -- let's actually do
it.
Upgrade Consideration
=====================
Erasure-coded storage policies using isa_l_rs_vand and nparity >= 5 must
be configured as deprecated, preventing any new containers from being
created with such a policy. This configuration is known to harm data
durability. Any data in such policies should be migrated to a new
policy. See https://bugs.launchpad.net/swift/+bug/1639691 for more
information.
UpgradeImpact
Related-Change: I50159c9d19f2385d5f60112e9aaefa1a68098313
Change-Id: I8f9de0bec01032d9d9b58848e2a76ac92e65ab09
Closes-Bug: 1639691
This patch adds methods to increase the partition power of an existing
object ring without downtime for the users using a 3-step process. Data
won't be moved to other nodes; objects using the new increased partition
power will be located on the same device and are hardlinked to avoid
data movement.
1. A new setting "next_part_power" will be added to the rings, and once
the proxy server reloaded the rings it will send this value to the
object servers on any write operation. Object servers will now create a
hard-link in the new location to the original DiskFile object. Already
existing data will be relinked using a new tool in the new locations
using hardlinks.
2. The actual partition power itself will be increased. Servers will now
use the new partition power to read from and write to. No longer
required hard links in the old object location have to be removed now by
the relinker tool; the relinker tool reads the next_part_power setting
to find object locations that need to be cleaned up.
3. The "next_part_power" flag will be removed.
This mostly implements the spec in [1]; however it's not using an
"epoch" as described there. The idea of the epoch was to store data
using different partition powers in their own namespace to avoid
conflicts with auditors and replicators as well as being able to abort
such an operation and just remove the new tree. This would require some
heavy change of the on-disk data layout, and other object-server
implementations would be required to adopt this scheme too.
Instead the object-replicator is now aware that there is a partition
power increase in progress and will skip replication of data in that
storage policy; the relinker tool should be simply run and afterwards
the partition power will be increased. This shouldn't take that much
time (it's only walking the filesystem and hardlinking); impact should
be low therefore. The relinker should be run on all storage nodes at the
same time in parallel to decrease the required time (though this is not
mandatory). Failures during relinking should not affect cluster
operations - relinking can be even aborted manually and restarted later.
Auditors are not quarantining objects written to a path with a different
partition power and therefore working as before (though they are reading
each object twice in the worst case before the no longer needed hard
links are removed).
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
[1] https://specs.openstack.org/openstack/swift-specs/specs/in_progress/
increasing_partition_power.html
Change-Id: I7d6371a04f5c1c4adbb8733a71f3c177ee5448bb
While we're at it, drop some unused expect_exception code from
test_versioned_writes.py
Change-Id: Id232880c7259839159c057471b61083d5862d189
Related-Change: I430c48c4a81e8392fa271160bcbc1817ef0a88f7
When the "copy" middleware tries to copy a segmented object which is
bigger than max_file_size, it immediatly returns "413 Request Entity Too
Large". But at that point, connections have already been established by
the proxy server to the object servers. These connections must be closed
before returning.
Closes-Bug: #1698207
Change-Id: I430c48c4a81e8392fa271160bcbc1817ef0a88f7
This changes:
- Add assertions for write_affinity values in load_app tests
- Add a test case that policy override read_affinity default by timing
strategy
- Avoid 'scope' but it's 'label'
Related-Change: I3f718f425f525baa80045ba067950c752bcaaefc
Change-Id: Ia8262490895d60da345f3679fc53653b2c2a2b3e
Revert tombstone only parts try to talk to all primary nodes - this
fixes it to randomize selection within part_nodes. Corresponding probe
test is modified to reflect this change.
The primary improvement of this patch is the reconstuctor at a handoff
node is being able to delete local tombstones when it succeeds to sync
to less than all primary nodes. (Before this patch, it requires all
nodes are responsible for the REVERT requests)
The number of primary nodes to communicate with the reconstructor can be
in dicsussion more but, right now with this patch, it's (replicas - k + 1)
that is able to prevent stale read.
*BONUS*
- Fix mis-testsetting (was setting less replicas than ec_k + ec_m)
for reconstructor ring in the unit test
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I05ce8fe75f1c4a7971cc8995b003df818b69b3c1
Closes-Bug: #1668857
Use a manager and config_number helper for clarity - then make a one
last assertion on the final consistent state.
Change-Id: I5030c314076003d17c41b8b136bcbda252474bad
Related-Change-Id: Icdf0a936fc456c5462471938cbc365bd012b05d4
Following OpenStack Style Guidelines:
[1] http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H203] Unit test assertions tend to give better messages for more specific
assertions. As a result, assertIsNone(...) is preferred over
assertEqual(None, ...) and assertIs(..., None)
Change-Id: If4db8872c4f5705c1fff017c4891626e9ce4d1e4
Following OpenStack Style Guidelines:
[1] http://docs.openstack.org/developer/hacking/#unit-tests-and-assertraises
[H203] Unit test assertions tend to give better messages for more specific
assertions. As a result, assertIsNone(...) is preferred over
assertEqual(None, ...) and assertIs(..., None)
Change-Id: I5b5900c509956cf13ec3d1f2105fc7c89531f22a
With the new default behavior of eventlet 0.20 to set SO_REUSEPORT on listening
sockets, it becomes easy to have a zero-downtime reload.
By binding the listening socket as late as we can, we can reduce the
time an incoming connection will stay in queue, waiting for the process
to be ready to handle the connection.
This commit moves the socket binding after the load of the application.
On some benchmarks, it reduced the wait-time for incoming connections from
about 1.1s to about 350ms.
Change-Id: I045f4c9aa2c07cdc5ed95afd028540f1b3874637
For various reasons, an operator might want to use specifics nameservers
instead of the systems ones to resolve CNAME in cname_lookup. This patch
creates a new configuration variable nameservers which accepts a list of
nameservers separated by commas. If not specified or empty, systems
namservers are used as previously.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I34219e6ab7e45678c1a80ff76a1ac0730c64ddde
* Only use one StringIO in ConfigString
* Rename the write_affinity_node_count function to be
write_affinity_node_count_fn
* Use comprehensions instead of six.moves.filter
* Rename OverrideConf to ProxyOverrideOptions
* Make ProxyOverrideOptions's __repr__ eval()able
* Various conf -> options renames
* Stop trying to handle a KeyError that should never come up
* Be explicit about how deep we need to copy in proxy/test_server.py
* Drop an unused return value
* Add a test for a non-"proxy-server" app name
* Combine bad-section-name tests
* Try to clean up (at least a little) a self-described "hokey test"
Related-Change: I3f718f425f525baa80045ba067950c752bcaaefc
Change-Id: I4e81175d5445049bc1f48b3ac02c5bc0f77e6f59
test_object_metadata_replication requires a replicated policy. We even
have it subclass ReplProbeTest, but then we hardcoded the policy index!
Stop doing that.
Change-Id: I8871cc0beceb0909abaf59babe40c3cafbcd0cc9
