Plumb the version from the ringbuilder through to the metadata at the
start of the ring. Recover this (if available) when running
swift-ring-builder <ring> write_builder
When we load the ring, track the count and MD5 of the bytes off disk, as
well as the number of uncompressed bytes.
Expose all this new information as properties on the Ring, along with
- device_count (number of non-None entries in self._devs),
- weighted_device_count (number of devices that have weight), and
- assigned_device_count (number of devices that actually have
partition assignments).
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: I73deaf6f1d9c1d37630c37c02c597b8812592351
If swift-ring-builder is erroneously given a composite builder
file, which it will fail to load, it will now print a hint
that the file is a composite builder file.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: If4517f3b61977a7f6ca3e08ed5deb182aa87a366
I can't imagine us *not* having a py3 proxy server at some point, and
that proxy server is going to need a ring.
While we're at it (and since they were so close anyway), port
* cli/ringbuilder.py and
* common/linkat.py
* common/daemon.py
Change-Id: Iec8d97e0ce925614a86b516c4c6ed82809d0ba9b
Displaying an empty ring should not be an error, thus
changing the exit code back to the former value of 0.
Closes-Bug: 1742417
Change-Id: I779c30cff1b4d24483f993221a8c6d944b7ae98d
This is a follow up patch to 530258 which will show
extra infromation on empty rings.
This patch goes one step further. On a completely empty ring:
$ swift-ring-builder my.builder create 8 3 1
$ swift-ring-builder my.builder
my.builder, build version 0, id 33b4e117056340feae7d40430180c6bb
256 partitions, 3.000000 replicas, 0 regions, 0 zones, 0 devices, 0.00 balance, 0.00 dispersion
The minimum number of hours before a partition can be reassigned is 1 (0:00:00 remaining)
The overload factor is 0.00% (0.000000)
Ring file my.ring.gz not found, probably it hasn't been written yet
Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta
There are no devices in this ring, or all devices have been deleted
It will still start the device list and then say no devices.. Why. let's
see what happens now on an empty ring with devices still marked as
deleted:
$ swift-ring-builder my.builder add r1z1-127.0.0.1:6010/sdb1 1
Device d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" with 1.0 weight got id 0
$ swift-ring-builder my.builder add r1z1-127.0.0.1:6010/sdb2 1
Device d1r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb2_"" with 1.0 weight got id 1
$ swift-ring-builder my.builder remove r1z1-127.0.0.1
Matched more than one device:
d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_""
d1r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb2_""
Are you sure you want to remove these 2 devices? (y/N) y
d0r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb1_"" marked for removal and will be removed next rebalance.
d1r1z1-127.0.0.1:6010R127.0.0.1:6010/sdb2_"" marked for removal and will be removed next rebalance.
$ swift-ring-builder my.builder
my.builder, build version 4, id 33b4e117056340feae7d40430180c6bb
256 partitions, 3.000000 replicas, 1 regions, 1 zones, 2 devices, 0.00 balance, 0.00 dispersion
The minimum number of hours before a partition can be reassigned is 1 (0:00:00 remaining)
The overload factor is 0.00% (0.000000)
Ring file my.ring.gz not found, probably it hasn't been written yet
Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta
0 1 1 127.0.0.1:6010 127.0.0.1:6010 sdb1 0.00 0 0.00 DEL
1 1 1 127.0.0.1:6010 127.0.0.1:6010 sdb2 0.00 0 0.00 DEL
There are no devices in this ring, or all devices have been deleted
Now even when all devices are removed we can still see them as they are still there, only marked as deleted.
Change-Id: Ib39f734deb67ad50bcdad5333cba716161a47e95
These were likely written before the first related change, or created
from an existing ring file.
Also, tolerate missing dispersion when rebalancing -- that may not exist
in the builder file.
Change-Id: I26e3b4429c747c23206e4671f7c86543bb182a15
Related-Change: Ib165cf974c865d47c2d9e8f7b3641971d2e9f404
Related-Change: Ie239b958fc7e0547ffda2bebf61546bd4ef3d829
Related-Change: I551fcaf274876861feb12848749590f220842d68
Since dispersion info is cached, this can easily happen if we make
changes to how dispersion info is calculated or stored (e.g. we extend
the dispersion calculation to consider dispersion of all part-replicas
in the related change)
Related-Change-Id: Ifefff0260deac0c3e8b369a1e158686c89936686
Change-Id: I714deb9e349cd114a21ec591216a9496aaf9e0d1
When the default display command for swift-ring-error encounters a
EmptyRingError trying to calculate balance it should not raise exception
and display the traceback in a command line environment.
Instead handle the exceptional condition and provide the user with
useful feedback.
Closes-Bug: #1737068
Change-Id: Ibadaf64748728a47a8f3f861ec1af601dbfeb9e0
With a sufficiently undispersed ring it's possible to move an entire
replicas worth of parts and yet the value of dispersion may not get any
better (even though in reality dispersion has dramatically improved).
The problem is dispersion will currently only represent up to one whole
replica worth of parts being undispersed.
However with EC rings it's possible for more than one whole replicas
worth of partitions to be undispersed, in these cases the builder will
require multiple rebalance operations to fully disperse replicas - but
the dispersion value should improve with every rebalance.
N.B. with this change it's possible for rings with a bad dispersion
value to measure as having a significantly smaller dispersion value
after a rebalance (even though they may not have had their dispersion
change) because the total amount of bad dispersion we can measure has
been increased but we're normalizing within a similar range.
Closes-Bug: #1697543
Change-Id: Ifefff0260deac0c3e8b369a1e158686c89936686
There are cases where a rebalance improves dispersion, but doesn't
improve balance. This is because the balance of a ring builder is
taken to be the balance of its least-balanced device, so if there's a
device that has no partitions, wants some, but can't get them, then
we'll never save the ring builder even if every other device in the
ring got better.
We can detect this situation by looking at the dispersion number; if it
changes, then the rebalance needs to be saved in order to continue to
make progress.
Partial-Bug: #1697543
Change-Id: Ie239b958fc7e0547ffda2bebf61546bd4ef3d829
Swift definitely can't make any use of empty rings, so it should
not be allowed to write them.
Replace warning with an error message & error exit.
Change-Id: I3a1b86368d363e67d1f91d7d8af4b391a0a53fff
Closes-Bug: #1396841
Currently, our integrity checking for objects is pretty weak when it
comes to object metadata. If the extended attributes on a .data or
.meta file get corrupted in such a way that we can still unpickle it,
we don't have anything that detects that.
This could be especially bad with encrypted etags; if the encrypted
etag (X-Object-Sysmeta-Crypto-Etag or whatever it is) gets some bits
flipped, then we'll cheerfully decrypt the cipherjunk into plainjunk,
then send it to the client. Net effect is that the client sees a GET
response with an ETag that doesn't match the MD5 of the object *and*
Swift has no way of detecting and quarantining this object.
Note that, with an unencrypted object, if the ETag metadatum gets
mangled, then the object will be quarantined by the object server or
auditor, whichever notices first.
As part of this commit, I also ripped out some mocking of
getxattr/setxattr in tests. It appears to be there to allow unit tests
to run on systems where /tmp doesn't support xattrs. However, since
the mock is keyed off of inode number and inode numbers get re-used,
there's lots of leakage between different test runs. On a real FS,
unlinking a file and then creating a new one of the same name will
also reset the xattrs; this isn't the case with the mock.
The mock was pretty old; Ubuntu 12.04 and up all support xattrs in
/tmp, and recent Red Hat / CentOS releases do too. The xattr mock was
added in 2011; maybe it was to support Ubuntu Lucid Lynx?
Bonus: now you can pause a test with the debugger, inspect its files
in /tmp, and actually see the xattrs along with the data.
Since this patch now uses a real filesystem for testing filesystem
operations, tests are skipped if the underlying filesystem does not
support setting xattrs (eg tmpfs or more than 4k of xattrs on ext4).
References to "/tmp" have been replaced with calls to
tempfile.gettempdir(). This will allow setting the TMPDIR envvar in
test setup and getting an XFS filesystem instead of ext4 or tmpfs.
THIS PATCH SIGNIFICANTLY CHANGES TESTING ENVIRONMENTS
With this patch, every test environment will require TMPDIR to be
using a filesystem that supports at least 4k of extended attributes.
Neither ext4 nor tempfs support this. XFS is recommended.
So why all the SkipTests? Why not simply raise an error? We still need
the tests to run on the base image for OpenStack's CI system. Since
we were previously mocking out xattr, there wasn't a problem, but we
also weren't actually testing anything. This patch adds functionality
to validate xattr data, so we need to drop the mock.
`test.unit.skip_if_no_xattrs()` is also imported into `test.functional`
so that functional tests can import it from the functional test
namespace.
The related OpenStack CI infrastructure changes are made in
https://review.openstack.org/#/c/394600/.
Co-Authored-By: John Dickinson <me@not.mn>
Change-Id: I98a37c0d451f4960b7a12f648e4405c6c6716808
When iterating over the (device, weight) tuples do not carry over the
device from the previous iteration.
Closes-Bug: 1454433
Change-Id: Iba82519b0b2bc80e2c1abbed308b651c4da4b06a
Use mock to force explicit RingBuilder rebalance results so that the
test is focussed on the ringbuilder.py rebalance command behavior
when balance is (or is not) achieved. Avoids assumptions about
RingBuilder behavior.
Change-Id: I242ffc2f1a4f7b69a679832a65790223642dcea8
Closes-Bug: #1499015
After the change to reuse device id's [1], the order of devices in the
output of swift-ring-builder is confusing. This patch list the devices
in order of (region, zone, ip, device).
The effect of this patch is as illustrated in [2].
This patch also partially fix Bug 1545016.
1. https://review.openstack.org/#/c/265461/
2. https://github.com/MicrowiseOnGitHub/tempfiles/blob/master/reorder_ring_output
Change-Id: I564ed1b8d0cd4a6250649689b1bce7ad3574fe57
Partial-Bug: 1545016
Closes-Bug: 1536743
This patch adds methods to increase the partition power of an existing
object ring without downtime for the users using a 3-step process. Data
won't be moved to other nodes; objects using the new increased partition
power will be located on the same device and are hardlinked to avoid
data movement.
1. A new setting "next_part_power" will be added to the rings, and once
the proxy server reloaded the rings it will send this value to the
object servers on any write operation. Object servers will now create a
hard-link in the new location to the original DiskFile object. Already
existing data will be relinked using a new tool in the new locations
using hardlinks.
2. The actual partition power itself will be increased. Servers will now
use the new partition power to read from and write to. No longer
required hard links in the old object location have to be removed now by
the relinker tool; the relinker tool reads the next_part_power setting
to find object locations that need to be cleaned up.
3. The "next_part_power" flag will be removed.
This mostly implements the spec in [1]; however it's not using an
"epoch" as described there. The idea of the epoch was to store data
using different partition powers in their own namespace to avoid
conflicts with auditors and replicators as well as being able to abort
such an operation and just remove the new tree. This would require some
heavy change of the on-disk data layout, and other object-server
implementations would be required to adopt this scheme too.
Instead the object-replicator is now aware that there is a partition
power increase in progress and will skip replication of data in that
storage policy; the relinker tool should be simply run and afterwards
the partition power will be increased. This shouldn't take that much
time (it's only walking the filesystem and hardlinking); impact should
be low therefore. The relinker should be run on all storage nodes at the
same time in parallel to decrease the required time (though this is not
mandatory). Failures during relinking should not affect cluster
operations - relinking can be even aborted manually and restarted later.
Auditors are not quarantining objects written to a path with a different
partition power and therefore working as before (though they are reading
each object twice in the worst case before the no longer needed hard
links are removed).
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
[1] https://specs.openstack.org/openstack/swift-specs/specs/in_progress/
increasing_partition_power.html
Change-Id: I7d6371a04f5c1c4adbb8733a71f3c177ee5448bb
* Adds a composite_builder module which provides the functionality to
build a composite ring from a number of component ring builders.
* Add id to RingBuilder to differentiate rings in composite.
A RingBuilder now gets a UUID when it is saved to file if
it does not already have one. A RingBuilder loaded from
file does NOT get a UUID assigned unless it was previously persisted in
the file. This forces users to explicitly assign an id to
existing ring builders by saving the state back to file.
The UUID is included in first line of the output from:
swift-ring-builder <builder-file>
Background:
This is another implementation for Composite Ring [1]
to enable better dispersion for global erasure coded cluster.
The most significant difference from the related-change [1] is that this
solution attempts to solve the problem as an offline tool rather than
dynamic compositing on the running servers. Due to the change, we gain
advantages such as:
- Less code and being simple
- No complex state validation on the running server
- Easy deployments with an offline tool
This patch does not provide a command line utility for managing
composite rings. The interface for such a tool is still under
discussion; this patch provides the enabling functionality first.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
[1] Related-Change: I80ef36d3ac4d4b7c97a1d034b7fc8e0dc2214d16
Change-Id: I0d8928b55020592f8e75321d1f7678688301d797
As described in bug #1558754 a rebalance after removing a device fails
if min_part_seconds_left > 0, despite the note that a rebalance should
remove partitions from removed devices on the next run.
This patch skips the early exit, and lets the builder itself handling
the rebalance. Partitions that shouldn't move now are still not moved
(except for removed devices), and there is still a warning if no
partition has been moved due to the fact that min_part_hours did not yet
pass.
A small test has been added to ensure rebalancing after removing a
device works without using the --force option (tests fails on current
master). Another test ensures that a rebalance after a recent change
(for example increasing a device's weight) does not move partitions and
still reports the former warning message.
Closes-Bug: 1558754
Change-Id: I083022d066338cbe6234bab491c7a8e8e0a7b517
Currently swift ignores a lot of the Hacking style guide. This patch
enables the H401 and H403 checks and fixes any violations. With this
we can get a little closer to following the OpenStack style guidelines.
Change-Id: I5109a052f2ceb2e6a9a174cded62f4231156d39b
I changed asserts with more specific assert methods.
e.g.: from assertTrue(sth == None) to assertIsNone(*) or
assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type)
or assertTrue(not sth) to assertFalse(sth).
The code gets more readable, and a better description will be shown on fail.
Change-Id: I39305808ad2349dc11a42261b41dbb347ac0618a
Adjusted width of ip and port columns in swift-ring-builder command
output to dynamically span to the longest ip or the longest port in
the devices list. Also combined the port and ip address columns for
better visual clarity. Took care of ipv6 format [ipv6]:port
Modified the corresponding test case with expected output.
Change-Id: I65837f8fed70be60b53d5a817a4ce529ad0f070e
Closes-Bug: #1567105
pretend_min_part_hours_passed do things like this:
self._last_part_moves[part] = 0xff
this will throw exception if self._last_part_moves is None.
this patch is to check self._last_part_moves to prevent exception.
Closes-bug: #1578835
Change-Id: Ic83c7a338b45bfcf61f5ab6100e6db335c3fa81a
In test_ringbuilder.py, there is one assertTrue should be
replaced with assertEqual.
Change-Id: I9a0e4a7363a5e16cc9b6df045953dfbb4f9dbd07
Closes-bug: #1604320
Also added a Timeout class to test.unit to wrap possible long-running
functions. For example, if there is some regression and the "--yes"
argument is no longer evaluated correctly and the test excepts some
keyboard input, it will be terminated after a few seconds to ensure
there is no long-running blocker on the gate.
Change-Id: I07b17d21d5af7fcc594ce5319ae2b6f7f58df4bb
If one has an object.builder file in the current directory and runs
test_ringbuilder, it will fail with an irritating error. That's because
test_use_ringfile_as_builderfile doesn't use self.tmpfile, but
object.builder - and that one might exist in the local directory.
This patch changes this, using self.tmpfile as argument name.
Closes-Bug: 1590356
Change-Id: I4b3287a36e8a5e469eb037128427dc7867910e53
Changing the recommended ports for Swift services
from ports 6000-6002 to unused ports 6200-6202;
so they do not conflict with X-Windows or other services.
Updated SAIO docs.
DocImpact
Closes-Bug: #1521339
Change-Id: Ie1c778b159792c8e259e2a54cb86051686ac9d18
Some of tests in test/unit/cli/test_ringbuilder doesn't assert
the exit code and unfortunately some of these passed even if the
statement fails for the assertion actually.
This patch enables to assert the exit code from ringbuider and
fixes some code/test bugs I noticed.
Change-Id: I18fa675ba8a90678e2b5ccb5f90eafab01d22787
This patch makes test_ringbuilder create a temporal directory,
run ring builder commands under it, and delete it after testing
for each test cases, to fix temp file leaking.
Change-Id: I6f59fe095ea6485af0e60b5a8e8fc3892e0a0f90
Try to find ring file, load and compare it with builder file, then show result state.
Examples:
Ring file object.ring.gz not found, probably it hasn't been written yet
Ring file object.ring.gz is up-to-date
Ring file object.ring.gz is obsolete
Ring file object.ring.gz is invalid: ValueError('string length not a multiple of item size',)
Change-Id: I4d769aa5fe1c2b1167ec088aa372874f7d13ae48
swift-ring-builder currently only displays min_part_hours and
not the amount of time remaining before a rebalance can occur.
This information is readily available and has been displayed
as a quality of life improvement.
Additionally, a bug where the time since the last rebalance
was always updated when rebalance was called regardless of
if any partitions were reassigned. This can lead to partitions
being unable to be reassigned as they never age according to
the time since last rebalance.
Change-Id: Ie0e2b5e25140cbac7465f31a26a4998beb3892e9
Closes-Bug: #1526017
Added new unit tests:
test_add_device_old_missing_region
test_create_ring_number_of_arguments
test_add_duplicate_devices
test_rebalance_with_seed
test_set_overload_number_of_arguments
test_main_no_arguments
test_main_single_argument
test_main_with_safe
Modified existing unit tests to create sample ring at start of test.
This change was needed to have unit tests run correctly and demonstrate
code coverage.
test_unknown
test_search_device_number_of_arguments
test_list_parts_number_of_arguments
test_set_weight_number_of_arguments
test_set_info_number_of_arguments
test_remove_device_number_of_arguments
test_set_min_part_hours_number_of_arguments
test_set_replicas_number_of_arguments
test_set_replicas_invalid_value
Updates to handled nested mocks.
Updates to handle no exception case when SystemExit is expected.
PEP8 corrections.
Moved new tests from try blocks to use of assertRaises or call to
run_srb using exp_results with specified exit codes.
Updated run_srb to accept a dictionary of expected results. Specifically,
look for 'valid_exit_codes' to test, default to (0,1).
Change-Id: I4cf3f5f055a9babba140c68a9c7ff90b9c50ea62
test_write_builder_after_device_removal() wasn't setting a
default min_part_hours so a warnign was printed. Explicitly
adding a min_part_hours suppresses the warning
Change-Id: I6f234b72c34e066abb91f28e6eacf50e29be8842
If a device with 0 weight is tried to remove, the following rebalance
does not write changes into builder file.
Scenario:
$ swift-ring-builder object.builder set_weight --id 1 0.00
$ swift-ring-builder object.builder rebalance
Wait for moving files out of the device id=1.
$ swift-ring-builder object.builder remove --id 1
$ swift-ring-builder object.builder rebalance
In fact, the device id=1 is not removed after rebalance (must be --force used).
Change-Id: Iad5a444023eae9882a3addd7f119ff4d18559ddd
