The repo is Python using both Python 2 and 3 now, so update hacking to
version 2.0 which supports Python 2 and 3. Note that latest hacking
release 3.0 only supports version 3.
Fix problems found.
Remove hacking and friends from lower-constraints, they are not needed
for installation.
Change-Id: I9bd913ee1b32ba1566c420973723296766d1812f
If we move it to constraints it's more globally accessible in our code,
but more importantly it's more obvious to ops that everything breaks if
you try to mis-configure different values per-service.
Change-Id: Ib8f7d08bc48da12be5671abe91a17ae2b49ecfee
There's still one problem, though: since swiftclient on py3 doesn't
support non-ASCII characters in metadata names, none of the tests in
TestReconstructorRebuildUTF8 will pass.
Change-Id: I4ec879ade534e09c3a625414d8aa1f16fd600fa4
Since Python 2.7, unittest in the standard library has included mulitple
facilities for skipping tests by decorators as well as an exception.
Switch to that directly, rather than importing nose.
Change-Id: I4009033473ea24f0d0faed3670db844f40051f30
I changed asserts with more specific assert methods.
e.g.: from assertTrue(sth == None) to assertIsNone(*) or
assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type) or
assertTrue(not sth) to assertFalse(sth).
The code gets more readable, and a better description will be shown on fail.
Change-Id: I3768faa568e3964e726ecc48ac8cb133cb088284
F812 list comprehension redefines <variable> from line ...
While the current violations were benign, this sort of code can easily
lead to subtle bugs. Seems worth checking, especially given how cheap it
is to bring existing code in line with it.
Change-Id: Ibdcf9f93b85a1f1411198001df6bdbfa8f92d114
Relocates some test infrastructure in preparation for
use with encryption tests, in particular moves the test
server setup code from test/unit/proxy/test_server.py
to a new helpers.py so that it can be re-used, and adds
ability to specify additional config options for the
test servers (used in encryption tests).
Adds unit test coverage for extract_swift_bytes and functional
test coverage for container listings. Adds a check on the content
and metadata of reconciled objects in probe tests.
Change-Id: I9bfbf4e47cb0eb370e7a74d18c78d67b6b9d6645
The probe test in question tries to make a manifest referencing
unavailable objects and expects that to fail with a 400. If the SLO
middleware is enabled, then it checks the segments, can't find some,
and returns the 400. If it's disabled, the PUT succeeds and makes an
object whose contents are some JSON blob. In the latter case, the
probe test would fail because it expected a real SLO manifest but
didn't find one.
Now we skip the remainder of the test when we detect that SLO is not
enabled.
Change-Id: I3e7e8e98107608e675efc24156e703bc167458bb
The assert_() method is deprecated and can be safely replaced by assertTrue().
This patch makes sure that running the tests does not generate warnings
all over the screen.
Change-Id: I74705c6498249337bfdf955d62e0ad972035bc1f
This patch adds the erasure code reconstructor. It follows the
design of the replicator but:
- There is no notion of update() or update_deleted().
- There is a single job processor
- Jobs are processed partition by partition.
- At the end of processing a rebalanced or handoff partition, the
reconstructor will remove successfully reverted objects if any.
And various ssync changes such as the addition of reconstruct_fa()
function called from ssync_sender which performs the actual
reconstruction while sending the object to the receiver
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
blueprint ec-reconstructor
Change-Id: I7d15620dc66ee646b223bb9fff700796cd6bef51
* move get_to_final_state into ProbeTest
* get rid of kill_servers
* add replicators manager and updaters manager to ProbeTest
(this is all going someplace, i promise)
Change-Id: I8393a2ebc0d04051cae48cc3c49580f70818dbf2
* refactor probe tests to use probe.common.ProbeTest
* move reset_environment functionality to ProbeTest.setUp()
* choose rings and policies that meet the criteria - raise SkipTest if
nothing matches
* replace all AssertionErrors in setup with SkipTest
Change-Id: Id56c497d58083f5fd55f5283cdd346840df039d3
A deprecated policy in swift.conf causes errors in
probe tests that may attempt to use that policy.
This patch introduces a list ENABLED_POLICIES in
test/probe/common.py and changes probe tests to only
use policies contained in that list.
Change-Id: Ie65477c15d631fcfc3a4a5772fbe6d7d171b22b0
This patch takes a first step towards support
for object system metadata by enabling headers
in the x-object-sysmeta- namespace to be
persisted when objects are PUT. This should be
useful for other pending patches such as on
demand migration and server side encryption
(https://review.openstack.org/#/c/64430/ and
https://review.openstack.org/#/c/76578/1).
The x-object-sysmeta- namespace is already
reserved/protected by the gatekeeper and
passed through the proxy. This patch modifies
the object server to persist these headers
alongside user metadata when an object is
PUT.
This patch will preserve existing object
system metadata and ignore any new system
metadata when handling object POSTs,
including POST-as-copy operations. Support
for modification of object system metadata
with a POST request requires further work
as discussed in the blueprint.
This patch will preserve existing object
system metadata and update it with new
system metadata when copying an object.
A new probe test is added which makes use of
the BrainSplitter class that has been moved
from test_container_merge_policy_index.py to
a new module brain.py.
blueprint object-system-metadata
Change-Id: If716bc15730b7322266ebff4ab8dd31e78e4b962
With the two vector timestamp change some resolution was lost in the queue
entries that could lead to the reconciler being unable to successfully remove
a processed item from the queue in pop_queue. To ensure the queue entries
with a significant offset can be successfully removed while still handling
the re-enqueued object case issue the DELETE with the timestamp slightly later
than the maximum of the queue entries last modified time (q_record) and
misplaced objects timestamp (q_ts).
Change-Id: I4726243b3f7c4c1e98f0c578e7ffdecf4ec22199
Replaced throughout code base & tox'd. Functional as well
as probe tests pass with and without policies defined.
POLICY --> 'X-Storage-Policy'
POLICY_INDEX --> 'X-Backend-Storage-Policy-Index'
Change-Id: Iea3d06de80210e9e504e296d4572583d7ffabeac
Add headers param to direct_client.direct_get_object, which is used in
probetests to passthrough the X-Storage-Policy-Index header.
DocImpact
Implements: blueprint storage-policies
Change-Id: I19adbbcefbc086c8467bd904a275d55cde596412
After a container database is replicated, a _post_replicate_hook will enqueue
misplaced objects for the container-reconciler into the .misplaced_objects
containers. Items to be reconciled are "batch loaded" into the reconciler
queue and the end of a container replication cycle by levering container
replication itself.
DocImpact
Implements: blueprint storage-policies
Change-Id: I3627efcdea75403586dffee46537a60add08bfda
Keep status_changed_at in container databases current with status changes that
occur as a result of container creation, deletion, or re-creation.
Merge container put/delete/created timestamps when handling replicate
responses from remote servers in addition to during the handling of the
REPLICATE request.
When storage policies are configured on a cluster send status_changed_at,
object_count and storage_policy_index as part of container replication sync
args.
Use status_changed_at during replication to determine the oldest active
container and merge storage_policy_index.
DocImpact
Implements: blueprint storage-policies
Change-Id: Ib9a0dd42c271145e641437dc04d0ebea1e11fc47
You can manually setup a split brain scenario for reconciler testing with the
enqueue script using the machinery from the included probetest. Evoke the
test as a script with with 'split-brain' command for more help.
DocImpact
Implements: blueprint storage-policies
Change-Id: I3a7b3167d674eba5f6e4072b176f6c4d29cdcd72
