Rather than having a comment declaring that the stack/history modes
only diverge after the delete(), move the common code to a separate
function called by tests for both modes.
Change-Id: I657106b11bf0697338a776513d2a51ac6e562c0a
From the related change of ECDuplication, Swift have a couple of Test
classes for EC policy, normal EC and EC Duplication, in the
test/unit/proxy/test_server.py. To enable the classes, the related change
abstracts the EC test cases as the ECTestMixin class to gather test
methods into one place but it was worse because TestObjectController did
still have both test cases for replication and for ec that may be hard
to understand the test class structure.
Hence, this patch attempts to refactor the structure as
From:
ECTestMixin
|
-------------------------------------
| |
TestObjectController TestObjectControllerECDuplication
(for replication and EC) (for EC Duplication Policy)
To:
BaseTestObjectController
|
--------------------------------------
| |
TestReplicatedObjectController BaseTestECObjectController
(for replication) |
---------------------------------
| |
TestECObjectController TestECDuplicationObjectController
(for EC policy) (for EC Duplication Policy)
Some more cleanups are in follow up patches because this patch shows a lot
of moving code chunks which could be hard to compare the diff. To make
the review easy, this patch forcus on ONLY the structure changes as
possible.
Related-Change: Idd155401982a2c48110c30b480966a863f6bd305
Related-Change: I25a3f8fc837706d78dca226fe282d9e5ead65a0d
Change-Id: Ifd3d0fa66773e640bb61cc528f7a1b2358e97d91
Also, use `==` instead of `is` for string comparison.
Change-Id: I12040d56afc239a7b7be1ae37b27cce708bf6f10
Related-Change: I1ccb2665b6cd2887659e548e55a26aa00de879e3
DiskFileMixin and DiskFileManagerMixin has almost
identical setUp() and tearDown() methods, and both
inherit BaseDiskFileTestMixin, so this moves the common
code into the abstract superclass.
Also remove repeated declaration of vars in
test_diskfile.py:run_quarantine_invalids
and a duplicated qualified import in obj/test_server.py
Change-Id: Id99ba151c7802c3b61e483a7e766bf6f2b2fe3df
Test that the actual configured options get reported in /info
Follow up to:
Related-Change: If1af43485b4708cab6c4b5d7f6f0a334d8752518
Change-Id: I1d76d40c70a38b959dd0103f3c56b1da4dae068f
Also removed a bunch of unnecessary unquotes. Just use path_info
instead (it's already unquoted).
Partial-Bug: #1670915
Change-Id: If1af43485b4708cab6c4b5d7f6f0a334d8752518
Before this change, subrequests made while servicing a copy would
result in logging the request type from the copy PUT/GET request
instead of the type from the subrequest being logged.
In order to have the correct requst type logged for subrequests:
- Changed subrequest environments to not inherit the orig_req_method
of the enclosing request.
- Changed copy to be more picky about when it sets orig_req_method
In addition, subrequest environments will no longer inherit the
swift.log_info from the enclosing request. That inheritance had
been added at Ic96a92e938589a2f6add35a40741fd062f1c29eb
along with swift.orig_req_method.
Change-Id: I1ccb2665b6cd2887659e548e55a26aa00de879e3
Closes-Bug: #1657246
Suffix hash invalidations in hashes.invalid can be lost when two
concurrent calls to get_hashes consolidate the hashes of a new
partition with no hashes.pkl:
- suffix S has been invalidated and is listed in hashes.invalid
- process X calls get_hashes when there is no existing hashes.pkl
- process X removes hashes.invalids file in consolidate_hashes
- process X calculates the hash of suffix S (note, process X has
not yet written hashes.pkl)
- process Y invalidates suffix S, appends S to hashes.invalid, so the
hash of suffix S *should* be recalculated at some point
- process Z calls get_hashes->consolidate_hashes, deletes hashes.invalid
because there is still no hashes.pkl
- process Z fails
- process X writes hashes.pkl with stale hash value for suffix S
- the invalidation of S that was made by process Y is lost
The solution is to never remove hashes.invalid during consolidate_hashes
without first recording any invalid suffixes in hashes and writing hashes
to disk in hashes.pkl. This is already the behaviour when hashes.pkl
exists. The cost of an additional write to hashes.pkl, introduced by this
patch, is only incurred once, when get_hashes first encounters a
partition with no hashes.pkl.
Related-Change: Ia43ec2cf7ab715ec37f0044625a10aeb6420f6e3
Change-Id: I08c8cf09282f737103e580c1f57923b399abe58c
The tests were lacking coverage for the chunk_transformer
reading multiple segment_size pieces from an input chunk.
This patch modifies test_chunk_transformer to exercise more
input chunk scenarios.
Also improve variable naming and comments in
_test_determine_chunk_destinations_prioritize
Change-Id: I4eb55ee3e87dae478828f7ccba86fec267492bd8
Related-Change: Ib9e8a6f67c2985164dd20b049c7f144f19fd1822
The Related-Change removed the frag_index expected from a
node response from the full_path included in log messages.
This made sense because the *expected* frag_index is not
necessarily the index that was actually received from the
node. However, it would be useful to include the *actual*
received frag_index in log messages.
This patch also:
- makes _full_path a module level function
- renames unique_index to be resp_frag_index to aid
understanding of the various indexes we deal with during
reconstruction.
Change-Id: Ic932835b3c1ed51a8456fce775fb59445fcb834b
Related-Change: I8096202f5f8d91296963f7a409a29d57fa7828e4
To address Alistair's comment at
https://review.openstack.org/#/c/219165.
This includes:
- Fix reconstructor log message to avoid redundant frag index info
- Fix incorrect FabricatedRing setting to have ec_k + ec_m replicas
- Use policy.ec_n_unique_fragments for testing frag index election
- Plus some various minor cleanup and docs additions
Huge refactoring around TestECMixin at the test/unit/proxy/test_server.py
is in https://review.openstack.org/#/c/440466/ to clarify the change.
Co-Authored-By: Alistair Coles <alistairncoles@gmail.com>
Change-Id: I8096202f5f8d91296963f7a409a29d57fa7828e4
test_GET_with_mixed_etags_at_same_timestamp in
test.unit.proxy.controllers.test_obj:TestECObjController
will fail intermittently when the randomly chosen bodies
of the two objects are identical. In conjunction with the
timestamps of the two objects being deliberately equal,
this makes the two objects identical.
Fix it by explicitly setting unique base strings for the
object bodies.
Change-Id: Idb1081edfa26b9f229f44b00c439cda33a7385fa
"swift.proxy.controllers.base.Controller" inherits "object", so the
Controller class and its sub classes (AccountController,
ContainerController, BaseObjectController, InfoController) are
"new style class". In new style class, if a class call super class's
constructor, "super(SubClass, self).__init__(foo, bar)" is recommended.
But, AccountController, ContainerController, BaseObjectController,
and InfoController use "Controller.__init__(self, app)", and it is
deprecated.
This patch fixes the calling super class constructor codes.
Change-Id: I4b94ec3131c7c7be4609716867a36490a70d5009
Closes-Bug: #1672285
I tried to move res.fix_conditional_response() into the get_log_line
method in utils but couldn't find a container request that was
*actually* conditional. We return 412 from the container on a couple
of bad requests where we really probably shouldn't - but regardless those
responses aren't really conditional.
Plus I'd have to make a synthetic test in utils somewhere.
So this seems better - leave the fix closer to the problem.
Change-Id: I76237d5d5025a653e879141b03df863205c59f3a
Related-Change-Id: I66c871f1f15fb43f19d59331b05cc7567ce10063
This patch moves some code from the crypto files
to a more common modules that will be used by symlinks
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I1758693c5dd428f9f2157966aac49d97c2c7ab12
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Add support for a 2+1 EC policy to be optionally used as default
policy when running in process functional tests.
The EC policy may be selected by setting the env var:
SWIFT_TEST_IN_PROCESS_CONF_LOADER=ec tox
when running .functests, or by using the new tox test env:
tox -e func-ec
Change-Id: I02e3553a74a024efdab91dcd609ac1cf4e4f3208
Replacing function without mocking creates weird behaviors as the results
of the tests sometimes depends on the order of the execution. Use real
mocking to avoid such behaviors.
Example: using real mock showed that test test_host_is_storage_domain was
wrong because it was not "reseting" the lookup_cname function at the
beginning of the test.
Change-Id: Ibc60697c9c850e6e3a108f0d906e8906c3f53ced
The policy of giving projects vanity domains stopped about 5 years ago.
swift.openstack.org is a redirect to the canonical location -
docs.openstack.org/developer/swift. While we are not aiming to remove
the redirect any time in the forseeable future due to existing published
links pointing to it, we should at the very least stop adding more of
those links to the world.
Change-Id: I10e92309f5d3b5f908fe4438f5cc0b184f161cba
SAIO docs do suggest using Ubuntu 14.04, but if using
16.04 then systemctl needs to be used to have rsync service
restart on reboot.
Change-Id: I4fb0d3d063df61fbdfca981f06911148f3c4dc04
Tighten up test to verify that 404 response results in a None
return from reconstructor _get_response. Merge this test case
into the test_get_response method to make use of the do_test()
infrastructure. Add check for 503 response status.
Also, use assertFalse to verify empty log lines, since a failure
will then result in any log lines being shown in failure message.
Related-Change: Iba86b495a14c15fc6eca8bf8a7df7d110256b0af
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Ia83517e6d4c2f5eeb136abd4c04ddab639d40b9e
Originally, ec duplication is designed to make duplicated copy at
chunk_transformer function for each node. However, it can cause doubled
(or more than, depends on ec_duplication_factor) etag calculation which
should be expensive for Swift proxy work.
Since this patch, chunk_transformer keeps only an unique fragments slot
to store, and then, send_chunk in ECObjectController._transfer_data
picks up the suitable chunk by unique backend fragment index assigned by
_determine_chunk_destination function to send to object server.
Note that, Putter still keeps node_index but the new putter_to_frag_index
dict and frag_hasher (chunk_index and chunk_hasher in the old names)
now refers the value by fragment index.
Change-Id: Ib9e8a6f67c2985164dd20b049c7f144f19fd1822
Unlinke DNSException, NXDOMAIN and NoAnswer are not errors. Thus, they
should be cached so that a misconfiguration from a cname_lookup user do
not generate too many requests on the nameservers.
This patch cache NXDOMAIN and NoAnswer response for 60 seconds.
Change-Id: I1d3002bceaf5f5bee364fea6afe52cbf2aeb5fd2
Layout the foundation for documenting the features which will enable
Global EC.
The formatting on the sections in our existing EC docs didn't follow
best practices [1] and it caused some sphinx build warnings.
1. http://www.sphinx-doc.org/en/stable/rest.html#sections
Change-Id: I2d164dafeb84629c75c3c2ff774329ee84270b7f
