Since Python 2.7, unittest in the standard library has included mulitple
facilities for skipping tests by decorators as well as an exception.
Switch to that directly, rather than importing nose.
Change-Id: I4009033473ea24f0d0faed3670db844f40051f30
It was deprecated and we discussed on this topic in Denver PTG
for Queen cycle. Main motivation for this work is that deprecated
post_as_copy option and its gate blocks future symlink work.
Change-Id: I411893db1565864ed5beb6ae75c38b982a574476
I think the probetest can be a little more pathological and I'm worried
about logging the response of each individual request at error when we
currently only log the entire failure at debug.
I have lots of async pendings sometimes.
Related-Change: Ic5d55ff368cec62ab2eb9fd883801ba4ab043dd4
Change-Id: I1c070b2b07d31552f36b647731b0b3ebed31871f
I changed asserts with more specific assert methods.
e.g.: from assertTrue(sth == None) to assertIsNone(*) or
assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type) or
assertTrue(not sth) to assertFalse(sth).
The code gets more readable, and a better description will be shown on fail.
Change-Id: I3768faa568e3964e726ecc48ac8cb133cb088284
Object-updater regards 404 from container-server as success.
This causes wrong giving up and generates object which never appears
at container, for example when all container replicas are placed in
handoff devices at the moment.
This patch makes object-updater try updating until it gets 201s from
all container servers, and ensures the update.
Change-Id: Ic5d55ff368cec62ab2eb9fd883801ba4ab043dd4
Closes-Bug: #1328735
Whatever container update override etag is sent to the object server
with a PUT must be used in container updates for subsequent
POSTs. Unfortunately the current container update override headers
(x-backend-container-update-override-*) are not persisted with the
object metadata so are not available when handling a POST.
For EC there is an ugly hack in the object server to use the
x-object-sysmeta-ec-[etag,size] values when doing a container update
for a POST.
With crypto, the encryption middleware needs to override the etag
(possibly overriding the already overridden EC etag value) with an
encrypted etag value. We therefore have a similar problem that this
override value is not persisted at the object server.
This patch introduces a new namespace for container override headers,
x-object-sysmeta-container-update-override-*, which uses object
sysmeta so that override values are persisted. This allows a general
mechanism in the object server to apply the override values (if any
have been set) from object sysmeta when constructing a container
update for a PUT or a POST. Middleware should use the
x-object-sysmeta-container-update-override-* namespace when setting
container update overrides. Middleware should be aware that other
middleware may have already set container override headers, in which
case consideration should be given to whether any existing value should
take precedence.
For backwards compatibility the existing
x-backend-container-update-override-* style headers are still
supported in the object server for EC override values, and the ugly
hack for EC etag/size override in POST updates remains in the object
server. That allows an older proxy server to be used with an upgraded
object server. The proxy server continues to use the
x-backend-container-update-override-* style headers for EC values so
that an older object server will continue to work with an upgraded
proxy server.
x-object-sysmeta-container-update-override-* headers take precedence
over x-backend-container-update-override-* headers and the use of
x-backend-container-update-override-* headers by middleware is
deprecated. Existing third party middleware that is using
x-backend-container-update-override-* headers should be modified to
use x-object-sysmeta-container-update-override-* headers in order to
be compatible with other middleware such as encryption and to ensure
that container updates during POST requests carry correct values. If
targeting multiple versions of Swift object servers it may be
necessary to send headers from both namespaces. However, in general it
is recommended to upgrade all backend servers, then upgrade proxy
servers before finally upgrading third party middleware.
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
UpgradeImpact
Change-Id: Ib80b4db57dfc2d37ea8ed3745084a3981d082784
When a PUT request is made to an EC object the resulting container
update must include the override values for the actual object
etag and size, as opposed to the fragment etag and size. When a POST
request is made the same override values should be included in the
container update, but currently the update includes the incorrect EC
fragment size (but the correct body etag).
This is ok so long as the update for the object PUT request arrives at
the container server first (whether by direct update or replication)
because the etag and size values in an update due to an object POST
will not have a newer timestamp that the PUT and will therefore be
ignored at the container server.
However, if the update due to the object PUT request has not arrived
at the container server when the update due to the object POST
arrives, then the etag and incorrect size sent with the POST update
will be recorded in the container server. If the update due to the PUT
subsequently arrives it will not fix this error because the timestamp
of its etag and size values is not greater than that of the already
recorded values.
Fortunately the correct object body size is persisted with the object
as X-Backend-Container-Update-Override-Size sysmeta so this patch
fixes the container update due to a POST to use that value instead of
the Content-Length metadata.
Closes-Bug: #1582723
Change-Id: Ide7c9c59eb41aa09eaced2acfd0700f882c6eab1
The replacement of assert_ with assertTrue [1] resulted in a number of
tests using calls of the form assertTrue(not <condition>). This patch
replaces those with assertFalse(<condition>).
[1] change I74705c6498249337bfdf955d62e0ad972035bc1f
Change-Id: I78b49558f4425c2335df187b1793d1e4b3c514b1
The assert_() method is deprecated and can be safely replaced by assertTrue().
This patch makes sure that running the tests does not generate warnings
all over the screen.
Change-Id: I74705c6498249337bfdf955d62e0ad972035bc1f
Enabled by a new > 0 integer config value, "servers_per_port" in the
[DEFAULT] config section for object-server and/or replication server
configs. The setting's integer value determines how many different
object-server workers handle requests for any single unique local port
in the ring. In this mode, the parent swift-object-server process
continues to run as the original user (i.e. root if low-port binding
is required), binds to all ports as defined in the ring, and forks off
the specified number of workers per listen socket. The child, per-port
servers drop privileges and behave pretty much how object-server workers
always have, except that because the ring has unique ports per disk, the
object-servers will only be handling requests for a single disk. The
parent process detects dead servers and restarts them (with the correct
listen socket), starts missing servers when an updated ring file is
found with a device on the server with a new port, and kills extraneous
servers when their port is found to no longer be in the ring. The ring
files are stat'ed at most every "ring_check_interval" seconds, as
configured in the object-server config (same default of 15s).
Immediately stopping all swift-object-worker processes still works by
sending the parent a SIGTERM. Likewise, a SIGHUP to the parent process
still causes the parent process to close all listen sockets and exit,
allowing existing children to finish serving their existing requests.
The drop_privileges helper function now has an optional param to
suppress the setsid() call, which otherwise screws up the child workers'
process management.
The class method RingData.load() can be told to only load the ring
metadata (i.e. everything except replica2part2dev_id) with the optional
kwarg, header_only=True. This is used to keep the parent and all
forked off workers from unnecessarily having full copies of all storage
policy rings in memory.
A new helper class, swift.common.storage_policy.BindPortsCache,
provides a method to return a set of all device ports in all rings for
the server on which it is instantiated (identified by its set of IP
addresses). The BindPortsCache instance will track mtimes of ring
files, so they are not opened more frequently than necessary.
This patch includes enhancements to the probe tests and
object-replicator/object-reconstructor config plumbing to allow the
probe tests to work correctly both in the "normal" config (same IP but
unique ports for each SAIO "server") and a server-per-port setup where
each SAIO "server" must have a unique IP address and unique port per
disk within each "server". The main probe tests only work with 4
servers and 4 disks, but you can see the difference in the rings for the
EC probe tests where there are 2 disks per server for a total of 8
disks. Specifically, swift.common.ring.utils.is_local_device() will
ignore the ports when the "my_port" argument is None. Then,
object-replicator and object-reconstructor both set self.bind_port to
None if server_per_port is enabled. Bonus improvement for IPv6
addresses in is_local_device().
This PR for vagrant-swift-all-in-one will aid in testing this patch:
https://github.com/swiftstack/vagrant-swift-all-in-one/pull/16/
Also allow SAIO to answer is_local_device() better; common SAIO setups
have multiple "servers" all on the same host with different ports for
the different "servers" (which happen to match the IPs specified in the
rings for the devices on each of those "servers").
However, you can configure the SAIO to have different localhost IP
addresses (e.g. 127.0.0.1, 127.0.0.2, etc.) in the ring and in the
servers' config files' bind_ip setting.
This new whataremyips() implementation combined with a little plumbing
allows is_local_device() to accurately answer, even on an SAIO.
In the default case (an unspecified bind_ip defaults to '0.0.0.0') as
well as an explict "bind to everything" like '0.0.0.0' or '::',
whataremyips() behaves as it always has, returning all IP addresses for
the server.
Also updated probe tests to handle each "server" in the SAIO having a
unique IP address.
For some (noisy) benchmarks that show servers_per_port=X is at least as
good as the same number of "normal" workers:
https://gist.github.com/dbishop/c214f89ca708a6b1624a#file-summary-md
Benchmarks showing the benefits of I/O isolation with a small number of
slow disks:
https://gist.github.com/dbishop/fd0ab067babdecfb07ca#file-results-md
If you were wondering what the overhead of threads_per_disk looks like:
https://gist.github.com/dbishop/1d14755fedc86a161718#file-tabular_results-md
DocImpact
Change-Id: I2239a4000b41a7e7cc53465ce794af49d44796c6
This patch adds the erasure code reconstructor. It follows the
design of the replicator but:
- There is no notion of update() or update_deleted().
- There is a single job processor
- Jobs are processed partition by partition.
- At the end of processing a rebalanced or handoff partition, the
reconstructor will remove successfully reverted objects if any.
And various ssync changes such as the addition of reconstruct_fa()
function called from ssync_sender which performs the actual
reconstruction while sending the object to the receiver
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
blueprint ec-reconstructor
Change-Id: I7d15620dc66ee646b223bb9fff700796cd6bef51
Crypto middleware needs to arrange for alternative
values of etag and content-type to be sent to container
servers with updates, since these will be encrypted with
a different key than the etag and content-type stored on
the object server.
Erasure coding apparently needs a similar capability.
This patch modifies the object server to overwrite the etag
and content-type values in the container update headers with
values that may optionally be specified by middleware in
X-Backend-Container-Update-Override-* headers.
Using the X-Backend- prefix ensures that these headers
cannot be sent or seen by clients.
A new probe test verifies the propagation of override
values from an internal client through the proxy, to
object server, to container server and then returned
in a container listing.
Change-Id: I7d846ed54ff173d08c66c6d5b0ecf7dff27f5a87
* refactor probe tests to use probe.common.ProbeTest
* move reset_environment functionality to ProbeTest.setUp()
* choose rings and policies that meet the criteria - raise SkipTest if
nothing matches
* replace all AssertionErrors in setup with SkipTest
Change-Id: Id56c497d58083f5fd55f5283cdd346840df039d3
Add headers param to direct_client.direct_get_object, which is used in
probetests to passthrough the X-Storage-Policy-Index header.
DocImpact
Implements: blueprint storage-policies
Change-Id: I19adbbcefbc086c8467bd904a275d55cde596412
Support separate replication ip address:
- Added new function in utils. This function provides ability
to select separate IP address for replication service.
- Db_replicator and object replicators were changed.
Replication process uses new function now.
Replication network parameters:
- Replication network fields (replication_ip, replication_port)
support was added to device dictionary in swift-ring-builder script.
- Changes were made to support new fields in search, show and set_info
functions.
Implementation of replication servers:
- Separate replication servers use the same code as normal replication
servers, but with replication_server parameter = True. When using a
separate replication network, the non-replication servers set
replication_server = False. When there is no separate replication
network (the default case), replication_server is not included in the config.
DocImpact
Change-Id: Ie9af5bdcdf9241c355e36053ca4adfe49dc35bd0
Implements: blueprint dedicated-replication-network
The probe tests were woefully out of date with all the changes that
have ocurred since they were written. I've updated most of them and
removed some that are hopeless outdated.
I also greatly improved the timing issues (hopefully completely
solved them? I ran them 25 times with no problems) and made them pep8
1.3.1 safe.
Change-Id: I8e9dbb6e7d6e04e293843b1dce1ded99d84e0348
Updated the imports and added a head_account to the "is the cluster
started yet?" checks. Hopefully this fixes the notorious timing
issues of these tests where auth answers requests just a bit before
the rest of the cluster is ready.
Fixes bug 1014931
Change-Id: Iea1d62db2317560371da49af5e94a0279b646294
