This commit makes it possible to PUT an object into Swift and have it
stored using erasure coding instead of replication, and also to GET
the object back from Swift at a later time.
This works by splitting the incoming object into a number of segments,
erasure-coding each segment in turn to get fragments, then
concatenating the fragments into fragment archives. Segments are 1 MiB
in size, except the last, which is between 1 B and 1 MiB.
+====================================================================+
| object data |
+====================================================================+
|
+------------------------+----------------------+
| | |
v v v
+===================+ +===================+ +==============+
| segment 1 | | segment 2 | ... | segment N |
+===================+ +===================+ +==============+
| |
| |
v v
/=========\ /=========\
| pyeclib | | pyeclib | ...
\=========/ \=========/
| |
| |
+--> fragment A-1 +--> fragment A-2
| |
| |
| |
| |
| |
+--> fragment B-1 +--> fragment B-2
| |
| |
... ...
Then, object server A gets the concatenation of fragment A-1, A-2,
..., A-N, so its .data file looks like this (called a "fragment archive"):
+=====================================================================+
| fragment A-1 | fragment A-2 | ... | fragment A-N |
+=====================================================================+
Since this means that the object server never sees the object data as
the client sent it, we have to do a few things to ensure data
integrity.
First, the proxy has to check the Etag if the client provided it; the
object server can't do it since the object server doesn't see the raw
data.
Second, if the client does not provide an Etag, the proxy computes it
and uses the MIME-PUT mechanism to provide it to the object servers
after the object body. Otherwise, the object would not have an Etag at
all.
Third, the proxy computes the MD5 of each fragment archive and sends
it to the object server using the MIME-PUT mechanism. With replicated
objects, the proxy checks that the Etags from all the object servers
match, and if they don't, returns a 500 to the client. This mitigates
the risk of data corruption in one of the proxy --> object connections,
and signals to the client when it happens. With EC objects, we can't
use that same mechanism, so we must send the checksum with each
fragment archive to get comparable protection.
On the GET path, the inverse happens: the proxy connects to a bunch of
object servers (M of them, for an M+K scheme), reads one fragment at a
time from each fragment archive, decodes those fragments into a
segment, and serves the segment to the client.
When an object server dies partway through a GET response, any
partially-fetched fragment is discarded, the resumption point is wound
back to the nearest fragment boundary, and the GET is retried with the
next object server.
GET requests for a single byterange work; GET requests for multiple
byteranges do not.
There are a number of things _not_ included in this commit. Some of
them are listed here:
* multi-range GET
* deferred cleanup of old .data files
* durability (daemon to reconstruct missing archives)
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I9c13c03616489f8eab7dcd7c5f21237ed4cb6fd2
* Get FakeConn ready for expect 100 continue
* Use debug_logger more and with better interfaces
* Fix patch_policies to be less annoying
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I28c0a3539d994cbb8e6b94d63a23ed4ea6cb956d
Extracting large chunks of the PUT method into smaller
methods to improve maintainability and reuse of code.
Based on the work that Clay Gerrard started:
https://review.openstack.org/#/c/77812/
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Id479fc5b159a2782361ac4a6e4a6d8bbaee4fe85
Signed-off-by: Thiago da Silva <thiago@redhat.com>
More memcache options can be set in the memcache.conf or proxy-server.conf
* connect_timeout
* pool_timeout
* tries
* io_timeout
Options set in proxy-server.conf are considered more specific to the memcache
middleware.
DocImpact
Change-Id: I194d0f4d88c6cb8c797a37dcab48f2d8473e7a4e
All GET or HEAD requests consistently error limit nodes that return 507
and increment errors for nodes responding with any other 5XX.
There were two places in the object PUT path where the proxy was error
limiting nodes and their behavior was inconsistent. During expect-100
connect we would only error_limit nodes on 507, and during response we
would increment errors for all 5XX series responses. This was pretty
hard to reason about and the divergence in behavior of questionable
value.
An audit of base controller highlighted where make_requests would apply
error_limit's on 507 but not increment errors on other 5XX responses.
Now anywhere we track errors on nodes we use error_limit on 507 and
error_occurred on any other 5XX series request. Additionally a Timeout
or Exception that is logged through exception_occurred will bump errors -
which is consistent with the approach in "Add Error Limiting to slow
nodes" [1].
1. https://review.openstack.org/#/c/112424/
Change-Id: I67e489d18afd6bdfc730bfdba76f85a2e3ca74f0
The proxy was storing the error count and last-error time in the
ring's internal data, specifically in the device dictionaries. This
works okay, but it means that whenever a ring changes, all the error
stats reset.
Now the error stats live in the proxy server object, so they survive a
ring reload.
Better yet, the error stats are now keyed off of the node's
IP/port/device triple, so if you have the same device in two rings
(like with multiple storage policies), then the error stats are
combined. If the proxy server sees a 507 for an objec request in
policy X, then that will now result in that particular object disk
being error-limited for requests in policies Y and Z as well.
Change-Id: Icc72b68b99f37367bb16d43688e7e45327e3e022
When a X-Backend-Timestamp is available it would generally preferred
over a less specific value and sorts correctly against any X-Timestamp
values anyway.
Change-Id: I08b7eb37ab8bd6eb3afbb7dee44ed07a8c69b57e
This commit lets the object server use splice() and tee() to move data
from disk to the network without ever copying it into user space.
Requires Linux. Sorry, FreeBSD folks. You still have the old
mechanism, as does anyone who doesn't want to use splice. This
requires a relatively recent kernel (2.6.38+) to work, which includes
the two most recent Ubuntu LTS releases (Precise and Trusty) as well
as RHEL 7. However, it excludes Lucid and RHEL 6. On those systems,
setting "splice = on" will result in warnings in the logs but no
actual use of splice.
Note that this only applies to GET responses without Range headers. It
can easily be extended to single-range GET requests, but this commit
leaves that for future work. Same goes for PUT requests, or at least
non-chunked ones.
On some real hardware I had laying around (not a VM), this produced a
37% reduction in CPU usage for GETs made directly to the object
server. Measurements were done by looking at /proc/<pid>/stat,
specifically the utime and stime fields (user and kernel CPU jiffies,
respectively).
Note: There is a Python module called "splicetee" available on PyPi,
but it's licensed under the GPL, so it cannot easily be added to
OpenStack's requirements. That's why this patch uses ctypes instead.
Also fixed a long-standing annoyance in FakeLogger:
>>> fake_logger.warn('stuff')
>>> fake_logger.get_lines_for_level('warn')
[]
>>>
This, of course, is because the correct log level is 'warning'. Now
you get a KeyError if you call get_lines_for_level with a bogus log
level.
Change-Id: Ic6d6b833a5b04ca2019be94b1b90d941929d21c8
A long, long time ago, on a GET request, the proxy would go look on 3*
nodes for the requested thing. If one of the primary nodes was
error-limited, it'd look on two primaries and a handoff. Since this
indicated some failure somewhere, the proxy would emit a warning:
"Handoff requested (1)". If two primaries were down, there'd be a
second message "Handoff requested (2)", and so on.
Some StatsD messages were emitted too.
A somewhat shorter time ago (commit d79a67eb), the proxy started
looking into handoffs if it got 404s from the primaries. While this
was a good idea, it resulted lots of "Handoff requested (N)" log spam;
you'd see these messages on every single 404. Also, the StatsD
handoff_count and handoff_all_count metrics shot way up and turned
into noise.
This commit restores the original intent (and usefulness) of the log
messages and StatsD metrics: if the proxy only looks at the normal
number of handoff nodes, nothing is logged. However, if a primary is
down, then the message "Handoff requested (1)" will be logged,
indicating that the proxy looked at one more handoff than it normally
would, and this happened because a primary node was error-limited.
Closes-Bug: 1297214
* or whatever the replica count was
Change-Id: If1b77c18c880b096e8ab1df3008db40ce313835d
The get_part method is fast and stable given a consistent hash path
suffix/prefix, so there's no absolute requirement for the fake
implementation other than convenience. OTOH, removing the fake
implementation and fixing the tests that were relying on it should make
it easier to write better tests going forward and harder to hide bugs
that don't show up when using the fakes.
There may be some overhead when writing new tests that use the ring if
you're making assertions on partitions or paths, but with a part power
of zero it's normally trivially obvious when a 1 needs to be a 0 or vice
versa. Or you can just drop the assertions around the parts you were
faking anyway.
Change-Id: I8bfc388a04eff6491038991cdfd7686c9d961545
Add comment to explain why we invoke the get_part method even if the
_part_shift is equal to 32.
Closes-Bug: #1335581
Change-Id: I160e9383b5e65f75ed5e89511cc7e63c51958a25
Timeout isn't an Exception, so Timeouts in tests weren't getting
raised. Instead, you'd sometimes have an HTTPResponse's .status be a
Timeout object, not an integer, which greatly confuses code that
expects an integer.
Also reorder the test that exposed the failure in the gate so it blows
up most times instead of sometimes do demonstrate the failure with out
this fix to FakeConn.
Change-Id: I76367a0575f84cad6b2f03e814f3f16bf96bc7d1
It's particularly interesting on writes (PUT, POST, DELETE) where the
current on-disk timestamp would prevent the object server from serving
the incoming request and returns 409 Conflict.
The FakeConn has also been updated to respond in kind for 409's on
expect and all responses generaly just cause it's good to keep fakes in
line with the reals - not that I expected any existing tests to break
because of the new headers.
Change-Id: Iac6fbd2f872a9521bb2db84a333365b69f54fb6c
In the proxy, container_info can return a 'storage_policy' of None. When
you set a header value on a swob.Request to None that effectively just
delete's the key. One path through the proxy during container sync was
counting on the the 'X-Backend-Storage-Policy-Index' being set which isn't
the case if the cached container_info if for a pre-policies container.
Also clean up some test cruft, tighten up the interface on FakeConn, and add
some object controller tests to exercise more interesting failure and handoff
code paths.
Change-Id: Ic379fa62634c226cc8a5a4c049b154dad70696b3
Discovered some tests that were coupling the code under test with the
storage policies configured in /etc/swift/swift.conf. There was some
tests that created fake rings in their tempdirs, but didn't reset or
patch the POLICIES global. So if your local config needed more rings
that the fake's were setting up (just 2) the tests would puke when they
loaded up an app that looked for rings. I think this probably started
happening when we added eager object ring loading back into the proxy.
* two TestCases in test_wsgi were missing @patch_policies
* fixed issue with patch_policies that could cause state to bleed
between tests
* patch_policies' legacy and default collections get a FakeRing by
default
* drive-by cleanup for test_loadapp_proxy() ring serialized path
handling
* drive-by cleanup for test_internal_client that was doing basically
the same thing as test_wsgi
Change-Id: Ia706000ba961ed24f2c22b81041e53a0c3f302fc
This daemon will take objects that are in the wrong storage policy and
move them to the right ones, or delete requests that went to the wrong
storage policy and apply them to the right ones. It operates on a
queue similar to the object-expirer's queue.
Discovering that the object is in the wrong policy will be done in
subsequent commits by the container replicator; this is the daemon
that handles them once they happen.
Like the object expirer, you only need to run one of these per cluster
see etc/container-reconciler.conf.
DocImpact
Implements: blueprint storage-policies
Change-Id: I5ea62eb77ddcbc7cfebf903429f2ee4c098771c9
Containers now have a storage policy index associated with them,
stored in the container_stat table. This index is only settable at
container creation time (PUT request), and cannot be changed without
deleting and recreating the container. This is because a container's
policy index will apply to all its objects, so changing a container's
policy index would require moving large amounts of object data
around. If a user wants to change the policy for data in a container,
they must create a new container with the desired policy and move the
data over.
Keep status_changed_at up-to-date with status changes.
In particular during container recreation and replication.
When a container-server receives a PUT for a deleted database an extra UPDATE
is issued against the container_stat table to notate the x-timestamp of the
request.
During replication if merge_timestamps causes a container's status to change
(from DELETED to ACTIVE or vice-versa) the status_changed_at field is set to
the current time.
Accurate reporting of status_changed_at is useful for container replication
forensics and allows resolution of "set on create" attributes like the
upcoming storage_policy_index.
Expose Backend container info on deleted containers.
Include basic container info in backend headers on 404 responses from the
container server. Default empty values are used as placeholders if the
database does not exist.
Specifically the X-Backend-Status-Changed-At, X-Backend-DELETE-Timestamp and
the X-Backend-Storage-Policy-Index value will be needed by the reconciler to
deal with reconciling out of order object writes in the face of recently
deleted containers.
* Add "status_changed_at" key to the response from ContainerBroker.get_info.
* Add "Status Timestamp" field to swift.cli.info.print_db_info_metadata.
* Add "status_changed_at" key to the response from AccountBroker.get_info.
DocImpact
Implements: blueprint storage-policies
Change-Id: Ie6d388f067f5b096b0f96faef151120ba23c8748
FakeLogger gets better log level handling
Parameterize logger on some daemons which were previously
unparameterized and try and use the interface in tests.
FakeRing use more real code
The existing FakeRing mock's implementation bit me on some pretty subtle
character encoding issue by-passing the hash_path code that is normally
part of get_part_nodes. This change tries to exercise more of the real
ring code paths when it makes sense and provide a better Fake for use in
testing.
Add write_fake_ring helper to test.unit for when you need a real ring.
DocImpact
Implements: blueprint storage-policies
Change-Id: Id2e3740b1dd569050f4e083617e7dd6a4249027e
The basic idea here is to replace the use of a single object ring in
the Application class with a collection of object rings. The
collection includes not only the Ring object itself but the policy
name associated with it, the filename for the .gz and any other
metadata associated with the policy that may be needed. When
containers are created, a policy (thus a specific obj ring) is
selected allowing apps to specify policy at container creation time
and leverage policies simply by using different containers for object
operations.
The policy collection is based off of info in the swift.conf file.
The format of the sections in the .conf file is as follows:
swift.conf format:
[storage-policy:0]
name = chicken
[storage-policy:1]
name = turkey
default = yes
With the above format:
- Policy 0 will always be used for access to existing containers
without the policy specified. The ring name for policy 0 is always
'object', assuring backwards compatiblity. The parser will always
create a policy 0 even if not specified
- The policy with 'default=yes' is the one used for new container
creation. This allows the admin to specify which policy is used without
forcing the application to add the metadata.
This commit simply introduces storage policies and the loading
thereof; nobody's using it yet. That will follow in subsequent
commits.
Expose storage policies in /info
DocImpact
Implements: blueprint storage-policies
Change-Id: Ica05f41ecf3adb3648cc9182f11f1c8c5c678985
A common pattern that we see clients do is send a HEAD request before a
PUT to see if it exists. This can slow things down quite a bit
especially since 404s on HEAD are currently a bit expensive.
This change will allow a client to include a "If-None-Match: *" header
with a PUT request. In combination with "Expect: 100-Continue" this
allows the server to return that it already has a copy of the object
before any data is sent.
I attempted to also include etag support with the If-None-Match header,
but that turned up having too many hairy edge cases, so was left as a
future excercise.
DocImpact
Change-Id: I94e3754923dbe5faba065719c7a9afa9969652dd
It seemed like some of the tests ment to exercise the proxy's timeout could
would occastionally race and the backend mock would be able to cough up a
chunk before the timeout fired in the proxy - typically resulting in a failed
assertion like " lalala" != 'lalala'.
We paramaterize the timeout value, and bump it up for those cases where we are
expecting to hit the timeout. Timing shows this change is just as fast for
passing tests. And if you want to break the node_timeout in
proxy.controller.base you can verify the tests are still just as effective -
if a bit slower to detect the failure path.
Fixes bug #1272509
Change-Id: Iaf91d9d551e94fc317a08e8c0ee02daeed331b60
The proxy.controllers.base's generate_request_headers will set an X-Timestamp
header for you if it didn't get populated by additional kwarg or the
transfer_headers method. This works fine if you only call it once per
request, but because of how proxy.controllers.obj and
proxy.controllers.container fill in the backend update header chains in
_backend_requests we need multiple independent copies and call the base
controllers generate_request_headers once of each backend request - which left
the ContainerController sending down different X-Timestamp values
(microseconds apart) for PUT and DELETE.
The ObjectController skirts the issue entirely because it always preloads a
X-Timestamp on the req used to generate backend headers, and it allows it to
be copied over via transfer_headers by including 'x-timestamp' in it's
pass_through_headers attribute.
Because the container-replicator is already does merge_timestamps the
differences would always eventaully even out and there is no consistency bug,
but this seems cleaner since they put_timestamp being stored on the three
replicas during a container PUT were all coming from the same client request.
Since both PUT and DELETE were effected, and the ContainerController doesn't
need to allow X-Timestamp to pass_through like the ObjectController does for
container-sync, it seemed cleanest to fix the issue in _backend_requests via
the additional kwarg to generate_request_headers.
There's a driveby fix for FakeLogger and update to the proxy_server's
ContainerController tests.
Change-Id: Idbdf1204da33f8fb356ae35961dbdc931b228b77
The "not" in front of the ismount() call got accidentally dropped in a
recent change. This patch adds it back along with a few more tests.
Note that this bug only showed up on an SAIO during probe tests because
I used actually-mounted (virtual) "disks". So keep that in mind when
building SAIOs for development/testing.
Change-Id: Ia193f3c4b73203605954036863575c22ddab6b03
If a source times out on read try another one of them with a
modified range. There had to be a lot of moved around code
to get this working but it should all make sense.
Change-Id: Ieaf045690a8823927a6f38098a95b37a4d4adb70
We also fix up any other pep8 failures that snuck in from merges along
the way.
Change-Id: I4ea984780ac2eac458c98fe181684eef4e04beaf
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Address all the "hacking" lines that are flagged, and all the modules
that just have one item flagged.
Change-Id: I372a4bdf9c7748f73e38c4fd55e5954f1afade5b
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Add a 'get_lines_for_level' method to FakeLogger in order to actually
validate the log line format string is valid for the given args.
Change-Id: Ib2559ddf5c9d89e17c6ac08d26cc46a4193a2df2
We currently have two methods for cleaning up object directories - unlinkold in
the object server and hash_suffix in the replicator.
This modifies the object server to reuse the more complete hash_suffix logic.
Change-Id: Id2fd83caad979bfd446d52a775a1fa373a670cf1
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Currently clients can not distinguish between non existing accounts
(which can be created) and accounts marked for deletion, which has
not yet been reaped and therefore cannot be re-created until reaped.
Following this patch, if an account is marked as deleted but hasn't
been reaped and is still on disk, responses will include a status
header:
'X-Account-Status' = 'Deleted'
Fixes:Bug #1188609
Change-Id: Ibd39965ae3f5d45fd78f130e0e31f5a0141a8633
The proxy can now be configured to prefer local object servers for PUT
requests, where "local" is governed by the "write_affinity". The
"write_affinity_node_count" setting controls how many local object
servers to try before giving up and going on to remote ones.
I chose to simply re-order the object servers instead of filtering out
nonlocal ones so that, if all of the local ones are down, clients can
still get successful responses (just slower).
The goal is to trade availability for throughput. By writing to local
object servers across fast LAN links, clients get better throughput
than if the object servers were far away over slow WAN links. The
downside, of course, is that data availability (not durability) may
suffer when drives fail.
The default configuration has no write affinity in it, so the default
behavior is unchanged.
Added some words about these settings to the admin guide.
DocImpact
Change-Id: I09a0bd00524544ff627a3bccdcdc48f40720a86e
Consolidate the different ways in which info of account/container
is gathered, cached, used, updated, etc.
This refactoring increases code reuse and is a basis for later
addition of account ACLs.
Changing the get_info users is left for future.
This staged approach ensures the behaviour is unchanged.
Change-Id: I67b58030d3f9e3bc86bcd7ece0f1dc693c4e08c3
Fixes: Bug #1162199
If you manage to break a proxy server so badly that it can't even
return headers, then in test/unit/proxy/test_server.py,
readuntil2crlfs() will spin forever, killing your laptop battery and
providing zero help figuring out what's wrong.
This test-only change makes it so that, should readuntil2crlfs() run
out of data before getting its two CRLFs, it raises an exception that
tells you what it did manage to get, and does so in a finite amount of
time.
Change-Id: Ieacd18ce7f4d35a4960100d5fc3a0f910cb406ac
Current codes will copy metadata headers when x-fresh-metadata:false, we
still need copy "x-delete-at" header and ensure expiring work at the same
time.
Change-Id: Ie31326b5f7b565e51e5aa249279bc1786f7bc847
Fixes: bug #1067528
