1999 Commits

Author SHA1 Message Date
Jenkins
dcaaa3d080 Merge "EC: don't 503 on marginally-successful PUT" 2015-05-26 23:16:28 +00:00
Samuel Merritt
666bf06c26 EC: don't 503 on marginally-successful PUT
On EC PUT in an M+K scheme, we require M+1 fragment archives to
durably land on disk. If we get that, then we go ahead and ask the
object servers to "commit" the object by writing out .durable
files. We only require 2 of those.

When we got exactly M+1 fragment archives on disk, and then one
connection timed out while writing .durable files, we should still be
okay (provided M is at least 3). However, we'd take our M > 2
remaining successful responses and pass that off to best_response()
with a quorum size of M+1, thus getting a 503 even though everything
worked well enough.

Now we pass 2 to best_response() to avoid that false negative.

There was also a spot where we were getting the quorum size wrong. If
we wrote out 3 fragment archives for a 2+1 policy, we were only
requiring 2 successful backend PUTs. That's wrong; the right number is
3, which is what the policy's .quorum() method says. There was a spot
where the right number wasn't getting plumbed through, but it is now.

Change-Id: Ic658a199e952558db329268f4d7b4009f47c6d03
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Closes-Bug: 1452468
2015-05-26 11:38:57 -07:00
Jenkins
41a210a8e5 Merge "Cleanup and extend end to end ssync tests" 2015-05-26 17:11:58 +00:00
Jenkins
2ea8bae389 Merge "Allow rsync to use compression" 2015-05-13 10:58:08 +00:00
Alistair Coles
98b725fec6 Cleanup and extend end to end ssync tests
Extends the existing end to end ssync tests with
a test using replication policy.

Also some cleanup and improvements to the test framework e.g. rather
than faking the connection between sender and receiver, use a real
connection and wrap it to capture traffic for verification.

Change-Id: Id71d2eb3fb8fa15c016ef151aacf95f97196a902
2015-05-13 11:05:13 +01:00
Samuel Merritt
29f4393d88 Remove workaround for old eventlet version
Swift now requires eventlet >= 0.16.1, so we can get rid of this
workaround for a bug in eventlet 0.9.16.

Change-Id: I4a1200b9bd9266896a704a840fda0d1b720bc86d
2015-05-08 17:48:48 -07:00
Jenkins
0997d2a506 Merge "Bump up a timeout in a test" 2015-05-08 20:43:19 +00:00
Jenkins
edb68ae9df Merge "Fix account replication during pre-storage-policy upgrade" 2015-05-06 10:17:42 +00:00
Jenkins
51565ab2fd Merge "Bulk upload: treat user xattrs as object metadata" 2015-05-05 00:03:42 +00:00
Samuel Merritt
94215049fd Bump up a timeout in a test
Got a slow crappy VM like I do? You might see this fail
occasionally. Bump up the timeout a little to help it out.

Change-Id: I8c0e5b99012830ea3525fa55b0811268db3da2a2
2015-05-04 15:08:51 -07:00
Jenkins
a042b8371e Merge "Fix intermittent container replicator test failure" 2015-05-04 20:48:06 +00:00
Jenkins
65f28ae105 Merge "Fix incorrect passing of file object to fsync()" 2015-05-01 17:06:48 +00:00
Jenkins
12e406a14e Merge "Fix tempauth acl checks when simplejson has no speedups" 2015-04-30 10:12:10 +00:00
Jenkins
d45b575534 Merge "Make the reaper use same timestamp for replica deletes" 2015-04-29 19:07:59 +00:00
Prashanth Pai
03536dbb55 Fix incorrect passing of file object to fsync()
swift.common.utils.fsync() requires a file descriptor as argument
but file object handle was being passed.

Change-Id: I316b58f6bc37de0945eff551e4e50565653664f5
Signed-off-by: Prashanth Pai <ppai@redhat.com>
2015-04-29 12:19:40 +05:30
Alistair Coles
bfbc94c3cb Fix intermittent container replicator test failure
Intermittent failure of this test could be due to
insufficient time elapsing between either the local
and remote db's being created or during the
debug_timing calls. This patch forces greater timestamp
separation and forces debug_timing to always log timings.

Also add message to the failign assertion so if this does
fail again we get some clue as to why.

Closes-Bug: 1369663

Change-Id: I4b69b2e759d586a14abd0931a68dbdf256d57c32
2015-04-28 10:53:05 +01:00
Clay Gerrard
8cdf0fdebe Fix account replication during pre-storage-policy upgrade
Old account schemas don't send the storage_policy_index key for container rows
during replication, and if the recieving end is already running an upgraded
server it is surprised with a KeyError.  Normally this would work itself out
if the old schema recieved any updates from container layer, or a new
container is created, or requires a row sync from another account database -
but if the account databases have rows out of sync and there's no activity in
the account otherwise, there's nothing to force the old schemas to be
upgraded.

Rather than force the old schema that already has a complete set of container
rows to migrate even in the absense of activity we can just fill in default
legacy value for the storage policy index and allow the accounts to get back
in sync and migrate the next time a container update occurs.

FWIW, I never able to get a cluster upgrade to get stuck in this state without
some sort of account failure that forced them to get their rows out of sync
(in my cause I just unlinked a pending and then made sure to force all my
account datbases to commit pending files before upgrading - leading to an
upgraded cluster that absolutly needed account-replication to solve a row
mismatch for inactive accounts with old schemas)

Closes-Bug #1424108

Change-Id: Iaf4ef834eb24f0e11a52cc22b93a864574fabf83
2015-04-27 14:01:32 -07:00
Jenkins
f2d4851b17 Merge "Make RingBuilders deep-copy-able" 2015-04-24 04:38:21 +00:00
Jenkins
69d2152301 Merge "Fix conflict SLO reponse" 2015-04-24 01:16:53 +00:00
Samuel Merritt
43ace3c628 Make RingBuilders deep-copy-able
We used to be able to deep-copy RingBuilder objects, but the addition
of debug logging (8d3b3b2) broke that since you can't deep-copy a
Python logger. This commit fixes that.

Swift doesn't really deep-copy RingBuilders anywhere, but third-party
code might.

Change-Id: If8bdadd93d9980db3d8a093f32d76ca604de9301
2015-04-22 15:35:32 -07:00
Samuel Merritt
215cd551df Bulk upload: treat user xattrs as object metadata
Currently, if you PUT a single object, then you can also associate
metadata with it by putting it in the request headers, prefixed with
"X-Object-Meta". However, if you're bulk-uploading objects, then you
have no way to assign any metadata.

The tar file format* allows for arbitrary UTF-8 key/value pairs to be
associated with each file in an archive (as well as with the archive
itself, but we don't care about that here). If a file has extended
attributes, then tar will store those as key/value pairs.

This commit makes bulk upload read those extended attributes, if
present, and convert those to Swift object metadata. Attributes
starting with "user.meta" are converted to object metadata, and
"user.mime_type"** is converted to Content-Type.

For example, if you have a file "setup.py":

    $ setfattr -n user.mime_type -v "application/python-setup" setup.py
    $ setfattr -n user.meta.lunch -v "burger and fries" setup.py
    $ setfattr -n user.meta.dinner -v "baked ziti" setup.py
    $ setfattr -n user.stuff -v "whee" setup.py

This will get translated to headers:

    Content-Type: application/python-setup
    X-Object-Meta-Lunch: burger and fries
    X-Object-Meta-Dinner: baked ziti

Swift will handle xattrs stored by both GNU and BSD tar***. Only
xattrs user.mime_type and user.meta.* are processed; others are
ignored.

This brings bulk upload much closer to feature-parity with non-bulk upload.

* The POSIX 1003.1-2001 (pax) format, at least. There are a few
  different, mutually-incompatible tar formats out there, because of
  course there are. This is the default format on GNU tar 1.27.1 or
  later.

** http://standards.freedesktop.org/shared-mime-info-spec/latest/ar01s02.html#idm140622087713936

*** Even with pax-format tarballs, different encoders store xattrs
    slightly differently; for example, GNU tar stores the xattr
    "user.rubberducky" as pax header "SCHILY.xattr.user.rubberducky",
    while BSD tar (which uses libarchive) stores it as
    "LIBARCHIVE.xattr.user.rubberducky". One might wonder if this is
    some programmer's attempt at job security.

Change-Id: I5e3ce87d31054f5239e86d47c45adbde2bb93640
2015-04-21 18:42:32 -07:00
Alistair Coles
2080f7dbd8 Fix tempauth acl checks when simplejson has no speedups
As documented in linked bug report, tempauth unit tests
were seen to fail on a system where simplejson was
installed but without the speedups extension. This
is because the tempauth account acl validation checks
that values are type str, but without the speedups
extension the json parser is returning unicode objects.

Fix is to have the acl validator tolerate those objects
being unicode or str.

Also change common/bufferedhttp.py to coerce ring device
to type str when constructing a path, in order to avoid
a UnicodeDecodeError when httplib sends a message that
has non-ascii header values.

Change-Id: I01524282cbaa25dc4b6dfa09f3f4723516cdba99
Closes-Bug: 1425776
2015-04-21 14:13:25 +01:00
Kota Tsuyuzaki
27f6fba5c3 Use reconstruct insetad of decode/encode
With bumping PyECLib up to 1.0.7 on global requirements,
we can use the "reconstruct" function directly instead
of the current hack doing decode/encode on reconstructor.
That is because the hack was for treating PyECLib < 1.0.7
(strictly jearsure scheme) reconstruction bug so we don't
have to do decode/encode anymore.

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: I69aae495670e3d0bdebe665f73915547a4d56f99
2015-04-20 16:48:10 -07:00
Jenkins
ad66801915 Merge "More user-friendly output for object metadata" 2015-04-16 22:15:05 +00:00
Jenkins
f90fe4fb3d Merge "Fix best response to return correct status" 2015-04-16 19:22:22 +00:00
Kazuhiro MIYAHARA
e4d326b5a7 Fix conflict SLO reponse
This patch fixes Swift to respond "409 Conflict"
when a segment object path of the manifest on PUT SLO
is same as requested object path. It is because
the request will overwrite the segment and then it
will absolutely cause "409 Conflict" on GET SLO.

e.g.:

request:
PUT "http://hostname/v1/AUTH_account/container/segment_object_00?multipart-manifest=put"

manifest file:
[{"path" : "container/segment_object_00", "etag" : "<etag of segment_object_00>", "size_bytes" : <size of segment_object_00>},
{"path" : "container/segment_object_01", "etag" : "<etag of segment_object_01>", "size_bytes" : <size of segment_object_01>},
{"path" : "container/segment_object_02", "etag" : "<etag of segment_object_02>", "size_bytes" : <size of segment_object_02>}]

Change-Id: I4f4f7b9dbeb6a7c355b801c7e0ae560aa19a70b4
Closes-Bug: 1417936
2015-04-16 11:41:55 -07:00
Alistair Coles
38ae7bb89c Make the reaper use same timestamp for replica deletes
The account reaper is using a unique timestamp when deleting
replicas of the same resource. This will result in unnecessary
replication traffic after reaping.

This patch makes the reaper use a single timestamp per resource.

Probe test is modified to check that delete times are equal
across replicas before replicators run.

test_direct_client.py is modified to check that it uses explicit
timestamp when passed to direct_delete_[object|container]
methods.

Drive-by bug fixes in the probe test e.g. it was not sending
X-Backend-Storage-Policy-Index when doing a direct GET to check
object state, so the 404s being verified could in fact be due
to diskfile not existing rather than diskfile being deleted.

Closes-Bug: 1442879
Change-Id: I8bab22d66308bb9d3294e1e0def017c784228423
2015-04-16 14:34:02 +01:00
Kota Tsuyuzaki
e440d6aed8 Fix best response to return correct status
Current best response could return "503 Internal Server Error".
However, "503" means "Service Unavailable". (The status int of
Internal Server Error is 500)

This patch fixes the response status as "503 Service Unavailable"

Change-Id: I88b8c52c26b19e9e76ba3375f1e16ced555ed54c
2015-04-16 12:55:40 +00:00
Ricardo Ferreira
57011d5699 More user-friendly output for object metadata
Split out system, user and other metadata in swift-object-info. Print
every position line by line instead of raw dict representation, so it
would be easier to parse with tools such as grep.

Co-Authored-By: Ricardo Ferreira <ricardo.sff@gmail.com>
Co-Authored-By: Kamil Rykowski <kamil.rykowski@intel.com>

Change-Id: Ia78da518c18f7e26016700aee87efb534fbd2040
Closes-Bug: #1428866
2015-04-16 09:20:14 +02:00
Clay Gerrard
52b102163e Don't apply the wrong Etag validation to rebuilt fragments
Because of the object-server's interaction with ssync sender's
X-Backend-Replication-Headers when a object (or fragment archive) is
pushed unmodified to another node it's ETag value is duped into the
recieving ends metadata as Etag.  This interacts poorly with the
reconstructor's RebuildingECDiskFileStream which can not know ahead of
time the ETag of the fragment archive being rebuilt.

Don't send the Etag from the local source fragment archive being used as
the basis for the rebuilt fragent archive's metadata along to ssync.

Change-Id: Ie59ad93a67a7f439c9a84cd9cff31540f97f334a
2015-04-15 23:33:32 +01:00
Jenkins
4223c58d71 Merge "Fix untranslatable string" 2015-04-15 12:47:58 +00:00
Christian Schwede
663ccd4e7a More tests for swift recon
Change-Id: I8d568c0f6fbe1c01d97491740aebf299deb63732
2015-04-15 06:36:06 +00:00
Clay Gerrard
5ef2e9ec00 fixup random test failure
Change-Id: I053cec441623e7556d6cdfbc1d271c5cdf21aa8b
2015-04-14 16:06:24 -07:00
Clay Gerrard
4aba2fbb25 Check if REST API version is valid
Swift doesn't check if the used API version is valid. Currently there
is only one valid REST API version, but that might change in the
future.

This patch enforces "v1" or "v1.0" as the version string when accessing
account, containers and objects.

The list of accepted version strings can be manually overridden using a
comma-separated list in swift.conf to make this backward-compatible.
The constraint loader has been modified slightly to accept strings as
well as integers.

Any request to an account, container, and object which does not
provide the correct version string will get a 400 BadRequest response.

The allowed api versions are by default excluded from /info.

Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: John Dickinson <me@not.mn>

Closes Bug #1437442

Change-Id: I5ab6e236544378abf2eab562ab759513d09bc256
2015-04-14 16:00:37 -07:00
Christian Schwede
e16df14a73 Add test for swift_recon.disk_usage
Change-Id: I4cab7aa6df3f0e1933e52ee5dbbb829f30604f10
2015-04-14 16:00:37 -07:00
Mike Fedosin
b76d8c5156 Fix broken test_policy_IO_override test
Since the existing check depends on how quickly the test
passes, it causes to the unpredictable behavior and may break
the test.

This commit removes the unnecessary dependency on the
current time and leaves only the check that timestamp is 
greater than 0.

Change-Id: I3321e9aba0df52cb78523710b26f310f35801674
Closes-Bug: #1441599
2015-04-14 16:00:37 -07:00
Andreas Jaeger
26bff69cb5 Fix untranslatable string
The word "number" is not marked for translation, we cannot just add
it this way to a string.
Instead create two separate strings.

Change-Id: I0396f1d7405987ed1c57f0344e1c5ebeca30a2e3
Closes-Bug: #1444327
2015-04-14 21:55:47 +01:00
Hisashi Osanai
749bdac1df Refactor the getting roles logic in _keystone_identity
The list_from_csv in swift.common.utils is used from the composite
auth token support and this method has been used in
_integral_keystone_identity. There is same logic in
_keystone_identity so the logic would be replaced with list_from_csv.

Change-Id: I3d72a34e6fc21fbe1d7331954695b1e3e1b67816
2015-04-14 13:15:15 -07:00
Donagh McCabe
8a1453e61e Support HTTP_X_SERVICE_IDENTITY_STATUS in keystoneauth
Keystone's auth_token now supports delay denial when
a service token is invalid. This change handles
the consequences for determining 401 vs 403 responses.

Related-Bug: #1422389

Change-Id: I90ba062e357cccea061c7101463b82b7c5e69293
2015-04-14 13:15:15 -07:00
Christian Schwede
d3213fb1fe Check if device name is valid when adding to the ring
Currently device names can be empty or start and/or end with spaces.
This can create unexpected results, for example these three commands
are all valid:

swift-ring-builder account.builder add "r1z1-127.0.0.1:6000/" 1
swift-ring-builder account.builder add "r1z1-127.0.0.1:6000/sda " 1
swift-ring-builder account.builder add "r1z1-127.0.0.1:6000/ meta" 1

This patch validates device names and prevents empty names or names
starting and/or ending with spaces.

Also fixed the test "test_warn_at_risk" - the test passed if the
exception was not raised.

Closes-Bug: 1438579

Change-Id: I811b0eae7db503279e6429d985275bbab8b29c9f
2015-04-14 13:15:15 -07:00
John Dickinson
e910f7e07d Merge EC feature into master
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I002787f558781bd4d884129b127bc9f108ea9ec4
2015-04-14 08:57:15 -07:00
paul luse
647b66a2ce Erasure Code Reconstructor
This patch adds the erasure code reconstructor. It follows the
design of the replicator but:
  - There is no notion of update() or update_deleted().
  - There is a single job processor
  - Jobs are processed partition by partition.
  - At the end of processing a rebalanced or handoff partition, the
    reconstructor will remove successfully reverted objects if any.

And various ssync changes such as the addition of reconstruct_fa()
function called from ssync_sender which performs the actual
reconstruction while sending the object to the receiver

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
blueprint ec-reconstructor
Change-Id: I7d15620dc66ee646b223bb9fff700796cd6bef51
2015-04-14 00:52:17 -07:00
Clay Gerrard
b2189ef47a Fix account reaper for > 3 replicas
There's a pre-existing IndexError in the pop from the container node
list in reaper's reap_object method for object rings with a replica
count greater than the container replica count.  Which is more likely
on EC storage policies.

When making the backend direct delete requests to the nodes once the
container node's list is exhausted the generic exception handler logs
the error and breaks out of any other backend object requests - but
the reaper marches forward and eventually the tombstones are
replicated.

This change just cycles the container headers across all the nodes -
which seems reasonable enough - but could certainly garner
bikeshedding.

Change-Id: I5897d00b0a8c1e05884945dd93d9ce891b207001
2015-04-14 00:52:17 -07:00
Yuan Zhou
61a9d35fd5 Update container sync to use internal client
This patch changes container sync to use Internal Client instead
of Direct Client.

In the current design, container sync uses direct_get_object to
get the newest source object(which talks to storage node directly).
This works fine for replication storage policies however in
erasure coding policies, direct_get_object would only return part
of the object(it's encoded as several pieces). Using Internal
Client can get the original object in EC case.

Note that for the container sync put/delete part, it's working in
EC since it's using Simple Client.

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

DocImpact
Change-Id: I91952bc9337f354ce6024bf8392046a1ecf6ecc9
2015-04-14 00:52:17 -07:00
Samuel Merritt
decbcd24d4 Foundational support for PUT and GET of erasure-coded objects
This commit makes it possible to PUT an object into Swift and have it
stored using erasure coding instead of replication, and also to GET
the object back from Swift at a later time.

This works by splitting the incoming object into a number of segments,
erasure-coding each segment in turn to get fragments, then
concatenating the fragments into fragment archives. Segments are 1 MiB
in size, except the last, which is between 1 B and 1 MiB.

+====================================================================+
|                             object data                            |
+====================================================================+

                                   |
          +------------------------+----------------------+
          |                        |                      |
          v                        v                      v

+===================+    +===================+         +==============+
|     segment 1     |    |     segment 2     |   ...   |   segment N  |
+===================+    +===================+         +==============+

          |                       |
          |                       |
          v                       v

     /=========\             /=========\
     | pyeclib |             | pyeclib |         ...
     \=========/             \=========/

          |                       |
          |                       |
          +--> fragment A-1       +--> fragment A-2
          |                       |
          |                       |
          |                       |
          |                       |
          |                       |
          +--> fragment B-1       +--> fragment B-2
          |                       |
          |                       |
         ...                     ...

Then, object server A gets the concatenation of fragment A-1, A-2,
..., A-N, so its .data file looks like this (called a "fragment archive"):

+=====================================================================+
|     fragment A-1     |     fragment A-2     |  ...  |  fragment A-N |
+=====================================================================+

Since this means that the object server never sees the object data as
the client sent it, we have to do a few things to ensure data
integrity.

First, the proxy has to check the Etag if the client provided it; the
object server can't do it since the object server doesn't see the raw
data.

Second, if the client does not provide an Etag, the proxy computes it
and uses the MIME-PUT mechanism to provide it to the object servers
after the object body. Otherwise, the object would not have an Etag at
all.

Third, the proxy computes the MD5 of each fragment archive and sends
it to the object server using the MIME-PUT mechanism. With replicated
objects, the proxy checks that the Etags from all the object servers
match, and if they don't, returns a 500 to the client. This mitigates
the risk of data corruption in one of the proxy --> object connections,
and signals to the client when it happens. With EC objects, we can't
use that same mechanism, so we must send the checksum with each
fragment archive to get comparable protection.

On the GET path, the inverse happens: the proxy connects to a bunch of
object servers (M of them, for an M+K scheme), reads one fragment at a
time from each fragment archive, decodes those fragments into a
segment, and serves the segment to the client.

When an object server dies partway through a GET response, any
partially-fetched fragment is discarded, the resumption point is wound
back to the nearest fragment boundary, and the GET is retried with the
next object server.

GET requests for a single byterange work; GET requests for multiple
byteranges do not.

There are a number of things _not_ included in this commit. Some of
them are listed here:

 * multi-range GET

 * deferred cleanup of old .data files

 * durability (daemon to reconstruct missing archives)

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I9c13c03616489f8eab7dcd7c5f21237ed4cb6fd2
2015-04-14 00:52:17 -07:00
Samuel Merritt
b1eda4aef8 Allow sending object metadata after data
This lets the proxy server send object metadata to the object server
after the object data. This is necessary for EC, as it allows us to
compute the etag of the object in the proxy server and still store it
with the object.

The wire format is a multipart MIME document. For sanity during a
rolling upgrade, the multipart MIME document is only sent to the
object server if it indicates, via 100 Continue header, that it knows
how to consume it.

Example 1 (new proxy, new obj server):

   proxy: PUT /p/a/c/o
          X-Backend-Obj-Metadata-Footer: yes

     obj: 100 Continue
        X-Obj-Metadata-Footer: yes

   proxy: --MIMEmimeMIMEmime...

Example2: (new proxy, old obj server)

   proxy: PUT /p/a/c/o
          X-Backend-Obj-Metadata-Footer: yes

     obj: 100 Continue

   proxy: <obj body>

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: Id38f7e93e3473f19ff88123ae0501000ed9b2e89
2015-04-14 00:52:17 -07:00
Alistair Coles
fa89064933 Per-policy DiskFile classes
Adds specific disk file classes for EC policy types.

The new ECDiskFile and ECDiskFileWriter classes are used by the
ECDiskFileManager.

ECDiskFileManager is registered with the DiskFileRouter for use with
EC_POLICY type policies.

Refactors diskfile tests into BaseDiskFileMixin and BaseDiskFileManagerMixin
classes which are then extended in subclasses for the legacy
replication-type DiskFile* and ECDiskFile* classes.

Refactor to prefer use of a policy instance reference over a policy_index
int to refer to a policy.

Add additional verification to DiskFileManager.get_dev_path to validate the
device root with common.constraints.check_dir, even when mount_check is
disabled for use in on a virtual swift-all-in-one.

Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: John Dickinson <me@not.mn>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I22f915160dc67a9e18f4738c1ddf068344e8ad5d
2015-04-14 00:52:16 -07:00
John Dickinson
dd9d97458e Prevent unauthorized delete in versioned container
An authenticated user can delete the most recent version of any
versioned object who's name is known if the user has listing access
to the x-versions-location container. Only Swift setups with
allow_version setting are affected.

This patch closes this bug, tracked as CVE-2015-1856

Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Christian Schwede <info@cschwede.de>
Co-Authored-By: Alistair Coles <alistair.coles@hp.com>

Closes-Bug: 1430645
Change-Id: Ibacc7413afe7cb6f77d92e5941dcfdf4768ffa18
2015-04-13 23:34:10 -07:00
Clay Gerrard
a707829334 Update test infrastructure
* Get FakeConn ready for expect 100 continue
 * Use debug_logger more and with better interfaces
 * Fix patch_policies to be less annoying

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Tushar Gohad <tushar.gohad@intel.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: I28c0a3539d994cbb8e6b94d63a23ed4ea6cb956d
2015-04-13 22:57:42 -07:00
Tushar Gohad
ed54066288 Add support for policy types, 'erasure_coding' policy
This patch extends the StoragePolicy class for non-replication storage
policies, the first one being "erasure coding".

Changes:

 - Add 'policy_type' support to BaseStoragePolicy class
 - Disallow direct instantiation of BaseStoragePolicy class

 - Subclass BaseStoragePolicy

   - "StoragePolicy":
     . Replication policy, default
     . policy_type = 'replication'

   - "ECStoragePolicy":
     . Erasure Coding policy
     . policy_type = 'erasure_coding'
     . Private member variables
       ec_type (EC backend),
       ec_num_data_fragments (number of fragments original
         data split into after erasure coding operation),
       ec_num_parity_fragments (number of parity fragments
       generated during erasure coding)
     . Private methods
       EC specific attributes and ring validator methods.

 - Swift will use PyECLib, a Python Erasure Coding library, for
   erasure coding operations. PyECLib is already an approved
   OpenStack core requirement.
   (https://bitbucket.org/kmgreen2/pyeclib/)

 - Add test cases for
   - 'policy_type' StoragePolicy member
   - policy_type == 'erasure_coding'

DocImpact

Co-Authored-By: Alistair Coles <alistair.coles@hp.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Christian Schwede <christian.schwede@enovance.com>
Co-Authored-By: Yuan Zhou <yuan.zhou@intel.com>
Change-Id: Ie0e09796e3ec45d3e656fb7540d0e5a5709b8386
Implements: blueprint ec-proxy-work
2015-04-13 22:57:42 -07:00