This is an alternative approach to that proposed in [1]
Adds support for optional per-policy config sections
to be added in proxy-server.conf. This is highly desirable
to allow per-policy affinity options to be set for use with
duplicated EC policies [2] and composite rings [3].
Certain options found in per-policy conf sections will
override their equivalents that may be set in the
[app:proxy-server] section. Currently the options
handled that way are:
sorting_method
read_affinity
write_affinity
write_affinity_node_count
For example:
[proxy-server:policy:0]
sorting_method = affinity
read_affinity = r1=100
write_affinity = r1
write_affinity_node_count = 1 * replicas
The corresponding attributes of the proxy-server Application
are now available from instances of an OverrideConf object
that is obtained from Application.get_policy_options(policy).
[1] Related-Change: I9104fc789ba85ab3ab5ccd34096125b482821389
[2] Related-Change: Idd155401982a2c48110c30b480966a863f6bd305
[3] Related-Change: I0d8928b55020592f8e75321d1f7678688301d797
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Change-Id: I3f718f425f525baa80045ba067950c752bcaaefc
Recently out gate started blowing up intermittently with a strange
case of ports mixed up. Sometimes a functional tests tries to
authorize on a port that's clearly an object server port, and
the like. As it turns out, eventlet developers added an unavoidable
SO_REUSEPORT into listen(), which makes listen(("localhost",0)
to reuse ports.
There's an issue about it:
https://github.com/eventlet/eventlet/issues/411
This patch is working around the problem while eventlet people
consider the issue.
Change-Id: I67522909f96495a6a30e1acdb79835dce2189549
The proxy server on occasion has error limited a node by the time the
test runs, causing the proxie's node_iter failing to iter out this
error limited node. As the test uses a default FakeRing with no
extra handoffs, on this occasion we only get 2 requests which is not
enough for quorum, causing it to return a 503.
This patch sets the error_suppression_interval to 0 when creating
the proxy server. Meaning a node effectively isn't error_limited.
Change-Id: I96cf4c4d63594f803cc1cd57e874d1624db8e249
Closes-Bug: #1682026
The refactoring in the Related-Change separated EC
specific object controller tests into EC specific TestCase
classes, but left two EC specific tests in the Replication
object controller test class. This patch moves them to the
appropriate test class.
Previously the tests were only executed once, now they are
executed in each of two subclasses using different EC
policies. As a result it was necessary to make the test
container name unique to the policy under test.
Related-Change: Ifd3d0fa66773e640bb61cc528f7a1b2358e97d91
Change-Id: Ie712ea91b5dd74c504a0dd6aa40c3d657277108c
Due to the refactoring of TestObjectController (related-change),
all of BaseTestECObjectController test methods are not being
needed to be unpatched because they are expected to run for test
setup-ed policies.
This patch works for items as follows:
- Move part of setUp/tearDown routines at BaseTestObjectController
needed by only TestReplicatedObjectController which affects
patch_policies
- Remove all unpatch_policies from BaseTestECObjectController
- Set up self.ec_policy to avoid to set policy index and
retrieve the policy for each test method.
The reason why I didn't squash this up to the related parent patch is
to clarify what was changed at those patches. The parent is for just
clustering the tests for each test class and this one attempts to
improve.
Related-Change: Idd155401982a2c48110c30b480966a863f6bd305
Change-Id: I25a3f8fc837706d78dca226fe282d9e5ead65a0d
From the related change of ECDuplication, Swift have a couple of Test
classes for EC policy, normal EC and EC Duplication, in the
test/unit/proxy/test_server.py. To enable the classes, the related change
abstracts the EC test cases as the ECTestMixin class to gather test
methods into one place but it was worse because TestObjectController did
still have both test cases for replication and for ec that may be hard
to understand the test class structure.
Hence, this patch attempts to refactor the structure as
From:
ECTestMixin
|
-------------------------------------
| |
TestObjectController TestObjectControllerECDuplication
(for replication and EC) (for EC Duplication Policy)
To:
BaseTestObjectController
|
--------------------------------------
| |
TestReplicatedObjectController BaseTestECObjectController
(for replication) |
---------------------------------
| |
TestECObjectController TestECDuplicationObjectController
(for EC policy) (for EC Duplication Policy)
Some more cleanups are in follow up patches because this patch shows a lot
of moving code chunks which could be hard to compare the diff. To make
the review easy, this patch forcus on ONLY the structure changes as
possible.
Related-Change: Idd155401982a2c48110c30b480966a863f6bd305
Related-Change: I25a3f8fc837706d78dca226fe282d9e5ead65a0d
Change-Id: Ifd3d0fa66773e640bb61cc528f7a1b2358e97d91
Follow up for related change:
- fix typos
- use common helper methods
- refactor some tests to reduce duplicate code
Related-Change: Idd155401982a2c48110c30b480966a863f6bd305
Change-Id: I2f91a2f31e4c1b11f3d685fa8166c1a25eb87429
An operator proposing a web UX to its customers might want to allow web
browser to access some headers by default (eg: X-Storage-Policy,
X-Container-Read, ...). This commit adds a new setting to the
proxy-server to allow some headers to be added cluster-wide to the CORS
header Access-Control-Expose-Headers.
Change-Id: I5ca90a052f27c98a514a96ee2299bfa1b6d46334
This patch enables efficent PUT/GET for global distributed cluster[1].
Problem:
Erasure coding has the capability to decrease the amout of actual stored
data less then replicated model. For example, ec_k=6, ec_m=3 parameter
can be 1.5x of the original data which is smaller than 3x replicated.
However, unlike replication, erasure coding requires availability of at
least some ec_k fragments of the total ec_k + ec_m fragments to service
read (e.g. 6 of 9 in the case above). As such, if we stored the
EC object into a swift cluster on 2 geographically distributed data
centers which have the same volume of disks, it is likely the fragments
will be stored evenly (about 4 and 5) so we still need to access a
faraway data center to decode the original object. In addition, if one
of the data centers was lost in a disaster, the stored objects will be
lost forever, and we have to cry a lot. To ensure highly durable
storage, you would think of making *more* parity fragments (e.g.
ec_k=6, ec_m=10), unfortunately this causes *significant* performance
degradation due to the cost of mathmetical caluculation for erasure
coding encode/decode.
How this resolves the problem:
EC Fragment Duplication extends on the initial solution to add *more*
fragments from which to rebuild an object similar to the solution
described above. The difference is making *copies* of encoded fragments.
With experimental results[1][2], employing small ec_k and ec_m shows
enough performance to store/retrieve objects.
On PUT:
- Encode incomming object with small ec_k and ec_m <- faster!
- Make duplicated copies of the encoded fragments. The # of copies
are determined by 'ec_duplication_factor' in swift.conf
- Store all fragments in Swift Global EC Cluster
The duplicated fragments increase pressure on existing requirements
when decoding objects in service to a read request. All fragments are
stored with their X-Object-Sysmeta-Ec-Frag-Index. In this change, the
X-Object-Sysmeta-Ec-Frag-Index represents the actual fragment index
encoded by PyECLib, there *will* be duplicates. Anytime we must decode
the original object data, we must only consider the ec_k fragments as
unique according to their X-Object-Sysmeta-Ec-Frag-Index. On decode no
duplicate X-Object-Sysmeta-Ec-Frag-Index may be used when decoding an
object, duplicate X-Object-Sysmeta-Ec-Frag-Index should be expected and
avoided if possible.
On GET:
This patch inclues following changes:
- Change GET Path to sort primary nodes grouping as subsets, so that
each subset will includes unique fragments
- Change Reconstructor to be more aware of possibly duplicate fragments
For example, with this change, a policy could be configured such that
swift.conf:
ec_num_data_fragments = 2
ec_num_parity_fragments = 1
ec_duplication_factor = 2
(object ring must have 6 replicas)
At Object-Server:
node index (from object ring): 0 1 2 3 4 5 <- keep node index for
reconstruct decision
X-Object-Sysmeta-Ec-Frag-Index: 0 1 2 0 1 2 <- each object keeps actual
fragment index for
backend (PyEClib)
Additional improvements to Global EC Cluster Support will require
features such as Composite Rings, and more efficient fragment
rebalance/reconstruction.
1: http://goo.gl/IYiNPk (Swift Design Spec Repository)
2: http://goo.gl/frgj6w (Slide Share for OpenStack Summit Tokyo)
Doc-Impact
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Change-Id: Idd155401982a2c48110c30b480966a863f6bd305
If a user sends a Range header with no satisfiable ranges, we send back
a 416 Requested Range Not Satisfiable response. Previously however,
there would be no indication of the size of the object they were
requesting, so they wouldn't know how to craft a satisfiable range. We
*do* send a Content-Length, but it is (correctly) the length of the
error message.
The RFC [1] has an answer for this:
> A server generating a 416 (Range Not Satisfiable) response to a
> byte-range request SHOULD send a Content-Range header field with an
> unsatisfied-range value, as in the following example:
>
> Content-Range: bytes */1234
>
> The complete-length in a 416 response indicates the current length of
> the selected representation.
Now, we'll send a Content-Range header for all 416 responses, including
those coming from the object server as well as those generated on a
proxy because of the Range mangling required to support EC policies.
[1] RFC 7233, section 4.2, although similar language was used in RFC
2616, sections 10.4.17 and 14.16
Change-Id: I80c7390fc6f84a10a212b0641bb07a64dfccbd45
Instead of using a separate .durable file to indicate
the durable status of a .data file, rename the .data
to include a durable marker in the filename. This saves
one inode for every EC fragment archive.
An EC policy PUT will, as before, first rename a temp
file to:
<timestamp>#<frag_index>.data
but now, when the object is committed, that file will be
renamed:
<timestamp>#<frag_index>#d.data
with the '#d' suffix marking the data file as durable.
Diskfile suffix hashing returns the same result when the
new durable-data filename or the legacy durable file is
found in an object directory. A fragment archive that has
been created on an upgraded object server will therefore
appear to be in the same state, as far as the consistency
engine is concerned, as the same fragment archive created
on an older object server.
Since legacy .durable files will still exist in deployed
clusters, many of the unit tests scenarios have been
duplicated for both new durable-data filenames and legacy
durable files.
Change-Id: I6f1f62d47be0b0ac7919888c77480a636f11f607
I don't think this is a real bug - just that the mocked iter wasn't
closing it subiters like the real iter does.
Change-Id: I44c8159f9eea8737bc86b6c7eb59a512e57e86c1
As mentioned in link[1], if we need filter on python3,
Raplace filter(lambda obj: test(obj), data) with:
[obj for obj in data if test(obj)].
[1] https://wiki.openstack.org/wiki/Python3
Change-Id: Ia1ea2ec89e4beb957a4cb358b0d0cef970f23e0a
This patch improves EC GET response handling:
- The proxy no longer requires all object servers to have a
durable file for the fragment archive that they return in
response to a GET. The proxy will now be satisfied if just
one object server has a durable file at the same timestamp
as fragments from other object servers.
This means that the proxy can now successfully GET an
object that had missing durable files when it was PUT.
- The proxy will now ensure that it has a quorum of *unique*
fragment indexes from object servers before considering a
GET to be successful.
- The proxy is now able to fetch multiple fragment archives
having different indexes from the same node. This enables
the proxy to successfully GET an object that has some
fragments that have landed on the same node, for example
after a rebalance.
This new behavior is facilitated by an exchange of new
headers on a GET request and response between the proxy and
object servers.
An object server now includes with a GET (or HEAD) response:
- X-Backend-Fragments: the value of this describes all
fragment archive indexes that the server has for the
object by encoding a map of the form: timestamp -> <list
of fragment indexes>
- X-Backend-Durable-Timestamp: the value of this is the
internal form of the timestamp of the newest durable file
that was found, if any.
- X-Backend-Data-Timestamp: the value of this is the
internal form of the timestamp of the data file that was
used to construct the diskfile.
A proxy server now includes with a GET request:
- X-Backend-Fragment-Preferences: the value of this
describes the proxy's current preference with respect to
those fragments that it would have object servers
return. It encodes a list of timestamp, and for each
timestamp a list of fragment indexes that the proxy does
NOT require (because it already has them).
The presence of a X-Backend-Fragment-Preferences header
(even one with an empty list as its value) will cause the
object server to search for the most appropriate fragment
to return, disregarding the existence or not of any
durable file. The object server assumes that the proxy
knows best.
Closes-Bug: 1469094
Closes-Bug: 1484598
Change-Id: I2310981fd1c4622ff5d1a739cbcc59637ffe3fc3
Co-Authored-By: Paul Luse <paul.e.luse@intel.com>
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Linux 3.11 introduced O_TMPFILE as a flag to open() sys call. This would
enable users to get a fd to an unnamed temporary file. As it's unnamed,
it does not require the caller to devise unique names. It is also not
accessible through any path. Hence, file creation is race-free.
This file is initially unreachable. It is then populated with data(write),
metadata(fsetxattr) and fsync'd before being atomically linked into the
filesystem in a fully formed state using linkat() sys call. Only after a
successful linkat() will the object file will be available for reference.
Caveats
* Unlike os.rename(), linkat() cannot overwrite destination path if it
already exists. If path exists, we unlink and try again.
* XFS support for O_TMPFILE was only added in Linux 3.15.
* If client disconnects during object upload, although there is no
incomplete/stale file on disk, the object directory would persist
and is not cleaned up immediately.
Change-Id: I8402439fab3aba5d7af449b5e465f89332f606ec
Signed-off-by: Prashanth Pai <ppai@redhat.com>
From the (non-normative) Implementation Considerations section of
https://www.w3.org/TR/cors/#resource-implementation :
> Resources that wish to enable themselves to be shared with multiple
> Origins but do not respond uniformly with "*" must in practice
> generate the Access-Control-Allow-Origin header dynamically in
> response to every request they wish to allow. As a consequence,
> authors of such resources should send a Vary: Origin HTTP header or
> provide other appropriate control directives to prevent caching of
> such responses, which may be inaccurate if re-used across-origins.
We do the first part (dynamic Access-Control-Allow-Origin: generation
based on the incoming Origin: header), but not the second (send a
Vary: Origin header). Consider this scenario:
1. Swift user Alice has some static content that should be available
from some (but not all) other domains. She creates a new container
with an appropriate X-Container-Meta-Access-Control-Allow-Origin
like "http://foo.example.comhttp://bar.example.com".
2. End user Bob pulls up a browser and visits http://foo.example.com,
which references a cross-origin resource. Seeing this, the browser
issues a preflight request and gets back a response that includes
headers like:
Access-Control-Allow-Origin: http://foo.example.com
Access-Control-Allow-Methods: HEAD, GET, PUT, POST, COPY,
OPTIONS, DELETE
Since the preflight succeeded, the browser follows through on the
cross-origin request and everything loads properly.
3. Now Bob visits http://bar.example.com, which references the same
resource. Ordinarily, the exact same thing would happen, but with
http://bar.example.com in the headers. However, if the browser
cached the preflight response (because it didn't want to make two
requests everytime it needed a resource), it would assume the server
would only allow resource-sharing with http://foo.example.com and
not load the resource.
Similar issues arise from the dynamically-generated
Access-Control-Allow-Headers header.
For more information on the Vary: header, see
http://tools.ietf.org/html/rfc7231#section-7.1.4
Change-Id: I9950e593312f654ee596b7f43f7ab9e5b684d8e5
In the object proxy controller, the POST method checked the metadata of an
object before calling swift.authorize. This could allow an auth middleware to
set metadata that violates constraints. Instead, checking the metadata should
take place after authorization.
Change-Id: I5f05039498c406473952e78c6a40ec11e8b53f8e
Closes-Bug: #1596944
Swift EC has a strong constraint about the ring must have a number of
replicas which fits ec_k + ec_m. That is validated when servers waking
up. However, Swift has more chance to load such an invalid ring when
a request comming, calling some node iteration like get_nodes,
get_part_nodes or so, and no ring validation is there.
This patch moves ring validation from policy validate_ring into the ring
instance as validation_hook that will run at ring reload. Since this patch,
ring instance will allow to use the old ring if the reload is not fourced.
Note that the exception if invalid ring found was changed from
RingValidationError to RingLoadError because RingValidationError is a
child of RingBuilderError but the ring reload is obviously outside of
"builder".
Closes-Bug: #1534572
Change-Id: I6428fbfb04e0c79679b917d5e57bd2a34f2a0875
I changed asserts with more specific assert methods.
e.g.: from assertTrue(sth == None) to assertIsNone(*) or
assertTrue(isinstance(inst, type)) to assertIsInstace(inst, type)
or assertTrue(not sth) to assertFalse(sth).
The code gets more readable, and a better description will be shown on fail.
Change-Id: If6aad8681aab7c9a41d65a4f449d8abbe3e64616
Change swift.proxy.server.Application.error_occurred()
to decode message as utf-8 in same way that the
exception_occurred() method was changed in [1].
This prevents a unicode error when logging error responses
in swift.proxy.controllers.base.Controller._make_request()
for paths that have non-ascii characters. Although the unicode
error is currently caught by a surrounding except clause, the
logging and error limiting treatment is different for ascii
vs non-ascii paths. This patch makes them consistent.
Fix the server type reported in _make_request() to be
the correct server type, not always 'Container Server'.
Fix path arg passed to _get_conn_response in
swift.proxy.controllers.obj.BaseObjectController to be req.path
rather than req.
Add unit tests for error_occurred() being called with non-ascii
paths and extend tests for exception_occurred() (see Related-Bug).
[1] Change-Id: Icb7284eb5abc9869c1620ee6366817112d8e5587
Related-Bug: #1597210
Change-Id: I285499d164bff94835bdddb25d2af6d73114c281
Adds a new form of system metadata for objects.
Sysmeta cannot be updated by an object POST because
that would cause all existing sysmeta to be deleted.
Crypto middleware will want to add 'system' metadata
to object metadata on PUTs and POSTs, but it is ok
for this metadata to be replaced en-masse on every
POST.
This patch introduces x-object-transient-sysmeta-*
that is persisted by object servers and returned
in GET and HEAD responses, just like user metadata,
without polluting the x-object-meta-* namespace.
All headers in this namespace will be filtered
inbound and outbound by the gatekeeper, so cannot
be set or read by clients.
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Janie Richling <jrichli@us.ibm.com>
Change-Id: I5075493329935ba6790543fc82ea6e039704811d
Before this patch, the proxy ObjectController supported sending
metadata from the proxy server to object servers in "footers" that
trail the body of HTTP PUT requests, but this support was for EC
policies only. The encryption feature requires that footers are sent
with both EC and replicated policy requests in order to persist
encryption specific sysmeta, and to override container update headers
with an encrypted Etag value.
This patch:
- Moves most of the functionality of ECPutter into a generic Putter
class that is used for replicated object PUTs without footers.
- Creates a MIMEPutter subclass to support multipart and multiphase
behaviour required for any replicated object PUT with footers and
all EC PUTs.
- Modifies ReplicatedObjectController to use Putter objects in place
of raw connection objects.
- Refactors the _get_put_connections method and _put_connect_node methods
so that more code is in the BaseObjectController class and therefore
shared by [EC|Replicated]ObjectController classes.
- Adds support to call a callback that middleware may have placed
in the environ, so the callback can set footers. The
x-object-sysmeta-ec- namespace is reserved and any footer values
set by middleware in that namespace will not be forwarded to
object servers.
In addition this patch enables more than one value to be added to the
X-Backend-Etag-Is-At header. This header is used to point to an
(optional) alternative sysmeta header whose value should be used when
evaluating conditional requests with If-[None-]Match headers. This is
already used with EC policies when the ECObjectController has
calculated the actual body Etag and sent it using a footer
(X-Object-Sysmeta-EC-Etag). X-Backend-Etag-Is-At is in that case set
to X-Object-Sysmeta-Ec-Etag so as to point to the actual body Etag
value rather than the EC fragment Etag.
Encryption will also need to add a pointer to an encrypted Etag value.
However, the referenced sysmeta may not exist, for example if the
object was created before encryption was enabled. The
X-Backend-Etag-Is-At value is therefore changed to support a list of
possible locations for alternate Etag values. Encryption will place
its expected alternative Etag location on this list, as will the
ECObjectController, and the object server will look for the first
object metadata to match an entry on the list when matching
conditional requests. That way, if the object was not encrypted then
the object server will fall through to using the EC Etag value, or in
the case of a replicated policy will fall through to using the normal
Etag metadata.
If your proxy has a third-party middleware that uses X-Backend-Etag-Is-At
and it upgrades before an object server it's talking to then conditional
requests may be broken.
UpgradeImpact
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Co-Authored-By: Samuel Merritt <sam@swiftstack.com>
Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>
Closes-Bug: #1594739
Change-Id: I12a6e41150f90de746ce03623032b83ed1987ee1
Relocates some test infrastructure in preparation for
use with encryption tests, in particular moves the test
server setup code from test/unit/proxy/test_server.py
to a new helpers.py so that it can be re-used, and adds
ability to specify additional config options for the
test servers (used in encryption tests).
Adds unit test coverage for extract_swift_bytes and functional
test coverage for container listings. Adds a check on the content
and metadata of reconciled objects in probe tests.
Change-Id: I9bfbf4e47cb0eb370e7a74d18c78d67b6b9d6645
This patch enables to show a x-put-timestamp as
a last-modified header in container-server.
Note that the last-modified header will be changed only when a
request for container (PUT container or POST container) comes into
Swift. i.e. some requests for objects (e.g. PUT object, POST object)
will never affect the last-modified value but only when using
python-swiftclient like as "swift upload", the last-modified will
be close to the upload time because python-swiftclient will make
a PUT container request for "swift upload" each time.
Change-Id: I9971bf90d24eee8921f67c02b7e2c80fd8995623
When we were caching directly to the WSGI environment, it made sense to
have different keys for the different caches. Now that we have a
separate data structure for the per-request cache, however, we ought to
be consistent.
Change-Id: I199cba6e5fc9ab4205bba369e6a2f34fc5ce22d4
get_account_info used to work like this:
* make an account HEAD request
* ignore the response
* get the account info by digging around in the request environment,
where it had been deposited by elves or something
Not actually elves, but the proxy's GETorHEAD_base method would take
the HEAD response and cache it in the response environment, which was
the same object as the request environment, thus enabling
get_account_info to find it.
This was extraordinarily brittle. If a WSGI middleware were to
shallow-copy the request environment, then any middlewares to its left
could not use get_account_info, as the left middleware's request
environment would no longer be identical to the response environment
down in GETorHEAD_base.
Now, get_account_info works like this:
* make an account HEAD request.
* if the account info is in the request environment, return it. This
is an optimization to avoid a double-set in memcached.
* else, compute the account info from the response headers, store it
in caches, and return it.
This is much easier to think about; get_account_info can get and cache
account info all on its own; the cache check and cache set are right
next to each other.
All the above is true for get_container_info as well.
get_info() is still around, but it's just a shim. It was trying to
unify get_account_info and get_container_info to exploit the
commonalities, but the number of times that "if container:" showed up
in get_info and its helpers really indicated that something was
wrong. I'd rather have two functions with some duplication than one
function with no duplication but a bunch of "if container:" branches.
Other things of note:
* a HEAD request to a deleted account returns 410, but
get_account_info would return 404 since the 410 came from the
account controller *after* GETorHEAD_base ran. Now
get_account_info returns 410 as well.
* cache validity period (recheck_account_existence and
recheck_container_existence) is now communicated to
get_account_info via an X-Backend header. This way,
get_account_info doesn't need a reference to the
swift.proxy.server.Application object.
* both logged swift_source values are now correct for
get_container_info calls; before, on a cold cache,
get_container_info would call get_account_info but not pass along
swift_source, resulting in get_account_info logging "GET_INFO" as
the source. Amusingly, there was a unit test asserting this bogus
behavior.
* callers that modify the return value of get_account_info or of
get_container_info don't modify what's stored in swift.infocache.
* get_account_info on an account that *can* be autocreated but has
not been will return a 200, same as a HEAD request. The old
behavior was a 404 from get_account_info but a 200 from
HEAD. Callers can tell the difference by looking at
info['account_really_exists'] if they need to know the difference
(there is one call site that needs to know, in container
PUT). Note: this is for all accounts when the proxy's
"account_autocreate" setting is on.
Change-Id: I5167714025ec7237f7e6dd4759c2c6eb959b3fca
Previously, if you called get_account_info, get_container_info, or
get_object_info, then the results of that call would be cached in the
WSGI environment as top-level keys. This is okay, except that if you,
in middleware, copy the WSGI environment and then make a subrequest
using the copy, information retrieved in the subrequest is cached
only in the copy and not in the original. This can mean lots of extra
trips to memcache for, say, SLO validation where the segments are in
another container; the object HEAD ends up getting container info for
the segment container, but then the next object HEAD gets it again.
This commit moves the cache for get_*_info into a dictionary at
environ['swift.infocache']; this way, you can shallow-copy the request
environment and still get the benefits from the cache.
Change-Id: I3481b38b41c33cd1e39e19baab56193c5f9bf6ac
Rewrite server side copy and 'object post as copy' feature as middleware to
simplify the PUT method in the object controller code. COPY is no longer
a verb implemented as public method in Proxy application.
The server side copy middleware is inserted to the left of dlo, slo and
versioned_writes middlewares in the proxy server pipeline. As a result,
dlo and slo copy_hooks are no longer required. SLO manifests are now
validated when copied so when copying a manifest to another account the
referenced segments must be readable in that account for the manifest
copy to succeed (previously this validation was not made, meaning the
manifest was copied but could be unusable if the segments were not
readable).
With this change, there should be no change in functionality or existing
behavior. This is asserted with (almost) no changes required to existing
functional tests.
Some notes (for operators):
* Middleware required to be auto-inserted before slo and dlo and
versioned_writes
* Turning off server side copy is not configurable.
* object_post_as_copy is no longer a configurable option of proxy server
but of this middleware. However, for smooth upgrade, config option set
in proxy server app is also read.
DocImpact: Introducing server side copy as middleware
Co-Authored-By: Alistair Coles <alistair.coles@hpe.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Change-Id: Ic96a92e938589a2f6add35a40741fd062f1c29eb
Signed-off-by: Prashanth Pai <ppai@redhat.com>
Signed-off-by: Thiago da Silva <thiago@redhat.com>
Follow up for change Ibbc89449e7878fc4215e47e3f7dfe4ae58a2d638
to add a test assertion that the ChunkWriteTimeout contexts are
exited without raising the timeout exception in
iter_bytes_from_response_part().
Change-Id: I6d323cb26779e457fb5940093a81b349b333a0af
This is a follow-up patch for https://review.openstack.org/#/c/258392/
That one added good unit test cases for various kinds of
allowe_origin like '*' or ''(empty). However, the result of handling
in Swift proxy will depend on strict_cors_mode option configuration.
This patch refactors the unit tests to split out for
strict_cors_mode = on/off and add some missing unit tests for
each case.
Change-Id: I55f7cd279436b5c9f71d81fecf06021380e35579
When eventlet.wsgi closes an ECAppIter on client disconnect we need to
make sure our sub-iterators are also closed. We already kill the
backend sockets, but the executing contexts are left running until
they timeout. A slow client can result in needlessly holding queued
backend fragments until the client_timeout (default 60s).
Update associated test that exposed the problem to demonstrate the
issue more quickly.
Change-Id: Ibbc89449e7878fc4215e47e3f7dfe4ae58a2d638
When using an EC policy, 304 responses to conditional GETs
are missing the Accept-Ranges header and have the wrong ETag
value. 412 responses also have the wrong etag.
416 responses to ranged GETs also have the wrong ETag.
This patch ensures behaviour with EC policy is consistent
with replication policy:
- 304 and 416 responses have correct etag and Accept-Ranges
- 412 responses have correct Etag but no Accept-Ranges
Co-Authored-By: Mahati Chamarthy <mahati.chamarthy@gmail.com>
Co-Authored-By: Thiago da Silva <thiago@redhat.com>
Closes-Bug: #1496234
Closes-Bug: #1558197
Closes-Bug: #1558193
Change-Id: Ic21317b9e4f632f0751133a3383eb5487379e11f
This change adds 2 new parameters to enable and control concurrent GETs
in swift, these are 'concurrent_gets' and 'concurrency_timeout'.
'concurrent_gets' allows you to turn on or off concurrent GETs, when
on it will set the GET/HEAD concurrency to replica count. And in the
case of EC HEADs it will set it to ndata.
The proxy will then serve only the first valid source to respond.
This applies to all account, container and object GETs except
for EC. For EC only HEAD requests are effected.
It achieves this by changing the request sending mechanism to using
GreenAsyncPile and green threads with a time out between each
request.
'concurrency_timeout' is related to concurrent_gets. And is the
amount of time to wait before firing the next thread. A value of 0
will fire at the same time (fully concurrent), setting another value
will stagger the firing allowing you the ability to give a node a
shorter chance to respond before firing the next. This value is a float
and should be somewhere between 0 and node_timeout. The default is
conn_timeout. Meaning by default it will stagger the firing.
DocImpact
Implements: blueprint concurrent-reads
Change-Id: I789d39472ec48b22415ff9d9821b1eefab7da867
There was a function in swift.common.utils that was importing
swob.HeaderKeyDict at call time. It couldn't import it at compilation
time since utils can't import from swob or else it blows up with a
circular import error.
This commit just moves HeaderKeyDict into swift.common.header_key_dict
so that we can remove the inline import.
Change-Id: I656fde8cc2e125327c26c589cf1045cb81ffc7e5
This patch makes a number of changes to enable content-type
metadata to be updated when using the fast-POST mode of
operation, as proposed in the associated spec [1].
* the object server and diskfile are modified to allow
content-type to be updated by a POST and the updated value
to be stored in .meta files.
* the object server accepts PUTs and DELETEs with older
timestamps than existing .meta files. This is to be
consistent with replication that will leave a later .meta
file in place when replicating a .data file.
* the diskfile interface is modified to provide accessor
methods for the content-type and its timestamp.
* the naming of .meta files is modified to encode two
timestamps when the .meta file contains a content-type value
that was set prior to the latest metadata update; this
enables consistency to be achieved when rsync is used for
replication.
* ssync is modified to sync meta files when content-type
differs between local and remote copies of objects.
* the object server issues container updates when handling
POST requests, notifying the container server of the current
immutable metadata (etag, size, hash, swift_bytes),
content-type with their respective timestamps, and the
mutable metadata timestamp.
* the container server maintains the most recently reported
values for immutable metadata, content-type and mutable
metadata, each with their respective timestamps, in a single
db row.
* new probe tests verify that replication achieves eventual
consistency of containers and objects after discrete updates
to content-type and mutable metadata, and that container-sync
sync's objects after fast-post updates.
[1] spec change-id: I60688efc3df692d3a39557114dca8c5490f7837e
Change-Id: Ia597cd460bb5fd40aa92e886e3e18a7542603d01
