Extensive refactor here to consolidate what nodes are contacted for
any request. This consolidation means reads will contact the same set
of nodes that writes would, giving a very good chance that
read-your-write behavior will succeed. This also means that writes
will not necessarily try all nodes in the cluster as it would
previously, which really wasn't desirable anyway. (If you really want
that, you can set request_node_count to a really big number, but
understand that also means reads will contact every node looking for
something that might not exist.)
* Added a request_node_count proxy-server conf value that allows
control of how many nodes are contacted for a normal request.
In proxy.controllers.base.Controller:
* Got rid of error_increment since it was only used in one spot by
another method and just served to confuse.
* Made error_occurred also log the device name.
* Made error_limit require an error message and also documented a bit
better.
* Changed iter_nodes to just take a ring and a partition and yield
all the nodes itself so it could control the number of nodes used
in a given request. Also happens to consolidate where sort_nodes is
called.
* Updated account_info and container_info to use all nodes from
iter_nodes and to call error_occurred appropriately.
* Updated GETorHEAD_base to not track attempts on its own and just
stop when iter_nodes tells it to stop. Also, it doesn't take the
nodes to contact anymore; instead it takes the ring and gets the
nodes from iter_nodes itself.
Elsewhere:
* Ring now has a get_part method.
* Made changes to reflect all of the above.
Change-Id: I37f76c99286b6456311abf25167cd0485bfcafac
In trying to override the content-type for the convenience
call I broke COPYs and POSTs for SLO. yaaa...
Change-Id: Ifbcda6d2dd0ee43e43d62e58a90301e7afd05e27
- We allow all headers requested in preflight request. The CORS
specification does leave the door open for this, as mentioned in
http://www.w3.org/TR/cors/#resource-preflight-requests
Note: Since the list of headers can be unbounded
simply returning headers can be enough.
- This is a followup to review:
https://review.openstack.org/#/c/24415/.
- Fixes bug 1155034.
Change-Id: If7b8f2f3a581c5209892d1ccc9f06ddb8fac92dd
DLO is supposed to sleep 1 second per segment after 10 (default) initial
segments are loaded to prevent abuse of the system. Currently, if the user
did a range request to the 11th segment the 1 sec delay would start immediately
on the 11th segment because the skipped segments were counted as initial
segments. This path fixes that.
Change-Id: I53bbc24af54921ddc04d9cb3b704366b51b3f740
If X-Copy-From is a manifest object, it can be bigger than MAX_FILE_SIZE.
Currently this fails with 503. It should be 413.
Bug #1158279
Change-Id: I7ec16088046c84e42d5be9c865e1338eb07845b6
We can set x-versions-location empty to remove this header in API, but
to keep consistent, we should allow x-remove-versions-location too. The
usage is post "x-remove-versions-locaion: x", just like ACL remove
headers.
Fixed: bug #1107592
Change-Id: I1271eec6401d4a0e8c1a7c2d63aeb8dfef00bf6d
The only place in the spec that I could see the Origin being checked was
during the pre-flight OPTIONS request. If it gets to the actual request
let auth decide. Please correct me if this is wrong.
Change-Id: Ic31b71746ec056091c7778ebff3db7becc32bd9c
There was a bug where account_info wasn't converting the
container_count value to an int. Causes max container count cap to
get hit pretty quick since '0' > 0.
Change-Id: Ibfc6eebbff5a00aaebb47e9731dd053b60e3caa4
This changes the way primary replicas can be sorted on GET requests.
Previously, replicas were shuffled. Now, if configured, the replicas
are sorted based on the most recent connection time data to that node.
This patch adds a config value that changes the sorting method.
get_more_nodes() (ie handoffs) is unaffected by this patch because
sorting by affinity would break the durability provided by the current
as-unique-as-possible handoff selection.
Timing data is collected for each node each time the proxy makes a
connection to that node (IP address). If timing data for a node doesn't
exist, then it is assumed at -1 (ie will sort earlier) so that timing
data can be collected for that node.
Change-Id: I837fa21c3a566b10cce33eb75788665e1d01cd8a
- Fixes bug 1119282.
- Allow middleware accessing metadata of an account without having to
store it separately in a new memcache namespace.
- Add tests for get_container_info that was previously missed.
- Add get_account_info method based on get_container_info, a function
for other middleware to query accounts.
- Rename container_info['count'] as container_info['object_count'].
Change-Id: I43787916c7a812cb08d278edf45370521f12c912
This patch fixes the Swift MemcacheRing set and set_multi
interface incompatible problem with python memcache. The fix
added two extra named parameters to both set and set_multi
method. When only time or timeout parameter is present, then one
of the value will be used. When both time and timeout are present,
the time parameter will be used.
Named parameter min_compress_len is added for pure compatibility
purposes. The current implementation ignores this parameter.
To make swift memcached methods all consistent cross the board,
method incr and decr have also been changed to include a new
named parameter time.
In future OpenStack releases, the named parameter timeout will be
removed, keep the named parameter timeout around for now is
to make sure that mismatched releases between client and server
will still work.
From now on, when a call is made to set, set_multi, decr, incr
by using timeout parametner, a warning message will be logged to
indicate the deprecation of the parameter.
Fixes: bug #1095730
Change-Id: I07af784a54d7d79395fc3265e74145f92f38a893
pyflakes itself can't be used in any automated gating way, because there are
two sets of false errors it raises. However, as an exercise, cleaning up the
'valid' ones uncovered three actual bugs. The other changes (mostly unused
variables) are included here for fun.
Command run: pyflakes swift | grep -v "undefined name '_'"
Change-Id: I18696bf047dedad1a9fdbde3463e214fba95f7c6
Fix for bug 1095130
* Added a wrapper function around public methods to handle
CORS actual requests. These requests need to return some
extra headers to be valid responses to a CORS request.
Access-Control-Expose-Headers and Access-Control-Allow-Origin.
* Added support for the CORS header Access-Control-Expose-Headers.
* Some refactoring of the OPTIONS method so the
"is_origin_allowed" logic can be reused.
* Added a little extra detail to the CORS documentation.
DocImpact
Change-Id: I68538e472a900775427f21a8a59e738a83dcc8bc
When the number of account/container or container/object replicas are
different, Swift had a few misbehaviors. This commit fixes them.
* On an object PUT/POST/DELETE, if there were 3 object replicas and
only 2 container replicas, then only 2 requests would be made to
object servers. Now, 3 requests will be made, but the third won't
have any X-Container-* headers in it.
* On an object PUT/POST/DELETE, if there were 3 object replicas and 4
container replicas, then only 3/4 container servers would receive
immediate updates; the fourth would be ignored. Now one of the
object servers will receive multiple (comma-separated) values in the
X-Container-* headers and it will attempt to contact both of them.
One side effect is that multiple async_pendings may be written for
updates to the same object. They'll have differing timestamps,
though, so all but the newest will be deleted unread. To trigger
this behavior, you have to have more container replicas than object
replicas, 2 or more of the container servers must be down, and the
headers sent to one object server must reference 2 or more down
container servers; it's unlikely enough and the consequences are so
minor that it didn't seem worth fixing.
The situation with account/containers is analogous, only without the
async_pendings.
Change-Id: I98bc2de93fb6b2346d6de1d764213d7563653e8d
When responding to a GET request for a manifest, it was intended that
the proxy server lazily fetch the pieces of the container
listing. That way, a single client request doesn't immediately turn
into a bunch of requests to backends. The additional requests should
only get made if the client is putting in the work of receiving the
object body.
However, commit 156f27c accidentally changed this so that all the
pieces of the container listing are eagerly fetched up-front. Better
yet, if an object has more than CONTAINER_LISTING_LIMIT (default
10,000) segments, the container listing is then fetched a second time,
albeit lazily, while streaming out the response.
This commit restores the laziness and adds tests for it.
Change-Id: I49840a7059e6f999ce19199ecb10cdb77358526b
Most of the test files set the HASH_PATH_SUFFIX so you can run the test
file stand alone. This change made it easier for me to run specific
proxy tests separately.
Change-Id: I87d70367dac7f240a2b6779649f8a02cf324ae0f
Ensures that the Allow header is set for 405 responses from the proxy,
as per RFC 2616.
CORS preflight requests now require both the Origin and Access-
Control-Request-Method headers to get a successful (200) response. The
draft spec defines errors as a response with a non-200 status code.
This patch sets the CORS error response code to 401 (Not Authorized).
A later patch may choose to make this configurable.
There is some ambiguity between RFC 2616 and the CORS draft spec
around what to do when a CORS request is made but the cluster has no
CORS information about the requested resource. This patch chooses to
return an error in this case because it is what would be simplest for
CORS client apps.
Further improvements to the OPTIONS verb not included in this patch
include support of more top-level resources (eg / or /v1/) or
sending the configured constraints in the reponse body.
Change-Id: I40be059e8bbf3737dafc4e6fefa7598d05669c60
Fix bug 1067677
When we delete a versioned object, the last version will be poped out from the
versiones container. When a versioned object is expired and deleted by object-
expirer, the last version is restored but remains in the versions container
instead of getting deleted. The reason is object-expirer will set
'X-If-Delete- At' header when deleteing an object. However this is for the
current version - not for the last version. When the object-server is trying
to delete the last version, the transaction will fail with error:
X-If-Delete-At and X-Delete-At do not match. Delete the 'X-If-Delete-At'
field in the later built delete request would help to solve this issue.
This patch, without the test, was first proposed by
Zhou Yuan <yuan.zhou@intel.com>.
Change-Id: I62c97e6a0f140888497e189129825865fb6f7966
This change adds multi range retrieval to OpenStack Swift. For non-
segmented data object, a client can use HTTP Range header to specify
multiple ranges to retrieve sections of the data object.
This implementation currently does not support segmented data object
multi range retrieval. When a client sends a multi range request
against a segmented data object, Swift will return HTTP status code
200. Support for segmented data multi range retrieval will be added
in near future.
This implementation is to bring Swift closer to CDMI multi range
data retrieval standard. Once support for segemented data multi
range is added, Swift will be compliant with CDMI standard in
this area.
DocImpact
Change-Id: I4ed1fb0a0a93c037ddb2f551ea62afe447945107
Swift proxy server, when communicating with
the back-end servers ALWAYS sends 100-continue.
Even if the length headers indicate that there
is no body (content-length: 0).
This behavior is not inline with the standard.
RFC2616:8.2.3 disallows 100-continue
for req.content_length==0
This fix removes 100-continue during PUT requests
without a body while maintining the ability for
handoff in case of faliure.
On branch bp/wsgi-application-interface-5
modified: swift/proxy/controllers/obj.py
Fixes: Bug #1070025
Implements Blueprint: wsgi-application-interface
Change-Id: Ia4eb8b886a74968cca4e8bde208641b37f2c104c
The proxy server's ObjectController was performing multiple GET
requests to the object server when processing requests with Range
headers. This was a workaround for a bug in the proxy server's
Controller.GETorHEAD_base method where a response code of 416 from the
object server was incorrectly treated as a bad response from the
backend, the same way a 404 or a 5xx would be.
A 416 (Requested Range Not Satisfiable) from the object server is now
considered a good response. Since the response headers from the object
server already include the X-Object-Manifest header, there's no need
to make a second request (sans Range header) to see if the object is a
manifest.
Bonus fix: updated message for status 416 to match RFC2616.
Bonus fix 2: removed a leftover debugging print() in
test/unit/proxy/test_server.py.
Fixes bug 1065869.
Change-Id: I156af0b463f02ef19a8cfe37092544a599d89b78
fixes bug 1049017
As per the bug report, the -remove- form of deletion only works with
meta headers and not with x-container-read and x-container-write.
This patch by Yukihiro KAWADA detects the container acls and sends
them through to the backend empty like the other metadata values.
patch2 extends metadata-helper in ContainerController tests so that
the new functionality can be tested.
patch3 changes the k.lower().startswith() commands for read/write to
a single k.lower - thanks David :)
patch4 fixes PEP8
patch5 fixes more than two hundred pep8 errors. Except one,
where the pep8 tool complains of overindent, or underindent
depending on the position on the line. No idea how to fix that one.
patch6 fixes the remaining pep8 error - thanks Darrell
Change-Id: I36c2dd72b4636136c5ce7db414bf2b61501090ad
This change replaces WebOb with a mostly compatible local library,
swift.common.swob. Subtle changes to WebOb's API over the years have been a
huge headache. Swift doesn't even run on the current version.
There are a few incompatibilities to simplify the implementation/interface:
* It only implements the header properties we use. More can be easily added.
* Casts header values to str on assignment.
* Response classes ("HTTPNotFound") are no longer subclasses, but partials
on Response, so things like isinstance no longer work on them.
* Unlike newer webob versions, will never return unicode objects.
Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f
This fixes an issue where Request objects (and related objects) were
not getting garbage collected when a 499 (client disconnect) occurred
for responses that still would have had more than the proxy server's
client chunk size left to send.
Fixed bug #1055834
Change-Id: Ib7b5fffe970dcdec60b90a9bdcd67c49b68fb914
I think not doing this was setting the SIGPIPE handler to signal.SIG_DFL
instead of the Python default of signal.SIG_IGN. This could cause other
tests which make a client stop reading before all data "chunks" are read
to fail harder than they should (i.e. the SIGPIPE there is benign and
even expected--the other side of the socket really did get closed
early).
Change-Id: I946a1913714a34ec677b701a2b5724bc8b715f70
This change refactors container_info to return a dict instead of a tuple
with 6 (and counting...) values.
Also adds container size to cached info.
Change-Id: If68a06ab170504fbf07323354f3d0e073850493d
Change previously hard-coded constants into config variables. This
allows deployers to tune their cluster more specifically based on
their needs. For example, a deployment that uses direct swift access
for public content may need to set a larger header value constraint to
allow for the full object name to be represented in the Content-
Disposition header (which browsers check to determine the name of a
downloaded object).
The new settings are set in the [swift-constraints] section of
/etc/swift/swift.conf. Comments were also added to this config file.
Cleaned up swift/common/constraints.py to pass pep8 1.3.3
Funtional tests now require constraints to be defined in /etc/test.conf or in
/etc/swift/swift.conf (in the case of running the functional tests against a
local swift cluster). To have any hope of tests passing, the defined
constraints must match the constraints on the tested cluster.
Removed a ton of "magic numbers" in both unit and functional tests.
Change-Id: Ie4588e052fd158314ddca6cd8fca9bc793311465
