Clients can construct tempurls for any method, but they only work if
they're in this list, so it's helpful for clients to see the list.
Change-Id: Id852f457d65b62c4fe79db01b1d7029a5fa5aa09
HEAD-only tempurls didn't work; tempurl only allowed a HEAD request if
the tempurl was generated for GET or PUT (that is, the method in the
HMAC-signed string was "GET" or "PUT").
The intent of the code was to allow a user with a GET or a PUT tempurl
to also perform a HEAD request; I think the breaking of HEAD tempurls
is just a bug.
Change-Id: I621ddaac03e0d058dd9e7c7c374cb5c4b6386d36
Swift can now optionally be configured to allow requests to '/info',
providing information about the swift cluster. Additionally a HMAC
signed requests to
'/info?swiftinfo_sig=<sign>&swiftinfo_expires=<expires>' can be
configured allowing privileged access to more sensitive information
not meant to be public.
DocImpact
Change-Id: I2379360fbfe3d9e9e8b25f1dc34517d199574495
Implements: blueprint capabilities
Closes-Bug: #1245694
Giving the inline query parameter will cause the tempurl
response to be given a "Content-Disposition: inline" header,
regardless of other query parameters or metadata. This allows
easy in-line viewing, eg in browsers.
DocImpact
Change-Id: Icd5c544d6a749d4f58e8a921968f4e432a2185db
In tempurl middleware, hmac uses the value of account metadata to
generate HMAC-SHA1 signature and hmac must accept a str-type string, not
a unicode string. The meta dict returned from get_info stroges special
chars as unicode however. So just encode it for tempurl using.
Closes-Bug: #1242644
Change-Id: I4be62eea014a573efc4748470de57dccf00e431d
Per http://www.ietf.org/rfc/rfc2616.txt, when a 401 error is returned, the
Www-Authenticate response header MUST also be returned. The format is
described in http://www.ietf.org/rfc/rfc2617.txt.
Swift supports and/or implements a number of authentication schemes
including tempauth, Keystone, tempurl, formpost and container sync. In
this fix, we use a catch-all, "Swift". The realm is the account (where
known) or "unknown" (bad path or where the 401 is returned from code
that does not have the request). Examples:
Www-Authenticate: Swift realm="AUTH_1234567889"
Www-Authenticate: Swift realm="unknown"
Fixes bug #1215491
Change-Id: I03362789318dfa156d3733ef9348795062a9cfc4
So, there's two cases here. In one case, we have memcache enabled;
account info gets cached in the usual place, and tempurl keys get
cached separately. In the other case, we have no cache. In neither
case is anything gained by having TempURL cache keys separately from
the account info.
This commit removes TempURL's custom caching logic and makes it rely
on get_account_info() instead.
Benefits include:
* immediate visibility of new keys on account POST
* less data in the cache per account --> more stuff fits in cache
* less code for bugs to hide in
Change-Id: Idb0b6c165da14196b4c79149c546f0159b54edcb
In other places in the codebase, we cache a negative result for less
time (usually a tenth) than we cache a positive result. This commit
makes TempURL do the same.
This creates a slightly nicer experience for someone trying to set up
TempURL; if they get the initial key setup wrong, try it, then fix it,
it now takes only 6 seconds until they can use signed URLs, not
60. That's a short enough time that they may not even notice the
caching of no keys.
Change-Id: I521f023e7cddaecd07f0ce32aedd4059bd0b8ec4
If an object name ends with a /, the content-disposition filename for
a tempurl ended up as an empty string. This fix just strips any
trailing slashes before calling basename.
Also, I added a test to verify that if the filename is passed as an
overriding query parameter, that it is used in full, no rstripping,
no basenaming.
Change-Id: I37725b6ded04ed3b91cdb21132490fd857276e2f
test_get_valid and test_get_valid_key2 test tempurl with a
single key and two keys respectively. This change moves common code
for these tests into a separate function.
Change-Id: Ifee211285e27df03d041a4c357b7c1f0c96ed6a8
Enhance internally logged messages to report referer and user-agent.
Pass the referering URL and METHOD between internal servers (when
known), and set the user-agent to be the server type (obj-server,
container-server, proxy-server, obj-updater, obj-replicator,
container-updater, direct-client, etc.) with the process PID. In
conjunction with the transaction ID, it helps to track down which PID
from a given system was responsible for initiating the request and
what that server was working on to make this request.
This has been helpful in tracking down interactions between object,
container and account servers.
We also take things a bit further performaing a bit of refactoring to
consolidate calls to transfer_headers() now that we have a helper
method for constructing them.
Finally we performed further changes to avoid header key duplication
due to string literal header key values and the various objects
representing headers for requests and responses. See below for more
details.
====
Header Keys
There seems to be a bit of a problem with the case of the various
string literals used for header keys and the interchangable way
standard Python dictionaries, HeaderKeyDict() and HeaderEnvironProxy()
objects are used.
If one is not careful, a header object of some sort (one that does not
normalize its keys, and that is not necessarily a dictionary) can be
constructed containing header keys which differ only by the case of
their string literals. E.g.:
{ 'x-trans-id': '1234', 'X-Trans-Id': '5678' }
Such an object, when passed to http_connect() will result in an
on-the-wire header where the key values are merged together, comma
separated, that looks something like:
HTTP_X_TRANS_ID: 1234,5678
For some headers in some contexts, this is behavior is desirable. For
example, one can also use a list of tuples which enumerate the multiple
values a single header should have.
However, in almost all of the contexts used in the code base, this is
not desirable.
This behavior arises from a combination of factors:
1. Header strings are not constants and different lower-case and
title-case header strings values are used interchangably in the
code at times
It might be worth the effort to make a pass through the code to
stop using string literals and use constants instead, but there
are plusses and minuses to doing that, so this was not attempted
in this effort
2. HeaderEnvironProxy() objects report their keys in ".title()"
case, but normalize all other key references to the form
expected by the Request class's environ field
swob.Request.headers fields are HeaderEnvironProxy() objects.
3. HeaderKeyDict() objects report their keys in ".lower()" case,
and normalize all other key references to ".lower()" case
swob.Response.headers fields are HeaderKeyDict() objects.
Depending on which object is used and how it is used, one can end up
with such a mismatch.
This commit takes the following steps as a (PROPOSED) solution:
1. Change HeaderKeyDict() to normalize using ".title()" case to
match HeaderEnvironProxy()
2. Replace standard python dictionary objects with HeaderKeyDict()
objects where possible
This gives us an object that normalizes key references to avoid
fixing the code to normalize the string literals.
3. Fix up a few places to use title case string literals to match
the new defaults
Change-Id: Ied56a1df83ffac793ee85e796424d7d20f18f469
Signed-off-by: Peter Portante <peter.portante@redhat.com>
This allows users to rotate their TempURL keys without invalidating
all existing signed URLs. This is handy if you have multiple systems
generating signed URLs, but you want to change your keys for some
reason (e.g. keys compromised, company policy, general paranoia).
Both the first and second keys are optional, so existing accounts'
signed URLs will continue to work as before.
This commit does change the memcache key used to store the fetched
TempURL keys. This is because we were storing the old key as a string
in memcached, but the new one is a list of keys. Since the key cache
lifetime here is only 60 seconds, it doesn't seem like too big a deal
to completely flush the TempURL cache.
Also, this commit adds caching of a negative TempURL result. If the
account HEAD reveals no TempURL keys at all, that result is now stored
for 60 seconds the same way that a positive result would be.
Change-Id: I40a02bd607283fbce11aa52a9bb8a5846ab17f5e
Folks have actually been asking for this. I think they're sending a
DELETE TempURL to someone way ahead of time and the someone issues it
when they're ready. Honestly, I'm not entirely sure of the use case,
but having the set of methods configurable wouldn't hurt.
Change-Id: Ibdb48f8a72077b045eeedddfae4c0a1f56098d7a
- Prior to this commit, a Content-Disposition header was always set
on responses to GET requests, with the filename based on the object
name. Now, the header will only be set for 2xx responses and the
filename can be overridden with a filename query parameter on the
request.
- Fixed a bug where all query parameters on the request were being
passed down the WSGI pipeline. Now, just the query parameters
useful in log-based debugging are included. This becomes important
with things like the Bulk middleware that act upon query
parameters.
- Fixed bug where the Content-Disposition header wasn't following RFC
spec.
DocImpact
Change-Id: I66ad809321dcdd03444324973c8b76869e3b0c8e
This patch fixes the Swift MemcacheRing set and set_multi
interface incompatible problem with python memcache. The fix
added two extra named parameters to both set and set_multi
method. When only time or timeout parameter is present, then one
of the value will be used. When both time and timeout are present,
the time parameter will be used.
Named parameter min_compress_len is added for pure compatibility
purposes. The current implementation ignores this parameter.
To make swift memcached methods all consistent cross the board,
method incr and decr have also been changed to include a new
named parameter time.
In future OpenStack releases, the named parameter timeout will be
removed, keep the named parameter timeout around for now is
to make sure that mismatched releases between client and server
will still work.
From now on, when a call is made to set, set_multi, decr, incr
by using timeout parametner, a warning message will be logged to
indicate the deprecation of the parameter.
Fixes: bug #1095730
Change-Id: I07af784a54d7d79395fc3265e74145f92f38a893
This change replaces WebOb with a mostly compatible local library,
swift.common.swob. Subtle changes to WebOb's API over the years have been a
huge headache. Swift doesn't even run on the current version.
There are a few incompatibilities to simplify the implementation/interface:
* It only implements the header properties we use. More can be easily added.
* Casts header values to str on assignment.
* Response classes ("HTTPNotFound") are no longer subclasses, but partials
on Response, so things like isinstance no longer work on them.
* Unlike newer webob versions, will never return unicode objects.
Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f
Based on PatchSet 3 of https://review.openstack.org/#/c/7569/ , make them to pass all funcional tests with both webob 1.x and 1.2.
The additional following compatibility issues were addressed:
- Until patch for range header issue is merged into official webob release, testRangedGetsWithLWSinHeader() should skip test against webob 1.2
(49c175aec2)
- common.constraints.check_utf8() can accept both utf8 str and unicode.
- To convert unicode to utf-8 str if necessary.
- Making proxy_logging can handle invalid utf-8 str
bug 888371
bug 959881
blueprint webob-support
Change-Id: I00e5fd04cd1653259606a4ffdd4926db3c84c496
TempURL/FormPOST is now at http://gholt.github.com/swift-tempurl/
For current users of TempURL/FormPOST, this will require installing the new
package and changing the "use" line of the tempurl and formpost conf section's
to:
[filter:tempurl]
use = egg:swifttempurl#tempurl
[filter:formpost]
use = egg:swifttempurl#formpost
And then 'swift-init proxy reload'.
Change-Id: I5bddf7f9e09ee07815530a41c46ff901fc21b447
