The X-Auth-Token is sensitive data. If revealed to an unauthozied person,
they can now make requests against an account until the token expires.
This implementation maintains current behavior (i.e, the token
is logged). Implementers can choose to set reveal_sensitive_prefix
to (e.g.) 12 so only first 12 characters of the token are logged.
Or, set to 0 to replace the token with "...".
DocImpact
Part of bug #1004114
Change-Id: Iecefa843d8f9ef59b9dcf0860e7a4d0e186a6cb5
Added logging of the x-delete-at value for object PUTs and POSTs and
made the additional log field generic for other similar use cases.
Logging the x-delete-at is useful because otherwise you might never
know why an object "disappeared" just by looking at saved proxy logs.
You'd see a successful PUT of the object followed some time by a GET
that 404s. If you still have the other logs, you could track down
what happened; but, at least with our clusters, we keep proxy logs
forever but only keep the other logs for a short while.
Change-Id: Ida559abcf8dc5666e2a7eea250a67d1c4b28b1be
Previous logging made a mess of ipv6 addresses. This just makes the
colon quote-safe so it passes through unscathed. The logs will still
be backward compatible because unquote don't care:
$ python
>>> from urllib import quote, unquote
>>> quote('2001:db8:85a3:8d3:1319:8a2e:370:7348')
'2001%3Adb8%3A85a3%3A8d3%3A1319%3A8a2e%3A370%3A7348'
>>> unquote(quote('2001:db8:85a3:8d3:1319:8a2e:370:7348'))
'2001:db8:85a3:8d3:1319:8a2e:370:7348'
>>> quote('2001:db8:85a3:8d3:1319:8a2e:370:7348', '/:')
'2001:db8:85a3:8d3:1319:8a2e:370:7348'
>>> unquote(quote('2001:db8:85a3:8d3:1319:8a2e:370:7348', '/:'))
'2001:db8:85a3:8d3:1319:8a2e:370:7348'
Change-Id: Ia13a9bc8a320cde5c56034a7f0b645913428bf21
This was an oustanding TODO for StatsD Swift metrics. A new timing
metric is tracked for (only) GET requests for accounts, containers,
and objects:
proxy-server.<req_type>.GET.<status_int>.first-byte.timing
Also updated StatsD documentation in the Admin Guide to clarify that
timing metrics are sent in units of milliseconds.
Change-Id: I5bb781c06cefcb5280f4fb1112a526c029fe0c20
The proxy_logging middleware was asserting that the response contained
either a Content-Length header or a Transfer-Encoding header. If not,
it would either add one (if app_iter was a list) or blow up
(otherwise). This blowing up is observable on a GET request to a
manifest object that references more than
swift.common.constraints.CONTAINER_LISTING_LIMIT segments.
If a response makes it up to eventlet.wsgi without a Content-Length
header, then a "Transfer-Encoding: chunked" header is automatically
stuffed into the response by eventlet. Therefore, it's not an error
for a response to not have a Content-Length header, and proxy_logging
should just let it happen.
Fixes bug 1078113.
Change-Id: I3751a8ae14dc68bab546f2746b61267a5115e252
This change replaces WebOb with a mostly compatible local library,
swift.common.swob. Subtle changes to WebOb's API over the years have been a
huge headache. Swift doesn't even run on the current version.
There are a few incompatibilities to simplify the implementation/interface:
* It only implements the header properties we use. More can be easily added.
* Casts header values to str on assignment.
* Response classes ("HTTPNotFound") are no longer subclasses, but partials
on Response, so things like isinstance no longer work on them.
* Unlike newer webob versions, will never return unicode objects.
Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f
Removed many StatsD logging calls in proxy-server and added
swift-informant-style catch-all logging in the proxy-logger middleware.
Many errors previously rolled into the "proxy-server.<type>.errors"
counter will now appear broken down by response code and with timing
data at: "proxy-server.<type>.<verb>.<status>.timing". Also, bytes
transferred (sum of in + out) will be at:
"proxy-server.<type>.<verb>.<status>.xfer". The proxy-logging
middleware can get its StatsD config from standard vars in [DEFAULT] or
from access_log_statsd_* config vars in its config section.
Similarly to Swift Informant, request methods ("verbs") are filtered
using the new proxy-logging config var, "log_statsd_valid_http_methods"
which defaults to GET, HEAD, POST, PUT, DELETE, and COPY. Requests with
methods not in this list use "BAD_METHOD" for <verb> in the metric name.
To avoid user error, access_log_statsd_valid_http_methods is also
accepted.
Previously, proxy-server metrics used "Account", "Container", and
"Object" for the <type>, but these are now all lowercase.
Updated the admin guide's StatsD docs to reflect the above changes and
also include the "proxy-server.<type>.handoff_count" and
"proxy-server.<type>.handoff_all_count" metrics.
The proxy server now saves off the original req.method and proxy_logging
will use this if it can (both for request logging and as the "<verb>" in
the statsd timing metric). This fixes bug 1025433.
Removed some stale access_log_* related code in proxy/server.py. Also
removed the BaseApplication/Application distinction as it's no longer
necessary.
Fixed up the sample config files a bit (logging lines, mostly).
Fixed typo in SAIO development guide.
Got proxy_logging.py test coverage to 100%.
Fixed proxy_logging.py for PEP8 v1.3.2.
Enhanced test.unit.FakeLogger to track more calls to enable testing
StatsD metric calls.
Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576
The old use of Eventlet's posthook process meant that responses that
forgot to include content-length or transfer-encoding headers would
get one tacked on, if Eventlet could guess what was probably meant. I
added a bit of that logic into proxy-logging now as we saw some
errors resulting from this.
Fixes Bug #1012714
Change-Id: I671453eaf3704eab814ff12c4625ba7d749cc7ed
