In some unit tests instead of self.fail(msg) statements
self.assertTrue(False, msg) were used, which might be ambiguous.
Using assertTrue(False, msg) gives the following message on fail:
File "C:\Python361\lib\unittest\case.py", line 678, in assertTrue
raise self.failureException(msg)
AssertionError: False is not true : msg
'False is not true' message implies that unit test failed (as the
result is False while we asserted True).
Replaced with self.fail(msg) is less ambiguous and more readable.
File "C:\Python361\lib\unittest\case.py", line 666, in fail
raise self.failureException(msg)
AssertionError: msg
TrivialFix
Change-Id: Ib56a0ed8549fd7af2724eb59222106888781e9c8
Relocates some test infrastructure in preparation for
use with encryption tests, in particular moves the test
server setup code from test/unit/proxy/test_server.py
to a new helpers.py so that it can be re-used, and adds
ability to specify additional config options for the
test servers (used in encryption tests).
Adds unit test coverage for extract_swift_bytes and functional
test coverage for container listings. Adds a check on the content
and metadata of reconciled objects in probe tests.
Change-Id: I9bfbf4e47cb0eb370e7a74d18c78d67b6b9d6645
assertEquals is deprecated in py3, replacing it.
Change-Id: Ida206abbb13c320095bb9e3b25a2b66cc31bfba8
Co-Authored-By: Ondřej Nový <ondrej.novy@firma.seznam.cz>
The urllib, urllib2 and urlparse modules of Python 2 were reorganized
into a new urllib namespace on Python 3. Replace urllib, urllib2 and
urlparse imports with six.moves.urllib to make the modified code
compatible with Python 2 and Python 3.
The initial patch was generated by the urllib operation of the sixer
tool on: bin/* swift/ test/.
Change-Id: I61a8c7fb7972eabc7da8dad3b3d34bceee5c5d93
Currently, statistics are organized by command. However, it would also be
useful to display statistics organized by policy. Different policies may be
based on different storage properties (ie, faster disks).
With this change, all the statistics for object timers will be sent per policy
as well.
Policy statistics reporting will use policy index and the name in Graphite will
show as proxy-server.object.policy.<policy-index>.<verb>, etc.
Updated unit tests for per-policy stat reporting and added new unit tests for
invalid cases.
Updated documentation in the Administrator's Guide to reflect this new
aggregation.
Change-Id: Id70491e4833791a3fb8ff385953d69018514cd9c
The assert_() method is deprecated and can be safely replaced by assertTrue().
This patch makes sure that running the tests does not create undesired
warnings.
Change-Id: I0602ba39ef93263386644ee68088d5f65fcb4a71
wsgi.input is a binary stream (bytes), not a text stream (unicode).
* Replace StringIO with BytesIO for WSGI input
* Replace StringIO('') with StringIO() and replace WsgiStringIO('') with
WsgiStringIO(): an empty string is already the default value
Change-Id: I09c9527be2265a6847189aeeb74a17261ddc781a
* replace "from cStringIO import StringIO"
with "from six.moves import cStringIO as StringIO"
* replace "from StringIO import StringIO"
with "from six import StringIO"
* replace "import cStringIO" and "cStringIO.StringIO()"
with "from six import moves" and "moves.cStringIO()"
* replace "import StringIO" and "StringIO.StringIO()"
with "import six" and "six.StringIO()"
This patch was generated by the stringio operation of the sixer tool:
https://pypi.python.org/pypi/sixer
Change-Id: Iacba77fec3045f96773d1090c0bd48613729a561
The iteritems() of Python 2 dictionaries has been renamed to items() on
Python 3. According to a discussion on the openstack-dev mailing list,
the overhead of creating a temporary list using dict.items() on Python 2
is very low because most dictionaries are small:
http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Patch generated by the following command:
sed -i 's,iteritems,items,g' \
$(find swift -name "*.py") \
$(find test -name "*.py")
Change-Id: I6070bb6c684be76e8e77222a7d280ec6edd43496
The Python 2 next() method of iterators was renamed to __next__() on
Python 3. Use the builtin next() function instead which works on Python
2 and Python 3.
Change-Id: Ic948bc574b58f1d28c5c58e3985906dee17fa51d
PEP 333 says that the WSGI framework will call .close() on the
iterator returned by a WSGI application once it's done, provided such
a method exists. So, if our code wraps an iterator, then we have to
call .close() on it once we're done with it. proxy_logging wasn't.
Since WSGIContext gets it right, I looked at making proxy_logging use
WSGIContext. However, WSGIContext is all about forcing the first chunk
out of the iterator so that it can capture the final HTTP status and
headers; it doesn't help if you want to look at every chunk.
proxy_logging wants every chunk so it can count the bytes sent.
This didn't hurt anything in Swift, but pconstantine was complaining
in IRC that our failure to call .close() was goofing up some other
middleware he had.
Change-Id: Ic6ea0795ccef6cda2b5c6737697ef7d58eac9ab4
To make it easier for Swift operators to specify problematic devices,
a policy index will be recorded in log files of proxy and storage servers
for each user request which is related to storage policy.
This patch simply adds 'storage_policy_index' field in a log format.
If there is no specified policy index, '-' is output in this field.
Extra fix: Doc about the log line of storage nodes now properly reflects
'server_pid' field.
DocImpact
Change-Id: I7286ae85bcbcec73b5377dc115cbdb0f57d1b025
Implements: blueprint logging-policy-number
Based on comments from deployers at the Juno OpenStack summit,
limiting the default logged token length (to, by default, prevent
tokens from being fully logged) is a good idea.
Change-Id: I58980e85329d99de41f1c08f75e85973452317b1
As seen on #1174809, changes use of mutable types as default
arguments and defaults them within the method. Otherwise, those
defaults can be unexpectedly persisted with the function between
invocations and erupt into mass hysteria on the streets.
There was indeed a test (TestSimpleClient.test_get_with_retries)
that was erroneously relying on this behavior. Since previous tests
had populated their own instantiations with a token, this test only
passed because the modified headers dict from previous tests was
being overridden. As expected, with the mutable defaults fix in
SimpleClient, this test begain to fail since it never specified any
token, yet it has always passed anyway. This change also now provides
the expected token.
Change-Id: If95f11d259008517dab511e88acfe9731e5a99b5
Related-Bug: #1174809
Prior to this patch the proxy_logging middleware always prepared to
read the response and process it for access logging, where the
proxy_logging instance first to handle the response bytes marked the
request as "logged".
This meant that the proxy_logging instance immediately before the
proxy-server app (right most in the pipeline, if properly setup) would
always log the responses for all the client requests, regardless if
middleware to the left of it still had processing to do for the
response. This would break SLO, where the slo middleware passed along
the GET of the manifest, unchanged from the client, so that the right
most proxy logging middleware would log the sponse for just the
manifest, but then the slo middleware would fetch all the segments of
the manifest, feeding them out in the response to the client, the
request now marked as "logged" so that the left most proxy logging
middleware would not log it.
This patch inverts this behavior so that now the first proxy_logging
middleware instance to receive a request in the pipeline marks that
request as handling it. So now, the left most proxy_logging middleware
handles logging for all client reaquests, and the right most
proxy_logging middleware handles all other requests initiated from
within the pipeline to its left.
Closes bug: 1297438
Change-Id: Ia3561523db76c693e4e0b2c38f461544dfcee086
Currently all headers are logged if access_log_headers are enabled in
proxy_logging middleware. By adding the option access_log_headers_only
it is now possible to limit the logged headers to the given header values.
DocImpact
Change-Id: I0a1e40567c9eddc9bb00dd00373dc6eeb33d347c
Add accurate timestamps to the proxy-logging middleware log lines for
the start and end of a request. We use 9 digits of precision since on
some systems the clock resolution can be as fine as 1 ns.
The goal is to allow for log processing that can use the more accurate
timestamps to correlate requests, computing number of requests in
flight at a given historical point in time.
Change-Id: I61e8784b1c455d629f1299207fc4fc7e4a134814
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Also bring unit test coverage to 100% (well, at least every line is
reported as "covered").
Change-Id: I659d0c02008368897b1307a7a5c9aaba73b80588
Signed-off-by: Peter Portante <peter.portante@redhat.com>
The X-Auth-Token is sensitive data. If revealed to an unauthozied person,
they can now make requests against an account until the token expires.
This implementation maintains current behavior (i.e, the token
is logged). Implementers can choose to set reveal_sensitive_prefix
to (e.g.) 12 so only first 12 characters of the token are logged.
Or, set to 0 to replace the token with "...".
DocImpact
Part of bug #1004114
Change-Id: Iecefa843d8f9ef59b9dcf0860e7a4d0e186a6cb5
Added logging of the x-delete-at value for object PUTs and POSTs and
made the additional log field generic for other similar use cases.
Logging the x-delete-at is useful because otherwise you might never
know why an object "disappeared" just by looking at saved proxy logs.
You'd see a successful PUT of the object followed some time by a GET
that 404s. If you still have the other logs, you could track down
what happened; but, at least with our clusters, we keep proxy logs
forever but only keep the other logs for a short while.
Change-Id: Ida559abcf8dc5666e2a7eea250a67d1c4b28b1be
Previous logging made a mess of ipv6 addresses. This just makes the
colon quote-safe so it passes through unscathed. The logs will still
be backward compatible because unquote don't care:
$ python
>>> from urllib import quote, unquote
>>> quote('2001:db8:85a3:8d3:1319:8a2e:370:7348')
'2001%3Adb8%3A85a3%3A8d3%3A1319%3A8a2e%3A370%3A7348'
>>> unquote(quote('2001:db8:85a3:8d3:1319:8a2e:370:7348'))
'2001:db8:85a3:8d3:1319:8a2e:370:7348'
>>> quote('2001:db8:85a3:8d3:1319:8a2e:370:7348', '/:')
'2001:db8:85a3:8d3:1319:8a2e:370:7348'
>>> unquote(quote('2001:db8:85a3:8d3:1319:8a2e:370:7348', '/:'))
'2001:db8:85a3:8d3:1319:8a2e:370:7348'
Change-Id: Ia13a9bc8a320cde5c56034a7f0b645913428bf21
This was an oustanding TODO for StatsD Swift metrics. A new timing
metric is tracked for (only) GET requests for accounts, containers,
and objects:
proxy-server.<req_type>.GET.<status_int>.first-byte.timing
Also updated StatsD documentation in the Admin Guide to clarify that
timing metrics are sent in units of milliseconds.
Change-Id: I5bb781c06cefcb5280f4fb1112a526c029fe0c20
The proxy_logging middleware was asserting that the response contained
either a Content-Length header or a Transfer-Encoding header. If not,
it would either add one (if app_iter was a list) or blow up
(otherwise). This blowing up is observable on a GET request to a
manifest object that references more than
swift.common.constraints.CONTAINER_LISTING_LIMIT segments.
If a response makes it up to eventlet.wsgi without a Content-Length
header, then a "Transfer-Encoding: chunked" header is automatically
stuffed into the response by eventlet. Therefore, it's not an error
for a response to not have a Content-Length header, and proxy_logging
should just let it happen.
Fixes bug 1078113.
Change-Id: I3751a8ae14dc68bab546f2746b61267a5115e252
This change replaces WebOb with a mostly compatible local library,
swift.common.swob. Subtle changes to WebOb's API over the years have been a
huge headache. Swift doesn't even run on the current version.
There are a few incompatibilities to simplify the implementation/interface:
* It only implements the header properties we use. More can be easily added.
* Casts header values to str on assignment.
* Response classes ("HTTPNotFound") are no longer subclasses, but partials
on Response, so things like isinstance no longer work on them.
* Unlike newer webob versions, will never return unicode objects.
Change-Id: I76617a0903ee2286b25a821b3c935c86ff95233f
Removed many StatsD logging calls in proxy-server and added
swift-informant-style catch-all logging in the proxy-logger middleware.
Many errors previously rolled into the "proxy-server.<type>.errors"
counter will now appear broken down by response code and with timing
data at: "proxy-server.<type>.<verb>.<status>.timing". Also, bytes
transferred (sum of in + out) will be at:
"proxy-server.<type>.<verb>.<status>.xfer". The proxy-logging
middleware can get its StatsD config from standard vars in [DEFAULT] or
from access_log_statsd_* config vars in its config section.
Similarly to Swift Informant, request methods ("verbs") are filtered
using the new proxy-logging config var, "log_statsd_valid_http_methods"
which defaults to GET, HEAD, POST, PUT, DELETE, and COPY. Requests with
methods not in this list use "BAD_METHOD" for <verb> in the metric name.
To avoid user error, access_log_statsd_valid_http_methods is also
accepted.
Previously, proxy-server metrics used "Account", "Container", and
"Object" for the <type>, but these are now all lowercase.
Updated the admin guide's StatsD docs to reflect the above changes and
also include the "proxy-server.<type>.handoff_count" and
"proxy-server.<type>.handoff_all_count" metrics.
The proxy server now saves off the original req.method and proxy_logging
will use this if it can (both for request logging and as the "<verb>" in
the statsd timing metric). This fixes bug 1025433.
Removed some stale access_log_* related code in proxy/server.py. Also
removed the BaseApplication/Application distinction as it's no longer
necessary.
Fixed up the sample config files a bit (logging lines, mostly).
Fixed typo in SAIO development guide.
Got proxy_logging.py test coverage to 100%.
Fixed proxy_logging.py for PEP8 v1.3.2.
Enhanced test.unit.FakeLogger to track more calls to enable testing
StatsD metric calls.
Change-Id: I45d94cb76450be96d66fcfab56359bdfdc3a2576
The old use of Eventlet's posthook process meant that responses that
forgot to include content-length or transfer-encoding headers would
get one tacked on, if Eventlet could guess what was probably meant. I
added a bit of that logic into proxy-logging now as we saw some
errors resulting from this.
Fixes Bug #1012714
Change-Id: I671453eaf3704eab814ff12c4625ba7d749cc7ed
