- Move statsd client into it's own module
- Move all logging functions into their own module
- Move all config functions into their own module
- Move all helper functions into their own module
Partial-Bug: #2015274
Change-Id: Ic4b5005e3efffa8dba17d91a41e46d5c68533f9a
Since we fake out all the greenthread stuff to run in the main thread,
we can (sometimes?) find that a transaction ID has already been set,
leading to failures in test_bad_request_app_logging like
AssertionError: b'X-Trans-Id: test-trans-id' not found
in b'X-Trans-Id: tx...'
By resetting the logger's txn_id, we're assured that our mock will be
run and the expected transaction ID will be used.
Change-Id: I465eed5372a2a5e591f80a09676f4b7f091cd444
The NOTICE log level (25) is added to logging when utils.monkey_patch
is executed. Depending on test execution order, this may not have
happened when some unit tests run, so LogRecords at NOTICE level may
have the default level name of 'Level 25'. FakeLogger doesn't have a
key in its lines dict for 'Level 25', so those tests would blow up.
Closes-Bug: #2034254
Related-Change: I6f35ad41414898fb7dc5da422f524eb52ff2940f
Change-Id: I2945ae725f97ee56e9c7ad580168ec8091f2049a
It adds another layer of indirection and state for the sake of labeling;
longer term it'll be easier to be explicit at the point of emission.
Related-Change: I0522b1953722ca96021a0002cf93432b973ce626
Change-Id: Ieebafb19c3fa60334aff2914ab1ae70b8f140342
Currently we simply mock calls in the FakeLogger for calls statsd calls,
and there are also some helper methods for counting and collating
metrics that were called. This Fakelogger is overloaded and doesn't
simulate the real world.
In real life we use a Statsdclient that is attached to the logger.
We've been in the situation where unit tests pass but the statsd client
stacktraces because we don't actually fake the statsdclient based off
the real one and let it's use its internal logic.
This patch creates a new FakeStatsdClient that is based off the real
one, this can then be used (like the real statsd client) and attached to
the FakeLogger.
There is quite a bit of churn in tests to make this work, because we now
have to looking into the fake statsd client to check the faked calls
made.
The FakeStatsdClient does everything the real one does, except overrides
the _send method and socket creation so no actual statsd metrics are
emitted.
Change-Id: I9cdf395e85ab559c2b67b0617f898ad2d6a870d4
Send some existing misplaced objects stats to statsd as well as the
recon file:
misplaced_placed
misplaced_unplaced
misplaced_success
misplaced_failure
Change-Id: I22ed81cce5d8c6fcb9028fcc0403c6e5e5110df8
Previously, objects updates that could not be sent immediately due to
per-container/bucket ratelimiting [1] would be skipped and re-tried
during the next updater cycle. There could potentially be a period of
time at the end of a cycle when the updater slept, having completed a
sweep of the on-disk async pending files, despite having skipped
updates during the cycle. Skipped updates would then be read from disk
again during the next cycle.
With this change the updater will defer skipped updates to an
in-memory queue (up to a configurable maximum number) until the sweep
of async pending files has completed, and then trickle out deferred
updates until the cycle's interval expires. This increases the useful
work done in the current cycle and reduces the amount of repeated disk
IO during the next cycle.
The deferrals queue is bounded in size and will evict least recently
read updates in order to accept more recently read updates. This
reduces the probablility that a deferred update has been made obsolete
by newer on-disk async pending files while waiting in the deferrals
queue.
The deferrals queue is implemented as a collection of per-bucket
queues so that updates can be drained from the queues in the order
that buckets cease to be ratelimited.
[1] Related-Change: Idef25cd6026b02c1b5c10a9816c8c6cbe505e7ed
Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com>
Co-Authored-By: Matthew Oliver <matt@oliver.net.au>
Change-Id: I95e58df9f15c5f9d552b8f4c4989a474f52262f4
Previously a DebugLogger was used when probe tests ran 'custom
daemons', which provided a means to inspect captured logs but logged
to console. This patch replaces that DebugLogger with a 'normal' swift
logger that is adapted to also capture logs and support log
inspection.
Change-Id: I25da3aa81018c5de7b63e5584ac6a9dbb73243db
Move DebugLogger and associated classes to its own module under test
so that it can be imported (for example in probe tests) without
requiring all the dependencies in test/unit/__init__.py.
Change-Id: I0ea3c26e54d91f27159805a45e49ad7f8f0e0431
