304 lines
16 KiB
XML
304 lines
16 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<section xmlns="http://docbook.org/ns/docbook"
|
|
xmlns:xi="http://www.w3.org/2001/XInclude"
|
|
xmlns:xlink="http://www.w3.org/1999/xlink"
|
|
version="5.0"
|
|
xml:id="ch_introduction-to-openstack-object-storage-monitoring">
|
|
<!-- ... Based on a blog, should be replaced with original material... -->
|
|
<title>Object Storage monitoring</title>
|
|
<?dbhtml stop-chunking?>
|
|
<para>Excerpted from a blog post by <link
|
|
xlink:href="http://swiftstack.com/blog/2012/04/11/swift-monitoring-with-statsd"
|
|
>Darrell Bishop</link></para>
|
|
<para>An OpenStack Object Storage cluster is a collection of many
|
|
daemons that work together across many nodes. With so many
|
|
different components, you must be able to tell what is going
|
|
on inside the cluster. Tracking server-level meters like CPU
|
|
utilization, load, memory consumption, disk usage and
|
|
utilization, and so on is necessary, but not
|
|
sufficient.</para>
|
|
<para>What are different daemons are doing on each server? What is
|
|
the volume of object replication on node8? How long is it
|
|
taking? Are there errors? If so, when did they happen?</para>
|
|
<para>In such a complex ecosystem, you can use multiple approaches
|
|
to get the answers to these questions. This section describes
|
|
several approaches.</para>
|
|
<section xml:id="monitoring-swiftrecon">
|
|
<title>Swift Recon</title>
|
|
<para>The Swift Recon middleware (see <link
|
|
xlink:href="http://swift.openstack.org/admin_guide.html#cluster-telemetry-and-monitoring"
|
|
>http://swift.openstack.org/admin_guide.html#cluster-telemetry-and-monitoring</link>)
|
|
provides general machine statistics, such as load average,
|
|
socket statistics, <code>/proc/meminfo</code> contents,
|
|
and so on, as well as Swift-specific meters:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>The MD5 sum of each ring file.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>The most recent object replication time.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>Count of each type of quarantined file: Account,
|
|
container, or object.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>Count of "async_pendings" (deferred container
|
|
updates) on disk.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>Swift Recon is middleware that is installed in the
|
|
object servers pipeline and takes one required option: A
|
|
local cache directory. To track
|
|
<literal>async_pendings</literal>, you must set up an
|
|
additional cron job for each object server. You access
|
|
data by either sending HTTP requests directly to the
|
|
object server or using the <command>swift-recon</command>
|
|
command-line client.</para>
|
|
<para>There are some good Object Storage cluster statistics
|
|
but the general server meters overlap with existing
|
|
server monitoring systems. To get the Swift-specific
|
|
meters into a monitoring system, they must be polled.
|
|
Swift Recon essentially acts as a middleware meters
|
|
collector. The process that feeds meters to your
|
|
statistics system, such as <literal>collectd</literal> and
|
|
<literal>gmond</literal>, probably already runs on the
|
|
storage node. So, you can choose to either talk to Swift
|
|
Recon or collect the meters directly.</para>
|
|
</section>
|
|
<section xml:id="monitoring-swift-informant">
|
|
<title>Swift-Informant</title>
|
|
<para>Florian Hines developed the Swift-Informant middleware
|
|
(see <link
|
|
xlink:href="https://github.com/pandemicsyn/swift-informant"
|
|
>https://github.com/pandemicsyn/swift-informant</link>)
|
|
to get real-time visibility into Object Storage client
|
|
requests. It sits in the pipeline for the proxy server,
|
|
and after each request to the proxy server, sends three
|
|
meters to a StatsD server (see <link
|
|
xlink:href="http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/"
|
|
>http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/</link>):</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>A counter increment for a meter like
|
|
<code>obj.GET.200</code> or
|
|
<code>cont.PUT.404</code>.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>Timing data for a meter like
|
|
<code>acct.GET.200</code> or
|
|
<code>obj.GET.200</code>. [The README says the
|
|
meters look like
|
|
<code>duration.acct.GET.200</code>, but I do
|
|
not see the <literal>duration</literal> in the
|
|
code. I am not sure what the Etsy server does but
|
|
our StatsD server turns timing meters into five
|
|
derivative meters with new segments appended, so
|
|
it probably works as coded. The first meter turns
|
|
into <code>acct.GET.200.lower</code>,
|
|
<code>acct.GET.200.upper</code>,
|
|
<code>acct.GET.200.mean</code>,
|
|
<code>acct.GET.200.upper_90</code>, and
|
|
<code>acct.GET.200.count</code>].</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>A counter increase by the bytes transferred for
|
|
a meter like
|
|
<code>tfer.obj.PUT.201</code>.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>This is good for getting a feel for the quality of
|
|
service clients are experiencing with the timing meters,
|
|
as well as getting a feel for the volume of the various
|
|
permutations of request server type, command, and response
|
|
code. Swift-Informant also requires no change to core
|
|
Object Storage code because it is implemented as
|
|
middleware. However, it gives you no insight into the
|
|
workings of the cluster past the proxy server. If the
|
|
responsiveness of one storage node degrades, you can only
|
|
see that some of your requests are bad, either as high
|
|
latency or error status codes. You do not know exactly why
|
|
or where that request tried to go. Maybe the container
|
|
server in question was on a good node but the object
|
|
server was on a different, poorly-performing node.</para>
|
|
</section>
|
|
<section xml:id="monitoring-statsdlog">
|
|
<title>Statsdlog</title>
|
|
<para>Florian's <link
|
|
xlink:href="https://github.com/pandemicsyn/statsdlog"
|
|
>Statsdlog</link> project increments StatsD counters
|
|
based on logged events. Like Swift-Informant, it is also
|
|
non-intrusive, but statsdlog can track events from all
|
|
Object Storage daemons, not just proxy-server. The daemon
|
|
listens to a UDP stream of syslog messages and StatsD
|
|
counters are incremented when a log line matches a regular
|
|
expression. Meter names are mapped to regex match
|
|
patterns in a JSON file, allowing flexible configuration
|
|
of what meters are extracted from the log stream.</para>
|
|
<para>Currently, only the first matching regex triggers a
|
|
StatsD counter increment, and the counter is always
|
|
incremented by one. There is no way to increment a counter
|
|
by more than one or send timing data to StatsD based on
|
|
the log line content. The tool could be extended to handle
|
|
more meters for each line and data extraction, including
|
|
timing data. But a coupling would still exist between the
|
|
log textual format and the log parsing regexes, which
|
|
would themselves be more complex to support multiple
|
|
matches for each line and data extraction. Also, log
|
|
processing introduces a delay between the triggering event
|
|
and sending the data to StatsD. It would be preferable to
|
|
increment error counters where they occur and send timing
|
|
data as soon as it is known to avoid coupling between a
|
|
log string and a parsing regex and prevent a time delay
|
|
between events and sending data to StatsD.</para>
|
|
<para>The next section describes another method for gathering
|
|
Object Storage operational meters.</para>
|
|
</section>
|
|
<section xml:id="monitoring-statsD">
|
|
<title>Swift StatsD logging</title>
|
|
<para>StatsD (see <link
|
|
xlink:href="http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/"
|
|
>http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/</link>)
|
|
was designed for application code to be deeply
|
|
instrumented; meters are sent in real-time by the code
|
|
that just noticed or did something. The overhead of
|
|
sending a meter is extremely low: a <code>sendto</code>
|
|
of one UDP packet. If that overhead is still too high, the
|
|
StatsD client library can send only a random portion of
|
|
samples and StatsD approximates the actual number when
|
|
flushing meters upstream.</para>
|
|
<para>To avoid the problems inherent with middleware-based
|
|
monitoring and after-the-fact log processing, the sending
|
|
of StatsD meters is integrated into Object Storage
|
|
itself. The submitted change set (see <link
|
|
xlink:href="https://review.openstack.org/#change,6058"
|
|
>https://review.openstack.org/#change,6058</link>)
|
|
currently reports 124 meters across 15 Object Storage
|
|
daemons and the tempauth middleware. Details of the
|
|
meters tracked are in the <link
|
|
xlink:href="http://docs.openstack.org/developer/swift/admin_guide.html"
|
|
>Administrator's Guide</link>.</para>
|
|
<para>The sending of meters is integrated with the logging
|
|
framework. To enable, configure
|
|
<code>log_statsd_host</code> in the relevant config
|
|
file. You can also specify the port and a default sample
|
|
rate. The specified default sample rate is used unless a
|
|
specific call to a statsd logging method (see the list
|
|
below) overrides it. Currently, no logging calls override
|
|
the sample rate, but it is conceivable that some meters
|
|
may require accuracy (sample_rate == 1) while others may
|
|
not.</para>
|
|
<literallayout class="monospaced">[DEFAULT]
|
|
...
|
|
log_statsd_host = 127.0.0.1
|
|
log_statsd_port = 8125
|
|
log_statsd_default_sample_rate = 1</literallayout>
|
|
<para>Then the LogAdapter object returned by
|
|
<code>get_logger()</code>, usually stored in
|
|
<code>self.logger</code>, has these new
|
|
methods:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para><code>set_statsd_prefix(self, prefix)</code>
|
|
Sets the client library stat prefix value which
|
|
gets prefixed to every meter. The default prefix
|
|
is the "name" of the logger such as "object-server",
|
|
"container-auditor", and so on. This is currently used to
|
|
turn "proxy-server" into one of "proxy-server.Account",
|
|
"proxy-server.Container", or "proxy-server.Object"
|
|
as soon as the Controller object is determined and
|
|
instantiated for the request.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><code>update_stats(self, metric, amount,
|
|
sample_rate=1)</code> Increments the supplied
|
|
metric by the given amount. This is used when you
|
|
need to add or subtract more that one from a
|
|
counter, like incrementing "suffix.hashes" by the
|
|
number of computed hashes in the object
|
|
replicator.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><code>increment(self, metric,
|
|
sample_rate=1)</code> Increments the given
|
|
counter metric by one.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><code>decrement(self, metric,
|
|
sample_rate=1)</code> Lowers the given counter
|
|
metric by one.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><code>timing(self, metric, timing_ms,
|
|
sample_rate=1)</code> Record that the given
|
|
metric took the supplied number of
|
|
milliseconds.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><code>timing_since(self, metric, orig_time,
|
|
sample_rate=1)</code> Convenience method to
|
|
record a timing metric whose value is "now" minus
|
|
an existing timestamp.</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>Note that these logging methods may safely be called
|
|
anywhere you have a logger object. If StatsD logging has
|
|
not been configured, the methods are no-ops. This avoids
|
|
messy conditional logic each place a meter is recorded.
|
|
These example usages show the new logging methods:</para>
|
|
<programlisting language="bash"># swift/obj/replicator.py
|
|
def update(self, job):
|
|
# ...
|
|
begin = time.time()
|
|
try:
|
|
hashed, local_hash = tpool.execute(tpooled_get_hashes, job['path'],
|
|
do_listdir=(self.replication_count % 10) == 0,
|
|
reclaim_age=self.reclaim_age)
|
|
# See tpooled_get_hashes "Hack".
|
|
if isinstance(hashed, BaseException):
|
|
raise hashed
|
|
self.suffix_hash += hashed
|
|
self.logger.update_stats('suffix.hashes', hashed)
|
|
# ...
|
|
finally:
|
|
self.partition_times.append(time.time() - begin)
|
|
self.logger.timing_since('partition.update.timing', begin)</programlisting>
|
|
<programlisting language="bash"># swift/container/updater.py
|
|
def process_container(self, dbfile):
|
|
# ...
|
|
start_time = time.time()
|
|
# ...
|
|
for event in events:
|
|
if 200 <= event.wait() < 300:
|
|
successes += 1
|
|
else:
|
|
failures += 1
|
|
if successes > failures:
|
|
self.logger.increment('successes')
|
|
# ...
|
|
else:
|
|
self.logger.increment('failures')
|
|
# ...
|
|
# Only track timing data for attempted updates:
|
|
self.logger.timing_since('timing', start_time)
|
|
else:
|
|
self.logger.increment('no_changes')
|
|
self.no_changes += 1</programlisting>
|
|
<para>The development team of StatsD wanted to use the <link
|
|
xlink:href="https://github.com/sivy/py-statsd"
|
|
>pystatsd</link> client library (not to be confused
|
|
with a <link
|
|
xlink:href="https://github.com/sivy/py-statsd"
|
|
>similar-looking project</link> also hosted on
|
|
GitHub), but the released version on PyPI was missing two
|
|
desired features the latest version in GitHub had: the
|
|
ability to configure a meters prefix in the client object
|
|
and a convenience method for sending timing data between
|
|
"now" and a "start" timestamp you already have. So they
|
|
just implemented a simple StatsD client library from
|
|
scratch with the same interface. This has the nice fringe
|
|
benefit of not introducing another external library
|
|
dependency into Object Storage.</para>
|
|
</section>
|
|
</section>
|