Browse Source

Counter-inspection for scaling controls

A spec to add counter-inspection to the gate to prevent scaling
problems.

Change-Id: I3ad971cfa33789ade733e8fb92ff51d487078638
changes/83/230183/8
Clint Byrum 6 years ago
parent
commit
b5dd6cfa2a
  1. 2
      doc/source/index.rst
  2. 103
      specs/devstack/counter-inspection.rst

2
doc/source/index.rst

@ -20,7 +20,7 @@ DevStack Project Specifications
:glob:
:maxdepth: 2
.. specs/devstack/*
specs/devstack/*
Implemented Specifications
--------------------------

103
specs/devstack/counter-inspection.rst

@ -0,0 +1,103 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
..
========================================================
Inspect counters for data collection and gating purposes
========================================================
Problem description
===================
OpenStack projects vary on their impact to the underlying infrastructure
that they rely on greatly. This is hard to measure without going to a
full scale deployment, but we should be able to measure the impact by
inspecting counters already maintained by the system.
Proposed change
===============
* Create a new "OpenStack QA Tools" repository to house small tools
written in python for purposes such as this.
* Create a tool, `os-collect-counters`, which collects relevant counters
from any backends it can reach using its own configuration and
outputs a JSON mapping with all counters. Includes ability to delta
with a previous run to allow showing impact on the counters for a
given time window.
* Initial counters will be at a minimum a set of MySQL counters (such
as Innodb_bytes_written) and published messages from the RabbitMQ
management interface, summarized by scope that can be inferred
from each queue name.
* Leverage existing subunit/statsd/graphite infrastructure to record results of
several tests in the devstack gate.
* For each run, the JSON from `os-collect-counters` will be added as an
attachment to the subunit stream.
* The counters in the attachment will be fed into statsd/graphite to
allow establishing trends.
* This will be facilitated by adding attachment storage plugins to
subunit2sql. The plugin used for OpenStack gate jobs will be
specific to OpenStack's infrastructure and look for the specifically
named attachment to push into statsd/graphite.
* Monitor counters for stable indicators and identify the best predictors of
problems.
* Once stable counters are identified, create an upper bounds for
these counters to help prevent new changes in the system from
accidentally introducing an inordinate amount of cost into the tested
code paths.
Since there are daunting social issues around failing gate tests
on global collisions, warnings and bugs about said warnings are
likely the only reasonable outcome we can achieve. It will take a
considerable amount of community agreement to make these limits hard.
Implementation
==============
A new python repo, `os-performance-tools`_, has already been created, and
will be maintained for the purposes of extracting and pushing counters
into statsd/graphite. This will include a subunit2sql attachments plugin
and code to output the counters as a subunit attachment.
.. _`os-performance-tools`: https://review.openstack.org/#/c/244428/
Assignee(s)
-----------
Primary assignee:
* Clint Byrum <clint@fewbar.com>
Milestones
----------
Target Milestone for completion:
Mitaka-2
Work Items
----------
* Create tools to emit counters from a running installation
* Modify devstack gate job output to include counters
* Add subunit2sql attachment plugin to subunit2sql workers
to push counters to infra graphite
* Analyze data for stable counters and useful trends
* Add upper bounds check to devstack gate
Dependencies
============
References
==========
Loading…
Cancel
Save