Remove readme info duplicates with the documentation

Improved overall quality of the overview chapter
Updated toolchain map

Change-Id: I5d629819aa294c11003ef238a07c07595ad3fd50
This commit is contained in:
Patrick Petit 2015-11-15 10:05:55 -08:00 committed by Swann Croiset
parent 40a3c1b6d1
commit 77c8090d3a
5 changed files with 134 additions and 121 deletions

123
README.md
View File

@ -1,107 +1,62 @@
Logging, Monitoring and Alerting (LMA) Collector Plugin for Fuel
================================================================
The Logging, Monitoring and Alerting (LMA) Collector Plugin for Fuel
====================================================================
The Logging, Monitoring & Alerting (LMA) *Collector* is a kind of advanced
monitoring agent that should be installed on each of the OpenStack nodes
you want to monitor.
The Collector is a key component of the LMA Toolchain since it is
individually responsible for supporting all the sensing, measurement,
collection, analysis and computation functions for the node it is running on.
The LMA Collector Plugin for Fuel is used to install and configure
the LMA Collector in OpenStack environments deployed by [Fuel](
https://wiki.openstack.org/wiki/Fuel).
A wealth of operational data are collected from a variety of sources including
the log files, collectd and RabbitMQ for the OpenStack notifications.
The Collector, which runs on the active controller of the control plane cluster, is
called the *Aggregator* because it performs additional aggregation and
multivariate correlation functions to compute service healthiness metrics at
the cluster level.
An important function of the Collector is to sanitize and transform the ingested
raw operational data into internal messages which uses the Heka
message structure. This structure is used to match, filter and route certain
types of messages to plugins written in Lua which perform the analysis and
computation functions of the toolchain.
The LMA Collector is an advanced monitoring agent solution that should
be installed on each of the OpenStack nodes you want to monitor.
Its main building blocks are:
* collectd which is bundled with a collection of standard and purpose-built
plugins for OpenStack.
* Heka which is the swiss army knife we use for data processing.
* A collection of Heka plugins written in Lua.
There are three types of Lua plugins running in the LMA Collector / Aggregator:
* The input plugins to collect, decode, and sanitize the operational data that
are transformed into internal messages which in turn are injected into the
Heka pipeline.
* The filter plugins to execute the alarms, the anomaly detection logic
and the correlation functions.
* The output plugins to encode and transmit the messages to external systems like
Elasticsearch, InfluxDB or Nagios where the information is persisted or further processed.
The output of the Collector / Aggregator is of four kinds:
* The logs and notifications that are sent to Elasticsearch for indexing.
Elasticsearch combined with Kibana provides an insightful log analytic dashboards.
* The metrics which are sent to InfluxDB.
InfluxDB combined with Grafana provides insightful time-series analytic dashboards.
* The health status checks that are sent to Nagios (or through SMTP) for all the OpenStack
services and clusters of nodes.
* The annotation messages that are sent to InfluxDB. The annotation messages contain
information about what caused a cluster of services or a cluster of nodes to change a state.
The annotation messages provide root cause analysis hints whenever possible.
The annotation messages are also used to construct the alert notifications sent via SMTP.
Please check the [LMA Collector Plugin for Fuel
](http://fuel-plugin-lma-collector.readthedocs.org/en/latest/index.html)
documentation for additional details.
Please check the [LMA Collector Plugin Overview](
http://fuel-plugin-lma-collector.readthedocs.org/en/latest/user/overview.html)
section of the plugin's documentation for additional details.
Release Notes
-------------
**0.8.0**
* Support for alerting in two different modes:
* Email notifications.
* Integration with Nagios.
* Upgrade to InfluxDB 0.9.4.
* Upgrade to Grafana 2.1
* Management of the LMA collector service by Pacemaker on the controller nodes
for improved reliability.
* Monitoring of the LMA toolchain components (self-monitoring).
* Support for configurable alarm rules in the Collector.
**0.7.0**
* Initial release of the plugin. This is a beta version.
The release notes of the LMA Collector Plugin are provided in the
[Release Notes](
http://fuel-plugin-lma-collector.readthedocs.org/en/latest/user/releases.html)
section of the plugin documentation.
Requirements
------------
The plugin's requirements are defined in the [LMA Collector Plugin Documentation](
The LMA Collector Plugin requirements are defined in the [Requirements](
http://fuel-plugin-lma-collector.readthedocs.org/en/latest/user/overview.html#requirements)
section of the plugin documentation.
Known issues
------------
No known issues so far.
The LMA Toolchain's related issues are listed on [Launchpad](
https://bugs.launchpad.net/lma-toolchain).
Limitations
-----------
The plugin is only compatible with OpenStack environments deployed with Neutron
for networking.
The LMA Collector Plugin limitations are described in the [Limitations](
http://fuel-plugin-lma-collector.readthedocs.org/en/latest/user/overview.html#limitations)
section of the plugin documentation.
Installation Guide
------------------
Installation
------------
Please follow the installation instructions of the [LMA Collector Plugin Documentation](
The installation instructions of the LMA Collector Plugin are provided
in the [Installation](
http://fuel-plugin-lma-collector.readthedocs.org/en/latest/user/installation.html#installation)
section of the plugin documentation.
User Guide
----------
How to configure and use the plugin is detailed in the [The LMA Collector Plugin Documentation](
http://fuel-plugin-lma-collector.readthedocs.org/en/latest/user/guide.html#user-guide)
Instructions for how to configure the plugin and the alarms
are provided in the [Configuration Guide](
http://fuel-plugin-lma-collector.readthedocs.org/en/latest/user/configuration.html)
and [Alarms Configuration Guide](
http://fuel-plugin-lma-collector.readthedocs.org/en/latest/user/alarms.html)
of the plugin documentation.
Communication
-------------
@ -111,21 +66,21 @@ with the members of the project.
Emails should be sent to `openstack-dev@lists.openstack.org` with the subject
prefixed by `[fuel][plugins][lma]`.
Reporting Bugs
--------------
Bugs should be filled on the [Launchpad fuel-plugins project](
https://bugs.launchpad.net/fuel-plugins) (not GitHub) with the tag `lma`.
Bugs should be filed against the [LMA Toolchain project](
https://launchpad.net/lma-toolchain) on Launchpad (not Github!).
Contributing
------------
If you would like to contribute to the development of this Fuel plugin you must
follow the [OpenStack development workflow](
http://docs.openstack.org/infra/manual/developers.html#development-workflow).
If you would like to contribute to the development of this plugin,
you must follow the [OpenStack development workflow](
http://docs.openstack.org/infra/manual/developers.html#development-workflow)
instructions.
Patch reviews take place on the [OpenStack gerrit](
Patch reviews take place on the [OpenStack Gerrit](
https://review.openstack.org/#/q/status:open+project:openstack/fuel-plugin-lma-collector,n,z)
system.

10
doc/source/dev/tests.rst Normal file
View File

@ -0,0 +1,10 @@
Running tests
-------------
You need to have `tox` and `bundler` installed for running the tests.
Quickstart for Ubuntu Trusty::
apt-get install tox ruby
gem install bundler
tox

View File

@ -28,6 +28,7 @@ Developer documentation
dev/notifications
dev/metrics
dev/outputs
dev/tests
Indices and Tables
==================

View File

@ -91,14 +91,3 @@ if you need further instructions about how to build the Fuel Plugin Builder.
7. Now that you have created the RPM file, you can install the plugin using the `fuel plugins --install` command::
[root@fuel ~] fuel plugins --install ./fuel-plugin-lma_collector/lma_collector-0.8-0.8.0-1.noarch.rpm
Running tests
-------------
You need to have `tox` and `bundler` installed for running the tests.
Quickstart for Ubuntu Trusty::
apt-get install tox ruby
gem install bundler
tox

View File

@ -3,36 +3,94 @@
Overview
========
The LMA Collector is best described as a pluggable message processing and
routing pipeline. Its core components are:
The Logging, Monitoring & Alerting (LMA) Collector is an advanced
monitoring agent solution that should be installed on each of the
OpenStack nodes you want to monitor.
* `collectd <https://collectd.org/>`_ that is bundled with a collection of
The LMA Collector (or Collector for short) is a key component
of the `LMA Toolchain project <https://launchpad.net/lma-toolchain>`_
as shown in the figure below::
+=====================================================+
|| LMA Collector Plugin ||
|| ||
|| measurement / collection / analysis / persistence ||
+=====================================================+
| | |
| | |
| | |
................................... | | | ................................
| InfluxDB Grafana Plugin | | | | | Elasticsearch Kibana Plugin |
| |<--' | '-->| |
| metrics / annotations analytics | | | logs / notifications analytics |
'...................................' | '................................'
v
................................
| Infrastructure Alerting Plugin |
| |
| alerting / escalation |
'................................'
Each Collector is individually responsible for supporting the sensing,
measurement, collection, analysis and alarm functions for the node
it is running on.
A wealth of operational data are collected from a variety of sources including
log files, collectd and RabbitMQ for the OpenStack notifications.
.. note:: The Collector which runs on the active controller of the control plane
cluster, is called the *Aggregator* because it performs additional
aggregation and multivariate correlation functions to compute service
healthiness metrics at the cluster level.
A primary function of the Collector is to sanitise and transform the ingested
raw operational data into internal messages using the
`Heka message structure <https://hekad.readthedocs.org/en/stable/message/index.html>`_.
This message structure is used within the Collector's framework to match, filter
and route messages to plugins written in
`Lua <http://www.lua.org/>`_ which perform various
data analysis and computation functions.
As such, the Collector may also be described as a pluggable framework
for operational data stream processing and routing.
Its main building blocks are:
* `collectd <https://collectd.org/>`_ which is bundled with a collection of
monitoring plugins. Many of them are purpose-built for OpenStack.
* `Heka <https://github.com/mozilla-services/heka>`_ that is written in Go is
the cornerstone component of the Collector. Heka supports out-of-the-box a
number of input and output plugins that allows to integrate the Collector
with a number of external systems using their native protocol like
Elasticsearch, InfluxDB, Nagios, SMTP, Whisper, Kafka, AMQP and Carbon to
name a few.
* A collection of Heka plugins written in Lua to decode and process the
acquired operational data that are then sent to external systems for further
processing.
* `Heka <https://github.com/mozilla-services/heka>`_ (a golang data processing
*swiss army knife* by Mozilla) which is the cornerstone component of the Collector.
Heka supports out-of-the-box a number of input and output plugins that allows
the Collector to integrate with a number of external systems' native
protocol like Elasticsearch, InfluxDB, Nagios, SMTP, Whisper, Kafka, AMQP and
Carbon to name a few.
* A collection of Heka plugins written in Lua to decode, process and encode the
operational data.
The Collector is installed on every OpenStack node by the Fuel plugin. Its role
is to gather all operational data that we think are relevant to increase the
operational visibility of an OpenStack environment. Those data are collected
from a variety of sources including log files, collectd, and RabbitMQ for the
OpenStack notifications. The Collector which runs on the active controller node
of the Mirantis OpenStack HA cluster is called the Aggregator because it
performs additional aggregation and correlation functions to detect service
availability problems.
There are three types of Lua plugins running in the Collector:
Another important function of the Collector is to transform raw data into
internal message representations that are based on the Heka message structure
which can be matched and filtered to, for example, create new metrics that can
be routed to a specific destination using simple matching rules. For example,
messages extracted from the OpenStack notifications are filtered and processed
to create new metrics like the creation time of an instance or volume.
* The input plugins which collect, sanitize and transform the raw
data into an internal message representation which is injected into the
Heka pipeline for further processing.
* The filter plugins which execute the analysis and correlation functions.
* The output plugins which encode and transmit the messages to external
systems like Elasticsearch, InfluxDB or Nagios where the data can
be further processed and persisted.
The output of the Collector / Aggregator is of four kinds:
* The logs and notifications which are sent to Elasticsearch for indexing.
Elasticsearch combined with Kibana provides insightful log analytics.
* The metrics which are sent to InfluxDB.
InfluxDB combined with Grafana provides insightful time-series analytics.
* The health status metrics for the OpenStack services which are sent to Nagios
(or via SMTP) for alerting and escalation purposes.
* The annotation messages which are sent to InfluxDB. The annotation messages contain
information about what caused a service cluster or node cluster to change a state.
The annotation messages provide root cause analysis hints whenever possible.
The annotation messages are also used to construct the alert notifications that are
sent via SMTP or to Nagios.
.. _plugin_requirements: