Improved the documentation.
Change-Id: I9596367862fe3b49b5fc6da56a37123b5d617e68
This commit is contained in:
Patrick Petit
Simon Pasquier
parent
23cae82af0
commit
06fdb2dddc
28
README.md
28
README.md
|
|
@ -1,5 +1,5 @@
|
|||
Logging, Monitoring and Alerting collector Plugin for Fuel
|
||||
==========================================================
|
||||
Logging, Monitoring and Alerting (LMA) Collector Plugin for Fuel
|
||||
================================================================
|
||||
|
||||
|
||||
Overview
|
||||
|
|
@ -23,27 +23,26 @@ Requirements
|
|||
Limitations
|
||||
-----------
|
||||
|
||||
The plugin is only compatible with environments using Neutron.
|
||||
The plugin is only compatible with OpenStack environments deployed with Neutron
|
||||
for networking.
|
||||
|
||||
Installation Guide
|
||||
==================
|
||||
|
||||
|
||||
ElasticSearch configuration
|
||||
---------------------------
|
||||
Prior to installing the LMA Collector Plugin, you may want to install
|
||||
ElasticSearch and Kibana.
|
||||
To install ElasticSearch and Kibana automatically using Fuel, you can refer to the
|
||||
[ElasticSearch-Kibana Fuel Plugin
|
||||
](https://github.com/stackforge/fuel-plugin-elasticsearch-kibana).
|
||||
|
||||
To install and configure ElasticSearch, you can refer to the
|
||||
[ElasticSearch/Kibana
|
||||
plugin](https://github.com/stackforge/fuel-plugin-elasticsearch-kibana) for
|
||||
Fuel.
|
||||
|
||||
You can also install the ElasticSearch server outside of Fuel as long as it
|
||||
meets the plugin's requirements.
|
||||
You can install ElasticSearch and Kibana outside of Fuel as long as
|
||||
your installation meets the LMA Collector plugin's requirements defined above.
|
||||
|
||||
**LMA collector plugin** installation
|
||||
-------------------------------------
|
||||
|
||||
To install the LMA collector plugin, follow these steps:
|
||||
To install the LMA Collector plugin, follow these steps:
|
||||
|
||||
1. Download the plugin from the [Fuel Plugins
|
||||
Catalog](https://software.mirantis.com/download-mirantis-openstack-fuel-plug-ins/).
|
||||
|
|
@ -68,7 +67,8 @@ User Guide
|
|||
|
||||
1. Create a new environment with the Fuel UI wizard.
|
||||
2. Click on the Settings tab of the Fuel web UI.
|
||||
3. Scroll down the page, select the LMA collector plugin checkbox and fill-in
|
||||
3. Scroll down the page, select the LMA collector plugin checkbox and
|
||||
fill-in
|
||||
the required fields.
|
||||
|
||||
Exploring the data
|
||||
|
|
|
|||
|
|
@ -1,23 +1,8 @@
|
|||
.. _collector_service:
|
||||
.. _common_message_format:
|
||||
|
||||
|
||||
=================
|
||||
Collector service
|
||||
=================
|
||||
|
||||
The collector service leverages:
|
||||
|
||||
* `collectd <http://www.collectd.org/>`_ for gathering system and operational :ref:`metrics` which are then
|
||||
sent to Heka.
|
||||
|
||||
* `Heka <http://hekad.readthedocs.org/en/latest/index.html>`_ for parsing
|
||||
:ref:`logs`, collecting OpenStack :ref:`notifications` and receiving metrics
|
||||
from collectd.
|
||||
|
||||
.. _message_format:
|
||||
|
||||
Message format
|
||||
==============
|
||||
=====================
|
||||
Common Message Format
|
||||
=====================
|
||||
|
||||
Heka turns the incoming data into Heka messages [#]_ with a well-defined format
|
||||
which is described below.
|
||||
|
|
@ -40,7 +25,7 @@ which is described below.
|
|||
|
||||
* **Fields**, array of Field structures (see below).
|
||||
|
||||
Field format
|
||||
Field Format
|
||||
============
|
||||
|
||||
Every message (either originating from logs, metrics or notifications) is
|
||||
|
|
@ -1,24 +1,70 @@
|
|||
=============================================
|
||||
Welcome to the LMA collector's documentation!
|
||||
=============================================
|
||||
===========================================
|
||||
Welcome to the LMA Collector Documentation!
|
||||
===========================================
|
||||
|
||||
The Logging, Monitoring and Alerting (LMA) collector is a service that collects
|
||||
logs, metrics and notifications from OpenStack environments, processes them and
|
||||
sends the processed data to storage backends such as ElasticSearch and
|
||||
InfluxDB.
|
||||
The Logging, Monitoring and Alerting (LMA) Collector, that we will refer hereafter as the LMA Collector or just the Collector,
|
||||
is a **Fuel plugin** which gathers raw operational data from a variety of sources including log messages,
|
||||
`collectd <https://collectd.org/>`_, and the `OpenStack notifications <https://wiki.openstack.org/wiki/SystemUsageData>`_
|
||||
to be sent to external systems that will take action on them.
|
||||
|
||||
Contents:
|
||||
Overview
|
||||
=========
|
||||
|
||||
The goal of the LMA Collector is to capture all **raw operational data** that we think are relevant to **increase the operational visibility**
|
||||
of your OpenStack cloud.
|
||||
|
||||
To achieve that goal, the raw operational data are parsed and sanitised to be turned into an internal
|
||||
`Heka <https://github.com/mozilla-services/heka>`_ message representation that can
|
||||
be further processed and routed to external systems that will take action on them.
|
||||
Examples of external systems handled by the LMA Collector out-of-the-box include:
|
||||
|
||||
* `ElasticSearch <http://www.elasticsearch.org/>`_, a powerful open source search server based on Lucene and analytics
|
||||
engine that makes data like log messages and notifications easy to explore and correlate.
|
||||
* `InfluxDB <http://influxdb.com/>`_, an open-source and distributed time-series database to store system metrics.
|
||||
|
||||
By combining the Collector with ElasticSearch and `Kibana <http://www.elasticsearch.org/overview/kibana/>`_,
|
||||
the LMA Toolchain provides an end-to-end solution that delivers real-time insights about all events in your OpenStack cloud.
|
||||
This can very useful to detect errors and search for their root cause.
|
||||
|
||||
Likewise, combining the Collector with InfluxDB and its `Grafana’s <http://grafana.org/>`_ metrics analytics front-end,
|
||||
allows you to identify service failures, troubleshoot performance bottlenecks and plan the capacity needed to meet changing demands
|
||||
for your OpenStack cloud.
|
||||
|
||||
The LMA Collector can be viewed as a **pluggable processing and routing pipeline** for operational data.
|
||||
Its core constituants are :
|
||||
|
||||
* Collectd that is provided with a large collection of service checks and system stats plugins
|
||||
* Heka is an open-source stream processing software written in Go developed by Mozilla.
|
||||
Heka is the cornerstone component of the LMA Collector.
|
||||
* A collection of Heka plugins written in Lua to turn the raw operational data into structured
|
||||
messages that can be further analyzed and routed by other Heka plugins.
|
||||
|
||||
Lastly, the LMA Collector is designed to be both insightful and adaptable to your own specific environment.
|
||||
|
||||
For example, thanks to Heka's extensibility, it is quite easy to plug an external monitoring system like Nagios into the LMA Collector.
|
||||
This is simply done through enabling the Nagios output plugin and define the appropriate
|
||||
`message matcher <https://hekad.readthedocs.org/en/v0.9.0/message_matcher.html#message-matcher>`_ criteria
|
||||
for the category of messages you want to send out to Nagios. You should obviously not do that through hacking the
|
||||
configuration of the nodes running production but through modifying and reapplying the Puppet manifests that shipped with the Fuel plugin.
|
||||
We also encourage you to read the Heka `documentation <https://hekad.readthedocs.org/en/v0.9.0/index.html>`_ to get familiar with the technology.
|
||||
|
||||
The rest of this documents is organised in several chapters that will take you through a description of the internal message
|
||||
format used for each category of operational data that are handled by the Collector.
|
||||
|
||||
|
||||
Table of Contents
|
||||
=================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
collector
|
||||
common_message_format
|
||||
logs
|
||||
notifications
|
||||
metrics
|
||||
outputs
|
||||
|
||||
Indices and tables
|
||||
Indices and Tables
|
||||
==================
|
||||
|
||||
* :ref:`genindex`
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
.. _logs:
|
||||
|
||||
====
|
||||
Logs
|
||||
====
|
||||
============
|
||||
Log Messages
|
||||
============
|
||||
|
||||
The Heka collector service is configured to tail the following log files:
|
||||
|
||||
|
|
@ -21,10 +21,10 @@ The Heka collector service is configured to tail the following log files:
|
|||
|
||||
* OpenStack logs.
|
||||
|
||||
Log message format
|
||||
==================
|
||||
Log Messages Format
|
||||
===================
|
||||
|
||||
In addition to the common :ref:`message_format`, log-based messages have
|
||||
In addition to the common :ref:`common_message_format`, log-based messages have
|
||||
additional properties.
|
||||
|
||||
Attributes in **bold** are always present in the messages while attributes in
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
.. _metrics:
|
||||
|
||||
=======
|
||||
Metrics
|
||||
=======
|
||||
================
|
||||
Metric Messages
|
||||
================
|
||||
|
||||
Metrics are extracted from several sources:
|
||||
|
||||
|
|
@ -12,10 +12,10 @@ Metrics are extracted from several sources:
|
|||
|
||||
* OpenStack notifications processed by the collector service.
|
||||
|
||||
Metric message format
|
||||
=====================
|
||||
Metric Messages Format
|
||||
======================
|
||||
|
||||
In addition to the common :ref:`message_format`, metric messages have
|
||||
In addition to the common :ref:`common_message_format`, metric messages have
|
||||
additional properties.
|
||||
|
||||
Attributes in **bold** are always present in the messages while attributes in
|
||||
|
|
|
|||
|
|
@ -1,18 +1,18 @@
|
|||
.. _notifications:
|
||||
|
||||
=============
|
||||
Notifications
|
||||
=============
|
||||
=====================
|
||||
Notification Messages
|
||||
=====================
|
||||
|
||||
OpenStack services can be configured to send notifications on the message bus
|
||||
about the executing task or the state of the cloud resources [#]_. These
|
||||
notifications are received by the LMA collector service and turned into Heka
|
||||
messages.
|
||||
|
||||
Notification message format
|
||||
===========================
|
||||
Notification Messages Format
|
||||
============================
|
||||
|
||||
In addition to the common :ref:`message_format`, notification-based messages
|
||||
In addition to the common :ref:`common_message_format`, notification-based messages
|
||||
have additional properties.
|
||||
|
||||
Attributes in **bold** are always present in the messages while attributes in
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
.. _outputs:
|
||||
|
||||
=======
|
||||
Outputs
|
||||
=======
|
||||
==================
|
||||
Supported Ouptputs
|
||||
==================
|
||||
|
||||
The LMA collector can forward part or all of the processed Heka messages to any
|
||||
kind of external system, provided that the system supports a protocol-based
|
||||
|
|
|
|||
Loading…
Reference in New Issue