Spec: Centralized logging support
bp tripleo-opstools-centralized-logging Change-Id: I4a26517732acb55f4e3a4f7347354116f257493b
This commit is contained in:
Martin Mágr
Lars Kellogg-Stedman
parent
736282b309
commit
f2cd57398d
|
|
@ -0,0 +1,147 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
========================================
|
||||
Enable deployment of centralized logging
|
||||
========================================
|
||||
|
||||
https://blueprints.launchpad.net/tripleo/+spec/tripleo-opstools-centralized-logging
|
||||
|
||||
TripleO should be deploying with an out-of-the-box centralized logging
|
||||
solution to serve the overcloud.
|
||||
|
||||
Problem Description
|
||||
===================
|
||||
|
||||
With a complex distributed system like OpenStack, identifying and
|
||||
diagnosing a problem may require tracking a transaction across many
|
||||
different systems and many different logfiles. In the absence of a
|
||||
centralized logging solution, this process is frustrating to both new
|
||||
and experienced operators and can make even simple problems hard to
|
||||
diagnose.
|
||||
|
||||
Proposed Change
|
||||
===============
|
||||
|
||||
We will deploy the Fluentd_ service in log collecting mode as a
|
||||
composable service on all nodes in the overcloud stack when configured
|
||||
to do so by the environment. Each composable service will have its
|
||||
own fluentd source configuration.
|
||||
|
||||
.. _fluentd: http://www.fluentd.org/
|
||||
|
||||
To receive these messages, we will deploy a centralized logging system
|
||||
running Kibana_, Elasticsearch_ and Fluentd on dedicated nodes to
|
||||
provide log aggregation and analysis. This will be deployed in a
|
||||
dedicated Heat stack that is separate from the overcloud stack using
|
||||
composable roles.
|
||||
|
||||
.. _kibana: https://www.elastic.co/products/kibana
|
||||
.. _elasticsearch: https://www.elastic.co/
|
||||
|
||||
We will also support sending messages to an external Fluentd
|
||||
instance not deployed by tripleo.
|
||||
|
||||
Summary of use cases
|
||||
--------------------
|
||||
|
||||
1. Elasticsearch, Kibana and Fluentd log relay/transformer deployed as
|
||||
a separate Heat stack in the overcloud stack; Fluentd log
|
||||
collector deployed on each overcloud node
|
||||
|
||||
2. ElasticSearch, Kibana and Fluentd log relay/transformer deployed in
|
||||
external infrastructure; Fluentd log collector deployed on each
|
||||
overcloud node
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
None
|
||||
|
||||
Security Impact
|
||||
---------------
|
||||
|
||||
Data collected from the logs of OpenStack services can contain
|
||||
sensitive information:
|
||||
|
||||
- Communication between the
|
||||
fluentd agent and the log aggregator should be protected with SSL.
|
||||
|
||||
- Access to the Kibana UI must have at least basic HTTP
|
||||
authentication, and client access should be via SSL.
|
||||
|
||||
- ElasticSearch should only allow collections over ``localhost``.
|
||||
|
||||
Other End User Impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
Additional resources will be required for running Fluentd on overcloud
|
||||
nodes. Log traffic from the overcloud nodes to the log aggregator
|
||||
will consume some bandwidth.
|
||||
|
||||
Other Deployer Impact
|
||||
---------------------
|
||||
|
||||
- Fluentd will be deployed on all overcloud nodes.
|
||||
- New parameters for configuring Fluentd collector.
|
||||
- New parameters for configuring log collector (Fluentd,
|
||||
ElasticSearch, and Kibana)
|
||||
|
||||
Developer Impact
|
||||
----------------
|
||||
|
||||
Support for the new node type should be implemented for tripleo-quickstart.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Martin Mágr <mmagr@redhat.com>
|
||||
Lars Kellogg-Stedman <lars@redhat.com>
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
- puppet-tripleo profile for fluentd service
|
||||
- tripleo-heat-templates composable role for FluentD collector deployment
|
||||
- tripleo-heat-templates composable role for FluentD aggregator deployment
|
||||
- tripleo-heat-templates composable role for ElasticSearch deployment
|
||||
- tripleo-heat-templates composable role for Kibana deployment
|
||||
- Support for logging node in tripleo-quickstart
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
- Puppet module for Fluentd: `konstantin-fluentd` [1]
|
||||
- Puppet module for ElasticSearch `elasticsearch-elasticsearch` [2]
|
||||
- Puppet module for Kibana (tbd)
|
||||
- CentOS Opstools SIG package repository
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Fluentd client deployment will be tested by current TripleO CI as soon as
|
||||
the patch is merged. Because the centralized logging features will not
|
||||
be enabled by default we may need to introduce specific tests for
|
||||
these features.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
Process of creating new node type and new options will have to be documented.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
[1] https://forge.puppet.com/srf/fluentd
|
||||
[2] https://forge.puppet.com/elasticsearch/elasticsearch
|
||||
Loading…
Reference in New Issue