..
 This work is licensed under a Creative Commons Attribution 3.0 Unported
 License.

 http://creativecommons.org/licenses/by/3.0/legalcode

============================================
Enable deployment of availability monitoring
============================================

https://blueprints.launchpad.net/tripleo/+spec/tripleo-opstools-availability-monitoring

TripleO should be deploying out-of-the-box availability monitoring solution
to serve the overcloud.

Problem Description
===================

Currently there is no such feature implemented except for possibility to deploy
sensu-server, sensu-api and uchiwa (Sensu dashboard) services in the undercloud
stack. Without sensu-client services deployed on overcloud nodes this piece
of code is useless. Due to potential of high resource consumption it is also
reasonable to remove current undercloud code to avoid possible problems
when high number of overcloud nodes is being deployed.

Instead sensu-server, sensu-api and uchiwa should be deployed on the separate
node(s) whether it is on the undercloud level or on the overcloud level.
And so sensu-client deployment support should be flexible enough to enable
connection to external monitoring infrastructure or with Sensu stack deployed
on the dedicated overcloud node.

Summary of use cases:

1. sensu-server, sensu-api and uchiwa deployed in external infrastructure;
sensu-client deployed on each overcloud node
2. sensu-server, sensu-api and uchiwa deployed as a separate Heat stack in
the overcloud stack; sensu-client deployed on each overcloud node

Proposed Change
===============

Overview
--------

The sensu-client service will be deployed as a composable service on
the overcloud stack when it is explicitly stated via environment file.
Sensu checks will have to be configured as subscription checks (see [0]
for details). Each composable service will have it's own subscription string,
which will ensure that checks defined on Sensu server node (wherever it lives)
are run on the correct overcloud nodes.

There will be implemented a possibility to deploy sensu-server, sensu-api
and uchiwa services on a stand alone node deployed by the undercloud.
This standalone node will have a dedicated purpose for monitoring
(not only for availability monitoring services, but in future also for
centralized logging services or performance monitoring services)

The monitoring node will be deployed as a separate Heat stack to the overcloud
stack using Puppet and composable roles for required services.

Alternatives
------------

None

Security Impact
---------------

Additional service (sensu-client) will be installed on all overcloud nodes.
These services will have open connection to RabbitMQ instance running
on monitoring node and are used to execute commands (checks) on the overcloud
nodes. Check definition will live on the monitoring node.

Other End User Impact
---------------------

None

Performance Impact
------------------

We might consider deploying separate RabbitMQ and Redis for monitoring purposes
if we want to avoid influencing OpenStack deployment in the overcloud.

Other Deployer Impact
---------------------

* Sensu clients will be deployed by default on all overcloud nodes except the monitoring node.
* New Sensu common parameters:

    * MonitoringRabbitHost

        * RabbitMQ host Sensu has to connect to

    * MonitoringRabbitPort

        * RabbitMQ port Sensu has to connect to

    * MonitoringRabbitUseSSL

        * Whether Sensu should connect to RabbitMQ using SSL

    * MonitoringRabbitPassword

        * RabbitMQ password used for Sensu to connect

    * MonitoringRabbitUserName

        * RabbitMQ username used for Sensu to connect

    * MonitoringRabbitVhost

        * RabbitMQ vhost used for monitoring purposes.

* New Sensu server/API parameters

    * MonitoringRedisHost

        * Redis host Sensu has to connect to

    * MonitoringRedisPassword

        * Redis password used for Sensu to connect

    * MonitoringChecks:

        * Full definition (for all subscriptions) of checks performed by Sensu

* New parameters for subscription strings for each composable service:

    * For example for service nova-compute MonitoringSubscriptionNovaCompute, which will default to 'overcloud-nova-compute'


Developer Impact
----------------

Support for new node type should be implemented for tripleo-quickstart.

Implementation
==============

Assignee(s)
-----------

Martin Mágr <mmagr@redhat.com>

Work Items
----------

* puppet-tripleo profile for Sensu services
* puppet-tripleo profile for Uchiwa service
* tripleo-heat-templates composable service for sensu-client deployment
* tripleo-heat-templates composable service for sensu-server deployment
* tripleo-heat-templates composable service for sensu-api deployment
* tripleo-heat-templates composable service for uchiwa deployment
* Support for monitoring node in tripleo-quickstart
* Revert patch(es) implementing Sensu support in instack-undercloud

Dependencies
============

* Puppet module for Sensu services: sensu-puppet [1]
* Puppet module for Uchiwa: puppet-uchiwa [2]
* CentOS Opstools SIG repo [3]

Testing
=======

Sensu client deployment will be tested by current TripleO CI as soon as
the patch is merged, as it will be deployed by default.

We should consider creating CI job for deploying overcloud with monitoring
node to test the rest of the monitoring components.

Documentation Impact
====================

Process of creating new node type and new options will have to be documented.

References
==========

[0] https://sensuapp.org/docs/latest/reference/checks.html#subscription-checks
[1] https://github.com/sensu/sensu-puppet
[2] https://github.com/Yelp/puppet-uchiwa
[3] https://wiki.centos.org/SpecialInterestGroup/OpsTools