Add collector documentation

This adds some documentation about collectors and metric collection.

Work items:

* Detail common and collector-specific options in cloudkitty.conf

* Detail common and collector-specific options in metrics.yml

* Explain how data is collected

Change-Id: I9fa330524160d8bc4ab33047c360ce8120ee3a3d
Story: 2004179
Task: 29064
This commit is contained in:
Luka Peschke 2019-01-24 16:29:33 +01:00
parent 1f7946ed38
commit 347992d8e9
2 changed files with 244 additions and 0 deletions

View File

@ -0,0 +1,243 @@
=========================
Collector configuration
=========================
Common options
==============
Options common to all collectors are specified in the ``[collect]`` section of
the configuration file. The following options are available:
* ``collector``: Defaults to ``gnocchi``. The name of the collector to load.
Must be one of [``gnocchi``, ``monasca``, ``prometheus``].
* ``period``: Default to 3600. Duration (in seconds) of the collect period.
* ``wait_periods``: Defaults to 2. Periods to wait before the current
timestamp. This is done to avoid missing some data that hasn't been
retrieved by the data source yet.
* ``metrics_conf``: Defaults to ``/etc/cloudkitty/metrics.yml``. Path of the
metric collection configuration file. See "Metric collection" section below
for details.
* ``scope_key``: Defaults to ``project_id``. Key at which the scope can be
found. The scope defines how data collection is split between the processors.
Collector-specific options
==========================
Collector-specific options must be specified in the
``collector_{collector_name}`` section of ``cloudkitty.conf``.
Gnocchi
-------
Section: ``collector_gnocchi``.
* ``gnocchi_auth_type``: Defaults to ``keystone``. Defines what authentication
method should be used by the gnocchi collector. Must be one of ``basic``
(for gnocchi basic authentication) or ``keystone`` (for classic keystone
authentication). If ``keystone`` is chosen, credentials can be specified
in a section pointed at by the ``auth_section`` parameter.
* ``gnocchi_user``: For gnocchi basic authentication only. The gnocchi user.
* ``gnocchi_endpoint``: For gnocchi basic authentication only. The gnocchi
endpoint.
* ``interface``: Defaults to ``internalURL``. For keystone authentication only.
The interface to use for keystone URL discovery.
* ``region_name``: Defaults to ``RegionOne``. For keystone authentication only.
Region name.
Monasca
-------
Section: ``collector_monasca``.
* ``interface``: Defaults to ``internal``. The interface to use for keystone
URL discovery.
* ``monasca_service_name``: Defaults to ``monasca``. Name of the Monasca
service in Keystone.
Metric collection
=================
Metric collection is highly configurable in cloudkitty. In order to keep the
main configuration file as clean as possible, metric collection is configured
in a yaml file. The path to this file defaults to
``/etc/cloudkitty/metrics.yml``, but can be configured:
.. code-block:: ini
[collect]
metrics_conf = /my/custom/path.yml
Minimal Configuration
---------------------
This config file has the following format:
.. code-block:: yaml
metrics: # top-level key
metric_one: # metric name
unit: squirrel
groupby: # attributes by which metrics should be grouped
- id
metadata: # additional attributes to retrieve
- color
At the top level of the file, a ``metrics`` key is required. It contains a dict
of metrics to collect, each key of the dict being the name of a metric as it is
called in the datasource (``volume.size`` or ``image.size`` for example).
For each metric, the following attributes are required:
* ``unit``: the unit in which the metric will be stored after conversion. This
is just an indication for humans and has absolutely no impact on metric
collection, conversion or rating.
* ``groupby``: A list of attributes by which metrics should be grouped
on collection. These will allow to re-group data when it is retrieved
through the v2 API. A typical usecase would be to group data by ID,
project ID, domain ID and user ID on collection, but only by user ID
on retrieval.
* ``metadata``: A list of additional attributes that should be collected for
the given metric. These can be used for rating rules and will appear in
monthly reports. However, it is not possible to group on these attributes.
If you need to group on a ``metadata`` attribute, move it to the ``groupby``
list.
.. note:: The ``scope_key`` is automatically added to ``groupby``.
Optional parameters
-------------------
Unit conversion
~~~~~~~~~~~~~~~
If you need to convert the collected qty (from MiB to GiB for example), it can
be done with the ``factor`` and ``offset`` options. ``factor`` defaults to 1
and ``offset`` to 0. These options are used to calculate the final result with
the following formula: ``qty = collected_qty * factor + offset``.
.. note:: ``factor`` and ``offset`` can be floats, integers or fractions.
Example from the default configuration file, conversion from B to MiB for the
``image.size`` metric:
.. code-block:: yaml
metrics:
image.size:
groupby:
- id
metadata:
- disk_format
unit: MiB # Final unit
factor: 1/1048576 # Dividing by 1024 * 1024
.. note::
Here we don't add anything, so there is no need to specify ``offset``.
Quantity mutation
~~~~~~~~~~~~~~~~~
It is also possible to mutate the collected qty with the ``mutate`` option.
Four values are accepted for this parameter:
* ``NONE``: This is the default. The collected data is not modifed.
* ``CEIL``: The qty is rounded up to the closest integer.
* ``FLOOR``: The qty is rounded down to the closest integer.
* ``NUMBOOL``: If the collected qty equals 0, leave it at 0. Else, set it to 1.
.. warning::
Quantity mutation is done **after** conversion. Example::
factor: 10
mutate: CEIL
In consequence, the configuration above will convert 9.9 to 99
(9.9 -> 99 -> 99) and not to 100 (9.9 -> 10 -> 100)
A typical usecase for the ``NUMBOOL`` conversion would be instance uptime
collection with the gnocchi collector: In order to know if an instance is
running or paused, you can use the ``cpu`` metric. This metric is at
0 when the instance is paused. Thus, the qty is mutated to a ``NUMBOOL``
because the ``cpu`` metric always represents one instance. Rating rules are
then defined based on the instance metadata. Example:
.. code-block:: yaml
metrics:
cpu:
unit: instance
mutate: NUMBOOL
groupby:
- id
metadata:
- flavor_id
Display name
~~~~~~~~~~~~
Sometimes, you'll want to use another name for a metric, either to shorten it a
bit or to make it more explicit. For example, the ``cpu`` metric from the
previous section could be called ``instance``. That's what the ``alt_name``
option does:
.. code-block:: yaml
metrics:
cpu:
unit: instance
alt_name: instance
mutate: NUMBOOL
groupby:
- id
metadata:
- flavor_id
Collector-specific configuration
--------------------------------
Some collectors require extra options. These must be specified through the
``extra_args`` option. Some options have defaults, other must be systematically
specified. The extra args for each collector are detailed below.
Gnocchi
~~~~~~~
* ``resource_type``: No default value. The resource type the current metric is
bound to.
* ``resource_key``: Defaults to ``id``. The attribute containing the unique
resource identifier. This is an advanced option, do not modify it
unless you know what you're doing.
* ``aggregation_method``: Defaults to ``max``. The aggregation method to use
when retrieving measures in gnocchi. Must be one of ``min``, ``max``,
``mean``.
Monasca
~~~~~~~
* ``resource_key``: Defaults to ``resource_id``. The attribute containing the
unique resource identifier. This is an advanced option, do not modify it
unless you know what you're doing.
* ``aggregation_method``: Defaults to ``max``. The aggregation method to use
when retrieving measures in gnocchi. Must be one of ``min``, ``max``,
``mean``.

View File

@ -6,4 +6,5 @@ Configuration Guide
:glob:
configuration
collector
policy