.. This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode ====================================== Reworking Prometheus Collector ====================================== CloudKitty's Prometheus collector needs some rework. This spec aims at defining what is to be done and why. Everything in this document is open to discussion, equally on the associated storyboard and gerrit. Problem Description =================== In the present state, Prometheus collector is not consistent with the other collectors present in CloudKitty in terms of configuration and operations. * The end user fully defines, on its own, the PromQL query to be executed in the ``metrics.yml`` file in a ``query`` option under ``extra_args`` subkey. Permitting the end user such complexity for building its request at the expense of genericity is not relevant regarding the use cases covered by CloudKitty. * The collector does not use native PromQL ``groupby`` capabilities. The process is done post Prometheus service response causing an overhead operational cost. * The collector ignores ``scope_key`` configuration option, disallowing scoping requests however it is the standard behaviour to be found in other collectors of the project. Besides, it is a major concern for orchestration. Let's say you deploy several CK processors. Without scoping, you will end up with data duplications likely to result in false reports. The collector also needs the following new features: * It needs authentication configuration options to dial into some secured Prometheus service. * It needs a configuration option to take in account a custom certificate authority file in order to authorize HTTPS requests signed with custom certificate. * It needs an insecure configuration option to explicitly trust HTTPS requests signed with an untrusted certificate. These features will be useful to end users for configuring CloudKitty to effectively connect to a Prometheus service. Proposed Change =============== In ``cloudkitty.conf``, under ``[collector_prometheus]`` section: * Add a ``prometheus_user`` and a ``prometheus_password`` option to provide credentials for authentication (both defaults: none). * Add a ``cafile`` option to allow custom certificate authority file (default: none). * Add an ``insecure`` option to explicitly allow insecure HTTPS requests (default: false). Possible deprecation of ``extra_args/query`` in ``metrics.yml`` for a defined metric (see Alternatives_ section). In ``metrics.yml``, for a defined metric, add an ``aggregation_method`` option under ``extra_args`` subkey to define which aggregation method to use collecting metric data. For instance: .. code-block:: YAML # metrics.yml metrics: # [...] volume_size: unit: GiB groupby: - id - project_id metadata: - volume_type extra_args: aggregation_method: max Valid aggregation methods will be the following : * ``avg``: the average value of all points in the time window. * ``min``: the minimum value of all points in the time window. * ``max``: the maximum value of all points in the time window. * ``sum``: the sum of all values in the time window. * ``count``: the count of all values in the time window. * ``stddev``: the population standard deviation of the values in the time window. * ``stdvar``: the population standard variance of the values in the time window. .. note:: Time window is computed from the ``period`` configuration option under ``[collect]`` section in ``cloudkitty.conf``. Other changes at this stage are mostly focused on the query building process happening inside the Prometheus collector: * ``[collect]/scope_key`` option value will be added to filter every query with the corresponding ``scope_id``. * The collector will now use the ``groupby`` capabilities offered by PromQL using the syntax ``by (scope_key, groupby, metadata)`` instead of deferring it to the collector implementation. Example ------- For the following ``cloudkitty.conf`` : .. code-block:: INI # Some options have been omitted for the sake of clarity [collect] collector = prometheus scope_key = namespace period = 3600 [collector_prometheus] prometheus_url = http://prometheus-dn.tld/api/v1 And the following ``metrics.yml`` : .. code-block:: YAML metrics: # [...] # Let's say we want to rate the following metric container_memory_usage_bytes: unit: GiB groupby: - container_id metadata: - volume_type extra_args: aggregation_method: max The PromQL request will look the following : ``max(max_over_time(container_memory_usage_bytes{namespace="foobar"}[3600s])) by (namespace, container_id, volume_type)`` Start and end timestamps will be computed from the ``period`` option. Let's say that we are starting from ``January 30th, 2019 at 1pm (timestamp: 1548853200)``. Then the end timestamp will be ``1548853200 + period (= 1548856800)``. The HTTP URL with the url-encoded PromQL will then look the following : ``http://prometheus-dn.tld/api/v1/query?query=max%28max_over_time%28container_memory_usage_bytes%7Bnamespace%3D%22foobar%22%7D%5B3600s%5D%29%29%20by%20%28namespace%2C%20container_id%2C%20volume_type%29&time=158856800`` Alternatives ------------ Instead of deprecation for the ``extra_args/query`` option defined for a metric in ``metrics.yml`` file, we could remove it. Data model impact ----------------- None REST API impact --------------- None Security impact --------------- None Notifications Impact -------------------- None Other end user impact --------------------- End users will probably have to reconfigure their metrics definition if they are using the Prometheus collector in the present state. However, after these changes are made, they will have less overhead configuration to do in order to switch from a collector to another, due to the consistency improvement in the way metrics are defined in ``metrics.yml`` for Prometheus collector. They will also benefit from the added configuration options for the ``[collector_prometheus]`` section in ``cloudkitty.conf``. Performance Impact ------------------ None Other deployer impact --------------------- None Developer impact ---------------- None Implementation ============== Assignee(s) ----------- Justin Ferrieu is assigned to work on the spec as well as to develop the evoked points. Primary assignee: jferrieu Other contributors: None Work Items ---------- * Change configuration schema, query building process and query response formatting process accordingly inside Prometheus collector. * Add HTTPS and authentication support. Dependencies ============ None Testing ======= The proposed changes will be tested with unit tests. Documentation Impact ==================== We will add an entry for detailing the Prometheus collector configuration in ``Administration Guide/Configuration Guide/Collector/Prometheus``. References ========== * Prometheus documentation: https://prometheus.io/docs/