.. This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode ====================================== Adding Prometheus fetcher ====================================== https://storyboard.openstack.org/#!/story/2005427 CloudKitty needs a fetcher describing how to get scopes (formerly tenants) from a Prometheus service. This spec aims at defining what is to be done and why. Everything in this document is open to discussion, equally on the associated storyboard and gerrit. Problem Description =================== In the present state, CloudKitty does not implement any method for discovering and fetching scope IDs from a Prometheus service and it needs one. Proposed Change =============== The fetcher will be configurable under a ``[fetcher_prometheus]`` section in ``cloudkitty.conf``. The configuration options will be the folllowing : * A ``metric`` option to provide a metric name from which we can infer the scope ID (default: none). * A ``scope_attribute`` option to provide the key referring the scope IDs (default: ``project_id``). * A ``filters`` option as a key-value dictionary to filter out the discovery query response with some metadata (optional). * A ``prometheus_url`` option to define the HTTP API endpoint URL of a Prometheus service. * A ``prometheus_user`` and a ``prometheus_password`` option to provide credentials for authentication (both defaults: none). * A ``cafile`` option to allow custom certificate authority file (default: none). * An ``insecure`` option to explicitly allow insecure HTTPS requests (default: false). The fetcher will request an aggregation on the specified ``metric`` using PromQL query and send it over HTTP to the specified Prometheus service API instant query endpoint. It will also group the result by the specified ``scope_attribute`` and filter out the response if some metadata ``filters`` have been specified in order to gather the scope_ids. Example ------- .. code-block:: ini # cloudkitty.conf # [...] [fetcher_prometheus] metric=container_memory_usage_bytes scope_attribute=namespace prometheus_url=https://prometheus-dn.tld/api/v1 prometheus_user=foobar prometheus_password=foobar insecure=true filters=label1:foo,label2:bar The PromQL request will look the following: ``max(container_memory_usage_bytes{label1="foo", label2="bar"}) by (namespace)`` The HTTP URL with the url-encoded PromQL will then look the following : ``http://prometheus-dn.tld/api/v1/query?query=max%28container_memory_usage_bytes%7Blabel1%3D%22foo%22%2C%20label2%3D%22bar%22%7D%29%20by%20%28namespace%29`` Data model impact ----------------- None REST API impact --------------- None Security impact --------------- None Notifications Impact -------------------- None Other end user impact --------------------- End users will benefit from new Prometheus fetcher to dynamically discover scope IDs. It will be particularly handy in conjunction with the Prometheus collector. Performance Impact ------------------ None Other deployer impact --------------------- None Developer impact ---------------- None Implementation ============== Assignee(s) ----------- Justin Ferrieu is assigned to work on the spec as well as to develop the evoked points. Primary assignee: jferrieu Other contributors: None Work Items ---------- * Implement Prometheus fetcher (with unit tests) * Add fetcher documentation Dependencies ============ None Testing ======= The proposed changes will be tested with unit tests. Documentation Impact ==================== We will add an entry detailing the Prometheus fetcher configuration in ``Administration Guide/Configuration Guide/Fetcher/Prometheus``. References ========== * Prometheus documentation: https://prometheus.io/docs/