Cloudkitty InfluxDB Storage backend via Kolla-ansible

NOTE: Backported feature to Stein, as part of a fix for bug 1838641.

This proposal will add support to Kolla-Ansible for Cloudkitty
 InfluxDB storage system deployment. The feature of InfluxDB as the
 storage backend for Cloudkitty was created with the following commit
 https://github.com/openstack/cloudkitty/commit/
 c4758e78b49386145309a44623502f8095a2c7ee

Problem Description
===================

With the addition of support for InfluxDB in Cloudkitty, which is
achieving general availability via Stein release, we need a method to
easily configure/support this storage backend system via Kolla-ansible.

Kolla-ansible is already able to deploy and configure an InfluxDB
system. Therefore, this proposal will use the InfluxDB deployment
configured via Kolla-ansible to connect to CloudKitty and use it as a
storage backend.

If we do not provide a method for users (operators) to manage
Cloudkitty storage backend via Kolla-ansible, the user has to execute
these changes/configurations manually (or via some other set of
automated scripts), which creates distributed set of configuration
files, "configurations" scripts that have different versioning schemas
and life cycles.

Proposed Change
===============

Architecture
------------

We propose a flag that users can use to make Kolla-ansible configure
CloudKitty to use InfluxDB as the storage backend system. When
enabling this flag, Kolla-ansible will also enable the deployment of
the InfluxDB via Kolla-ansible automatically.

CloudKitty will be configured accordingly to [1] and [2]. We will also
externalize the "retention_policy", "use_ssl", and "insecure", to
allow fine granular configurations to operators. All of these
configurations will only be used when configured; therefore, when they
are not set, the default value/behavior defined in Cloudkitty will be
used. Moreover, when we configure "use_ssl" to "true", the user will
be able to set "cafile" to a custom trusted CA file. Again, if these
variables are not set, the default ones in Cloudkitty will be used.

Implementation
--------------
We need to introduce a new variable called
`cloudkitty_storage_backend`. Valid options are `sqlalchemy` or
`influxdb`. The default value in Kolla-ansible is `sqlalchemy` for
backward compatibility. Then, the first step is to change the
definition for the following variable:
`/ansible/group_vars/all.yml:enable_influxdb: "{{ enable_monasca |
bool }}"`

We also need to enable InfluxDB when CloudKitty is configured to use
it as the storage backend. Afterwards, we need to create tasks in
CloudKitty configurations to create the InfluxDB schema and configure
the configuration files accordingly.

Alternatives
------------
The alternative would be to execute the configurations manually or
handle it via a different set of scripts and configurations files,
which can become cumbersome with time.

Security Impact
---------------
None identified by the author of this spec

Notifications Impact
--------------------
Operators that are already deploying CloudKitty with InfluxDB as
storage backend would need to convert their configurations to
Kolla-ansible (if they wish to adopt Kolla-ansible to execute these
tasks).

Also, deployments (OpenStack environments) that were created with
Cloudkitty using storage v1 will need to migrate all of their data to
V2 before enabling InfluxDB as the storage system.

Other End User Impact
---------------------
None.

Performance Impact
------------------
None.

Other Deployer Impact
---------------------
New configuration options will be available for CloudKitty.
* cloudkitty_storage_backend
* cloudkitty_influxdb_retention_policy
* cloudkitty_influxdb_use_ssl
* cloudkitty_influxdb_cafile
* cloudkitty_influxdb_insecure_connections
* cloudkitty_influxdb_name

Developer Impact
----------------
None

Implementation
==============

Assignee
--------
* `Rafael Weingärtner <rafaelweingartne>`

Work Items
----------
 * Extend InfluxDB "enable/disable" variable
 * Add new tasks to configure Cloudkitty accordingly to these new
 variables that are presented above
 * Write documentation and release notes

Dependencies
============
None

Documentation Impact
====================
New documentation for the feature.

References
==========
[1] `https://docs.openstack.org/cloudkitty/latest/admin/configuration/storage.html#influxdb-v2`
[2] `https://docs.openstack.org/cloudkitty/latest/admin/configuration/collector.html#metric-collection`

Change-Id: I65670cb827f8ca5f8529e1786ece635fe44475b0
Signed-off-by: Rafael Weingärtner <rafael@apache.org>
Related-Bug: #1838641
(cherry picked from commit 97cb30cdd8)

ansible/group_vars/all.yml

@@ -569,7 +569,7 @@ enable_horizon_vitrage: "{{ enable_vitrage | bool }}"
 enable_horizon_watcher: "{{ enable_watcher | bool }}"
 enable_horizon_zun: "{{ enable_zun | bool }}"
 enable_hyperv: "no"
-enable_influxdb: "{{ enable_monasca | bool }}"
+enable_influxdb: "{{ enable_monasca | bool or (cloudkitty_storage_backend is defined and cloudkitty_storage_backend == 'influxdb') }}"
 enable_ironic: "no"
 enable_ironic_ipxe: "no"
 enable_ironic_neutron_agent: "{{ enable_neutron | bool and enable_ironic | bool }}"
@@ -1031,3 +1031,8 @@ prometheus_elasticsearch_exporter_interval: "60s"
 # Vitrage
 ############
 enable_vitrage_prometheus_datasource: "{{ enable_prometheus | bool }}"
+
+####################
+# InfluxDB options
+####################
+influxdb_address: "{{ kolla_internal_fqdn }}"

ansible/roles/cloudkitty/defaults/main.yml

@@ -90,3 +90,29 @@ cloudkitty_git_repository: "{{ kolla_dev_repos_git }}/{{ project_name }}"
 cloudkitty_dev_repos_pull: "{{ kolla_dev_repos_pull }}"
 cloudkitty_dev_mode: "{{ kolla_dev_mode }}"
 cloudkitty_source_version: "{{ kolla_source_version }}"
+
+####################
+# Storage backend
+####################
+# Valid options are 'sqlalchemy' or 'influxdb'. The default value in
+# Kolla-ansible is 'sqlalchemy' (for backwards compatibility).
+# When cloudkitty_storage_backend is define with "influxdb" as its value,
+# we also trigger the deployment of Influxdb via Kolla-ansible.
+# Also, when using 'influxdb' as the backend, we trigger the configuration/use
+# of Cloudkitty storage backend version 2.
+cloudkitty_storage_backend: "sqlalchemy"
+
+# InfluxDB retention policy to use (defaults to autogen).
+# cloudkitty_influxdb_retention_policy: "autogen"
+
+# Set to true to use SSL for InfluxDB connections.
+# cloudkitty_influxdb_use_ssl: false
+
+# Path of the CA certificate to trust for HTTPS connections.
+# cloudkitty_influxdb_cafile: "/full/qualified/path/to/CAs/certificates"
+
+# Set to true to authorize insecure HTTPS connections to InfluxDB.
+# This means, HTTPS connections without validating the certificate used by InfluxDB
+# cloudkitty_influxdb_insecure_connections: false
+
+cloudkitty_influxdb_name: "cloudkitty"

ansible/roles/cloudkitty/tasks/bootstrap.yml

@@ -33,4 +33,14 @@
   when:
     - not use_preconfigured_databases | bool
 
+- name: Creating Cloudkitty influxdb database
+  become: true
+  kolla_toolbox:
+    module_name: influxdb_database
+    module_args:
+      hostname: "{{ influxdb_address }}"
+      port: "{{ influxdb_http_port }}"
+      database_name: "{{ cloudkitty_influxdb_name }}"
+  when: cloudkitty_storage_backend == 'influxdb'
+
 - include_tasks: bootstrap_service.yml

ansible/roles/cloudkitty/templates/cloudkitty.conf.j2

@@ -55,3 +55,36 @@ port = {{ cloudkitty_api_port }}
 
 [orchestrator]
 coordination_url = mysql://{{ cloudkitty_database_user }}:{{ cloudkitty_database_password }}@{{ cloudkitty_database_address }}/{{ cloudkitty_database_name }}
+
+[storage]
+backend = {{ cloudkitty_storage_backend }}
+{% if cloudkitty_storage_backend == 'influxdb' %}
+version = 2
+{% endif %}
+{% if cloudkitty_storage_backend == 'sqlalchemy' %}
+version = 1
+{% endif %}
+
+{% if cloudkitty_storage_backend == 'influxdb' %}
+[storage_influx]
+database = {{ cloudkitty_influxdb_name }}
+host = {{ influxdb_address }}
+port = {{ influxdb_http_port }}
+
+{% if cloudkitty_influxdb_retention_policy is defined %}
+retention_policy = {{ cloudkitty_influxdb_retention_policy }}
+{% endif %}
+
+{% if cloudkitty_influxdb_use_ssl is defined %}
+use_ssl = {{ cloudkitty_influxdb_use_ssl }}
+{% endif %}
+
+{% if cloudkitty_influxdb_insecure_connections is defined %}
+insecure = {{ cloudkitty_influxdb_insecure_connections }}
+{% endif %}
+
+{% if cloudkitty_influxdb_cafile is defined %}
+cafile = {{ cloudkitty_influxdb_cafile }}
+{% endif %}
+
+{% endif %}

ansible/roles/monasca/defaults/main.yml

@@ -170,7 +170,7 @@ monasca_database_address: "{{ database_address }}"
 monasca_database_port: "{{ database_port }}"
 
 monasca_influxdb_name: "monasca"
-monasca_influxdb_address: "{{ kolla_internal_fqdn }}"
+monasca_influxdb_address: "{{ influxdb_address }}"
 monasca_influxdb_http_port: "{{ influxdb_http_port }}"
 monasca_influxdb_retention_policy:
   name: 'monasca_metrics'

releasenotes/notes/cloudKitty-v2-influxdb-07cb8b0051ac9ea0.yaml

@@ -0,0 +1,4 @@
+---
+features:
+  - |
+    Add support to Kolla-Ansible for Cloudkitty InfluxDB storage system deployment.