From 97cb30cdd8041210a09167b8458a08f985892f1f Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Rafael=20Weing=C3=A4rtner?= <rafael@apache.org>
Date: Fri, 10 May 2019 13:40:46 -0300
Subject: [PATCH] Cloudkitty InfluxDB Storage backend via Kolla-ansible
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This proposal will add support to Kolla-Ansible for Cloudkitty
 InfluxDB storage system deployment. The feature of InfluxDB as the
 storage backend for Cloudkitty was created with the following commit
 https://github.com/openstack/cloudkitty/commit/
 c4758e78b49386145309a44623502f8095a2c7ee

Problem Description
===================

With the addition of support for InfluxDB in Cloudkitty, which is
achieving general availability via Stein release, we need a method to
easily configure/support this storage backend system via Kolla-ansible.

Kolla-ansible is already able to deploy and configure an InfluxDB
system. Therefore, this proposal will use the InfluxDB deployment
configured via Kolla-ansible to connect to CloudKitty and use it as a
storage backend.

If we do not provide a method for users (operators) to manage
Cloudkitty storage backend via Kolla-ansible, the user has to execute
these changes/configurations manually (or via some other set of
automated scripts), which creates distributed set of configuration
files, "configurations" scripts that have different versioning schemas
and life cycles.

Proposed Change
===============

Architecture
------------

We propose a flag that users can use to make Kolla-ansible configure
CloudKitty to use InfluxDB as the storage backend system. When
enabling this flag, Kolla-ansible will also enable the deployment of
the InfluxDB via Kolla-ansible automatically.

CloudKitty will be configured accordingly to [1] and [2]. We will also
externalize the "retention_policy", "use_ssl", and "insecure", to
allow fine granular configurations to operators. All of these
configurations will only be used when configured; therefore, when they
are not set, the default value/behavior defined in Cloudkitty will be
used. Moreover, when we configure "use_ssl" to "true", the user will
be able to set "cafile" to a custom trusted CA file. Again, if these
variables are not set, the default ones in Cloudkitty will be used.

Implementation
--------------
We need to introduce a new variable called
`cloudkitty_storage_backend`. Valid options are `sqlalchemy` or
`influxdb`. The default value in Kolla-ansible is `sqlalchemy` for
backward compatibility. Then, the first step is to change the
definition for the following variable:
`/ansible/group_vars/all.yml:enable_influxdb: "{{ enable_monasca |
bool }}"`

We also need to enable InfluxDB when CloudKitty is configured to use
it as the storage backend. Afterwards, we need to create tasks in
CloudKitty configurations to create the InfluxDB schema and configure
the configuration files accordingly.

Alternatives
------------
The alternative would be to execute the configurations manually or
handle it via a different set of scripts and configurations files,
which can become cumbersome with time.

Security Impact
---------------
None identified by the author of this spec

Notifications Impact
--------------------
Operators that are already deploying CloudKitty with InfluxDB as
storage backend would need to convert their configurations to
Kolla-ansible (if they wish to adopt Kolla-ansible to execute these
tasks).

Also, deployments (OpenStack environments) that were created with
Cloudkitty using storage v1 will need to migrate all of their data to
V2 before enabling InfluxDB as the storage system.

Other End User Impact
---------------------
None.

Performance Impact
------------------
None.

Other Deployer Impact
---------------------
New configuration options will be available for CloudKitty.
* cloudkitty_storage_backend
* cloudkitty_influxdb_retention_policy
* cloudkitty_influxdb_use_ssl
* cloudkitty_influxdb_cafile
* cloudkitty_influxdb_insecure_connections
* cloudkitty_influxdb_name

Developer Impact
----------------
None

Implementation
==============

Assignee
--------
* `Rafael Weingärtner <rafaelweingartne>`

Work Items
----------
 * Extend InfluxDB "enable/disable" variable
 * Add new tasks to configure Cloudkitty accordingly to these new
 variables that are presented above
 * Write documentation and release notes

Dependencies
============
None

Documentation Impact
====================
New documentation for the feature.

References
==========
[1] `https://docs.openstack.org/cloudkitty/latest/admin/configuration/storage.html#influxdb-v2`
[2] `https://docs.openstack.org/cloudkitty/latest/admin/configuration/collector.html#metric-collection`

Change-Id: I65670cb827f8ca5f8529e1786ece635fe44475b0
Signed-off-by: Rafael Weingärtner <rafael@apache.org>
---
 ansible/group_vars/all.yml                    |  7 +++-
 ansible/roles/cloudkitty/defaults/main.yml    | 26 +++++++++++++++
 ansible/roles/cloudkitty/tasks/bootstrap.yml  | 10 ++++++
 .../cloudkitty/templates/cloudkitty.conf.j2   | 32 +++++++++++++++++++
 ansible/roles/monasca/defaults/main.yml       |  2 +-
 ...oudKitty-v2-influxdb-07cb8b0051ac9ea0.yaml |  4 +++
 6 files changed, 79 insertions(+), 2 deletions(-)
 create mode 100644 releasenotes/notes/cloudKitty-v2-influxdb-07cb8b0051ac9ea0.yaml

diff --git a/ansible/group_vars/all.yml b/ansible/group_vars/all.yml
index 9ae3db5665..9b5922f3e6 100644
--- a/ansible/group_vars/all.yml
+++ b/ansible/group_vars/all.yml
@@ -573,7 +573,7 @@ enable_horizon_vitrage: "{{ enable_vitrage | bool }}"
 enable_horizon_watcher: "{{ enable_watcher | bool }}"
 enable_horizon_zun: "{{ enable_zun | bool }}"
 enable_hyperv: "no"
-enable_influxdb: "{{ enable_monasca | bool }}"
+enable_influxdb: "{{ enable_monasca | bool or (cloudkitty_storage_backend is defined and cloudkitty_storage_backend == 'influxdb') }}"
 enable_ironic: "no"
 enable_ironic_ipxe: "no"
 enable_ironic_neutron_agent: "{{ enable_neutron | bool and enable_ironic | bool }}"
@@ -1034,3 +1034,8 @@ prometheus_elasticsearch_exporter_interval: "60s"
 # Vitrage
 ############
 enable_vitrage_prometheus_datasource: "{{ enable_prometheus | bool }}"
+
+####################
+# InfluxDB options
+####################
+influxdb_address: "{{ kolla_internal_fqdn }}"
diff --git a/ansible/roles/cloudkitty/defaults/main.yml b/ansible/roles/cloudkitty/defaults/main.yml
index 6e1bad1ac1..342c7c4360 100644
--- a/ansible/roles/cloudkitty/defaults/main.yml
+++ b/ansible/roles/cloudkitty/defaults/main.yml
@@ -96,3 +96,29 @@ cloudkitty_dev_repos_pull: "{{ kolla_dev_repos_pull }}"
 cloudkitty_dev_mode: "{{ kolla_dev_mode }}"
 cloudkitty_source_version: "{{ kolla_source_version }}"
 cloudkitty_custom_metrics_yaml_file: "metrics.yml"
+
+####################
+# Storage backend
+####################
+# Valid options are 'sqlalchemy' or 'influxdb'. The default value in
+# Kolla-ansible is 'sqlalchemy' (for backwards compatibility).
+# When cloudkitty_storage_backend is define with "influxdb" as its value,
+# we also trigger the deployment of Influxdb via Kolla-ansible.
+# Also, when using 'influxdb' as the backend, we trigger the configuration/use
+# of Cloudkitty storage backend version 2.
+cloudkitty_storage_backend: "sqlalchemy"
+
+# InfluxDB retention policy to use (defaults to autogen).
+# cloudkitty_influxdb_retention_policy: "autogen"
+
+# Set to true to use SSL for InfluxDB connections.
+# cloudkitty_influxdb_use_ssl: false
+
+# Path of the CA certificate to trust for HTTPS connections.
+# cloudkitty_influxdb_cafile: "/full/qualified/path/to/CAs/certificates"
+
+# Set to true to authorize insecure HTTPS connections to InfluxDB.
+# This means, HTTPS connections without validating the certificate used by InfluxDB
+# cloudkitty_influxdb_insecure_connections: false
+
+cloudkitty_influxdb_name: "cloudkitty"
diff --git a/ansible/roles/cloudkitty/tasks/bootstrap.yml b/ansible/roles/cloudkitty/tasks/bootstrap.yml
index 1da6991465..154ff75ca7 100644
--- a/ansible/roles/cloudkitty/tasks/bootstrap.yml
+++ b/ansible/roles/cloudkitty/tasks/bootstrap.yml
@@ -34,5 +34,15 @@
   when:
     - not use_preconfigured_databases | bool
 
+- name: Creating Cloudkitty influxdb database
+  become: true
+  kolla_toolbox:
+    module_name: influxdb_database
+    module_args:
+      hostname: "{{ influxdb_address }}"
+      port: "{{ influxdb_http_port }}"
+      database_name: "{{ cloudkitty_influxdb_name }}"
+  when: cloudkitty_storage_backend == 'influxdb'
+
 - include_tasks: bootstrap_service.yml
   when: database.changed or use_preconfigured_databases | bool
diff --git a/ansible/roles/cloudkitty/templates/cloudkitty.conf.j2 b/ansible/roles/cloudkitty/templates/cloudkitty.conf.j2
index e4e38307a1..b397a4a33e 100644
--- a/ansible/roles/cloudkitty/templates/cloudkitty.conf.j2
+++ b/ansible/roles/cloudkitty/templates/cloudkitty.conf.j2
@@ -62,3 +62,35 @@ port = {{ cloudkitty_api_port }}
 coordination_url = mysql://{{ cloudkitty_database_user }}:{{ cloudkitty_database_password }}@{{ cloudkitty_database_address }}/{{ cloudkitty_database_name }}
 max_workers = {{ openstack_service_workers }}
 
+[storage]
+backend = {{ cloudkitty_storage_backend }}
+{% if cloudkitty_storage_backend == 'influxdb' %}
+version = 2
+{% endif %}
+{% if cloudkitty_storage_backend == 'sqlalchemy' %}
+version = 1
+{% endif %}
+
+{% if cloudkitty_storage_backend == 'influxdb' %}
+[storage_influx]
+database = {{ cloudkitty_influxdb_name }}
+host = {{ influxdb_address }}
+port = {{ influxdb_http_port }}
+
+{% if cloudkitty_influxdb_retention_policy is defined %}
+retention_policy = {{ cloudkitty_influxdb_retention_policy }}
+{% endif %}
+
+{% if cloudkitty_influxdb_use_ssl is defined %}
+use_ssl = {{ cloudkitty_influxdb_use_ssl }}
+{% endif %}
+
+{% if cloudkitty_influxdb_insecure_connections is defined %}
+insecure = {{ cloudkitty_influxdb_insecure_connections }}
+{% endif %}
+
+{% if cloudkitty_influxdb_cafile is defined %}
+cafile = {{ cloudkitty_influxdb_cafile }}
+{% endif %}
+
+{% endif %}
diff --git a/ansible/roles/monasca/defaults/main.yml b/ansible/roles/monasca/defaults/main.yml
index 1f10f2a89f..7648b3bdfd 100644
--- a/ansible/roles/monasca/defaults/main.yml
+++ b/ansible/roles/monasca/defaults/main.yml
@@ -171,7 +171,7 @@ monasca_database_address: "{{ database_address }}"
 monasca_database_port: "{{ database_port }}"
 
 monasca_influxdb_name: "monasca"
-monasca_influxdb_address: "{{ kolla_internal_fqdn }}"
+monasca_influxdb_address: "{{ influxdb_address }}"
 monasca_influxdb_http_port: "{{ influxdb_http_port }}"
 monasca_influxdb_retention_policy:
   name: 'monasca_metrics'
diff --git a/releasenotes/notes/cloudKitty-v2-influxdb-07cb8b0051ac9ea0.yaml b/releasenotes/notes/cloudKitty-v2-influxdb-07cb8b0051ac9ea0.yaml
new file mode 100644
index 0000000000..92f694f703
--- /dev/null
+++ b/releasenotes/notes/cloudKitty-v2-influxdb-07cb8b0051ac9ea0.yaml
@@ -0,0 +1,4 @@
+---
+features:
+  - |
+    Add support to Kolla-Ansible for Cloudkitty InfluxDB storage system deployment.