70 Commits

Author SHA1 Message Date
Doug Szumski
b39a0f805a Switch to Monasca API for logs
The Monasca Log API has been removed and in this change we switch
to using the unified API. If dedicated log APIs are required then
this can be supported through configuration. Out of the box the
Monasca API is used for both logs and metrics which is envisaged to
work for most use cases.

In order to use the unified API for logs, we need to disable the
legacy Kafka client. We also rename the Monasca API config file
to remove a warning about using the old style name.

Depends-On: https://review.opendev.org/#/c/728638
Change-Id: I9b6bf5b6690f4b4b3445e7d15a40e45dd42d2e84
2020-05-23 17:49:32 +01:00
Doug Szumski
776253c436 Fix registration of Monasca Grafana datasource
The refactor in change I500cc8800c412bc0e95edb15babad5c1189e6ee4
broke the task `Enable Monasca Grafana datasource for control
plane organisation`. This change fixes the brackets.

Change-Id: I9167a312be107fbfddfd07740f67845c2eaafc3d
Closes-Bug: 1878878
2020-05-15 10:16:22 +01:00
Zuul
7a193d1f06 Merge "Ansible lint: lines longer than 160 chars" 2020-04-17 09:29:00 +00:00
Zuul
87984f5425 Merge "Add Ansible group check to prechecks" 2020-04-16 15:33:46 +00:00
Michal Nasiadka
d403690b88 Ansible lint: lines longer than 160 chars
Change-Id: I500cc8800c412bc0e95edb15babad5c1189e6ee4
2020-04-16 15:59:06 +02:00
James Kirsch
4d155d69cd Refactor copy certificates task
Refactor service configuration to use the copy certificates task. This
reduces code duplication and simplifies implementing encrypting backend
HAProxy traffic for individual services.

Change-Id: I0474324b60a5f792ef5210ab336639edf7a8cd9e
2020-04-14 17:26:19 +00:00
Zuul
31fe3c0651 Merge "Fix invalid JSON body in Elasticsearch API requests" 2020-03-18 10:47:57 +00:00
Zuul
2a2ce059dc Merge "Add notify restart container when cert changed" 2020-03-10 12:12:55 +00:00
yj.bai
d3cc2f670e Add notify restart container when cert changed
When change the cert file in /etc/kolla/certificate/.
The certificate in the container has not changed.
So I think can use kolla-ansible deploy when certificate is
changed. restart <container>

Partially-Implements: blueprint custom-cacerts

Change-Id: Iaac6f37e85ffdc0352e8062ae5049cc9a6b3db26
Signed-off-by: yj.bai <bai.yongjun@99cloud.net>
2020-03-10 16:23:09 +08:00
Radosław Piliszek
266fd61ad7 Use "name:" instead of "role:" for *_role modules
Both include_role and import_role expect role's name to be given
via "name" param instead of "role".
This worked but caused errors with ansible-lint.
See: https://review.opendev.org/694779

Change-Id: I388d4ae27111e430d38df1abcb6c6127d90a06e0
2020-03-02 10:01:17 +01:00
Mark Goddard
49fb55f182 Add Ansible group check to prechecks
We assume that all groups are present in the inventory, and quite obtuse
errors can result if any are not.

This change adds a precheck that checks for the presence of all expected
groups in the inventory for each service. It also introduces a common
service-precheck role that we can use for other common prechecks.

Change-Id: Ia0af1e7df4fff7f07cd6530e5b017db8fba530b3
Partially-Implements: blueprint improve-prechecks
2020-02-28 16:23:14 +00:00
James Kirsch
256322a8fe Construct service configuration urls using kolla_internal_fqdn
Service configuration urls should be constructed using
kolla_internal_fqdn instead of kolla_internal_vip_address. Otherwise SSL
validation will fail when certificates are issued using domain names.

Change-Id: I21689e22870c2f6206e37c60a3c33e19140f77ff
Closes-Bug: 1862419
2020-02-22 08:28:01 -08:00
Mark Goddard
5db9eab042 Fix invalid JSON body in Elasticsearch API requests
The kibana, elasticsearch and monasca roles all use the uri module to
perform Elasticsearch configuration tasks via its API. The body of the
request should be JSON formatted, but these tasks now fail because it is
not.

The following error is seen:

TASK [monasca : Create default control plane organisation if it doesn't
exist]

invalid character '\\'' looking for beginning of object key string

The 'JSON' body in this case was:

{'name': 'monasca_control_plane@default'}

This was probably caused by the recent change to execute these tasks in
the kolla_toolbox container, but may also be caused by an Ansible
version bump (or something else).

This change fixes the issue by ensuring that the body is JSON-encoded in
all cases.

Change-Id: I7acc097381dd9a4af4e014525c1c88213abbde93
Closes-Bug: #1864177
2020-02-21 10:09:51 +00:00
James Kirsch
88418cbaa9 Use kolla_toolbox to execute REST methods
Delegate executing uri REST methods to the current module containers
using kolla_toolbox. This will allow self signed certificate that are
already copied into the container to be automatically validated. This
circumvents requiring Kolla Ansible to explicitly disable certificate
validation in the ansible uri module.

Partially-Implements: blueprint custom-cacerts

Change-Id: I2625db7b8000af980e4745734c834c5d9292290b
2020-01-28 14:03:32 -08:00
James Kirsch
511ba9f6a2 Copy CA into containers.
When kolla_copy_ca_into_containers is set to "yes", the Certificate
Authority in /etc/kolla/certificates will be copied into service
containers to enable trust for that CA. This is especially useful when
the CA is self signed, and would not be trusted by default.

Partially-Implements: blueprint custom-cacerts

Change-Id: I4368f8994147580460ebe7533850cf63a419d0b4
2020-01-28 14:03:32 -08:00
Michal Nasiadka
1009931162 Change local_action to delegate_to: localhost
As part of the effort to implement Ansible code linting in CI
(using ansible-lint) - we need to implement recommendations from
ansible-lint output [1].

One of them is to stop using local_action in favor of delegate_to -
to increase readability and and match the style of typical ansible
tasks.

[1]: https://review.opendev.org/694779/

Partially implements: blueprint ansible-lint

Change-Id: I46c259ddad5a6aaf9c7301e6c44cd8a1d5c457d3
2019-11-22 15:04:44 +00:00
Mark Goddard
c6392ed1aa Perform database migration in monasca upgrade
Currently the database is only synced during deployment. This change
performs the sync during upgrade as well.

Change-Id: Ia45fc733a1ab69de9d4762f5d9c8767041eeaed3
Closes-Bug: #1832020
2019-10-21 16:26:08 +01:00
Radosław Piliszek
bc053c09c1 Implement IPv6 support in the control plane
Introduce kolla_address filter.
Introduce put_address_in_context filter.

Add AF config to vars.

Address contexts:
- raw (default): <ADDR>
- memcache: inet6:[<ADDR>]
- url: [<ADDR>]

Other changes:

globals.yml - mention just IP in comment

prechecks/port_checks (api_intf) - kolla_address handles validation

3x interface conditional (swift configs: replication/storage)

2x interface variable definition with hostname
(haproxy listens; api intf)

1x interface variable definition with hostname with bifrost exclusion
(baremetal pre-install /etc/hosts; api intf)

neutron's ml2 'overlay_ip_version' set to 6 for IPv6 on tunnel network

basic multinode source CI job for IPv6

prechecks for rabbitmq and qdrouterd use proper NSS database now

MariaDB Galera Cluster WSREP SST mariabackup workaround
(socat and IPv6)

Ceph naming workaround in CI
TODO: probably needs documenting

RabbitMQ IPv6-only proto_dist

Ceph ms switch to IPv6 mode

Remove neutron-server ml2_type_vxlan/vxlan_group setting
as it is not used (let's avoid any confusion)
and could break setups without proper multicast routing
if it started working (also IPv4-only)

haproxy upgrade checks for slaves based on ipv6 addresses

TODO:

ovs-dpdk grabs ipv4 network address (w/ prefix len / submask)
not supported, invalid by default because neutron_external has no address
No idea whether ovs-dpdk works at all atm.

ml2 for xenapi
Xen is not supported too well.
This would require working with XenAPI facts.

rp_filter setting
This would require meddling with ip6tables (there is no sysctl param).
By default nothing is dropped.
Unlikely we really need it.

ironic dnsmasq is configured IPv4-only
dnsmasq needs DHCPv6 options and testing in vivo.

KNOWN ISSUES (beyond us):

One cannot use IPv6 address to reference the image for docker like we
currently do, see: https://github.com/moby/moby/issues/39033
(docker_registry; docker API 400 - invalid reference format)
workaround: use hostname/FQDN

RabbitMQ may fail to bind to IPv6 if hostname resolves also to IPv4.
This is due to old RabbitMQ versions available in images.
IPv4 is preferred by default and may fail in the IPv6-only scenario.
This should be no problem in real life as IPv6-only is indeed IPv6-only.
Also, when new RabbitMQ (3.7.16/3.8+) makes it into images, this will
no longer be relevant as we supply all the necessary config.
See: https://github.com/rabbitmq/rabbitmq-server/pull/1982

For reliable runs, at least Ansible 2.8 is required (2.8.5 confirmed
to work well). Older Ansible versions are known to miss IPv6 addresses
in interface facts. This may affect redeploys, reconfigures and
upgrades which run after VIP address is assigned.
See: https://github.com/ansible/ansible/issues/63227

Bifrost Train does not support IPv6 deployments.
See: https://storyboard.openstack.org/#!/story/2006689

Change-Id: Ia34e6916ea4f99e9522cd2ddde03a0a4776f7e2c
Implements: blueprint ipv6-control-plane
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-10-16 10:24:35 +02:00
Kris Lindgren
2fe0d98ebb Add a job that *only* deploys updated containers
Sometimes as cloud admins, we want to only update code that is running
in a cloud.  But we dont need to do anything else.  Make an action in
kolla-ansible that allows us to do that.

Change-Id: I904f595c69f7276e71692696471e32fd1f88e6e8
Implements: blueprint deploy-containers-action
2019-09-26 17:51:14 +01:00
Mark Goddard
741f6d9be9 Create and grant all keystone roles in service-ks-register
This ensures we execute the keystone os_* modules in one place.

Also rework some of the task names and loop item display.

Change-Id: I6764a71e8147410e7b24b0b73d0f92264f45240c
2019-09-24 08:50:04 +01:00
Mark Goddard
3522d235bd Refactor service, endpoint and user registration
Use upstream Ansible modules for registration of services, endpoints,
users, projects, roles, and role grants.

Change-Id: I7c9138d422cc91c177fd8992347176bb54156b5a
2019-09-17 10:13:56 -07:00
Zuul
42aef5a50f Merge "Support configuration of trusted CA certificate file" 2019-08-28 07:48:51 +00:00
Scott Solkhon
09e02ef8f1 Support configuration of trusted CA certificate file
This commit adds the functionality for an operator to specify
their own trusted CA certificate file for interacting with the
Keystone API.

Implements: blueprint support-trusted-ca-certificate-file
Change-Id: I84f9897cc8e107658701fb309ec318c0f805883b
2019-08-16 12:47:42 +00:00
Doug Szumski
65b9756127 Add support for using custom Logstash patterns
A user may want to define and use Logstash patterns. This
commit adds support to copy them into the Monasca Log
Transformer container. In the future support could be
added for other Logstash containers.

Change-Id: Id8cde14af6dc7f49714f6b1cb878882d0048d293
2019-08-08 10:48:35 +01:00
Radosław Piliszek
6a737b1968 Fix handling of docker restart policy
Docker has no restart policy named 'never'. It has 'no'.
This has bitten us already (see [1]) and might bite us again whenever
we want to change the restart policy to 'no'.

This patch makes our docker integration honor all valid restart policies
and only valid restart policies.
All relevant docker restart policy usages are patched as well.

I added some FIXMEs around which are relevant to kolla-ansible docker
integration. They are not fixed in here to not alter behavior.

[1] https://review.opendev.org/667363

Change-Id: I1c9764fb9bbda08a71186091aced67433ad4e3d6
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
2019-07-18 13:39:06 +00:00
Mark Goddard
d5e5e885d1 During deploy, always sync DB
A common class of problems goes like this:

* kolla-ansible deploy
* Hit a problem, often in ansible/roles/*/tasks/bootstrap.yml
* Re-run kolla-ansible deploy
* Service fails to start

This happens because the DB is created during the first run, but for some
reason we fail before performing the DB sync. This means that on the second run
we don't include ansible/roles/*/tasks/bootstrap_service.yml because the DB
already exists, and therefore still don't perform the DB sync. However this
time, the command may complete without apparent error.

We should be less careful about when we perform the DB sync, and do it whenever
it is necessary. There is an argument for not doing the sync during a
'reconfigure' command, although we will not change that here.

This change only always performs the DB sync during 'deploy' and
'reconfigure' commands.

Change-Id: I82d30f3fcf325a3fdff3c59f19a1f88055b566cc
Closes-Bug: #1823766
Closes-Bug: #1797814
2019-07-12 08:56:54 +00:00
Zuul
af8ae0aa41 Merge "Simplify handler conditionals" 2019-07-04 21:34:14 +00:00
Will Szumski
9074da56a7 Specify endpoint when creating monasca user
otherwise I'm seeing:

TASK [monasca : Creating the monasca agent user] ****************************************************************************************************************************
fatal: [monitor1]: FAILED! => {"changed": false, "module_stderr": "Shared connection to 172.16.3.24 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  F
ile \"/tmp/ansible_I0RmxQ/ansible_module_kolla_toolbox.py\", line 163, in <module>\r\n    main()\r\n  File \"/tmp/ansible_I0RmxQ/ansible_module_kolla_toolbox.py\", line 141,
 in main\r\n    output = client.exec_start(job)\r\n  File \"/opt/kayobe/venvs/kolla-ansible/lib/python2.7/site-packages/docker/utils/decorators.py\", line 19, in wrapped\r\n
    return f(self, resource_id, *args, **kwargs)\r\n  File \"/opt/kayobe/venvs/kolla-ansible/lib/python2.7/site-packages/docker/api/exec_api.py\", line 165, in exec_start\r\
n    return self._read_from_socket(res, stream, tty)\r\n  File \"/opt/kayobe/venvs/kolla-ansible/lib/python2.7/site-packages/docker/api/client.py\", line 377, in _read_from_
socket\r\n    return six.binary_type().join(gen)\r\n  File \"/opt/kayobe/venvs/kolla-ansible/lib/python2.7/site-packages/docker/utils/socket.py\", line 75, in frames_iter\r\
n    n = next_frame_size(socket)\r\n  File \"/opt/kayobe/venvs/kolla-ansible/lib/python2.7/site-packages/docker/utils/socket.py\", line 62, in next_frame_size\r\n    data = read_exactly(socket, 8)\r\n  File \"/opt/kayobe/venvs/kolla-ansible/lib/python2.7/site-packages/docker/utils/socket.py\", line 47, in read_exactly\r\n    next_data = read(socket, n - len(data))\r\n  File \"/opt/kayobe/venvs/kolla-ansible/lib/python2.7/site-packages/docker/utils/socket.py\", line 31, in read\r\n    return socket.recv(n)\r\nsocket.timeout: timed out\r\n", "msg": "MODULE FAILURE", "rc": 1}

when the monitoring nodes aren't on the public API network.

Change-Id: I7a93f69da0e02c9264da0b081d2e60626f899e3a
2019-06-28 18:36:24 +01:00
Mark Goddard
de00bf491d Simplify handler conditionals
Currently, we have a lot of logic for checking if a handler should run,
depending on whether config files have changed and whether the
container configuration has changed. As rm_work pointed out during
the recent haproxy refactor, these conditionals are typically
unnecessary - we can rely on Ansible's handler notification system
to only trigger handlers when they need to run. This removes a lot
of error prone code.

This patch removes conditional handler logic for all services. It is
important to ensure that we no longer trigger handlers when unnecessary,
because without these checks in place it will trigger a restart of the
containers.

Implements: blueprint simplify-handlers

Change-Id: I4f1aa03e9a9faaf8aecd556dfeafdb834042e4cd
2019-06-27 15:57:19 +00:00
Zuul
2af1df6c99 Merge "Fix issue finding custom, host specific plugins" 2019-06-13 18:58:32 +00:00
Doug Szumski
4be219fcb7 Fix issue finding custom, host specific plugins
The results from the find operation need to be registered per host,
because they depend on the host which runs the search. This bug
impacts users specifying custom plugins for specific hosts.

Change-Id: I41b2986b2f4ccd8fdc6553e83737e4106b6a2c07
2019-06-07 14:01:45 +01:00
Zuul
888e50f01b Merge "Use become for all docker tasks" 2019-06-07 10:47:23 +00:00
Mark Goddard
b123bf6621 Use become for all docker tasks
Many tasks that use Docker have become specified already, but
not all. This change ensures all tasks that use the following
modules have become:

* kolla_docker
* kolla_ceph_keyring
* kolla_toolbox
* kolla_container_facts

It also adds become for 'command' tasks that use docker CLI.

Change-Id: I4a5ebcedaccb9261dbc958ec67e8077d7980e496
2019-06-06 19:04:58 +01:00
Isaac Prior
84edfd09b6 Fix monasca grafana organisation check
"Create default control plane organisation if it doesn't exist" task
fails when organisation already exists.
The list organisation task currently returns project domain id.
The create organisation task currently provides project domain name.
Change the create task to use default_project_domain_id instead.

TrivialFix

Change-Id: Ice70d55e6729fe55164dcf85e98acdc1d7925209
2019-05-31 12:25:21 +01:00
Bartosz Zurkowski
10d33f82bb Find Monasca agent plugins locally
Find module searches paths on managed server. Since role path and custom
Kolla config is located on deployment node and deployment node is not
considered to be a managed server, Monasca plugin files cannot be found.
After the deployment container running Monasca agent collector stucks in
restart mode due to missing plugin files.

The problem does not occur if deployment was started from a managed
server (eg. OSC). The problem occurs if the deployment was started from
a separate deployment server - a common case.

This change enforces running find module locally on deployment node.

Change-Id: Ia25daafe2f82f5744646fd2eda2d255ccead814e
Signed-off-by: Bartosz Zurkowski <b.zurkowski@samsung.com>
2018-12-16 16:51:54 +01:00
Bartosz Zurkowski
c5d1e1d5f2 Call Grafana APIs only once
In multinode deployments creating default Grafana organization failed,
because Ansible attempted to call Grafana API in the context of each
host in the inventory. After creating organization via the first host,
subsequent attempts via the remaining hosts failed due to already
existing organization. This change enforces creating default
organization only once.

Other tasks using Grafana API have been enforced to be ran only once as
well.

Change-Id: I3a93a719b3c9b4e55ab226d3b22d571d9a0f489d
Signed-off-by: Bartosz Zurkowski <b.zurkowski@samsung.com>
2018-12-16 16:30:58 +01:00
Eduardo Gonzalez
1a682fab28 Support stop specific containers
With this change, an operator may be able to stop a
service container without stopping all services in a host.
This change is the starting point to start
fast-forward upgrades support.
In next changes new flags will be introducced to disable
stop dataplane services during upgrades.

Change-Id: Ifde7a39d7d8596ef0d7405ecf1ac1d49a459d9ef
Implements: blueprint support-stop-containers
2018-11-26 08:07:01 +00:00
Zuul
445e4f7640 Merge "Support custom monasca-notification templates" 2018-11-20 09:41:21 +00:00
Zuul
42ab513ec9 Merge "Automatically configure Monasca Grafana datasource" 2018-11-13 15:18:40 +00:00
Zuul
8bbe9011e2 Merge "Don't set recurse on config folders" 2018-11-12 13:50:24 +00:00
Doug Szumski
75d095b64e Automatically configure Monasca Grafana datasource
In Kolla, an OpenStack project is created to store logs and metrics
harvested from the control plane by Monasca. This commit enables
the Monasca Datasource in the Grafana organisation which maps to
this OpenStack control plane project. What this means in practice
is that if a user logs into Monasca Grafana, and has access to the
the control plane project, they will immediately be able to create
dashboards using data from Monasca which has been gathered from the
control plane.

Support to enable creation of this datasource for other OpenStack
projects can be added in a separate commit.

Partially-Implements: blueprint monasca-grafana
Change-Id: I03e741ddb1c582b7280c64637ed3e3683df6419b
2018-11-07 20:24:41 +00:00
Doug Szumski
712c89760c Add support for deploying Monasca Grafana
The Monasca Grafana fork allows users to log into Grafana with their
OpenStack user credentials and see metrics associated with their
OpenStack project. The long term goal is to enable Keystone support
in upstream Grafana, but this work seems to have stalled.

Partially-Implements: blueprint monasca-grafana
Change-Id: Icc04613b2571c094ae23b66d0bcc38b58c0ee4e1
2018-11-02 13:35:35 +00:00
Doug Szumski
6cbb5cbdb4 Support using external DBs in Monasca
This changes allows the user to configure a Monasca database
which may be different from the default database.

Partially-Implements: blueprint monasca-roles
Change-Id: Ia905190b8037ecb1782a758c0b65581fe9024bf6
2018-11-02 13:04:06 +00:00
Doug Szumski
b7b45effed Support deploying the Monasca Agent
The Monasca Agent collects metrics and in this change is deployed
across the control plane. These metrics are collected into an OpenStack
project. It supports configuring a small number of plugins, which can
be extended in later commits. It also makes the Monasca Agent credentials
available to other roles, such as the common role to allow forwarding
logs to Monasca.

Partially-Implements: blueprint monasca-roles
Change-Id: I76b34fc5e1c76407a45fcf272268d5798b473ca2
2018-11-02 13:04:05 +00:00
Doug Szumski
4896031757 Don't set recurse on config folders
A small number of services set the recurse flag when they create
their config directory. This can change permission of files within
the directory, which are later set back to the original state. The
side effect is that the service is then restarted, even though the
net change to the config files amounts to nothing. The expected
behaviour is that a service only restarts if the config *has*
changed. This patch fixes this issue.

Change-Id: Ib6f1ca7b416247f8d455fb25892f4a3b27de03ba
Closes-Bug: 1800480
2018-10-29 15:35:32 +00:00
Doug Szumski
1feb6b6aaa Support custom monasca-notification templates
Jira, Slack and possiblly other plugins allow custom templates
for defining the format of notifications. This change lets you
provide these in a templates folder which is copied into the
monasca-notification container.

Partially-Implements: blueprint monasca-roles
Change-Id: Ibc5ba3944d51f6c8ffc8bdc9ed60f43dd91ca7e0
2018-10-09 15:16:17 +01:00
Zuul
867fe4c6b3 Merge "Improve registration of Monasca InfluxDB database" 2018-10-05 08:14:43 +00:00
Zuul
9355f17f2d Merge "Support deploying Monasca Persister" 2018-10-03 14:19:50 +00:00
Zuul
5d2d270eee Merge "Support deploying Monasca Notification engine" 2018-10-03 14:07:40 +00:00
Zuul
c969dac19d Merge "Support deploying Monasca Thresh" 2018-10-03 14:07:39 +00:00