Emilien Macchi ec4351c566 [SQUASH] backport tripleo-container-manage to stable/train ...

This is a squash of 32 commits to facilitate the backport of
tripleo-container-manage and its dependencies.

Introduce tripleo-container-manage role

This is a first ieration of the role, but there is still a long TODO,
that will come later in separated patches:
- Add molecule testing
- In podman.yaml, add cpuset_cpus with parity of what is in paunch
- Remove containers that are:
  - managed by tripleo-ansible (using the container_label flag)
  - not in the container-startup-config
- Print stdout when containers start as it was done with paunch

Story: 2006732
Task: 37165

Co-Authored-By: Kevin Carter <kecarter@redhat.com>
Co-Authored-By: Alex Schultz <aschultz@redhat.com>

Depends-On: https://review.opendev.org/#/c/702144/
Change-Id: I2f88caa8e1c230dfe846a8a0dd9f939b98992cd5
(cherry picked from commit a191a2d600)

tripleo-container-manage: set some defaults

Set defaults that are needed to use the role outside of THT more easily.

Change-Id: Id67cf06c85a2a6b50e6494b1a66f534ccb06c4a7
(cherry picked from commit 609d7895a1)

Move the filters plugin to the core plugins location

This change is a workaround for a zuul issue which moves the
nested ansible role plugin to the core plugins directory so
that it is not creating a gate conflict.

Change-Id: I9f959803381063502b4d15980b14c3416ffa153f
Signed-off-by: Kevin Carter <kecarter@redhat.com>
(cherry picked from commit e2719131db)

Revert "Workaround for ansible-lint installation failure"

Backport note: this is a second backport of the same patch
since now it includes the change in tripleo-container-manage
role that is being backported to stable/train.

This reverts the disabling of the ansible-lint test from
commit cffd4fc9d4 and updates
ansible-lint to the fixed version.

Included are fixes for ansible-lint test failures which
got merged as part of I2f88caa8e1c230dfe846a8a0dd9f939b98992cd5
while the lint test was disabled.

Change-Id: I37100f5e1764a5cd2cb8df82ae963e673ca0a8da
(cherry picked from commit 28e105c056)

tripleo-container-manage: few improvements

- Add and use variables to make the role more flexible:
  tripleo_container_manage_config,
  tripleo_container_manage_config_id
  tripleo_container_manage_debug
  tripleo_container_manage_config_pattern (and rename hashed_files var)

  With these vars, the role can pretty much be used outside of TripleO.

- Show logs of config data generation if debug is enabled

- Do not run the "podman exec" tasks in check mode

- Remove the dependency on the "step" variable

Change-Id: I28ee31b723f27c392f880676aaae9368906cf45f
(cherry picked from commit 9c69840640)

Adds new molecule testing for tripleo-container-manage

This test ensure the "create" part is working fine with an easy and
simple container. The container-create and default scenario have been
consoldated so its running one complete test for now.

Change-Id: I9139c7b63c15739a1a95d913acb1128af299ce97
Co-Authored-By: Emilien Macchi <emilien@redhat.com>
Signed-off-by: Kevin Carter <kecarter@redhat.com>
(cherry picked from commit 414f47cc32)

tripleo-container-manage: add check tasks to the molecule playbook

Verify that the "fedora" container exists and has the right infos.

Change-Id: If78a254564cff502b49769b0b401d2efdac8cb23
(cherry picked from commit c20ab42015)

Molecule job for testing plugins and modules

and remove ansible_facts from podman_container_info module
Change-Id: I0c768bc6168363fa3758562f9f053aa9ab85236b

(cherry picked from commit 2d42082737)

tripleo-container-manage - first support for idempotency

- Implements a new helper to figure out if the existing containers on
  the hosts need to be removed (and re-created later).
  The helper will remove the container if:
    - the container is managed by tripleo_ansible (+ other conditions later)
    - a running container isn't in the config
    - the container has no config_data Label
    - the config_data changed for the container

- Restart the systemd service for the container if the podman_container
  module reported as changed (note the podman_container isn't yet
  idempotent but we're working on it in a separated patch)

- Fix the healthcheck & timer systemd files to be attached to the right
  service (with tripleo_ prefix)

Story: #2006732
Task: #37163
Change-Id: I5081c918b47dcb9f3629a3649fdf33d17668c1ff
(cherry picked from commit 7e41e0642d)

tripleo-container-manage: include_tasks to speed things up

Change-Id: I79d48fef7552f72c8fc0ebbad35d98cfc618b114
(cherry picked from commit 57411934b7)

tripleo-container-manage: introduce concurrency

Co-Authored-By: Alex Schultz <aschultz@redhat.com>
Change-Id: I1e5e941558c492b050e4db542703e322707dbbbd
(cherry picked from commit 2f8f0fc027)

tripleo-container-manage: fix log_opt

Put the right variables to have proper logging.

Change-Id: I60a14721ed978dddbebfd28f830fb2c50f326ce0
(cherry picked from commit e7f71c352e)

tripleo-container-manage: some nits fixed

Change-Id: I4006db3f5e3092aefeb8a0e50819dda11931630b
(cherry picked from commit 73e49888f7)

tripleo-container-manage: fix exec check

Fix the task which check if the container where the exec happens is
actually running.

Also podman_containers needs to be refreshed at every start_order to get
new container informations status.
So moving podman_containers from main, and putting it where we need it:
- before processing the list of containers to delete
- when start_order playbook starts

Co-Authored-By: Sagi Shnaidman <sshnaidm@redhat.com>
Change-Id: I2afb71288208c8b97763caa832c94e06e1b9457c
(cherry picked from commit e68d4b42a2)

tripleo-container-manage: port paunch-services

paunch-services used to be useful for container start/stop ordering,
when on the host some containers are managed by Pacemaker and some
others by Paunch.

We need to keep that feature so we are now porting these scripts and
services into tripleo-ansible.

These files where managed by Paunch:
%{_libexecdir}/paunch-container-shutdown
%{_libexecdir}/paunch-start-podman-container
%{_unitdir}/paunch-container-shutdown.service
%{_presetdir}/91-paunch-container-shutdown.preset
%{_unitdir}/netns-placeholder.service
%{_presetdir}/91-netns-placeholder.preset

Now we handle them via Ansible now, and cleanup the Paunch version.
It creates the exact same files from:
https://github.com/rdo-packages/paunch-distgit

This feature is disabled by default and will be explicitely enabled by
THT later. Molecule tests enable it though for testing coverage.

Story: 2006732
Task: 37382
Change-Id: I4f79429baab50bc0199fb65fe84641908d83935d
(cherry picked from commit 6e76f444df)

tripleo-container-manage: use async

- Add a new filter 'haskey' which returns container data with a specific
  config key. The filter takes a list of dicts and returns the dicts
  which have a certain key given in parameter with 'attribute'.
  If 'reverse' is set to True, the returned list won't contains dicts
  which have the 'attribute' value.
  If 'any' is set to True, the returned list will match any value in the
  list of 'value' parameter which has to be a list.

- Make the exec and create playbook using ansible "async".
  (WIP is to re-add the podman_container.changed in systemd playbook).

Note: we don't manage Systemd resources with async, since it exposes
race conditions at the systemd level.

Change-Id: Ice92cd5f90039e685c171e9035f929349a67ff2c
(cherry picked from commit e26f817597)

tripleo-container-manage: fix duplicated loop vars

So we don't have the Ansible warning saying the loop var is already used
for another loop.

Change-Id: Ie28857cb712d2133aad4a67ae6942fcbbb1a6aee
(cherry picked from commit 80e0476f78)

tripleo-container-manage: skip some tasks in check mode

Change-Id: I5e68fdb2d872c741f00a1a24bac33112ba630f69
(cherry picked from commit f506dd6994)

tripleo-container-manage: add no_log to podman create

So we don't leak confidential informations like passwords and others in
the logs.
Note that if debug is enabled, the infos are displayed in the ansible
logs.

Change-Id: I8473c9118dbce2b04eeb2c01bcc1e55232325a67
(cherry picked from commit ac8cff4ebb)

Improve molecule tests for tripleo-container-manage (systemd)

Change-Id: Id14ad4876bdf7dcc979d5ecf4fe2b4751c635b88
(cherry picked from commit 484faa7e80)

fix typo

Change-Id: I0a970ae73f01c6ff181a67367cf668091748fc9c
Signed-off-by: Kevin Carter <kecarter@redhat.com>
(cherry picked from commit 3720b41a7f)

tripleo-container-manage: restart systemd service if container changed

If a podman_container resource changed, it'll be added to a fact, that
later will be used when it comes to figure our if a systemd service
needs to be restarted or not.

It introduces a new filter: list_of_keys.
This filter takes in input a list of dictionaries and for each of them
it will add the key to list_of_keys and returns it.

Change-Id: I0285b006c015f6cd223615ebdad52286f7683a87
(cherry picked from commit 0a037c65f7)

tripleo-container-manage: some improvements

- Improve logging in some tasks, to display what container is managed
- Use no_log for the tasks which leak all infos about containers
  (config_data). If debug is enabled, the logs will show all the
  config_data.
- Move the /etc/sysconfig/podman_drop_in tasks into shutdown.yaml so it
  runs once and it's staying close to the other tasks related to why we
  have this file (manage shutdown/start order when reboot).

Change-Id: I56896f92d58db900fd2ea06281d89f75e8d53a17
(cherry picked from commit 6743f75863)

tripleo-container-manage: create ansible-managed dropin file

If a deployment has containers managed by tripleo-ansible and not
paunch, and an operator would try to run paunch on the host, we want to
send a warning because paunch will remove all containers managed by
tripleo-ansible and then redeploy the containers with managed_by=paunch.

We could change the default of manage_by in Paunch (currently
managed_by=paunch) to something else e.g. tripleo, which would also be
the managed_by of tripleo-ansible; so both paunch and tripleo-ansible
could be used on the same host. However, we decided that when
tripleo-ansible is used to deploy the containers, paunch could not be
used anymore on that host.

Therefore, we create a file that will be checked by paunch and if
present, paunch CLI will show a warning.

Change-Id: I722cb8faa3b7eee81b418da83451bf802351dd79
(cherry picked from commit db1b13c962)

tripleo-container-manage: remove no_log for create/exec tasks

1. Remove no_log for the create and exec tasks. It's always useful to
   see what is being run.

2. Enable podman_container debug, so we see what commands are being run.

Change-Id: If728788293dd64622cf95da840b60e271197e9a0
(cherry picked from commit 89e4adf2ae)

Add missing ExecReload in container service unit file

It may happen that we want to just reload the container. Before this
patch, it was a "stop and start", while podman has the "podman kill"
available, accepting the HUP signal.

Doing so allows other automated tools to actually just "reload" the
container as we would do for a standard service.

Change-Id: I35eff80f7637b013d3a1a831289ec9b1e0f81431
(cherry picked from commit 2a2bed6f5c)

Introduce unit tests structure for tripleo-ansible filters

Change-Id: Ie2fea14d2cbfb2c0b78cdc3064df0a558fa28a4c
(cherry picked from commit f90a6d42b3)

Fix case in filters when Labels is None

When Labels is None it's not managed by tripleo container,
then skip it.
Change-Id: Ib82c4d28c462abb3f1a5ccb7d5137ec6059b2665

(cherry picked from commit 1acc95211d)

tripleo-container-manage: include more arguments for container exec

Container execs can be more complex than just a user and a command.
It can also use --env and --privileged.

- container_exec_cmd is a new filter that will help to build the
  container exec command from the container data.
- list_or_dict_arg is an utility taken from openstack/paunch which
  allows to build a command and its arguments with list or dict data.

This patch will allow the container exec to work fine when they have
environment variables, like it's the case for Keystone bootstrap.

Change-Id: I15e0b518936b37e26799dbda9677a248cf17ff3c
(cherry picked from commit 55d5363b4a)

Add SyslogIdenfier to healthcheck systemd unit

Adding this new field will allow to filter all healthcheck logs using
the Idenfier value.

For instance, using journalctl, you would be able to run this:
`journalctl -t healthcheck_collectd'

It will also allow to get a dedicated file out of (r)syslog if needed.

Change-Id: Icdc5caf4cedc46291a807c39c0a31c74955a4a74
Closes-Bug: #1856573
(cherry picked from commit 49858c5265)

Allow to run tripleo-container-manage in check mode

Change-Id: I3350a43805b4a148f64de393716c26b0158fcff4
(cherry picked from commit 6513a4bed8)

tripleo-container-manage: fix config_data in Config/Labels

config_data should not contain the container name, just the actual
config data which is the value of the container_data dict.

It removes the add of start_order into the compare, this isn't necessary
and breaks idempotency for containers which don't have start_order in
their config_data.
Also adds more unit tests to cover all situations handled by the filter.

Change-Id: I0b64bf34c8f7498128b3b1fceb7c727f8544cec6
(cherry picked from commit 19c5d7e77e)

tripleo-container-manage: search container configs recursively

If the user doesn't override the default value for
tripleo_container_manage_config, which is set to
'/var/lib/tripleo-config/', to something deeper, we want Ansible to
recursively search for the container config until it finds it.

Change-Id: I7d23fec91ffb813f0ab6f11b85d811ef3897f9e0
(cherry picked from commit 205f7c9b0c)

Idempotency for podman_container

Introduce partial idepmpotency for running podman_container

- Replace all options by their defaults it they're not set
- Force lower case on all podman inspection data
- Add a class with methods for every parameter to check its value
and compare it with module argument
- Add check_mode support and podman_actions

Change-Id: I1ae93dff1e10a1a696bb171996a75a0db6c34fa3
(cherry picked from commit 1212544a28)

Fix case where there're no effective caps

When no effective caps in container, it should be list, not None
Change-Id: I5412edc844ad43223c4b3bda35662a7f7ee43f3a

(cherry picked from commit 3a6690c904)

Add idempotency for networks parameter in podman_container

Change-Id: Ib7235d1cb64ad0f42e7a0201008536e1a35bd696
(cherry picked from commit 25f8abbc5a)

use version parsing from distutils

use LooseVersion for version comparisons

Change-Id: I609920a96c725c49f1623f60f8295d89ae4f3141
(cherry picked from commit 8c83219fbb)

Improve idempotency for podman containers module

Fix some pep8 issues
Change-Id: If4233e57edeec10ccac965d61c78f30688cd5531

(cherry picked from commit 0609c16d10)

Improve idempotency for podman container module

Strip all registry values from "image" parameter in input.
Add this case to test.

Change-Id: I78656e48cf85a1a39f873ee40193765eecf02c56
(cherry picked from commit 87d9e9d9a6)

2020-02-03 17:46:55 +00:00

0 lines

Python

Raw Blame History