ec4351c566
This is a squash of 32 commits to facilitate the backport of tripleo-container-manage and its dependencies. Introduce tripleo-container-manage role This is a first ieration of the role, but there is still a long TODO, that will come later in separated patches: - Add molecule testing - In podman.yaml, add cpuset_cpus with parity of what is in paunch - Remove containers that are: - managed by tripleo-ansible (using the container_label flag) - not in the container-startup-config - Print stdout when containers start as it was done with paunch Story: 2006732 Task: 37165 Co-Authored-By: Kevin Carter <kecarter@redhat.com> Co-Authored-By: Alex Schultz <aschultz@redhat.com> Depends-On: https://review.opendev.org/#/c/702144/ Change-Id: I2f88caa8e1c230dfe846a8a0dd9f939b98992cd5 (cherry picked from commita191a2d600
) tripleo-container-manage: set some defaults Set defaults that are needed to use the role outside of THT more easily. Change-Id: Id67cf06c85a2a6b50e6494b1a66f534ccb06c4a7 (cherry picked from commit609d7895a1
) Move the filters plugin to the core plugins location This change is a workaround for a zuul issue which moves the nested ansible role plugin to the core plugins directory so that it is not creating a gate conflict. Change-Id: I9f959803381063502b4d15980b14c3416ffa153f Signed-off-by: Kevin Carter <kecarter@redhat.com> (cherry picked from commite2719131db
) Revert "Workaround for ansible-lint installation failure" Backport note: this is a second backport of the same patch since now it includes the change in tripleo-container-manage role that is being backported to stable/train. This reverts the disabling of the ansible-lint test from commitcffd4fc9d4
and updates ansible-lint to the fixed version. Included are fixes for ansible-lint test failures which got merged as part of I2f88caa8e1c230dfe846a8a0dd9f939b98992cd5 while the lint test was disabled. Change-Id: I37100f5e1764a5cd2cb8df82ae963e673ca0a8da (cherry picked from commit28e105c056
) tripleo-container-manage: few improvements - Add and use variables to make the role more flexible: tripleo_container_manage_config, tripleo_container_manage_config_id tripleo_container_manage_debug tripleo_container_manage_config_pattern (and rename hashed_files var) With these vars, the role can pretty much be used outside of TripleO. - Show logs of config data generation if debug is enabled - Do not run the "podman exec" tasks in check mode - Remove the dependency on the "step" variable Change-Id: I28ee31b723f27c392f880676aaae9368906cf45f (cherry picked from commit9c69840640
) Adds new molecule testing for tripleo-container-manage This test ensure the "create" part is working fine with an easy and simple container. The container-create and default scenario have been consoldated so its running one complete test for now. Change-Id: I9139c7b63c15739a1a95d913acb1128af299ce97 Co-Authored-By: Emilien Macchi <emilien@redhat.com> Signed-off-by: Kevin Carter <kecarter@redhat.com> (cherry picked from commit414f47cc32
) tripleo-container-manage: add check tasks to the molecule playbook Verify that the "fedora" container exists and has the right infos. Change-Id: If78a254564cff502b49769b0b401d2efdac8cb23 (cherry picked from commitc20ab42015
) Molecule job for testing plugins and modules and remove ansible_facts from podman_container_info module Change-Id: I0c768bc6168363fa3758562f9f053aa9ab85236b (cherry picked from commit2d42082737
) tripleo-container-manage - first support for idempotency - Implements a new helper to figure out if the existing containers on the hosts need to be removed (and re-created later). The helper will remove the container if: - the container is managed by tripleo_ansible (+ other conditions later) - a running container isn't in the config - the container has no config_data Label - the config_data changed for the container - Restart the systemd service for the container if the podman_container module reported as changed (note the podman_container isn't yet idempotent but we're working on it in a separated patch) - Fix the healthcheck & timer systemd files to be attached to the right service (with tripleo_ prefix) Story: #2006732 Task: #37163 Change-Id: I5081c918b47dcb9f3629a3649fdf33d17668c1ff (cherry picked from commit7e41e0642d
) tripleo-container-manage: include_tasks to speed things up Change-Id: I79d48fef7552f72c8fc0ebbad35d98cfc618b114 (cherry picked from commit57411934b7
) tripleo-container-manage: introduce concurrency Co-Authored-By: Alex Schultz <aschultz@redhat.com> Change-Id: I1e5e941558c492b050e4db542703e322707dbbbd (cherry picked from commit2f8f0fc027
) tripleo-container-manage: fix log_opt Put the right variables to have proper logging. Change-Id: I60a14721ed978dddbebfd28f830fb2c50f326ce0 (cherry picked from commite7f71c352e
) tripleo-container-manage: some nits fixed Change-Id: I4006db3f5e3092aefeb8a0e50819dda11931630b (cherry picked from commit73e49888f7
) tripleo-container-manage: fix exec check Fix the task which check if the container where the exec happens is actually running. Also podman_containers needs to be refreshed at every start_order to get new container informations status. So moving podman_containers from main, and putting it where we need it: - before processing the list of containers to delete - when start_order playbook starts Co-Authored-By: Sagi Shnaidman <sshnaidm@redhat.com> Change-Id: I2afb71288208c8b97763caa832c94e06e1b9457c (cherry picked from commite68d4b42a2
) tripleo-container-manage: port paunch-services paunch-services used to be useful for container start/stop ordering, when on the host some containers are managed by Pacemaker and some others by Paunch. We need to keep that feature so we are now porting these scripts and services into tripleo-ansible. These files where managed by Paunch: %{_libexecdir}/paunch-container-shutdown %{_libexecdir}/paunch-start-podman-container %{_unitdir}/paunch-container-shutdown.service %{_presetdir}/91-paunch-container-shutdown.preset %{_unitdir}/netns-placeholder.service %{_presetdir}/91-netns-placeholder.preset Now we handle them via Ansible now, and cleanup the Paunch version. It creates the exact same files from: https://github.com/rdo-packages/paunch-distgit This feature is disabled by default and will be explicitely enabled by THT later. Molecule tests enable it though for testing coverage. Story: 2006732 Task: 37382 Change-Id: I4f79429baab50bc0199fb65fe84641908d83935d (cherry picked from commit6e76f444df
) tripleo-container-manage: use async - Add a new filter 'haskey' which returns container data with a specific config key. The filter takes a list of dicts and returns the dicts which have a certain key given in parameter with 'attribute'. If 'reverse' is set to True, the returned list won't contains dicts which have the 'attribute' value. If 'any' is set to True, the returned list will match any value in the list of 'value' parameter which has to be a list. - Make the exec and create playbook using ansible "async". (WIP is to re-add the podman_container.changed in systemd playbook). Note: we don't manage Systemd resources with async, since it exposes race conditions at the systemd level. Change-Id: Ice92cd5f90039e685c171e9035f929349a67ff2c (cherry picked from commite26f817597
) tripleo-container-manage: fix duplicated loop vars So we don't have the Ansible warning saying the loop var is already used for another loop. Change-Id: Ie28857cb712d2133aad4a67ae6942fcbbb1a6aee (cherry picked from commit80e0476f78
) tripleo-container-manage: skip some tasks in check mode Change-Id: I5e68fdb2d872c741f00a1a24bac33112ba630f69 (cherry picked from commitf506dd6994
) tripleo-container-manage: add no_log to podman create So we don't leak confidential informations like passwords and others in the logs. Note that if debug is enabled, the infos are displayed in the ansible logs. Change-Id: I8473c9118dbce2b04eeb2c01bcc1e55232325a67 (cherry picked from commitac8cff4ebb
) Improve molecule tests for tripleo-container-manage (systemd) Change-Id: Id14ad4876bdf7dcc979d5ecf4fe2b4751c635b88 (cherry picked from commit484faa7e80
) fix typo Change-Id: I0a970ae73f01c6ff181a67367cf668091748fc9c Signed-off-by: Kevin Carter <kecarter@redhat.com> (cherry picked from commit3720b41a7f
) tripleo-container-manage: restart systemd service if container changed If a podman_container resource changed, it'll be added to a fact, that later will be used when it comes to figure our if a systemd service needs to be restarted or not. It introduces a new filter: list_of_keys. This filter takes in input a list of dictionaries and for each of them it will add the key to list_of_keys and returns it. Change-Id: I0285b006c015f6cd223615ebdad52286f7683a87 (cherry picked from commit0a037c65f7
) tripleo-container-manage: some improvements - Improve logging in some tasks, to display what container is managed - Use no_log for the tasks which leak all infos about containers (config_data). If debug is enabled, the logs will show all the config_data. - Move the /etc/sysconfig/podman_drop_in tasks into shutdown.yaml so it runs once and it's staying close to the other tasks related to why we have this file (manage shutdown/start order when reboot). Change-Id: I56896f92d58db900fd2ea06281d89f75e8d53a17 (cherry picked from commit6743f75863
) tripleo-container-manage: create ansible-managed dropin file If a deployment has containers managed by tripleo-ansible and not paunch, and an operator would try to run paunch on the host, we want to send a warning because paunch will remove all containers managed by tripleo-ansible and then redeploy the containers with managed_by=paunch. We could change the default of manage_by in Paunch (currently managed_by=paunch) to something else e.g. tripleo, which would also be the managed_by of tripleo-ansible; so both paunch and tripleo-ansible could be used on the same host. However, we decided that when tripleo-ansible is used to deploy the containers, paunch could not be used anymore on that host. Therefore, we create a file that will be checked by paunch and if present, paunch CLI will show a warning. Change-Id: I722cb8faa3b7eee81b418da83451bf802351dd79 (cherry picked from commitdb1b13c962
) tripleo-container-manage: remove no_log for create/exec tasks 1. Remove no_log for the create and exec tasks. It's always useful to see what is being run. 2. Enable podman_container debug, so we see what commands are being run. Change-Id: If728788293dd64622cf95da840b60e271197e9a0 (cherry picked from commit89e4adf2ae
) Add missing ExecReload in container service unit file It may happen that we want to just reload the container. Before this patch, it was a "stop and start", while podman has the "podman kill" available, accepting the HUP signal. Doing so allows other automated tools to actually just "reload" the container as we would do for a standard service. Change-Id: I35eff80f7637b013d3a1a831289ec9b1e0f81431 (cherry picked from commit2a2bed6f5c
) Introduce unit tests structure for tripleo-ansible filters Change-Id: Ie2fea14d2cbfb2c0b78cdc3064df0a558fa28a4c (cherry picked from commitf90a6d42b3
) Fix case in filters when Labels is None When Labels is None it's not managed by tripleo container, then skip it. Change-Id: Ib82c4d28c462abb3f1a5ccb7d5137ec6059b2665 (cherry picked from commit1acc95211d
) tripleo-container-manage: include more arguments for container exec Container execs can be more complex than just a user and a command. It can also use --env and --privileged. - container_exec_cmd is a new filter that will help to build the container exec command from the container data. - list_or_dict_arg is an utility taken from openstack/paunch which allows to build a command and its arguments with list or dict data. This patch will allow the container exec to work fine when they have environment variables, like it's the case for Keystone bootstrap. Change-Id: I15e0b518936b37e26799dbda9677a248cf17ff3c (cherry picked from commit55d5363b4a
) Add SyslogIdenfier to healthcheck systemd unit Adding this new field will allow to filter all healthcheck logs using the Idenfier value. For instance, using journalctl, you would be able to run this: `journalctl -t healthcheck_collectd' It will also allow to get a dedicated file out of (r)syslog if needed. Change-Id: Icdc5caf4cedc46291a807c39c0a31c74955a4a74 Closes-Bug: #1856573 (cherry picked from commit49858c5265
) Allow to run tripleo-container-manage in check mode Change-Id: I3350a43805b4a148f64de393716c26b0158fcff4 (cherry picked from commit6513a4bed8
) tripleo-container-manage: fix config_data in Config/Labels config_data should not contain the container name, just the actual config data which is the value of the container_data dict. It removes the add of start_order into the compare, this isn't necessary and breaks idempotency for containers which don't have start_order in their config_data. Also adds more unit tests to cover all situations handled by the filter. Change-Id: I0b64bf34c8f7498128b3b1fceb7c727f8544cec6 (cherry picked from commit19c5d7e77e
) tripleo-container-manage: search container configs recursively If the user doesn't override the default value for tripleo_container_manage_config, which is set to '/var/lib/tripleo-config/', to something deeper, we want Ansible to recursively search for the container config until it finds it. Change-Id: I7d23fec91ffb813f0ab6f11b85d811ef3897f9e0 (cherry picked from commit205f7c9b0c
) Idempotency for podman_container Introduce partial idepmpotency for running podman_container - Replace all options by their defaults it they're not set - Force lower case on all podman inspection data - Add a class with methods for every parameter to check its value and compare it with module argument - Add check_mode support and podman_actions Change-Id: I1ae93dff1e10a1a696bb171996a75a0db6c34fa3 (cherry picked from commit1212544a28
) Fix case where there're no effective caps When no effective caps in container, it should be list, not None Change-Id: I5412edc844ad43223c4b3bda35662a7f7ee43f3a (cherry picked from commit3a6690c904
) Add idempotency for networks parameter in podman_container Change-Id: Ib7235d1cb64ad0f42e7a0201008536e1a35bd696 (cherry picked from commit25f8abbc5a
) use version parsing from distutils use LooseVersion for version comparisons Change-Id: I609920a96c725c49f1623f60f8295d89ae4f3141 (cherry picked from commit8c83219fbb
) Improve idempotency for podman containers module Fix some pep8 issues Change-Id: If4233e57edeec10ccac965d61c78f30688cd5531 (cherry picked from commit0609c16d10
) Improve idempotency for podman container module Strip all registry values from "image" parameter in input. Add this case to test. Change-Id: I78656e48cf85a1a39f873ee40193765eecf02c56 (cherry picked from commit87d9e9d9a6
)
138 lines
4.4 KiB
INI
138 lines
4.4 KiB
INI
[tox]
|
|
minversion = 2.0
|
|
envlist = docs, linters
|
|
skipdist = True
|
|
|
|
[testenv]
|
|
usedevelop = True
|
|
install_command = pip install -c{env:UPPER_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/train} {opts} {packages}
|
|
passenv = *
|
|
setenv =
|
|
VIRTUAL_ENV={envdir}
|
|
ANSIBLE_ACTION_PLUGINS={toxinidir}/tripleo_ansible/roles.galaxy/config_template/action:{toxinidir}/tripleo_ansible/ansible_plugins/action
|
|
ANSIBLE_CALLBACK_PLUGINS={toxinidir}/tripleo_ansible/ansible_plugins/callback
|
|
ANSIBLE_FILTER_PLUGINS={toxinidir}/tripleo_ansible/ansible_plugins/filter
|
|
ANSIBLE_LIBRARY={toxinidir}/tripleo_ansible/roles.galaxy/config_template/library:{toxinidir}/tripleo_ansible/ansible_plugins/modules
|
|
ANSIBLE_MODULE_UTILS={toxinidir}/tripleo_ansible/ansible_plugins/module_utils
|
|
ANSIBLE_ROLES_PATH={toxinidir}/tripleo_ansible/roles.galaxy:{toxinidir}/tripleo_ansible/roles
|
|
ANSIBLE_INVENTORY={toxinidir}/tests/hosts.ini
|
|
ANSIBLE_NOCOWS=1
|
|
ANSIBLE_RETRY_FILES_ENABLED=0
|
|
ANSIBLE_STDOUT_CALLBACK=debug
|
|
ANSIBLE_LOG_PATH={envlogdir}/ansible-execution.log
|
|
VIRTUAL_ENV={envdir}
|
|
# pip: Avoid 2020-01-01 warnings: https://github.com/pypa/pip/issues/6207
|
|
# paramiko CryptographyDeprecationWarning: https://github.com/ansible/ansible/issues/52598
|
|
PYTHONWARNINGS=ignore:DEPRECATION::pip._internal.cli.base_command,ignore::UserWarning
|
|
PIP_DISABLE_PIP_VERSION_CHECK=1
|
|
sitepackages = True
|
|
deps =
|
|
-r {toxinidir}/test-requirements.txt
|
|
-r {toxinidir}/ansible-requirements.txt
|
|
commands = stestr run {posargs}
|
|
whitelist_externals =
|
|
bash
|
|
tox
|
|
|
|
[testenv:bindep]
|
|
# Do not install any requirements. We want this to be fast and work even if
|
|
# system dependencies are missing, since it's used to tell you what system
|
|
# dependencies are missing! This also means that bindep must be installed
|
|
# separately, outside of the requirements files.
|
|
deps = bindep
|
|
commands = bindep test
|
|
|
|
[testenv:pep8]
|
|
envdir = {toxworkdir}/linters
|
|
commands =
|
|
python -m pre_commit run flake8 -a
|
|
|
|
[testenv:ansible-lint]
|
|
envdir = {toxworkdir}/linters
|
|
deps =
|
|
{[testenv:linters]deps}
|
|
commands =
|
|
bash -c "ANSIBLE_ROLES_PATH='{toxinidir}/tripleo_ansible/roles.galaxy' \
|
|
ansible-galaxy install -fr {toxinidir}/tripleo_ansible/ansible-role-requirements.yml"
|
|
python -m pre_commit run ansible-lint -a
|
|
|
|
[testenv:yamllint]
|
|
envdir = {toxworkdir}/linters
|
|
deps = {[testenv:linters]deps}
|
|
commands =
|
|
python -m pre_commit run yamllint -a
|
|
|
|
[testenv:bashate]
|
|
envdir = {toxworkdir}/linters
|
|
deps = {[testenv:linters]deps}
|
|
commands =
|
|
python -m pre_commit run bashate -a
|
|
|
|
[testenv:whitespace]
|
|
envdir = {toxworkdir}/linters
|
|
deps = {[testenv:linters]deps}
|
|
commands =
|
|
python -m pre_commit run trailing-whitespace -a
|
|
|
|
[testenv:shebangs]
|
|
envdir = {toxworkdir}/linters
|
|
deps = {[testenv:linters]deps}
|
|
commands =
|
|
python -m pre_commit run check-executables-have-shebangs -a
|
|
|
|
[testenv:linters]
|
|
deps =
|
|
-r {toxinidir}/requirements.txt
|
|
-r {toxinidir}/test-requirements.txt
|
|
-r {toxinidir}/molecule-requirements.txt
|
|
commands =
|
|
{[testenv:pep8]commands}
|
|
{[testenv:ansible-lint]commands}
|
|
{[testenv:bashate]commands}
|
|
{[testenv:yamllint]commands}
|
|
{[testenv:whitespace]commands}
|
|
{[testenv:shebangs]commands}
|
|
bash -c '{toxinidir}/scripts/detect-broken-symlinks.sh'
|
|
|
|
[testenv:releasenotes]
|
|
basepython = python3
|
|
deps = -r{toxinidir}/doc/requirements.txt
|
|
commands =
|
|
sphinx-build -a -E -W -d releasenotes/build/doctrees -b html releasenotes/source releasenotes/build/html
|
|
|
|
[testenv:docs]
|
|
basepython = python3
|
|
deps =
|
|
-r {toxinidir}/doc/requirements.txt
|
|
commands=
|
|
doc8 doc
|
|
sphinx-build -a -E -W -d doc/build/doctrees -b html doc/source doc/build/html -T
|
|
|
|
[doc8]
|
|
# Settings for doc8:
|
|
extensions = .rst
|
|
ignore = D001
|
|
|
|
[testenv:venv]
|
|
commands = {posargs}
|
|
|
|
[testenv:role-addition]
|
|
deps=
|
|
{[testenv:linters]deps}
|
|
commands =
|
|
bash -c "ansible-playbook -i localhost, role-addition.yml -e role_name=skeleton_test"
|
|
bash -c "if (podman ps 2> /dev/null || docker ps 2> /dev/null); then \
|
|
cd {toxinidir}/tripleo_ansible/roles/skeleton_test; \
|
|
molecule test --all; \
|
|
else \
|
|
echo -e '\n\nNo molecule tests have been executed\nSee https://docs.openstack.org/tripleo-ansible/latest/contributing.html#local-testing-of-new-roles\n\n'; \
|
|
fi"
|
|
{[testenv:linters]commands}
|
|
|
|
[testenv:modules]
|
|
deps=
|
|
{[testenv:linters]deps}
|
|
commands =
|
|
bash -c "cd {toxinidir}/tripleo_ansible/ansible_plugins/tests; molecule test --all;"
|
|
{[testenv:linters]commands}
|