368 Commits

Author SHA1 Message Date
Mihai Plasoianu
f14007220f Fix node scaling
The playbook fails when removing unreachable nodes from deployment with
`openstack overcloud delete node`. Some `ignore_unreachable: true` are
missing. Also, one cannot use `any_errors_fatal: true` when ignoring
unreachable nodes, otherwise the playbook execution will stop after the
current task.

Change-Id: Ibcb84e58bac1975490df281c0de950cdf74337b2
2020-05-19 16:26:53 +02:00
Jose Luis Franco Arza
2c85bde000 Add common_deploy_steps to post_upgrade_steps.
When refactoring deploy-steps to add a common playbook [0] it seems that
the post_upgrade_steps_playbook block was missed. As a consequence, when
executing the post_upgrade_tasks some of the common Ansible variables
are not available.

[0] - Ib00e8aa9f7d06517290543a8aaf8a2527969bd3c

Change-Id: I04704a14a8b932e21d21348e10014c707a87eeeb
Closes-Bug: #1875579
2020-05-15 11:49:48 +00:00
Zuul
dfbf9f03d0 Merge "Split out selinux management" 2020-04-28 22:55:42 +00:00
Oliver Walsh
122eccbe28 Default to stack name inventory group for deployment_target_hosts param
Since I2cc721676005536b14995980f7a042991c92adcc we can no longer assume that
an overcloud group exists in the inventory. Default to the <stack_name> group
instead.

Change-Id: I895e315ff3984ebf1806288a8275a8b0d74bef49
Closes-bug: #1875429
2020-04-27 16:54:59 +01:00
Alex Schultz
07106c501e Split out selinux management
Currently if you have selinux enabled on the undercloud but disable it
for the overcloud, selinux is disabled on the undercloud during the
deployment. This can be resolved by only managing the selinux setting
for the deployment target hosts rather than the all.

Change-Id: I94b81ea0b954cdba7704720a145b752fa58d4308
Closes-Bug: #1874828
2020-04-24 08:54:30 -06:00
Cédric Jeanneret
cabed543fa Introduce new HideSensitiveLogs parameter
This one toggles the no_log parameter. Directly related to #1873770 in
order to allow a deeper debug within CI.

Change-Id: I27f677467263c0e6cc78d775edff55b3811fec1f
Related-Bug: #1873770
2020-04-20 16:44:09 +02:00
Bogdan Dobrelya
3b8e6f78e1 Use lists for storing host entries in Heat
This simplifies all the split/join transformations and improves the
memory footprint to a reduced list of unique entries for
HostsEntryValue (originally required for storing the ultimate data for
hosts entries in a form of a quite long single-line string value).

That improves the hosts entries processing for large scale deployments
and removes possible limitations to the sizes of strings.

Closes-bug: #1869375
Change-Id: I5ac498621e9e3c49def565744a7b521cb2cc5c25
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2020-03-27 15:20:49 +01:00
Bogdan Dobrelya
341ec7b9cc Drop unused remnants of the hosts-config bits
Hosts entries are used to be configured via tripleo_ansible's
tripleo_hosts_entries.

Ifd4bc4ce5618587c341ecbf37f82777ae6fc2f4a removed the use
of WRITE_HOSTS, which currently makes hosts-config.yaml "headless" and
taking no real data for the hosts-config.sh template that generates
outputs for OS::TripleO::Hosts::SoftwareConfig.

Also I606e0f27f9f9ae9d85bc0fc653f8985eb734d004 removed the use of
HOST_ENTRY, which makes the hosts-config.sh taking an empty value for
it.

Probably that all makes it safe now to remove any use of
hosts-config.sh and hosts-config.yaml and corresponding
OS::TripleO::Hosts::SoftwareConfig completely.

Change-Id: Id04767ae0c32caf62271cf564608350974fefd1b
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
2020-03-24 10:41:35 +01:00
Zuul
21c907f1de Merge "Use jinja raw tag instead of quoted concatenation" 2020-03-16 21:28:49 +00:00
Zuul
67bdcba7ab Merge "Use exists filter instead of stat where possible" 2020-03-16 15:02:25 +00:00
Zuul
7ede3a3543 Merge "Fail NetworkConfig task on timeout" 2020-03-16 14:33:56 +00:00
Jesse Pretorius (odyssey4me)
42059a1031 Use jinja raw tag instead of quoted concatenation
Many of the lines are difficult to grasp due to the crazy
quotation and concatenation implemented to get the desired
result from generating playbooks via a jinja template.

We can make it easier to read and easier to understand by
using the jinja raw tag instead. This eases the maintenace
burden on us all and helps us sleep better at night.

Change-Id: I82c4de4a63817707a2b0ed0ced827be37c0d0463
2020-03-13 11:18:11 +00:00
Jesse Pretorius (odyssey4me)
9666a7e645 Use exists filter instead of stat where possible
When checking for the existance of a file on the host where
Ansible is being executed, using the stat module with localhost
delegation is rather heavy-handed. We can instead just make use
of the 'exists' test.

This should improve execution time just a little bit and reduces
the number of tasks for us to maintain. We also remove the
repetition of the task file path by using a variable.

Change-Id: I8b278ca83b2afb07575dbae2496ec265c3a06473
2020-03-12 10:05:19 +00:00
Jesse Pretorius (odyssey4me)
8e1171d01a Use a common playbook import for common plays
It would appear that we use many of the same plays in several
of the playbooks. In this patch we extract these into a common
playbook file so that we only need to repeat a single import
in each of the playbooks. This way we reduce the maintenance
burden because we only need to maintain it in one file.

Change-Id: Ib00e8aa9f7d06517290543a8aaf8a2527969bd3c
2020-03-09 16:20:29 +00:00
Emilien Macchi
bfb8b2b7fd Cleanup tasks for container-puppet.py
- When Paunch is disabled, don't create container-puppet.py and if the
  file exists, make sure we remove it so operators don't run it by
  accident.
- Remove the reference of that script from the README and the commands,
  to make it clear there is a new tool now.

Change-Id: I5032eef6567b37c02fe53dea852aadff3e185eec
2020-03-02 13:16:49 +00:00
Zuul
58cd2e915a Merge "Rename tripleo-hieradata to tripleo_hieradata" 2020-02-29 02:39:44 +00:00
Zuul
bb231974c2 Merge "Use a jinja block to reduce str_replace repetition" 2020-02-28 09:43:03 +00:00
Zuul
42f130ecc7 Merge "[update/upgrade] Use include_tasks instead of import_tasks" 2020-02-28 09:43:01 +00:00
Emilien Macchi
bb1c568857 Rename tripleo-hieradata to tripleo_hieradata
The role is now tripleo_hieradata for the new versions of Ansible.

Change-Id: Idd2606db44cd7514537ee870be189760fdf70ae4
2020-02-25 10:01:40 -05:00
Jesse Pretorius (odyssey4me)
dda2030a6a Use a jinja block to reduce str_replace repetition
It would appear that we use the same str_replace params most
of the time, and there's no harm in using aliases where only
some of them are used. This way we reduce the maintenance
burden because we only need to maintain it in one place.

Change-Id: Ib034405a15ade9e9fb234a9875ebbe922abfdfc6
2020-02-24 17:49:37 +00:00
Jesse Pretorius (odyssey4me)
3040a61411 [update/upgrade] Use include_tasks instead of import_tasks
include_tasks is dynamic and the tasks are either included (or not) at
runtime. This has the advantage that if a "when" keyword excludes the
include_tasks, then all the tasks are excluded as a group.

This is opposed to import_tasks which happen at playbook parse time. The
"when" keyword is inherited by each individual task that was imported.

While the two are functionally equivalent for these use cases,
import_tasks ends up being much slower, since ansible then has to
compute a much larger set of tasks to skip at runtime. Using
include_tasks is much faster, even at small scale (~50 hosts).

This is applying what was done in https://review.opendev.org/697510
to the update/upgrade tasks.

When doing include_tasks, we ensure that we also apply the 'always' tag
so that we have access to use the tags in the included task files. See
[a] for further details.

[a] https://odyssey4.me/2019/11/26/ansible-include-tags.html

Change-Id: I2eab008ca27546acbd2b1275f07bcca0b84b858c
2020-02-24 17:49:37 +00:00
Emilien Macchi
9018e4e42d Disable Paunch by default Standalone/Overcloud
Except for standalone004 and ovb-ha jobs which still run on Docker,
let's enable tripleo-ansible to manage the containers instead of paunch.

Depends-On: https://review.opendev.org/#/c/709043
Change-Id: Ib29e7c9ce4028e1cb6f6ea6c0ae77890aefde93b
2020-02-22 18:35:48 +00:00
Zuul
c9fbf190b8 Merge "Generate /etc/hosts early on both under and overcloud" 2020-02-20 15:48:10 +00:00
Sofer Athlan-Guyot
536230b323 [update] Ensure we get fresh hiera data before running update_steps.
We need to make sure that the hiera data are fresh before the update
step so that anyone using those data during those steps are seeing the
latest information from heat.

Factor out the hiera generation and include it in deployment and
update playbooks.

The double tasks definition in the deployment playbook seemed to be
redundant so It has been removed.

Change-Id: I6b6c676880ccc8cbed23af135e5865c222a8f1d0
Closes-Bug: #1861799
2020-02-10 23:57:40 +00:00
Saravanan KR
f17e06a778 Fail NetworkConfig task on timeout
Running os-net-config as async, with failed_when as false
results in undefined variable error if async task times out.
Instead of ignoring failure of task, check for the
presense of results of the command execution 'rc', if it
is not defined, then rest of the tasks are not useful.
Closes-Bug: #1862627

Change-Id: Ibbcde856ac69bf73a47086d95a52c3b1a0d10911
2020-02-10 20:53:02 +05:30
Zuul
b130f78076 Merge "Replace svirt_sandbox_file_t by container_file_t" 2020-02-10 13:58:31 +00:00
Cédric Jeanneret
0875895553 Replace svirt_sandbox_file_t by container_file_t
While they are, at SELinux level, exactly the same (one is an alias to
the other), the "container_file_t" name is easier to understand (and
shorter to write).

A second pass in a couple of days or weeks will be needed in order to
change files that were merged after this first pass.

Change-Id: Ib4b3e65dbaeb5894403301251866b9817240a9d5
2020-02-07 13:33:20 +01:00
Jose Luis Franco Arza
ee5d5e55ee Create /var/lib/config-data if it doesn't exist.
When performing the Undercloud upgrade from OSP13
to OSP16, we start from an almost empty Undercloud
node which has been upgraded to RHEL8 via Leapp.
The /var/lib/config-data is being lost during the
upgrade procedure, so this task makes sure that the
folder exists before checking the selinux state.

Change-Id: I760a4e532e0c299efcf57cee68e8e8f93795ea29
2020-02-07 10:17:29 +01:00
Zuul
d578748bab Merge "Dynamically include container-puppet tasks" 2020-02-07 01:19:17 +00:00
Zuul
7101e5f615 Merge "Remove unnecessary block/whitespace from step1 tasks" 2020-02-07 01:11:08 +00:00
James Slagle
1602d68daa Dynamically include container-puppet tasks
The container-puppet tasks only need to be run if tasks actually exist, which
is already being checked on the ansible control node.

A "when" statement is then applied to the set of tasks necessary to run
the container-puppet tasks, when the tasks are actually defined.

This patch moves that set of tasks to a separate tasks file and uses a
dynamic include.  This results in less tasks being skipped, which can
save several minutes at scale. This results in 3 less tasks that need to
be skipped at steps 1-5, which equates to 15 tasks overall, when no
container-puppet tasks actually exist.

When container-puppet tasks do exist, all the tasks will be executed as
necessary.

Change-Id: Ifad32bf79942cde58295fd9aae7e23e2f62c1ae2
2020-02-06 18:00:18 +00:00
Cédric Jeanneret
96c40f89be Generate /etc/hosts early on both under and overcloud
Prior to this patch, the /etc/hosts was generated only on the overcloud
nodes, leading to some issues when it comes to TLS-Everywhere, as raised
in associated bug.

Depends-On: https://review.opendev.org/706242
Change-Id: I836ab1a23c8aea35c0cea54d0765c7313a4b9038
Closes-Bug: 1861782
2020-02-06 12:36:37 +01:00
Zuul
c2eeafdd2b Merge "Dynamically include generate-config tasks" 2020-02-05 09:13:21 +00:00
Zuul
38ded7a003 Merge "Use action plugin for all_nodes data" 2020-02-05 09:13:19 +00:00
James Slagle
c8bc412e4f Dynamically include generate-config tasks
The generate-config tasks which run on the host to generate config data
under /var/lib/config-data, only run at step1.

There are several tasks that used a when statement to only run the
related tasks at step1. This patch moves all the related generate-config
tasks to a separate tasks file, which can then be dynamically included
at step1.

Using a dynamic include results in less tasks being skipped, which can
save several minutes at scale. This results in 4 less tasks that need to
be skipped at steps 2-5, which equates to 16 tasks overall.

Change-Id: Ifdddcb13362e26babedd47e674089fb0e2a37994
2020-02-05 02:37:58 +00:00
James Slagle
c8dcce7089 Use action plugin for all_nodes data
Update deploy_steps_playbook.yaml to use the new action plugin for
rendering the all_nodes data. The native python is much faster than the
jinja2 template:

Change-Id: I3ac05c30f7c5d136c5da9441faf7890cb6fb9d05
2020-02-05 02:37:50 +00:00
Kevin Carter
80f44a4388 Rename roles that we're missed
These roles were not renamed when we removed all of the hyphens.
This change removes the remaining hyphenated roles.

Change-Id: I10a0064fa0bdb80957a3ef7acfe376c745d8512b
Signed-off-by: Kevin Carter <kecarter@redhat.com>
2020-02-04 20:53:58 +00:00
Zuul
10cee0c174 Merge "Rename common deploy step 1 tasks" 2020-02-04 00:26:43 +00:00
Jesse Pretorius (odyssey4me)
134795a13d Remove unnecessary block/whitespace from step1 tasks
In Id5985ce8ac741baa9adc9f5874df0459fd4c24b2 the step1 tasks were moved
into their own file with the leading whitespace and block statement left
exactly as they were.

This patch removes the unnecessary block statement, moves the block name
to the parent inclusion and removes all the leading whitespace.

Change-Id: I243c761a88f746a6abb4ddb13845e813eaf7155c
2020-02-03 12:14:46 +00:00
James Slagle
5d8425ddaf Rename common deploy step 1 tasks
There was already a play/task for "common deploy step tasks 1", so this
was a bad name to begin with. Use common bootstrap tasks instead.

Also adds a missing debug task.

Change-Id: I9840a26f10d8ad72b5fa187e56b1b3dbfd63e40d
2020-01-29 17:19:52 -05:00
Jesse Pretorius (odyssey4me)
bc4df9c5a9 Use YAML anchors/aliases to reduce playbook task repetition
It would appear that we use many of the same tasks in several
of the plays. We can use anchors/alaises to reduce this
repetition This way we reduce the maintenance burden because
we only need to maintain it in one place.

Change-Id: I2c8a4a0270c99d76500ac42d90fffdc0475cb995
2020-01-29 09:50:45 +00:00
Jesse Pretorius (odyssey4me)
2092b1303f Update ffwd-upgrade branch names
The next iteration of fast-forward-upgrade will be
from queens through to train, so we update the names
accordingly.

Change-Id: Ia6d73c33774218b70c1ed7fa9eaad882fde2eefe
2020-01-27 19:42:40 +00:00
Zuul
d2cd8acf31 Merge "Update all roles to use the new role name" 2020-01-24 07:12:44 +00:00
Jose Luis Franco Arza
14db20baee Force facts cache refreshing before upgrade.
When upgrading from Rocky to Stein, an upgrade of the operating system is
performed. This upgrade from RHEL7 to RHEL8 implies the removal of the
default /usr/bin/python binary. As the facts cache is enabled, Ansible's
strategy does not consider to upgrade facts and therefore we try to run the
ansible playbook using the old python binary when running the upgrade.
This fails with the error: /usr/bin/python: No such file or directory.

This patch makes use of the setup task in combination with gather_facts
false, to ensure that the facts are gathered and refreshed for the
Overcloud nodes. This way, we make sure that we are using the right
python binary. As during scale, a similar situation is occuring, this
patch adds the same logic in scale_playbook.

Closes-Bug: #1856313
Change-Id: I87974e88c38b42e90bc3cd801fcf1deaf268720c
2020-01-22 12:25:46 +01:00
Kevin Carter
9a2a36437d
Update all roles to use the new role name
Ansible has decided that roles with hypens in them are no longer supported
by not including support for them in collections. This change renames all
the roles we use to the new role name.

Depends-On: Ie899714aca49781ccd240bb259901d76f177d2ae
Change-Id: I4d41b2678a0f340792dd5c601342541ade771c26
Signed-off-by: Kevin Carter <kecarter@redhat.com>
2020-01-20 10:32:23 -06:00
Emilien Macchi
1046bf6359 container-puppet: remove deprecated directory & json files
- Remove /var/lib/docker-puppet which was depecrated in Stein, and not
  used since. The new directory is /var/lib/container-puppet.
- Remove /var/lib/tripleo-config/container-startup-config-*.json
  generation, since it's not done per step and per container in
  /var/lib/tripleo-config/container-startup-config/step_X/*.json

- Adapt container-puppet.py to point to the right json file by default.
- Adapt deploy-steps.j2 to check the step configs directory instead of
  the deprecated json file.

Change-Id: I98963941c9d969ab1dfd92d70f973013f84e1c25
Note: this patch won't be backported to Train.
2020-01-16 09:37:13 -05:00
Zuul
2d48f121ea Merge "Add missing any_errors_fatal" 2020-01-13 09:44:57 +00:00
James Slagle
c8d53f86c1 Add missing any_errors_fatal
Each play in deploy_steps_playbook.yaml should set
any_errors_fatal=true. This patch adds that argument where it was
missing.

We don't yet have the capability in TripleO to continue the deployment
if some hosts fail because we don't have per-role logic specifying which
roles need 100% success and which ones don't. Once that is available,
any_errors_fatal=true could be removed.

Change-Id: I1b6dc3cec6199fd50a779cde3a8199ba19297191
Closes-Bug: #1859175
2020-01-10 07:57:35 -05:00
Emilien Macchi
7f40baabcd Manage all Keystone resources with Ansible
Depends-On: I557d8f33c9c699aed14b3b6fc1d1c0407365cd08
Depends-On: Ia68f8852662fb4abbd194954a246afb740bf3f71

Change-Id: I96a3351fca26cd8bb122a86cb4c3a58d5f88573e
2020-01-06 22:33:05 +00:00
Zuul
0a8c7702bb Merge "Do not configure Hiera and Hieradata in Ansible check mode" 2020-01-02 23:46:39 +00:00