368 Commits

Author SHA1 Message Date
Jiri Stransky
22f26d7a94 Fix MariaDB staged upgrade
This patch fixes 2 issues on MariaDB staged upgrade:

* After we landed service stopping on the data transfer step, the
  actual MariaDB staged upgrade moved from step 1 to step 2 of
  external_upgrade_tasks, but we only ran steps 0 and 1 during the
  upgrade. Increase the number of steps to run step 2 as well.

* The mysql_upgrade container was being re-run on each upgrade-scaleup
  even on nodes where it already ran before and MariaDB was already
  running there, conflicting with the upgrade container. We fix this
  by only running the MariaDB upgrade if there's no MariaDB running
  yet on the particular node.

Change-Id: I2feba6c22cdf07ea6af406c5edd27f62ff285d87
Co-Authored-By: Damien Ciabrini <dciabrin@redhat.com>
Closes-Bug: #1841415
2019-08-26 10:26:52 +02:00
Jose Luis Franco Arza
6c675af9ba Add tags always into external update tasks.
After bumping the Ansible version in Tripleo to 2.6, it was needed
to perform a change in the include module into include_tasks or
import_tasks, as include was getting deprecated [0]. The external
update/upgrades tasks got impacted by that change, but as they use
a loop to execute the tasks we couldn't use import_tasks.
The way include_tasks handles the tasks execution depending on the
tags differs from import_tasks (dynamic vs static) and as a consequence
when running the external upgrade run passing --tags container_image_prepare
we didn't see any tasks running. This behavior got fix in [1], which
seems to be the right way to preserve the import_tasks tags handling as
explained in [2]. But the external update tasks were missed to patch.

This patch includes the tags: always statement inside the external
update tasks and also syncs the variables content we pass into external_update_tasks
as many of those variables, which we do pass for the external_upgrade_playbook,
were missing in the external_update_playbook.

[0] - https://review.opendev.org/#/c/579844/
[1] - https://review.opendev.org/#/c/639642/1
[2] - https://github.com/ansible/ansible/issues/30882#issuecomment-380596557
Closes-Bug: #1839520
Change-Id: If7e7b4bbb3cead0887384cc543ce37e9ee5396ab
2019-08-14 12:43:25 +00:00
Rabi Mishra
7e3ead74ff Use hiera config from ansible group_vars
Also drops unnecessary for loop from 'Server hieradata from vars'
task.

Change-Id: I0450bbdc9d2e24e8449d6a302cc672631f8312f6
2019-08-13 12:01:27 +05:30
Rabi Mishra
6ff7c512c3 Use ansible group_vars for extraconfig hieradata
Change-Id: I582fad2bd7b058a573c6e0398f05906582461861
Depends-On: https://review.opendev.org/#/c/673728
2019-08-06 23:21:47 +05:30
Rabi Mishra
5b2d2795a8 Use ansible group_vars for service_configs hieradata
Change-Id: Ifa05dc0bce634ad06efa7f4658845c8d6d044818
Depends-On: https://review.opendev.org/#/c/673727
2019-08-06 23:21:47 +05:30
Zuul
880838c325 Merge "Fix misspelling of "errors" in deploy-steps.j2" 2019-07-31 03:23:53 +00:00
Zuul
b881a5e8e7 Merge "Move the Hiera symlink task from post configuration to deployment steps." 2019-07-31 02:27:36 +00:00
Luke Short
3b813d845c Move the Hiera symlink task from post configuration to deployment steps.
This helps to consolidate the Undercloud and Standalone deployments. It
also avoids an issue where the Ansible Python interperter cannot be
found.

Change-Id: I01a95be975011d2419a523da572503e0ebcfa49b
Resolves: rhbz#1733608
Signed-off-by: Luke Short <ekultails@gmail.com>
2019-07-29 16:48:03 -04:00
John Fulton
efae2298ad Fix misspelling of "errors" in deploy-steps.j2
Change-Id: I0e2429dc22a4ed03619b9c3f52b0dd1108aea56b
Closes-Bug: #1838099
2019-07-26 19:48:46 +00:00
James Slagle
87e3491265 Restore AllNodesExtraMapData
Commit a7661065743086961b8ef93056b810e7d2a49eda removed the
all-nodes-config.j2.yaml Heat template. This template had the support
for specifying hieradata to merge into the all_nodes hieradata via the
AllNodesExtraMapData. This commit restores the parameter which is now
set as an Ansible variable which will be used by the tripleo-hieradata
role.

Change-Id: Ib3838d404b45ae6684ef7e0aa211cb8015271866
2019-07-24 16:08:33 -04:00
Zuul
b8de91a534 Merge "Set HieradataVars as group vars instead" 2019-07-19 07:32:15 +00:00
Zuul
742abc27a2 Merge "Remove tripleo_ssh_known_hosts_use_template var" 2019-07-19 07:31:43 +00:00
Zuul
cf21af0b95 Merge "Use template for NetworkConfig script" 2019-07-15 19:03:38 +00:00
Zuul
5ead29be3b Merge "Set nova_additional_cell as global_vars" 2019-07-15 18:37:16 +00:00
James Slagle
5c53029232 Set HieradataVars as group vars instead
Instead of writing a set of vars to hieradata_vars.yaml and then passing
that into tasks with vars_files, just set the data as group vars.

This makes for a simpler interface without having to remember to pass in
the data with vars_files on any task/play that might require them.

Change-Id: I851d9404861d55791bde1346c58bc1c94f591569
2019-07-13 11:50:30 +00:00
James Slagle
4a457fa529 Remove tripleo_ssh_known_hosts_use_template var
This var is no longer required to pass into the tripleo-ssh-known-hosts
role as using the template will be the only mode going forward.

Change-Id: I0bde2ac03edf26f5594381caa9542aa1eefd8f20
Depends-On: I4a2002fa42337e115f9679fcf2f2db71308389d6
2019-07-13 11:50:23 +00:00
James Slagle
2f6f519149 Use Ansible for ssh known hosts config
Migrate to using the template mode of the tripleo-ssh-known-hosts role.

Detailed Changes:
common/deploy-steps.j2:
  Remove ssh_known_hosts_hostnames parameter
  Remove ssh_known_hosts variable in config-download output
  Set tripleo_ssh_known_hosts_use_template var to true when including
    the tripleo-ssh-known-hosts role to activate template mode.
extraconfig/tasks/ssh/host_public_key.yaml:
  Template removed as it was deprecated and is no longer used.
extraconfig/tasks/ssh/known_hosts_config.yaml:
  Template removed as it was deprecated and is no longer used.
overcloud-resource-registry-puppet.j2.yaml:
  Mappings for Ssh known hosts resources removed
overcloud.j2.yaml:
  Resources removed: SshKnownHostsConfig, SshKnownHostsHostnames,
    {{role.name}}SshKnownHostsDeployment
  Removed passing in ssh_known_hosts_hostnames parameter to
    common/deploy-steps.j2
puppet/role.role.j2.yaml:
  Removed SshHostPubKey and SshKnownHostsHostnames resources
  Removed known_hosts_entry and known_hosts_hostnames stack outputs
  Add role_networks to set Ansible group vars which is a list of enabled
    networks for the role

Change-Id: I10d0de42acc0b88b7947b464e976b5d0b9067ca2
2019-07-13 11:50:01 +00:00
Martin Schuppert
6413ef838e Set nova_additional_cell as global_vars
I9a37d1faec73a81a28d8f89d86375fb15ee765c7 moved to use ansible
for all_nodes hieradata. This also removed to create the
SERVICE_cell_node_names in in net_ip_list_map.j2.yaml.
This change addes nova_additional_cell to the global_vars that
the all_nodes hieradata via ansible can write cell service
hostnames.

Change-Id: I58c635b0fd92fa5a437cd6abd7c954d9cad720c9
2019-07-12 13:39:06 +02:00
James Slagle
4f9b32a9f4 Lookup pre/post deployment vars with hostvars
The lookup function that was used previously still failed when these
vars were not defined. Using hostvars instead allows these to be
defaulted to [] when the vars are not actually defined.

Change-Id: I51f3c21f07a893848483a14e399a6d402a6205b8
2019-07-11 13:16:54 -04:00
James Slagle
0aa06134c6 Use template for NetworkConfig script
Use the first_found Ansible lookup plugin to look for the NetworkConfig
script in both the role directory and server directory. This allows for
using a role generic NetworkConfig script in the role dir, while still
being backwards compatible.

The NetworkConfig script is also rendered from a template on the remote
node instead of just copied. Treating it as a template allows it to be
made generic per role instead of node specific.

Depends-On: I6bbca8eb256f387c4f74378baa5253890cfe1abf
Change-Id: Ibbb5dd2ccbdaec083ec636ac44dfdc75b65f89a7
2019-07-10 14:49:28 -04:00
James Slagle
c5ba2cec81 Use Ansible for {{role}} and host_extraconfig hieradata
Migrates these hieradata files to use the cloud_domain and fqdn
templates from tripleo-hieradata. The templates have more meaningful
names to actually represent the data that these hieradata files contain.

Detailed changes:
common/deploy-steps.j2:
  Add parameter for CloudDomain and add it to the Ansible global vars
  Remove the Per-host hieradata tasks and parameters as they are no longer used
  Add cloud_domain and fqdn to list of templates for tripleo-hieradata
overcloud.j2.yaml:
  Add role_networks to HieradataVars for each role
  Replace {{role}} and host_extraconfig from hieradata hierarchy with
    cloud_domain and fqdn.
puppet/role.role.j2.yaml:
  Remove {{role}} hieradata from hieradata Datafiles as the template is
    now used instead.
  Remove per host hieradata resources and output as they are no longer
    used.

Depends-On: I9c038399fccc4730b73e1a52281d7256ea689ee9
Change-Id: I136fb7aa864a2a3668f5a3845f3ded28b13bfe43
2019-07-10 18:13:03 +00:00
James Slagle
27e7005ad4 Use Ansible for net_ip_map hieradata
Migrates the net_ip_map hieradata from Heat to the tripleo-hieradata
role.

Detailed changes:
network/ports/port.j2:
  cidr output added for the network subnet cidr of the port
  port for a role
network/ports/noop.yaml:
  cidr output for compatiblity with port template interface
puppet/role.role.j2.yaml:
  {{network.name}}_cidr output added for the each networks subnet cidr
  Remove the net_ip_map hieradata from the per host datafiles generated
  by Heat as tripleo-hieradata template will be used instead.
overcloud.j2.yaml:
  consume cidr outputs from the roles, as well as the ctlplane subnet
  cidr and set them as hieradata vars that get passed into the
  templating process for tripleo-hieradata
common/deploy-steps.j2:
  Add a networks data structure to the global vars setting some needed
  vars for tripleo-heiradata.
  Use the net_ip_map template from tripleo-hieradata

Change-Id: Ib30e8347e1a50efec2b6a97482dc98c383cb7cf7
Depends-On: Iaec3654900f250eacf39c7748df166b716937358
2019-07-10 18:12:48 +00:00
Cédric Jeanneret
c36433e34e Run NetworkConfig only if configuration script exists
Script is first located/generated on the undercloud node (aka ansible
host), meaning we have to use local_action.
We also deactivate the "become" in order to avoid useless privilege
escalation.

Change-Id: I8c1ed334dc5b578a87307a47656ee2d87f1e3688
Depends-On: https://review.opendev.org/668560
Related-Bug: #1834094
2019-07-08 15:36:59 +02:00
James Slagle
02fe40cd8b Run NetworkConfig if never run previously
Update the condition to test if NetworkConfig should be run to also
include logic to run if it was never run previously. Matches prior
behavior when using a SoftwareDeployment resource

Change-Id: Ibd2c7d96a6ad8d207cbb484b7e211909f897e43f
2019-06-28 10:24:15 -04:00
James Slagle
a766106574 Use Ansible for vip_data hieradata
Migrates the vip_data hieradata to the tripleo-hieradata role which will
use a jinja2 template to create this file on the deployed nodes. Allows
for removing the AllNodesConfig resource/template completely.

Change-Id: I5412f26423a1739088f12705fa47bac2377b9817
Depends-On: Iaf91f36a189d8e00627c21f14ca992a4d60fa10f
2019-06-28 10:21:56 -04:00
James Slagle
4c34147cc1 Use Ansible for all_nodes hieradata
Migrate the generation of the all_nodes hieradata from Heat to using the
tripleo-hieradata role instead.

Change-Id: I9a37d1faec73a81a28d8f89d86375fb15ee765c7
Depends-On: I6e1e1c28dc09c9e04119db910068d62409a5afc8
2019-06-28 10:21:56 -04:00
James Slagle
f44f339399 Use Ansible for bootstrap_node hieradata
Instead of generating and saving the hieradata in the bootstrap_node
hieradata file in Heat, use the tripleo-hieradata role instead.

Change-Id: Id6a7b10acc65ca6cd2135796a80fad0723078871
Depends-On: I9ccccbfee44fc7c1375dec7357ccad6cba909c16
2019-06-28 10:21:56 -04:00
Emilien Macchi
e115e14e53 Use Ansible to configure the Hiera config
Replace the Heat resources which used to manage the hiera config file
and now use Ansible with the new tripleo-hieradata role.

Remove {{server_resource_name}}Deployment and
{{server_resource_name}}Config resources not needed anymore.

Change-Id: I88fe3460af93b83cc086afeb1c2b959bbb720470
implements: blueprint reduce-deployment-resources
Depends-On: Ia5f11ea1a2f16b736ed7053c5182c7f5a7eb1f4b
2019-06-21 10:15:34 -04:00
Emilien Macchi
cb767a97b9 Use Ansible for {{server_resource_name}}Deployment
Use the new tripleo-hieradata Ansible role to configure
the remaining Hieradata files that were created by Heat before.

{{server_resource_name}} isn't yet fully converted to Ansible, and
another patch will remove the resource when Ansible controls the
hiera.yaml file in /etc/puppet/hieradata.

Change-Id: Ib946b566d66300d23e7abc12498cc11d3f5fa5f8
implements: blueprint reduce-deployment-resources
2019-06-21 10:14:37 -04:00
Emilien Macchi
17e16e5107 Interface for Ansible Host variables
The new AnsibleHostVarsMap output is a map of roles to Ansible host
vars, which later will be in config-download and populate the Ansible
host vars for each host in each role.

Change-Id: I3880f0f72beb24ee0b0868dca48afce6328144d3
2019-06-18 02:47:17 +00:00
James Slagle
79d9113e5d Use ansible for HostsConfig
Replace the HostsConfig SoftwareDeployment with an ansible task.

implements: blueprint reduce-deployment-resources
Change-Id: Ifd4bc4ce5618587c341ecbf37f82777ae6fc2f4a
2019-06-17 12:05:27 +00:00
James Slagle
b31b7fd4e1 Use ansible for ArtifactsConfig
Replace the Heat SoftwareDeployment with an ansible task for running the
deploy-artifacts.sh script.

Change-Id: Ib5f114440fcfcb7a7652111f0651ef01761d2601
implements: blueprint reduce-deployment-resources
2019-06-15 03:50:17 +00:00
James Slagle
0cd9b72541 Use ansible for NetworkConfig
Use ansible to apply the NetworkConfig resource instead of the
NetworkDeployment resource. The script module is used to run the same
rendered script as NetworkDeployment. In the future the script could be
refactored into a more proper ansible role, but the first step is to
remove the SoftwareDeployment for NetworkConfig and just use an ansible
task.

Change-Id: Ie60ddb90562f9630c24711337473f9e3acdcf4c1
Depends-On: Ie543782569de14d56bc41740611f7512e8357a22
implements: blueprint reduce-deployment-resources
2019-06-15 03:49:17 +00:00
James Slagle
ec68f1740c Use ansible for AllNodesValidationConfig
Replaces the Heat SoftwareDeployment resoure for
AllNodesValidationConfig with an Ansible stask instead.

Depends-On: I0b3188ef45dc379f83c36561616b422418845b66
Change-Id: I6e8f8f29cf9ef906b9be55f27aa536d8cc200190
implements: blueprint reduce-deployment-resources
2019-06-15 03:49:06 +00:00
Zuul
42dd7da0a6 Merge "Temporarily add .yaml extension for all-nodes-deployment-hieradata.j2" 2019-06-14 08:47:56 +00:00
Zuul
4adbf118e6 Merge "Replace hardcoded gather_facts:no with variable" 2019-06-13 18:51:04 +00:00
Zuul
40eca6ca87 Merge "Explicitly set ansible_python_interpreter in external upgrade playbook" 2019-06-13 15:18:53 +00:00
James Slagle
4373d6461f Temporarily add .yaml extension for all-nodes-deployment-hieradata.j2
Write both paths with and without the .yaml extension until
https://review.opendev.org/#/c/663736 merges. This patch will allow the
tripleo-common patch to merge, then we can come back and clean this up.

Needed-By: I0b3188ef45dc379f83c36561616b422418845b66
Change-Id: I1434da248bb9bc06e38381a47f274e1480fd450c
2019-06-12 14:58:05 +00:00
James Slagle
203418529a Replace hardcoded gather_facts:no with variable
Use a gather_facts variable to control fact gathering at the play level
instead of just the hardcoded gather_facts:no.

This allows for forcing facts to be gathered during a play if the
earlier fact gathering tasks had been skipped (such as when using
--start-at-task).

Change-Id: I2b7625baac4a09413cc50b84d2df1c00ec14b00a
2019-06-12 14:57:30 +00:00
Jiri Stransky
770b578bfb Explicitly set ansible_python_interpreter in external upgrade playbook
During the run of external upgrade playbook, we connect to systems
with different OS releases via delegate_to, which seems to confuse the
autodetection of Python interpreter. Set it explicitly to
platform-python which should work both on EL7 and EL8.

Co-Authored-By: Sergii Golovatiuk <sgolovat@redhat.com>
Change-Id: Ieb486f0c3e4de7229cdd2be49d4d0f038668f06d
Closes-Bug: #1830713
2019-06-11 15:22:41 +00:00
Mathieu Bultel
a59a188a6b Force ansible serial to 1 for the Controller
By introducing update_serial variable we parallelize update
execution on non-pacemaker enabled nodes. Custom role data users
need to update their role files. By default we do serial 1 making
sure nothing changes for users who didn't update their role data.

Resolves: rhbz#1652057
Closes-Bug: #1831617

Change-Id: I4ee0110a6c2b9466d81e37e5df27f5f81a6eceb5
2019-06-07 14:25:54 +02:00
Zuul
b5cf51665d Merge "Use ansible for AllNodesDeployment" 2019-06-06 02:19:25 +00:00
James Slagle
9be6c1c935 Use ansible for AllNodesDeployment
Replaces the SoftwareDeployment resource for AllNodesDeployment with the
usage of the new tripleo-hieradata role.

Change-Id: Ic174699ef038ca3028c21a49373658a5d43844f6
Depends-On: I28b5c2bedbba1c27da628c6bda4c7a57ca0f3ad7
implements: blueprint reduce-deployment-resources
2019-06-03 13:34:35 -04:00
Jiri Stransky
6d9560e177 Respect tags in upgrade tasks
There seems to be a difference between how include_tasks and
import_tasks work.

`include_tasks` applies properties on the inclusion itself, and if we
want to apply something on the included tasks, we need to use `apply`
dict. We previously had to add `always` tag onto `include_tasks` to
make the inclusion happen when we ran upgrade with `--tags`.

`import_tasks`, on the other hand, is processed more like a block than
an individual task, and all its properties get applied on the tasks
inside. This meant that the `always` tag got applied on all tags in
the upgrade playbook, instead of on the import itself, which broke use
of tags in `upgrade_tasks`. This is now fixed by removing the tags
from `import_tasks`. The import should happen always regardless if
there are any `tags` on the import.

Change-Id: I66a4ed99f9e0cc199899494813073b4a085d99e7
Closes-Bug: #1830892
2019-05-30 07:35:27 +00:00
Emilien Macchi
016279b71e standalone/undercloud - post: use EndpointMap to fetch Keystone URL
Using EndpointMap to ensure we get the hostname/fqdn if possible
otherwise it fallbacks to the IP for Keystone public endpoint.

This is useful when the operator uses a certificate based on
hostname/fqdn and not an IP address.

Closes-Bug #1763776
Change-Id: Ifa9d55cca90caf5be0c83507cb47447e25311fce
2019-05-21 08:41:22 -04:00
Michele Baldessari
ef6c23ef64 Fix haproxy firewall rules
The problem we want to selve is that the change
https://review.opendev.org/#/c/631486/ (moving iptables creation to the
host) never really worked.

The reason it never worked and we never noticed is two-fold:
A) It ran: -e include ::tripleo::profile::base::haproxy
the problem is that without quoting puppet basically does a noop

B) Once the quoting is fixed it breaks because 'export FACTER_step'
exports a custom fact but does not export a hiera key per-se (so calls
to hiera('step') would fail

So we add proper quoting only on the variables that are arguments to a
parameter so that there is no risk of ansible doing the wrong thing and
puppet gets the correct arguments.

We also explicitely set the step for hiera in the deploy_steps_tasks.
The reason we need it is because in non-HA the iptables rules would
be created at step 1. But since the deploy_steps_tasks run before the
actual tasks that set the step hieradata.we would get the following
error:
Error: Function lookup() did not find a value for the name 'step'

We can just write out the step hiera key during the deploy_steps_tasks,
it will be enforced again shortly afterwards once the
common/deploy-steps-tasks.yaml gets invoked.

We also switch back to puppet_execute: ::tripleo::profile::base::haproxy
even for the pacemaker profile. This was broken by the flattening of the
haproxy service (Id55ae44a7b1b5f08b40170f7406e14973fa93639)

Co-Authored-By: Luca Miccini <lmiccini@redhat.com>

Change-Id: Iab310207ca17a6c596470dda30a39e029c4fe09c
Closes-Bug: #1828250
2019-05-10 17:42:39 +02:00
Mathieu Bultel
9755a1b2d6 Enable serial execution for ansible host
Allow to override serial parameter for parallel execution
in Ansible

Needed-By: Iafd7c58aaf2f1b30cf46cd16122b5821c96c497c
Needed-By: I15511b4f36260292e0ea4100b15b8e65a701b38b

Change-Id: I707282bf3cfcc7e370af41f987700e39c1b0b268
2019-05-07 14:03:27 +00:00
Zuul
17f70ba91e Merge "Split upgrade_steps_playbook into different plays." 2019-05-03 13:41:34 +00:00
Zuul
06362bcdc2 Merge "Add ServiceNetMap to global_vars" 2019-05-02 01:13:20 +00:00
James Slagle
e0d26441f1 Add ServiceNetMap to global_vars
This adds the ServiceNetMap value to global_vars for config-download.
This will make the value consumable from ansible tasks when running
config-download.

Additional values can be added in a similar fashion in the future to
allow for less hardcoded data coming out of Heat, and instead using
jinja expressions to consume data from Ansible variables when
config-download runs.

Change-Id: I8c442caac140f1c96123c1be47e858949419fd8f
2019-04-24 14:10:30 -04:00