openstack
/
tripleo-upgrade
Code Issues Proposed changes

Merge tripleo-upgrade repo from redhat-openstack namespace 

Make sure scripts are created with the executable bit set.

Change-Id: I731902411e987b4ea7c2aa84fef869fe5e1c25ae

Add a oooq comptatibility layer, documentation and example.

Change-Id: I30fe6359f1c0098ff9bcdd5939724491d94ef199

Add support for applying w/a before and after upgrade

This change adds the ability to apply workarounds before and after
the overcloud upgrade process has finished. This allows the user
to workaround particular issues that show up after the upgrade
process has finished.

Change-Id: I21a7e885bcc466af6bf80410ba2cc8d03865cb33

Fix missing quotes

This changes adds missing quotes to the node_upgrade_script.yml
templates. Currently the task is failing because of the missing quotes.

Change-Id: Ied9217df374d09bf90e6878a7c79e22042f44d99

Add support for applying workarounds post undercloud upgrade

This change adds support for applying workarounds after the undercloud
upgrade process has finished as part of the undercloud upgrade.

Change-Id: If85900969c0d591cf6024408d958a82fa8c8a534

Adjust overcloud_converge_upgrade_script script

This change adjusts the overcloud_converge_upgrade_script to allow
running the upgrade converge stage. In addition it adjust the ssh
config file to skip host key check so the non-controller script does
not get stuck waiting for user input.

Change-Id: Ic38f325c61e90165a5322ef754f7e5514ed8e687

Append working_dir to logs generated

Change-Id: I6bc9f0c58ad8684ed03dee042e9cfb2bdc6835f6

Install ceph-ansible during undercloud upgrade

ceph-ansible is required to be installed manually for deployments
with ceph nodes. This change installs the ceph-ansible package
before the undercloud upgrade.

Change-Id: If8918a38250a10681d965d0715ebc17078166336

Use openstack overcloud container image prepare command

This change adds the use of openstack overcloud container image prepare
command for generating the environment file containing the container
image names and local registry address.

Change-Id: I174f7e3aae415d51224cf73da83a859e90eed095

Do not rely on ansible inventory for upgrading non controller nodes

Currently we are relying on the ansible inventory to provide groups
containing compute nodes and their facts when creating the upgrade
scripts. In order to remove this requirement and provide easier
integration this change discovers the compute and swift storage nodes
from the undercloud. In addition it adds a wait loop for instance live
migration to complete before and after upgrading compute nodes and adds
support for swift storage nodes upgrade.

Change-Id: Ia4b2e81845c3fec9036c4695f0dd1746d4c5c6b8

Add silent and nobuffer options to curl command

Change-Id: I0f4bd71b67d4827717d9f7d3fc075fc396eb6363

Adjust tht environment files and custom roles data during upgrade

This change switches the environment files used in the overcloud deploy
command to their variants used for deploying containers. In addition, if
using a custom roles data file for composable roles deployments, this
change adjusts the local roles data copy with the changes introduced in
Pike.

Change-Id: Icd3c3b67342c0bd10fc3d28ed89a94fc9f714db4

Fix overcloud_deploy_script var location in oooq_test playbook

Change-Id: I01f18a5413ff223cbca900521037e739ebc43d5d

Disconnect ssh session before uploading images to local registry

This change disconnects the running ssh session in order to allow
the stack user to connect to the docker daemon. This acts as a
workaround for https://bugs.launchpad.net/tripleo/+bug/1699477

Change-Id: Ia06fe8581ba17525cbcb2d955d7947b7c546811d

Add undercloud_reboot var and reboot only on ovs or kernel updates

This change adds the undercloud_reboot var to manipulate if undercloud
reboots should be made or not and does the reboot only when kernel or
ovs updates occur.

Change-Id: Ic89291c5791bbd098204b2c85793335e0faa8d94

Add InfraRed docs

This change adds the steps required to run the tripleo-upgrade role
as an InfraRed plugin.

Change-Id: I08fde6c8954ec4eeedc47971607bec592fc801a1

Add docs for running the role manually from the undercloud

Change-Id: I6d5bb6479b6e1fb45e34aa776d3c315cfae98ae4

Do not stop services before undercloud upgrade

According to the docs stopping services should be done by the undercloud
upgrade process so we do not need to manually do it.

Change-Id: I7f14257b4e90d3a0610c0f90e856096643325861

Specifically become_user: root when running ovs command

Currently become_user is set to the undercloud user, usually
stack, when invoking the undercloud_validate_upgrade.yaml playbook.
This change overrides become_user and sets it to root when running
ovs-vsctl command as it requires privilege escalation.

Change-Id: Ia93245f4a2d09d73849c03401d38ffb25e7be802

Use ip address instead of name when rebooting

OOOQ doesn't set a name for the undercloud in hosts so in case the
reboot is triggered the virthost cannot reach the undercloud. This
change switches the wait condition and calls the ip address instead
of the name.

Change-Id: I2121e2dabe9794d31fe70bbfa0ff8e53da6b3b1b

Add reload ssh tag to the Kill SSH task

Add a tag to the Kill SSH task so it can be avoided when
running the role manually from the undercloud.

Change-Id: Iec088cc174d7e8270fbea0698ee76227b45842f7

Add option to create script with --setup-heat-output option

According to BZ#1477962 in order to be able to run non controller
upgrade scripts after major upgrade composable steps we need to run
the deploy command with the --setup-heat-outputs option.

Change-Id: I8137ff18047a130c5ea8dca2ce11378eabc30329

Add deprecated params to custom roles data file

In Pike there were additional flags assigned to the default roles
data file. This change adds these flags during the upgrade process
to the roles defined in the custom roles data file.

Change-Id: I58c2f30ff74d2302027d7488dc03c5146c371649

Use a default value for the HOME env var

Change-Id: I40e2f46f1311dc4b797977c8136ec4037505ef05

Update python-openstackclient before undercloud upgrade

Updating python-openstackclient before undercloud upgrade is a
requirement to get undercloud upgrade passing.

Change-Id: I4ab67f9af68036a29e11bb2457d70535bd94b7b0

Create environment file for injecting undercloud certificate

When undercloud is SSL enabled the overcloud nodes need to be aware
of the undercloud CA cert. This change creates this file during upgrade
and adds it to the overcloud deploy commands when the overcloud nodes
are not able to reach the undercloud ssl enabled public endpoints.

Change-Id: I79a03299bc28d0ca2dbd83c28087a4c56f6b2271

Convert puppet ceph parameters to ceph ansible during upgrade

This is a workaround for BZ#1488855.

Change-Id: I58ac44b2166abddf56a327a4ee09457139c831da

Remove setup heat outputs workaround

https://review.openstack.org/#/c/502470/ allows us to obtain the
RoleConfig output so there's no need to run the setup heat outputs
step anylonger. This change remove this step.

Change-Id: I7235b8625eea4f77a055d0ab1862d0318f3a776f

Echo bug number in workarounds script

This change replaces the bug number comments with an echo statement.
This provides easier way to debug what workarounds failed to apply
when the workarounds script is run.

Change-Id: I02c5534427ee5cf7b7af618f36577c1008d50992

Fix uc_keystone_conn condition

This change adds an addition condition to inject the undercloud
certificate in the CAMap only when  undercloud ssl is enabled.

Change-Id: Iae023922bf1ed0bb22e86710b122c65a8f1568a3

Gather facts only from undercloud node

Facts are only required for the undercloud node. This change adjusts
the existing playbooks to gather facts only from undercloud node in
order to save some time and not rely on nodes which are not required
to be reachable.

Change-Id: Idceec3d10d84ef112da558109d9904a1d8c6ed93

Do not include docker.yaml and docker-ha.yaml environments

As described in BZ#1466744 the docker.yaml and docker-ha.yaml
environments are currently included by default so we do not need
to specifically include them.

Change-Id: I44a72ddd65cf816003ceca21ef33470a3ab125a7

Reboot compute nodes post upgrade

As a post upgrade requirement we need to reboot the nodes in case
of an OVS upgrade. This change runs a post upgrade check for non
controller nodes and reboots the nodes if an ovs upgrade has been
detected. It also adds additional validation for compute nodes to
make sure that the nova-compute service is enabled after reboot.

Change-Id: I583e589118aabae84f8e1dc9ec2c4b43ca17a250

Add L3 agent connectivity check during upgrade

This change adds a check which validates that ICMP
connectivity with a floating IP is not interrupted
during the major upgrade composable step.

Change-Id: Iee55af85b9a2c3ece86731e043130d191ff6a821

Run pre docker composable upgrade workarounds at correct position

This change moves the pre docker composable upgrade workarounds to
be run right before the docker composable upgrade step.

Change-Id: I604ea2eb6202d48b0f771ea80e5e731df687600e

Use bool with ansible booleans

use bool filter when using ansible booleans.

Change-Id: Ibeb59772e935cc28a661ccddcaa4773388ce296d

Add option for creating workloads before upgrade

This change adds the option to launch an instance before
starting upgrade. This operation is useful when doing
tests such as instance live migration during upgrade or
floating ip connectivity testing during upgrade. The
script requires a network defined in the external_network_name
var which provides external connectivity to exist beforehand.

Change-Id: Ib39e41b36fac7794ea515c8a9d56141866dcfeed

Fix pre compute upgrade check

This change adds the MIGRATING state to be checked before the
compute nodes upgrade.

Change-Id: I0073a7e69a71a044882d4760dbb49cd4f455dd89

Fix workload_launch position

Change-Id: I6193ac6a60165bb20ece6277067c05696ed6d3b1

Run non controller node pre upgrade script

This change runs the non controller node pre upgrade script.
In addition it exposes the option to run instances migration
between compute nodes during upgrade.

Change-Id: Ief55eecdc85bb620f637c4ed4d9b5bc3243b37d1

Update roles_data adjustments to latest changes

This change updates the roles_data file adjustments to the latest
changes.

Change-Id: Ic787b135cdf96b33829e05140e069b398df7196f

Use docker and docker-ha environment files for upstream deployments

Change-Id: I70bb9767e97f616729adff983fff065858a6dcdc

Convert services environments to services-docker only for upstream

Per BZ#116463 in downstream the environments used for extra service
enablement now point to docker resources.

Change-Id: I379622ec2749ac8b485aec79a7500308ef74214e

Echo debug message to differentiate live migration from block

Change-Id: I69e7f381543d84fcd308acbd3a90f5d0ac23ae1b

Accept <= 5% ICMP packet loss during upgrade connectivity check

Change-Id: I34d1de225c0e391035e22e18f63356e04afbbfd5

Reorganize playbooks to separate upgrade/update

This change adds separate directories for upgrade/update which
provides a better separation between updates and upgrades.

Change-Id: Icf1a09514fb0e6236535ae32265bbd3805918478

Run block migrate multiple times

Block migrate doesn't work seem to work if triggered once but it
does if the command is run for a second time. This change runs the
block migrate command multiple times to make sure that the instance
gets migrated.

Change-Id: I8b9a9ecae21f7ce49a03945afec66b9e671622b7

Ensure files/ are part of the setup.cfg files to copy

When installing tripleo-upgrade into a .quickstart
environment, the files/ folder wasn't getting copied, which is necessary
at least for "adjust ssh config to skip host key check" in
create-upgrade-scripts.yaml.

Change-Id: I7d862ec5c13ba719923c90cc40790b842b582999

Add tasks for undercloud minor update

Start adding the minor undercloud update tasks

Change-Id: I33705b270e2d5e6a28f1cad8179e1f4b3e4ea975

Remove timeouts from upgrade scripts

Depending on the number of deployed nodes upgrade could take longer
and we want to rely on the heat stack timeout. This change clears
any manual timeouts set in the upgrade scripts.

Change-Id: I5d141e2cc13621d3be5fb0c27b0ac3c3fc30d424

Minor updates of RHOS 12.

Manage minor update workflow from within tripleo-upgrade repo.

Change-Id: I8c6771af4825ce166e8470413ca4687be0a58cb9

Reboot controller nodes post upgrade

This change adds the option to reboot controller nodes post upgrade
and performs basic verifications that the clustered services are
reported as up.

Change-Id: I370d421e5968ae50bd1ff140cdfcf98a4db03a5f

Don't force ssh_config on everybody.

This add an option to be able to not overwrite the ssh_config file.
As a side note the ssh_config is missing from the repo, so by default
this task is broken.

Change-Id: Idfb78e2b7226a7e6295acd3f250bbfb48d0a103d

Fix filter used in the node upgrade scripts

This change is fixes the current filter used for node upgrade
scripts so that deployment with $domain.tld format are supported.

Change-Id: I18f43c440bb93e0fcefb664c7d716ff9368673a4

Run live migration multiple times

Change-Id: I7c028defd3cb9080efa7bdbe9daa6ed201df8640

Manually inject undercloud SSL cert to overcloud nodes

Per BZ#1501779 the compute nodes do not get their trusted store
updated when using a CAMap and upgrade fails. This change updates
the overcloud nodes trusted store manually so the overcloud nodes
are prepared for update. This should translate in a documentation
step that before upgrade starts the user needs to make  sure the
overcloud nodes are able to reach the undercloud SSL public
endpoint.

Change-Id: Ib95a29c608803504a866ae71cbc0082faf3c194f

Replace puppet external ceph environment with ceph-ansible one

This change replaces during the upgrade the external ceph puppet
environment file with its ceph-ansible equivalent.

Change-Id: I9020e8f7c43f91259b551caa2e20f03be1424106

Append deprecated params only to predefined roles

We should append the deprecated params only to predefined roles
in order to avoid failures such as reported in BZ#1501237.

Change-Id: I3a7c332b35da9639fb6f8e5b38234dc0c55d8499

Split the post controller scripts into per services scripts

This change splits the post upgrade controller scripts into
per service checks and adds them to a common directory so
they can be shared between update and upgrade.

Change-Id: I8f2fb6162a5acb8a92057400a7b04e6e2388abaa

Add the ability to specify a remote docker registry

This change adds the ability to specify a remote docker registry
to be used for downloading the Docker images on the undercloud or
be used directly by the overcloud nodes during upgrade.

Change-Id: I132a8b94f9a101d1c9c624d202bb01527dc2b844

Fix BZ#1499677 workaround condition

In addition to empty gvwstate.dat file there might be situations
where the gvwstate.dat file is missing after reboot. This patch
addresses this condition for BZ#1499677 workaround.

Change-Id: I295b133248f48ab41b1748225cbe9359662b280d

Cleanup galera resource instead of rebooting node for BZ#1499677

Instead of rebooting the node while implementing the workaround for
BZ#1499677 we should simply cleanup the Galera resource. This way
we can save some time and potential issues caused by an additional
reboot.

Change-Id: I391daeae41321baec1cbd8c458132a3161cd96d5

[UPDATES] Introduce option for minor updates workarounds.

To speed up testing of minor updates it might be required
 to apply some patches before they are landed.
Hence we need a flag to differentiate if workarounds are required
 or not

Change-Id: I642e4ade204f5fd30ec9433f1d90a2d539287c5e

Do not pipe curl output in container images environment script

Curl can sometimes exit with exit code 23 when its output is piped
into another command. To avoid this errors we save the curl output
to a file.

Change-Id: I4123b6c66ae2873c11631f229cb8e3eec5a5a66b

Use service environment files when generating the images environment

In the last build openstack overcloud container image prepare only
generates the parameters for the services included by default. In
order to make it work when extra non-default services are enabled
we need to pass the environment files to the prepare command. This
change addresses this issue.

Change-Id: I86ab6faaffcd4c7cc1a07e9a6ed1e890cb5cf980

Place the oooq deploy command into overcloud_deploy_script var

This change places the openstack overcloud deploy command with its
arguments in the location defined by the overcloud_deploy_script
var. This way we don't require oooq users to specifically set the
overcloud_deploy_script to a hardcoded location and make it less
confusing.

Change-Id: Id2b14fcffbd169c342df4b5b9105dff81e18e3a0

Replace ceph radosgw environment during upgrade

Change-Id: I489211f39941bba5b1ca2ddf1b635c3bdb0151fe

Avoid losing undercloud connection in TripleO CI.

When running the role in the TripleO upstream CI
the connectivity to the undercloud gets lost when
rebooting the undercloud after upgrade or killing
the ssh service, this makes the playbook fail.

As a solution, a flag tripleo_ci has been added.
This flag will default to false, and when set to
true no undercloud reboot, nor ssh killing will
be executed.

Change-Id: If4a303fff49bbe55cdfb7142d8dd69264ab47ab4

Align deployment-files option with IR.

deployment-files option is not a list of choices in IR,
adjust it accordingly.

Change-Id: I6889e9b75f842cc466278fed5dbf85a80cb58ee0

Append docker-ha only when needed.

Before appending the docker-ha.yaml env
file, we need to check if the overcloud
was deployed with pacemaker. If so, then
we'll add the env file to the upgrade
script.

Change-Id: I9867d86b6d23385c576d2f8c5a25ab3333f7113d

Specify tht directory used in upgrade script.

When deploying with oooq, the generated script
overcloud-deploy.sh is reused in order to append
the cooresponding env files for the upgrade.

However, if the location of the tripleo-heat-templates
directory is different from the used when deploying
then the upgrade does not succeed.

This change modifies the tripleo-heat-templates
location used to upgrade when the directory found
in overcloud-deploy.sh is different. If it is the
same no change is done.

Change-Id: I55ada3e75b7463b1c14c8734410d2591cf162e67

Don't append DockerInsecureRegistryAddress

This is no longer required as the prepare command detects whether the
registry is secure and DockerInsecureRegistryAddress as necessary.

Depends on upstream https://review.openstack.org/#/c/514473/
Related-Bug: #1722632

Change-Id: Ia9d91f6280600c59d0079c5d1f26a00f04040426

Fix the controller regex during roles_data conversion

The Ocata roles_data controller role might include a comment for
the controller role name. This change adjusts the regexp used during
the roles data conversion to take into consideration that comment.

Change-Id: I7b43b1fcb9e477de8e1265ef6aa6ba5149e82d47

Use prepare --set for ceph image parameter

This is more maintainable, and consistent with other uses of prepare.

Change-Id: Ieec88e271973a248192c2b247cd2c5e0cccbfb7c

Add storage environment files to be used for containers prepare

This change adds the storage environments files to be matched when
creating the environment file containing the Docker images names
which gets created via container image prepare.

Change-Id: I34c3cdaab1b63ce3f43d748372d35143bc12b8b4

Add environment file containing required DPDK changes during upgrade

Change-Id: Iabaf16e18ee7546bd6275f8f84226892423a6c95

Create failed_upgrade log files.

Most of the stack failures in the TripleO
CI are registered inside two log files
failed_upgrade_list.log and failed_upgrade.log.

This patch adds the option of inject the
stack failures list command into these two
log files, as well as priting out the detailed
stack failure list (--long).

Change-Id: I4fad989818f67ad0a73e45b47f835750f18c3bb6

Replace storage-environemnt.yaml for upstream only

Per BZ#1502862 the Ceph environment files switch during upgrade is
only needed for upstream deployments. This change does the changes
to accommodate this.

Change-Id: Ia3adb120c9b524c66d069593b0779b3399295fd4

Do not update python-openstackclient before upgrade

BZ#1488471 was fixed so there's no need to update the python-
openstackclient package before upgrade.

Change-Id: I2aba7515ec43926ec4f8de5c701467dea31dba1b

Be more aggressive on accepted packet loss during upgrade

Tests have shown that the packet loss shouldn't eceed 1% during
the upgrade steps. This change adjusts the accepted level to this.

Change-Id: I9e47ea56a78e4e9eab40fca609cbedaecfcf1e14

Add more tags for the upgrade process.

This enable one to either do only a small part of the whole process.
This can be useful for debugging or development.

Change-Id: Ic6cc9a1e6aa2793fde65636d2ad92bc174173252

Use new method of discovering tag and adjust local registry upload

This change uses the new mechanism of retrieving the tag from the
latest image provided in the registry.

Change-Id: I7e063f13c7d4812e9986452774881235b620bd0e

Swap baremetal environment file for containerized.

In oooq, when upgrading from baremetal
to containerized overcloud two different
environment files are used. These env
files are located in [0].

When upgrading using tripleo-upgrade, we
need to convert that environment file name
to its corresponding containers version.

[0] https://github.com/openstack/tripleo-heat-templates/tree/master/ci/environments

Change-Id: I6c9fad2f402a162cf663c5089e79c2e10f3d0928

Add condition to create local docker registry.

The only way to not execute the docker
registry environment file creation task
is via tags, which is not easy to handle
in TripleO CI.

As tripleo-quickstart already prepares
the local docker registry file, there
is no need to execute it. So a new
parameter 'create_docker_registry' is
been added.

Also, the 'force' option is being added
to avoid overwritting a provided script
with the role template.

Change-Id: I800d6696b8dbb83f05f3d9381c6e5689558f4b77

Prepare workloads before update/upgrade.

Prepare scripts for managing workload on oc before running
 update/upgrade operation.

Allow to run ping test during minor update.

Change-Id: I1d5754f36f53588c97c646aa4e1380e9ca5938bc

Remove tag parsing from the image prepare command

Parsing the tag is not needed anymore and the one returned by the
container image prepare command can be used.

Change-Id: If782fc655da7b22e4d4a803509e9cc8c49774368

[UPDATES] Run minor update per role.

With recent changes it's advised to perform minor update in batches:
role-by-role.
This change limits the scope of update with '--nodes <Role>' option.

Change-Id: I0bc03873b749dc9c15b13cacbfff78cead4360d8
This commit is contained in:
Sofer Athlan-Guyot 2017-08-01 15:30:22 +02:00 committed by Emilien Macchi
parent c321b1b112
commit 4572055ffb
64 changed files with 2097 additions and 229 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

170
README.md
View File

@ -35,6 +35,30 @@ Role Variables
Available variables are listed below:
upgrade_noop: false
Only create upgrade scripts without running them
update_noop: false
Only create update scripts without running them
undercloud_upgrade: false
Run undercloud upgrade
overcloud_upgrade: false
Run overcloud upgrade
undercloud_update: false
Run undercloud update
overcloud_update: false
Run overcloud update
overcloud_deploy_script: "~/overcloud_deploy.sh"
Location of the initial overcloud deploy script.
@ -63,6 +87,30 @@ Location of the overcloud credentials file.
Allows the user to apply known issues workarounds during the upgrade process. The list of patches/commands used for workarounds should be passed via --extra-vars and it should include dictionaries for undercloud/overcloud workarounds.
use_oooq: false
Set to true when the deployment has been done by tripleo quickstart.
workload_launch: false
Set to true to launch an instance before starting upgrade. This can be useful for running tests during upgrade such as live migration or floating IP connectivity checks.
external_network_name: "public"
Name of the external network providing floating IPs for instance connectivity. This provides external connectivity and needs to exist beforehand, created by the user.
workload_image_url: "http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img"
URL of the image used for the workload instance.
workload_memory: "512"
Amount of memory assigned for the workload instance.
tripleo_ci: false
Set to true when running the role in the TripleO CI jobs. It avoids losing connectivity to the undercloud by skipping reboot and ssh kill tasks.
Dependencies
------------
@ -73,17 +121,131 @@ Example Playbook
An example playbook is provided in tests/test.yml:
- hosts: all
gather_facts: true
- hosts: undercloud
gather_facts: false
gather_facts: true
become: yes
become_method: sudo
become_user: stack
roles:
- tripleo-upgrade
Usage with tripleo Quickstart
-----------------------------
After a successful deployment with OOOQ, you can create the necessary
scripts using this example playbook (duplicate from
./tests/oooq-test.yaml):
---
- hosts: undercloud
gather_facts: true
become: yes
become_method: sudo
become_user: stack
roles:
- { role: tripleo-upgrade, use_oooq: 'true'}
And then you run it like this (adjust the paths to your oooq specific
one)
ANSIBLE_SSH_ARGS="-F $(pwd)/ssh.config.ansible" \
ANSIBLE_CONFIG=$PWD/ansible.cfg \
ansible-playbook -i hosts -vvv tripleo-upgrade/tests/oooq-test.yaml
This will only create the file (without running the actual upgrade):
- undercloud_upgrade.sh
- container_images_download.sh
- local_docker_registry_env.sh
- composable_docker_upgrade.sh
- overcloud-compute-*_upgrade_pre.sh
- overcloud-compute-*_upgrade.sh
- overcloud-compute-*_upgrade_post.sh
- converge_docker_upgrade.sh
with the correct parameters.
Usage with InfraRed
-----------------------------
tripleo-upgrade comes preinstalled as an InfraRed plugin. After a successful InfraRed
overcloud deployment you need to run the following steps to upgrade the deployment:
Symlink roles path:
ln -s $(pwd)/plugins $(pwd)/plugins/tripleo-upgrade/infrared_plugin/roles
Set up undercloud upgrade repositories:
infrared tripleo-undercloud \
--upgrade yes \
--mirror ${mirror_location} \
--ansible-args="tags=upgrade_repos"
Set up undercloud update repositories:
infrared tripleo-undercloud \
--update-undercloud yes \
--mirror ${mirror_location} \
--build latest \
--version 12 \
--ansible-args="tags=upgrade_repos"
Upgrade undercloud:
infrared tripleo-upgrade \
--undercloud-upgrade yes
Update undercloud:
infrared tripleo-upgrade \
--undercloud-update yes
Set up overcloud upgrade repositories:
infrared tripleo-overcloud \
--deployment-files virt \
--upgrade yes \
--mirror ${mirror_location} \
--ansible-args="tags=upgrade_collect_info,upgrade_repos"
Set up overcloud update repositories/containers:
infrared tripleo-overcloud \
--deployment-files virt \
--ocupdate True \
--build latest \
--ansible-args="tags=update_collect_info,update_undercloud_validation,update_repos,update_prepare_containers"
Upgrade overcloud:
infrared tripleo-upgrade \
--overcloud-upgrade yes
Update overcloud:
infrared tripleo-upgrade \
--overcloud-update yes
Running the role manually from the undercloud
---------------------------------------------
This role can be run manually from the undercloud by doing the following steps:
Note: before starting the upgrade process make sure that both the undercloud
and overcloud nodes have the repositories with upgraded packages set up
Clone this repository
git clone https://github.com/redhat-openstack/tripleo-upgrade.git
Set ansible roles path
ANSIBLE_ROLES_PATH=$(pwd)
Create inventory file
printf "[undercloud]\nlocalhost ansible_connection=local" > hosts
Run the playbook including this role
ansible-playbook -i hosts tripleo-upgrade/tests/test.yml
License
-------

74
defaults/main.yml
View File

@ -1,9 +1,14 @@
---
# defaults file for tripleo-upgrade
# main vars:
working_dir: "{{ ansible_env.HOME }}"
working_dir: "{{ (ansible_env|default({})).HOME|default('/home/stack') }}"
# TODO: those variable can be changed for the explicit keyword in tag
# when https://github.com/ansible/ansible/issues/11045 is merged.
# enable update/upgrade
upgrade_noop: false
update_noop: false
undercloud_upgrade: false
overcloud_upgrade: false
undercloud_update: false
@ -12,26 +17,69 @@ overcloud_update: false
# enable upgrade workarounds
upgrade_workarounds: false
# enable update workarounds
updates_workarounds: false
# use oooq
use_oooq: false
# Running in tripleo ci
tripleo_ci: false
#rc files:
undercloud_rc: "{{ working_dir }}/stackrc"
overcloud_rc: "{{ working_dir }}/overcloudrc"
# launch workload before update/upgrade
workload_launch: false
external_network_name: "public"
workload_image_url: "http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img"
workload_memory: "512"
# upgrade jinja template name:
undercloud_upgrade_template: undercloud_upgrade.sh.j2
undercloud_update_template: undercloud_update.sh.j2
container_images_download_template: download_images.sh.j2
local_docker_registry_env_template: create_registry_env.sh.j2
workload_launch_template: workload_launch.sh.j2
# upgrade scripts name:
undercloud_upgrade_script: "{{ working_dir }}/undercloud_upgrade.sh"
undercloud_update_script: "{{ working_dir }}/undercloud_update.sh"
overcloud_deploy_script: "{{ working_dir }}/overcloud_deploy.sh"
overcloud_composable_upgrade_script: "{{ working_dir }}/composable_docker_upgrade.sh"
overcloud_converge_upgrade_script: "{{ working_dir }}/converge_docker_upgrade.sh"
container_images_download_script: "{{ working_dir }}/container_images_download.sh"
local_docker_registry_env_script: "{{ working_dir }}/local_docker_registry_env.sh"
workload_launch_script: "{{ working_dir }}/workload_launch.sh"
# overcloud jinja template name
overcloud_pre_update_workarounds_template: overcloud_pre_update_workarounds.sh.j2
update_workarounds_template: update_workarounds.sh.j2
overcloud_update_setup_template: overcloud_update_setup.sh.j2
overcloud_update_template: overcloud_update.sh.j2
# overcloud update scripts
pre_undercloud_update_workarounds_script: "{{ working_dir }}/pre_undercloud_update_workarounds.sh"
post_undercloud_update_workarounds_script: "{{ working_dir }}/post_undercloud_update_workarounds.sh"
pre_overcloud_update_workarounds_script: "{{ working_dir }}/pre_overcloud_update_workarounds.sh"
post_overcloud_update_workarounds_script: "{{ working_dir }}/post_overcloud_update_workarounds.sh"
overcloud_update_setup_script: "{{ working_dir }}/overcloud_update_setup.sh"
overcloud_update_script_base: "{{ working_dir }}/overcloud_update"
# container registry file
container_registry_file: "docker-images.yaml"
# enable local docker registry
use_local_docker_registry: true
# create local docker registry env file
create_docker_registry: true
# url of the remote docker registry to be used
docker_registry_url: 'registry.example.local'
# use upstream or downstream container images
upstream_container_images: true
@ -43,3 +91,27 @@ containers_default_parameters: "{{ working_dir }}/docker-osp12.yaml"
# container_images.yaml file location
container_images_location: "{{ working_dir }}/container_images.yaml"
# reboot nodes post upgrade
undercloud_reboot: false
controller_reboot: false
force_reboot: false
# time to wait for nodes to reboot in seconds
node_reboot_timeout: 300
# enable l3 agent connectivity check during upgrade
l3_agent_connectivity_check: false
l3_agent_connectivity_check_start_script: "{{ working_dir }}/l3_agent_start_ping.sh"
l3_agent_connectivity_check_stop_script: "{{ working_dir }}/l3_agent_stop_ping.sh"
l3_agent_connectivity_check_start_template: l3_agent_start_ping.sh.j2
l3_agent_connectivity_check_stop_template: l3_agent_stop_ping.sh.j2
# migrate instances between compute nodes during upgrade
compute_evacuate: false
# enable post upgrade checks
controller_upgrade_post: false
# Provide a custom ssh-config file
need_ssh_config: true

69
infrared_plugin/main.yml
View File

@ -1,22 +1,39 @@
---
# This is file and plugin.spec are required by Infrared project
- hosts: all
gather_facts: true
- hosts: undercloud
gather_facts: false
gather_facts: true
become: yes
become_method: sudo
become_user: stack
pre_tasks:
- name: Set upgrade workload launch
set_fact:
workload_launch: true
when: install.upgrade.workload
- name: Set upgrade workload image
set_fact:
workload_image_url: "{{ install.upgrade.workloadimage }}"
when: install.upgrade.workload
- name: Set upgrade workload memory
set_fact:
workload_memory: "{{ install.upgrade.workloadmemory }}"
when: install.upgrade.workload
- name: Set undercloud upgrade
set_fact:
undercloud_upgrade: true
when: install.undercloud.upgrade
- name: Set undercloud reboot
set_fact:
undercloud_reboot: true
when: install.undercloud.reboot
- name: Set overcloud upgrade
set_fact:
overcloud_upgrade: true
@ -32,6 +49,15 @@
upstream_container_images: false
when: not install.upstream.container.images
- name: Set use docker local registry
set_fact:
use_local_docker_registry: false
when: not install.upgrade.docker.local.registry
- name: Set docker registry url
set_fact:
docker_registry_url: "{{ install.upgrade.docker.registry.url }}"
- name: Set undercloud update
set_fact:
undercloud_update: true
@ -42,5 +68,40 @@
overcloud_update: true
when: install.overcloud.get('update', {})
- name: Set updates workarounds
set_fact:
updates_workarounds: true
when: install.updates.workarounds
- name: Set upgrade floating ip check
set_fact:
l3_agent_connectivity_check: true
when: install.upgrade.floatingip.check
- name: Set upgrade compute host evacuate
set_fact:
compute_evacuate: true
when: install.upgrade.compute.evacuate
- name: Set deployment-files base
set_fact:
container_registry_file: "{{ install.deployment.files }}/docker-images.yaml"
when: install.deployment.files
- name: Set upgrade controller reboot
set_fact:
controller_reboot: true
when: install.upgrade.controller.reboot
- name: Set upgrade controller post
set_fact:
controller_upgrade_post: true
when: install.upgrade.controller.post
- name: Set upgrade force reboot
set_fact:
force_reboot: true
when: install.upgrade.reboot.force
roles:
- tripleo-upgrade

64
infrared_plugin/plugin.spec
View File

@ -29,6 +29,60 @@ subparsers:
help: |
Use upstream or downstream container images during upgrade
default: false
undercloud-reboot:
type: Bool
help: |
Reboot undercloud post upgrade when ovs or kernel get upgraded
default: false
upgrade-floatingip-check:
type: Bool
help: |
Check floating ip connectivity during upgrade.
Note: This requires a running instance with attached floating ip and allowed icmp traffic.
default: false
upgrade-workload:
type: Bool
help: |
Launch workload before starting upgrade
default: false
upgrade-workloadimage:
type: Value
help: |
Image URL to be used for spawning instance before upgrade.
default: http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img
upgrade-workloadmemory:
type: Value
help: |
Memory assigned to the instance spawned before upgrade
default: 512
upgrade-compute-evacuate:
type: Bool
help: |
Migrate instances between compute nodes during upgrade.
default: true
upgrade-controller-reboot:
type: Bool
help: |
Reboot controller nodes post upgrade
default: true
upgrade-controller-post:
type: Bool
help: |
Run controller post upgrade checks
default: true
upgrade-reboot-force:
type: Bool
help: |
Hard reboot nodes during upgrade
default: false
upgrade-docker-local-registry:
type: Bool
help: Use local docker registry on the undercloud
default: false
upgrade-docker-registry-url:
type: Value
help: The alternative docker registry to use for deployment.
default: 'registry.example.local'
- title: TripleO Update
options:
overcloud-update:
@ -41,3 +95,13 @@ subparsers:
help: |
Update Undercloud
default: false
updates-workarounds:
type: Bool
help: |
Apply updates workarounds
default: false
deployment-files:
type: Value
help: |
Directory containing the templates of the overcloud deployment.
default: virt

1
setup.cfg
View File

@ -27,6 +27,7 @@ data_files =
usr/local/share/ansible/roles/tripleo-upgrade/templates = templates/*
usr/local/share/ansible/roles/tripleo-upgrade/tests = tests/*
usr/local/share/ansible/roles/tripleo-upgrade/vars = vars/*
usr/local/share/ansible/roles/tripleo-upgrade/files = files/*
playbooks = playbooks/*
[wheel]

18
tasks/common/controller_post_script.yml Normal file
View File

@ -0,0 +1,18 @@
---
- name: create a directory to store post scripts for controller nodes
file:
path: "{{working_dir}}/{{ node_name | splitext | first }}_post"
state: directory
- name: create post scripts for {{ node_name }}
template:
src: "check_service_{{ item }}.sh.j2"
dest: "{{working_dir}}/{{ node_name | splitext | first }}_post/{{ item }}.sh"
mode: 0775
with_items:
- 'reboot'
- 'haproxy'
- 'rabbitmq'
- 'galera'
- 'redis'
- 'haproxy_backend'

10
tasks/common/controller_post_scripts.yml Normal file
View File

@ -0,0 +1,10 @@
- name: register controller nodes
shell: |
source {{ undercloud_rc }}
openstack server list -f json | jq -r -c '.[] | select(.Name | contains("controller")) | .Name'
register: controllers
- include: controller_post_script.yml
loop_control:
loop_var: node_name
with_items: "{{ controllers.stdout_lines }}"

19
tasks/common/create_workload.yml Normal file
View File

@ -0,0 +1,19 @@
---
- name: create workload launch script
template:
src: "{{ workload_launch_template }}"
dest: "{{ workload_launch_script }}"
mode: 0775
- name: create start l3 agent connectivity check scripts
template:
src: "{{ l3_agent_connectivity_check_start_template }}"
dest: "{{ l3_agent_connectivity_check_start_script }}"
mode: 0775
- name: create stop l3 agent connectivity check scripts
template:
src: "{{ l3_agent_connectivity_check_stop_template }}"
dest: "{{ l3_agent_connectivity_check_stop_script }}"
mode: 0775

3
tasks/container_images.yaml
View File

@ -1,3 +0,0 @@
---
- name: download container images
shell: "bash {{ container_images_download_script }}"

51
tasks/create-scripts.yaml
View File

@ -1,51 +0,0 @@
---
- name: create undercloud upgrade script
template:
src: "{{ undercloud_upgrade_template }}"
dest: "{{ undercloud_upgrade_script }}"
- block:
- name: create undercloud upgrade workarounds script
template:
src: upgrade_undercloud_workarounds.sh.j2
dest: ~/upgrade_undercloud_workarounds.sh
- name: create overcloud upgrade workarounds script
template:
src: upgrade_overcloud_workarounds.sh.j2
dest: ~/upgrade_overcloud_workarounds.sh
when: upgrade_workarounds
- name: create container images download script
template:
src: "{{ container_images_download_template }}"
dest: "{{ container_images_download_script }}"
- name: create registry environment file script
template:
src: "{{ local_docker_registry_env_template }}"
dest: "{{ local_docker_registry_env_script }}"
- name: create composable upgrade scripts
include: step_upgrade.yml
loop_control:
loop_var: ugstage
with_items:
- step: "Docker containers composable upgrade"
script: "{{ overcloud_composable_upgrade_script }}"
environment_file:
- "{{ tht_directory }}/environments/docker.yaml"
- "{{ tht_directory }}/environments/docker-ha.yaml"
- "{{ tht_directory }}/environments/major-upgrade-composable-steps-docker.yaml"
- "{% if not upstream_container_images or (upstream_container_images and use_local_docker_registry) %}{{ containers_default_parameters }}{% endif %}"
- step: "Docker containers converge upgrade"
script: "{{ overcloud_converge_upgrade_script }}"
environment_file:
- "{{ tht_directory }}/environments/docker.yaml"
- "{{ tht_directory }}/major-upgrade-converge-docker.yaml"
- "{% if not upstream_container_images or (upstream_container_images and use_local_docker_registry) %}{{ containers_default_parameters }}{% endif %}"
- name: Nova compute nodes upgrade scripts
include: node_upgrade_script.yml
with_items: "{{ groups.compute|default([]) }}"
loop_control:
loop_var: node_name

16
tasks/docker_composable_upgrade.yml
View File

@ -1,16 +0,0 @@
---
- name: run docker upgrade composable step
shell: |
source {{ undercloud_rc }}
bash {{ overcloud_composable_upgrade_script }} &> overcloud_composable_upgrade.log
register: overcloud_composable_upgrade
ignore_errors: yes
- name: print stack failures
shell: |
source {{ undercloud_rc }}
openstack stack failures list overcloud
when: overcloud_composable_upgrade.rc != 0
- fail: msg="Overcloud upgrade composable step failed... :("
when: overcloud_composable_upgrade.rc != 0

49
tasks/main.yml
View File

@ -1,48 +1,17 @@
---
# tasks file for tripleo-upgrade
- name: create upgrade scripts
include: create-scripts.yaml
tags: create_upgrade_scripts
- block:
- name: upgrade undercloud
shell: "bash {{ undercloud_upgrade_script }} &> undercloud_upgrade.log"
tags: undercloud_upgrade
- name: prepare workload scripts
include: common/create_workload.yml
- name: validate undercloud upgrade
include: undercloud_validate_upgrade.yaml
tags: undercloud_upgrade_validate
- name: launch workload
command: "{{ workload_launch_script }}"
- name: apply undercloud upgrade workarounds
shell: "bash ~/upgrade_undercloud_workarounds.sh"
when: upgrade_workarounds
when: undercloud_upgrade or undercloud_update
when: workload_launch
- block:
- name: apply upgrade overcloud upgrade workarounds
shell: "bash ~/upgrade_overcloud_workarounds.sh"
when: upgrade_workarounds
- include: upgrade/main.yml
when: upgrade_noop|bool or undercloud_upgrade|bool or overcloud_upgrade|bool
# TODO: move this out of upgrade
- name: download container images
include: container_images.yaml
tags: container_images
- name: create local docker registry environment file
include: local_docker_registry_env.yaml
tags: local_docker_registry
- include: docker_composable_upgrade.yml
tags: docker_composable_upgrade
- name: upgrade nova compute nodes
include: node_upgrade.yml
with_items: "{{ groups.compute|default([]) }}"
loop_control:
loop_var: node_name
tags: nova_compute_upgrade
- name: run docker upgrade converge step
shell: "bash {{ overcloud_converge_upgrade_script }}"
tags: docker_converge_upgrade
when: overcloud_upgrade
- include: update/main.yml
when: update_noop|bool or undercloud_update|bool or overcloud_update|bool

3
tasks/node_upgrade.yml
View File

@ -1,3 +0,0 @@
---
- name: Upgrade {{ node_name }}
shell: "bash ~/{{ node_name }}_upgrade.sh"

21
tasks/node_upgrade_script.yml
View File

@ -1,21 +0,0 @@
---
- name: register instances running on {{ node_name }}
shell: |
source {{ overcloud_rc }}
openstack server list --host {{ hostvars[node_name].ansible_fqdn }} -f json | jq -r -c '.[] | select(.Status | contains("ACTIVE") or contains("PAUSED")) | .Name'
register: node_instances
- name: create pre upgrade script for {{ node_name }}
template:
src: node_upgrade_pre.sh.j2
dest: ~/{{ node_name }}_upgrade_pre.sh
- name: create script for upgrading {{ node_name }}
template:
src: node_upgrade.sh.j2
dest: ~/{{ node_name }}_upgrade.sh
- name: create post upgrade script for {{ node_name }}
template:
src: node_upgrade_post.sh.j2
dest: ~/{{ node_name }}_upgrade_post.sh

41
tasks/undercloud_validate_upgrade.yaml
View File

@ -1,41 +0,0 @@
---
- name: reboot the undercloud
shell: "sleep 2 && shutdown -r now"
async: 1
poll: 0
ignore_errors: true
become: true
become_user: root
tags: undercloud_reboot
- block:
- name: waiting for the undercloud to be available
become: no
wait_for:
port: 22
host: "{{ ansible_ssh_host }}"
search_regex: OpenSSH
delay: 10
delegate_to: localhost
when: "'hypervisor' not in groups and 'virthost' not in groups"
- name: waiting for the undercloud to be available
become: no
wait_for:
port: 22
host: "{{ ansible_ssh_host }}"
search_regex: OpenSSH
delay: 10
delegate_to: hypervisor
when: "'hypervisor' in groups"
- name: waiting for the undercloud to be available
become: no
wait_for:
port: 22
host: "{{ ansible_ssh_host }}"
search_regex: OpenSSH
delay: 10
delegate_to: virthost
when: "'virthost' in groups"
tags: undercloud_reboot

43
tasks/update/create-update-scripts.yaml Normal file
View File

@ -0,0 +1,43 @@
---
- name: create undercloud update script
template:
src: "{{ undercloud_update_template }}"
dest: "{{ undercloud_update_script }}"
mode: 0775
- name: create scripts with workarounds
template:
src: "{{ update_workarounds_template }}"
dest: "{{ working_dir }}/{{ item.script }}"
mode: 0755
with_items:
- '{{ pre_undercloud_update_workarounds|default([]) }}'
- '{{ post_undercloud_update_workarounds|default([]) }}'
- '{{ pre_overcloud_update_workarounds|default([]) }}'
- '{{ post_overcloud_update_workarounds|default([]) }}'
when: updates_workarounds|bool
- name: generate inventory file
shell: |
source {{ undercloud_rc }} ;
tripleo-ansible-inventory --static-inventory /tmp/fake-oc-hosts ;
grep role_name /tmp/fake-oc-hosts | awk -F '=' '{ print $2 }' 2>/dev/null | sed -e 's/^[ \t]*//' -e 's/[ \t]*$//'
register: oc_roles
- name: store roles
set_fact:
oc_roles: "{{ oc_roles }}"
- name: create overcloud update setup script
template:
src: "{{ overcloud_update_setup_template }}"
dest: "{{ overcloud_update_setup_script }}"
mode: 0775
- name: create overcloud update script
template:
src: "{{ overcloud_update_template }}"
dest: "{{ overcloud_update_script_base }}-{{ item }}.sh"
mode: 0775
with_items:
- "{{ oc_roles.stdout_lines|default('all') }}"

74
tasks/update/main.yml Normal file
View File

@ -0,0 +1,74 @@
---
- name: create update scripts
include: create-update-scripts.yaml
tags: create_update_scripts
when: update_noop|bool or undercloud_update|bool or overcloud_update|bool
- block:
- name: undercloud pre-update workarounds
shell: |
source {{ undercloud_rc }}
bash {{ pre_undercloud_update_workarounds_script }} &> {{ working_dir }}/uc_pre_update_workarounds.log
when: updates_workarounds|bool
- name: update undercloud
shell: "bash {{ undercloud_update_script }} &> {{ working_dir }}/undercloud_update.log"
tags: undercloud_update
- name: undercloud post-update workarounds
shell: |
source {{ undercloud_rc }}
bash {{ post_undercloud_update_workarounds_script }} &> {{ working_dir }}/uc_pre_update_workarounds.log
when: updates_workarounds|bool
- name: validate undercloud update
include: ../upgrade/undercloud_validate_upgrade.yaml
tags: undercloud_update_validate
when: undercloud_update|bool
- block:
- name: start l3 agent connectivity check
shell: |
source {{ overcloud_rc }}
{{ l3_agent_connectivity_check_start_script }}
when: l3_agent_connectivity_check
async: 21660
poll: 0
- name: overcloud pre-update workarounds
shell: |
source {{ undercloud_rc }}
bash {{ pre_overcloud_update_workarounds_script }} &> {{ working_dir }}/pre_overcloud_update_workarounds.log
when: updates_workarounds|bool
- name: setup HEAT outputs
shell: |
source {{ undercloud_rc }}
bash {{ overcloud_update_setup_script }} &> {{ working_dir }}/overcloud_update_setup.log
tags:
- overcloud_update
- overcloud_update_setup
- name: update overcloud
shell: |
source {{ undercloud_rc }}
bash {{ overcloud_update_script_base }}-{{ item }}.sh &> {{ working_dir}}/oc-update-{{ item }}.log
with_items:
- "{{ oc_roles.stdout_lines|default('all') }}"
tags:
- overcloud_update
- name: overcloud post-update workarounds
shell: |
source {{ undercloud_rc }}
bash {{ post_overcloud_update_workarounds_script }} &> {{ working_dir }}/post_overcloud_update_workarounds.log
when: updates_workarounds|bool
- name: stop l3 agent connectivity check
shell: |
source {{ overcloud_rc }}
{{ l3_agent_connectivity_check_stop_script }}
when: l3_agent_connectivity_check
when: overcloud_update|bool

10
tasks/upgrade/compute_upgrade.yml Normal file
View File

@ -0,0 +1,10 @@
- name: list compute nodes
shell: |
source {{ overcloud_rc }}
openstack hypervisor list -f json | jq -r -c '.[] | .["Hypervisor Hostname"]'
register: hypervisors
- include: node_upgrade.yml
with_items: "{{ hypervisors.stdout_lines }}"
loop_control:
loop_var: node_name

10
tasks/upgrade/container_images.yaml Normal file
View File

@ -0,0 +1,10 @@
---
- name: Kill SSH
shell: sleep 1; pkill -u {{ ansible_ssh_user }} sshd
async: 3
poll: 2
tags: reload_ssh
when: not tripleo_ci
- name: download container images
shell: "bash {{ container_images_download_script }}"

10
tasks/upgrade/controller_node_upgrade.yml Normal file
View File

@ -0,0 +1,10 @@
---
- name: Running post upgrade scripts for {{ node_name | splitext | first }}
shell: "{{working_dir}}/{{ node_name | splitext | first }}_post/{{ item }}.sh"
with_items:
- 'reboot'
- 'haproxy'
- 'rabbitmq'
- 'galera'
- 'redis'
- 'haproxy_backend'

10
tasks/upgrade/controller_post_upgrade.yml Normal file
View File

@ -0,0 +1,10 @@
- name: register controller nodes
shell: |
source {{ undercloud_rc }}
openstack server list -f json | jq -r -c '.[] | select(.Name | contains("controller")) | .Name'
register: controllers
- include: controller_node_upgrade.yml
with_items: "{{ controllers.stdout_lines }}"
loop_control:
loop_var: node_name

41
tasks/upgrade/convert_ceph_params.yaml Normal file
View File

@ -0,0 +1,41 @@
---
- name: check if ceph ansible is enabled
command: "grep -Eq 'ceph-ansible.yaml|storage-environment.yaml' {{ overcloud_deploy_script }}"
register: ceph_ansible
ignore_errors: true
- block:
- name: register environment files
shell: |
grep '\-e\ \|\-\-environment-file' {{ overcloud_deploy_script }} | awk {'print $2'} | grep -v '\$'
register: envs
- name: look for ceph osd extra config
shell: |
grep 'ceph::profile::params::osds' {{ item }}
ignore_errors: true
register: osd_env
with_items:
- "{{ envs.stdout_lines }}"
- set_fact:
ceph_env: "{{ item.item }}"
when: "{{ item.stdout|length > 0 }}"
with_items:
- "{{ osd_env.results }}"
- block:
- name: register environment file
command: "cat {{ ceph_env }}"
register: ceph_puppet
- set_fact:
ceph_osds: "{{ (ceph_puppet.stdout | from_yaml).parameter_defaults.ExtraConfig['ceph::profile::params::osds'] }}"
- name: generate new environment file
template:
src: cephosd.yaml.j2
dest: "{{working_dir}}/ceph-ansible-env.yaml"
when: ceph_env is defined
when: ceph_ansible|succeeded

239
tasks/upgrade/convert_roles_data.yaml Normal file
View File

@ -0,0 +1,239 @@
- name: Register roles data file location if exists
shell: "grep '\\-r\\ \\|\\-\\-roles' {{ overcloud_deploy_script }} | awk {'print $2'}"
register: custom_roles_file
ignore_errors: true
- name: Check if roles data has already been adjusted
stat:
path: "{{ custom_roles_file.stdout }}.pre_pike_upgrade"
register: custom_roles_adjusted
- block:
- name: Make a copy of the custom roles data file
copy:
src: "{{ custom_roles_file.stdout }}"
dest: "{{ custom_roles_file.stdout }}.pre_pike_upgrade"
remote_src: true
- name: Assigns deprecated params to Controller role
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: "^(- name: Controller( # the 'primary' role goes first)?$)"
replace: "{{ item }}"
with_items:
- '\1\n deprecated_param_image: "controllerImage"'
- '\1\n deprecated_param_flavor: "OvercloudControlFlavor"'
- '\1\n deprecated_param_extraconfig: "controllerExtraConfig"'
- '\1\n uses_deprecated_params: True'
- name: Assigns network attributes to Controller role or custom controller roles
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '^(- name: Controller.*)'
replace: "{{ item }}"
with_items:
- '\1\n networks:\n - External\n - InternalApi\n - Storage\n - StorageMgmt\n - Tenant'
- '\1\n tags:\n - primary\n - controller'
- '\1\n description: |\n Controller role that has all the controler services loaded and handles\n Database, Messaging and Network functions.'
- name: Assigns deprecated params to Compute role
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '^(- name: Compute$)'
replace: "{{ item }}"
with_items:
- '\1\n deprecated_server_resource_name: "NovaCompute"'
- '\1\n deprecated_param_ips: "NovaComputeIPs"'
- '\1\n deprecated_param_scheduler_hints: "NovaComputeSchedulerHints"'
- '\1\n deprecated_param_metadata: "NovaComputeServerMetadata"'
- '\1\n deprecated_param_extraconfig: "NovaComputeExtraConfig"'
- '\1\n deprecated_param_image: "NovaImage"'
- '\1\n uses_deprecated_params: True'
- name: Assigns network attributes to Compute role or custom compute roles
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '^(- name: Compute.*)'
replace: "{{ item }}"
with_items:
- '\1\n networks:\n - InternalApi\n - Storage\n - Tenant'
- '\1\n description: |\n Basic Compute Node role'
- name: Assigns new attributes to AltCompute role
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '^(- name: AltCompute.*)'
replace: "{{ item }}"
with_items:
- '\1\n networks:\n - InternalApi\n - Storage\n - Tenant'
- '\1\n description: |\n Basic Compute Node role'
- name: Assigns new attributes to BlockStorage role
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '^(- name: BlockStorage.*)'
replace: "{{ item }}"
with_items:
- '\1\n networks:\n - InternalApi\n - Storage\n - StorageMgmt'
- '\1\n description: |\n Cinder Block Storage node role'
- name: Assigns deprecated params to ObjectStorage role
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '^(- name: ObjectStorage$)'
replace: "{{ item }}"
with_items:
- '\1\n deprecated_param_flavor: "OvercloudSwiftStorageFlavor"'
- '\1\n deprecated_param_image: "SwiftStorageImage"'
- '\1\n deprecated_param_ips: "SwiftStorageIPs"'
- '\1\n deprecated_param_metadata: "SwiftStorageServerMetadata"'
- '\1\n uses_deprecated_params: True'
- name: Assigns network attributes to ObjectStorage role or custom object storage roles
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '^(- name: ObjectStorage.*)'
replace: "{{ item }}"
with_items:
- '\1\n networks:\n - InternalApi\n - Storage\n - StorageMgmt'
- '\1\n description: |\n Swift Object Storage node role'
- name: Assigns new attributes to CephStorage role
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '^(- name: CephStorage.*)'
replace: "{{ item }}"
with_items:
- '\1\n networks:\n - Storage\n - StorageMgmt'
- '\1\n description: |\n Ceph OSD Storage node role'
- name: Assigns new attributes to Database role
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '^(- name: Database.*)'
replace: "{{ item }}"
with_items:
- '\1\n networks:\n - InternalApi'
- '\1\n description: |\n Standalone database role with the database being managed via Pacemaker'
- name: Assigns new attributes to Galera role
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '^(- name: Galera.*)'
replace: "{{ item }}"
with_items:
- '\1\n networks:\n - InternalApi'
- '\1\n description: |\n Standalone database role with the database being managed via Pacemaker'
- name: Assigns new attributes to Networker role
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '^(- name: Networker.*)'
replace: "{{ item }}"
with_items:
- '\1\n networks:\n - InternalApi\n - Tenant'
- '\1\n description: |\n Standalone networking role to run Neutron agents'
- name: Assigns new attributes to Messaging role
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '^(- name: Messaging.*)'
replace: "{{ item }}"
with_items:
- '\1\n networks:\n - InternalApi'
- '\1\n description: |\n Standalone messaging role with RabbitMQ being managed via Pacemaker'
- name: Add services common to all roles introduced in Pike
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '(- OS::TripleO::Services::Timezone)'
replace: "{{ item }}"
with_items:
- '\1\n - OS::TripleO::Services::CertmongerUser'
- '\1\n - OS::TripleO::Services::Docker'
- '\1\n - OS::TripleO::Services::Securetty'
- '\1\n - OS::TripleO::Services::Tuned'
- '\1\n - OS::TripleO::Services::ContainersLogrotateCrond'
- name: Add CinderBackend services introduced in Pike
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '(- .*CinderVolume)'
replace: "{{ item }}"
with_items:
- '\1\n - OS::TripleO::Services::CinderBackendVRTSHyperScale'
- '\1\n - OS::TripleO::Services::CinderBackendDellEMCUnity'
- '\1\n - OS::TripleO::Services::CinderBackendDellEMCVMAXISCSI'
- name: Add Clustercheck service introduced in Pike
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '(- OS::TripleO::Services::MySQL$)'
replace: '\1\n - OS::TripleO::Services::Clustercheck'
- name: Add ExternalSwiftProxy service introduced in Pike
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '(- OS::TripleO::Services::SwiftProxy)'
replace: '\1\n - OS::TripleO::Services::ExternalSwiftProxy'
- name: Add Iscsid service introduced in Pike
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: "{{ item }}"
replace: '\1\n - OS::TripleO::Services::Iscsid'
with_items:
- '(- .*CinderVolume)'
- '(- OS::TripleO::Services::NovaCompute)'
- name: Add Neutron API services introduced in Pike
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '(- OS::TripleO::Services::NeutronApi)'
replace: "{{ item }}"
with_items:
- '\1\n - OS::TripleO::Services::NeutronBgpVpnApi'
- '\1\n - OS::TripleO::Services::NeutronL2gwApi'
- name: Add Neutron agents introduced in Pike
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '(- OS::TripleO::Services::NeutronL3Agent)'
replace: "{{ item }}"
with_items:
- '\1\n - OS::TripleO::Services::NeutronL2gwAgent'
- '\1\n - OS::TripleO::Services::NeutronLbaasv2Agent'
- name: Add Neutron agents introduced in Pike
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '(- .*NeutronOvsAgent)'
replace: "{{ item }}"
with_items:
- '\1\n - OS::TripleO::Services::NeutronVppAgent'
- '\1\n - OS::TripleO::Services::NeutronLinuxbridgeAgent'
- '\1\n - OS::TripleO::Services::Vpp'
- name: Add NovaMigrationTarget service introduced in Pike
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '(- OS::TripleO::Services::NovaCompute)'
replace: '\1\n - OS::TripleO::Services::NovaMigrationTarget'
- name: Add OVNController service introduced in Pike
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '(- OS::TripleO::Services::OVNDBs)'
replace: '\1\n - OS::TripleO::Services::OVNController'
- name: Add Manila backend services introduced in Pike
replace:
dest: '{{ custom_roles_file.stdout }}'
regexp: '(- OS::TripleO::Services::ManilaShare)'
replace: "{{ item }}"
with_items:
- '\1\n - OS::TripleO::Services::ManilaBackendIsilon'
- '\1\n - OS::TripleO::Services::ManilaBackendUnity'
- '\1\n - OS::TripleO::Services::ManilaBackendVMAX'
- '\1\n - OS::TripleO::Services::ManilaBackendVNX'
when: custom_roles_file.stdout|length > 0 and not custom_roles_adjusted.stat.exists

178
tasks/upgrade/create-upgrade-scripts.yaml Normal file
View File

@ -0,0 +1,178 @@
---
- name: create a comptatible deployment scripts from oooq
include: use_oooq.yaml
tags: use_oooq
when: use_oooq|bool
- name: make a copy of the initial overcloud deploy script
copy:
remote_src: yes
src: "{{ overcloud_deploy_script }}"
dest: "{{ overcloud_deploy_script }}.orig"
- name: replace services environment files with services-docker correspondents
replace:
dest: "{{ overcloud_deploy_script }}"
regexp: environments/services/
replace: environments/services-docker/
when: upstream_container_images|bool
- name: replace storage environment with ceph ansible environment
replace:
dest: "{{ overcloud_deploy_script }}"
regexp: environments/storage-environment.yaml
replace: environments/ceph-ansible/ceph-ansible.yaml
when: upstream_container_images|bool
- name: replace ceph radosgw with ceph ansible radosgw environment
replace:
dest: "{{ overcloud_deploy_script }}"
regexp: environments/ceph-radosgw.yaml
replace: environments/ceph-ansible/ceph-rgw.yaml
- name: replace external ceph environment with ceph ansible environment
replace:
dest: "{{ overcloud_deploy_script }}"
regexp: environments/puppet-ceph-external.yaml
replace: environments/ceph-ansible/ceph-ansible-external.yaml
when: upstream_container_images|bool
- name: replace external ceph environment with new external ceph environment
replace:
dest: "{{ overcloud_deploy_script }}"
regexp: environments/puppet-ceph-external.yaml
replace: environments/storage-environment-external.yaml
when: not upstream_container_images|bool
- name: Replace baremetal environment file to containerized used in OOOQ
replace:
dest: "{{ overcloud_deploy_script }}"
regexp: ci/environments/(\w+).yaml
replace: ci/environments/\1-containers.yaml
when: tripleo_ci
- name: Convert Ceph parameters to ceph ansible
include: convert_ceph_params.yaml
- name: Adjust custom roles data file during upgrade
include: convert_roles_data.yaml
- name: check if dpdk is enabled
command: "grep -Fq environments/neutron-ovs-dpdk.yaml {{ overcloud_deploy_script }}"
register: dpdk_env
ignore_errors: true
- name: create dpdk env required for upgrade
template:
src: "dpdk-upgrade-env.yaml.j2"
dest: "{{working_dir}}/dpdk-upgrade-env.yaml"
when: dpdk_env|succeeded
- name: check if undercloud is ssl enabled
command: "grep -Fq OS_AUTH_URL=https {{ undercloud_rc }}"
register: undercloud_ssl
ignore_errors: true
- include: undercloud_ssl_camap.yaml
when: undercloud_ssl|succeeded
- name: create workload launch script
template:
src: "{{ workload_launch_template }}"
dest: "{{ workload_launch_script }}"
mode: 0775
- name: create undercloud upgrade script
template:
src: "{{ undercloud_upgrade_template }}"
dest: "{{ undercloud_upgrade_script }}"
mode: 0775
- name: create upgrade workaround scripts
template:
src: workarounds.sh.j2
dest: "{{working_dir}}/{{ item }}.sh"
mode: 0775
with_items:
- 'pre_undercloud_upgrade_workarounds'
- 'post_undercloud_upgrade_workarounds'
- 'pre_docker_composable_upgrade_workarounds'
- 'pre_nova_compute_upgrade_workarounds'
- 'pre_docker_converge_upgrade_workarounds'
- 'post_docker_converge_upgrade_workarounds'
when: upgrade_workarounds
- block:
- name: create start l3 agent connectivity check scripts
template:
src: "{{ l3_agent_connectivity_check_start_template }}"
dest: "{{ l3_agent_connectivity_check_start_script }}"
mode: 0775
- name: create stop l3 agent connectivity check scripts
template:
src: "{{ l3_agent_connectivity_check_stop_template }}"
dest: "{{ l3_agent_connectivity_check_stop_script }}"
mode: 0775
- name: register used service environment files
shell: |
grep '\-e\ \|\-\-environment-file\ ' {{ overcloud_deploy_script }} | awk {'print $2'} | grep -E 'environments/services|environments/storage|environments/ceph-ansible'
register: services
ignore_errors: true
- name: create container images download script
template:
src: "{{ container_images_download_template }}"
dest: "{{ container_images_download_script }}"
mode: 0775
force: no
- name: create registry environment file script
template:
src: "{{ local_docker_registry_env_template }}"
dest: "{{ local_docker_registry_env_script }}"
mode: 0775
force: no
- name: Check if overcoud deployed with pacemaker
shell: grep "\-pacemaker" "{{ overcloud_deploy_script }}"
register: deployment_with_ha
ignore_errors: True
- name: create composable upgrade scripts
include: step_upgrade.yml
loop_control:
loop_var: ugstage
with_items:
- step: "Docker containers composable upgrade"
script: "{{ overcloud_composable_upgrade_script }}"
environment_file:
- "{% if upstream_container_images %}{{ tht_directory }}/environments/docker.yaml{% endif %}"
- "{% if upstream_container_images and (deployment_with_ha.rc == 0) %}{{ tht_directory }}/environments/docker-ha.yaml{% endif %}"
- "{{ tht_directory }}/environments/major-upgrade-composable-steps-docker.yaml"
- "{% if ceph_env is defined %}{{ working_dir }}/ceph-ansible-env.yaml{% endif %}"
- "{% if not upstream_container_images or (upstream_container_images and use_local_docker_registry) %}{{ containers_default_parameters }}{% endif %}"
- "{% if dpdk_env|succeeded %}{{working_dir}}/dpdk-upgrade-env.yaml{% endif %}"
- step: "Docker containers converge upgrade"
script: "{{ overcloud_converge_upgrade_script }}"
environment_file:
- "{% if upstream_container_images %}{{ tht_directory }}/environments/docker.yaml{% endif %}"
- "{% if upstream_container_images %}{{ tht_directory }}/environments/docker-ha.yaml{% endif %}"
- "{{ tht_directory }}/environments/major-upgrade-converge-docker.yaml"
- "{% if ceph_env is defined %}{{ working_dir }}/ceph-ansible-env.yaml{% endif %}"
- "{% if not upstream_container_images or (upstream_container_images and use_local_docker_registry) %}{{ containers_default_parameters }}{% endif %}"
- "{% if dpdk_env|succeeded %}{{working_dir}}/dpdk-upgrade-env.yaml{% endif %}"
- name: adjust ssh config to skip host key check
copy:
src: ssh_config
dest: "~/.ssh/config"
mode: 0600
when: need_ssh_config|bool
- name: Create upgrade scripts for non controller nodes
include: non_controller_upgrade_scripts.yml
- name: Create post upgrade scripts for controller nodes
include: ../common/controller_post_scripts.yml

36
tasks/upgrade/docker_composable_upgrade.yml Normal file
View File

@ -0,0 +1,36 @@
---
- name: start l3 agent connectivity check
shell: |
source {{ overcloud_rc }}
{{ l3_agent_connectivity_check_start_script }}
when: l3_agent_connectivity_check
async: 21660
poll: 0
- name: run docker upgrade composable step
shell: |
source {{ undercloud_rc }}
bash {{ overcloud_composable_upgrade_script }} &> {{ working_dir }}/overcloud_composable_upgrade.log
register: overcloud_composable_upgrade
ignore_errors: yes
- name: print stack failures
shell: |
source {{ undercloud_rc }}
openstack stack failures list --long overcloud | tee {{ working_dir}}/overcloud_composable_failed_upgrade.log
when: overcloud_composable_upgrade.rc != 0
- name: print resource list
shell: |
source {{ undercloud_rc }}
openstack stack resource list --filter status=FAILED --nested-depth 5 overcloud | tee {{ working_dir}}/overcloud_composable_failed_resources.log
when: overcloud_composable_upgrade.rc != 0
- fail: msg="Overcloud upgrade composable step failed... :("
when: overcloud_composable_upgrade.rc != 0
- name: stop l3 agent connectivity check
shell: |
source {{ overcloud_rc }}
{{ l3_agent_connectivity_check_stop_script }}
when: l3_agent_connectivity_check

30
tasks/upgrade/docker_converge_upgrade.yml Normal file
View File

@ -0,0 +1,30 @@
---
- name: start l3 agent connectivity check
shell: |
source {{ overcloud_rc }}
{{ l3_agent_connectivity_check_start_script }}
when: l3_agent_connectivity_check
async: 21660
poll: 0
- name: run docker upgrade converge step
shell: |
source {{ undercloud_rc }}
bash {{ overcloud_converge_upgrade_script }} &> overcloud_converge_upgrade.log
register: overcloud_converge_upgrade
ignore_errors: yes
- name: print stack failures
shell: |
source {{ undercloud_rc }}
openstack stack failures list overcloud
when: overcloud_converge_upgrade.rc != 0
- fail: msg="Overcloud upgrade converge step failed... :("
when: overcloud_converge_upgrade.rc != 0
- name: stop l3 agent connectivity check
shell: |
source {{ overcloud_rc }}
{{ l3_agent_connectivity_check_stop_script }}
when: l3_agent_connectivity_check

0
tasks/local_docker_registry_env.yaml → tasks/upgrade/docker_registry_images_env.yaml
View File

3
tasks/upgrade/local_docker_registry_env.yaml Normal file
View File

@ -0,0 +1,3 @@
---
- name: create local registry environment file
shell: "bash {{ local_docker_registry_env_script }}"

76
tasks/upgrade/main.yml Normal file
View File

@ -0,0 +1,76 @@
---
- name: create upgrade scripts
include: create-upgrade-scripts.yaml
tags: create_upgrade_scripts
when: upgrade_noop|bool or undercloud_upgrade|bool or overcloud_upgrade|bool
- block:
- name: apply pre undercloud upgrade workarounds
command: "{{working_dir}}/pre_undercloud_upgrade_workarounds.sh"
when: upgrade_workarounds
- name: upgrade undercloud
shell: "bash {{ undercloud_upgrade_script }} &> {{ working_dir }}/undercloud_upgrade.log"
tags: undercloud_upgrade
- name: validate undercloud upgrade
include: undercloud_validate_upgrade.yaml
tags: undercloud_upgrade_validate
- name: apply post undercloud upgrade workarounds
command: "{{working_dir}}/post_undercloud_upgrade_workarounds.sh"
when: upgrade_workarounds
when: undercloud_upgrade|bool
- block:
# TODO: move this out of upgrade
- name: download container images
include: container_images.yaml
when: use_local_docker_registry
tags: container_images
- name: create local docker registry environment file
include: docker_registry_images_env.yaml
when: create_docker_registry
tags: docker_registry_images_env
- name: apply pre docker composable upgrade workarounds
command: "{{working_dir}}/pre_docker_composable_upgrade_workarounds.sh"
when: upgrade_workarounds
tags: docker_composable_upgrade
- include: docker_composable_upgrade.yml
tags: docker_composable_upgrade
- name: apply pre nova compute upgrade workarounds
command: "{{working_dir}}/pre_nova_compute_upgrade_workarounds.sh"
when: upgrade_workarounds
tags: nova_compute_upgrade
- name: upgrade nova compute nodes
include: compute_upgrade.yml
tags: nova_compute_upgrade
- name: upgrade swift storage nodes
include: swiftstorage_upgrade.yml
tags: swift_storage_upgrade
- name: apply pre docker upgrade converge workarounds
command: "{{working_dir}}/pre_docker_converge_upgrade_workarounds.sh"
when: upgrade_workarounds
tags: docker_converge_upgrade
- include: docker_converge_upgrade.yml
tags: docker_converge_upgrade
- name: apply post docker upgrade converge workarounds
command: "{{working_dir}}/post_docker_converge_upgrade_workarounds.sh"
when: upgrade_workarounds
tags: docker_converge_upgrade
- name: run controller post upgrade steps
include: controller_post_upgrade.yml
tags: controller_post_upgrade
when: controller_upgrade_post|bool
when: overcloud_upgrade|bool

11
tasks/upgrade/node_upgrade.yml Normal file
View File

@ -0,0 +1,11 @@
---
- name: Running pre upgrade for {{ node_name | splitext | first }}
shell: "{{working_dir}}/{{ node_name | splitext | first | splitext | first }}_upgrade_pre.sh"
tags: node_upgrade_pre
- name: Upgrade {{ node_name | splitext | first }}
shell: "{{working_dir}}/{{ node_name | splitext | first | splitext | first }}_upgrade.sh"
- name: Running post upgrade for {{ node_name | splitext | first }}
shell: "{{working_dir}}/{{ node_name | splitext | first | splitext | first }}_upgrade_post.sh"
tags: node_upgrade_post

26
tasks/upgrade/node_upgrade_script.yml Normal file
View File

@ -0,0 +1,26 @@
---
- block:
- name: register instances running on {{ node_name }}
shell: |
source {{ overcloud_rc }}
openstack server list --host {{ node_name }} -f json | jq -r -c '.[] | select(.Status | contains("ACTIVE") or contains("PAUSED")) | .Name'
register: node_instances
- name: create pre upgrade script for {{ node_name }}
template:
src: node_upgrade_pre.sh.j2
dest: "{{working_dir}}/{{ node_name | splitext | first | splitext | first }}_upgrade_pre.sh"
mode: 0775
when: "'compute' in node_name"
- name: create script for upgrading {{ node_name }}
template:
src: node_upgrade.sh.j2
dest: "{{working_dir}}/{{ node_name | splitext | first | splitext | first }}_upgrade.sh"
mode: 0775
- name: create post upgrade script for {{ node_name }}
template:
src: node_upgrade_post.sh.j2
dest: "{{working_dir}}/{{ node_name | splitext | first | splitext | first }}_upgrade_post.sh"
mode: 0775

18
tasks/upgrade/non_controller_upgrade_scripts.yml Normal file
View File

@ -0,0 +1,18 @@
- name: register compute nodes
shell: |
source {{ overcloud_rc }}
openstack hypervisor list -f json | jq -r -c '.[] | .["Hypervisor Hostname"]'
register: hypervisors
- name: register swift storage nodes
shell: |
source {{ undercloud_rc }}
openstack server list -f json | jq -r -c '.[] | select(.Name | contains("swift") or contains("objectstorage")) | .Name'
register: swift_nodes
- include: node_upgrade_script.yml
loop_control:
loop_var: node_name
with_items:
- "{{ hypervisors.stdout_lines }}"
- "{{ swift_nodes.stdout_lines }}"

14
tasks/step_upgrade.yml → tasks/upgrade/step_upgrade.yml
View File

@ -18,8 +18,22 @@
with_items: "{{ ugstage.environment_file }}"
when: item|length > 0
- name: Append options to {{ ugstage.step }} script
lineinfile:
dest: "{{ ugstage.script }}"
insertbefore: "{{ deploy_lastline.stdout }}"
line: '--{{ item }} \'
with_items: "{{ ugstage.option|default('') }}"
when: item|length > 0
- name: Change log file name of {{ ugstage.step }} script
lineinfile:
dest: "{{ ugstage.script }}"
regexp: '^--log-file.*'
state: absent
- name: Remove timeout from {{ ugstage.step }} script
replace:
dest: "{{ ugstage.script }}"
regexp: 'timeout 100m '
replace: ''

11
tasks/upgrade/swiftstorage_upgrade.yml Normal file
View File

@ -0,0 +1,11 @@
- name: register swift storage nodes
shell: |
source {{ undercloud_rc }}
openstack server list -f json | jq -r -c '.[] | select(.Name | contains("swift") or contains("objectstorage")) | .Name'
register: swift_nodes
- include: node_upgrade.yml
with_items: "{{ swift_nodes.stdout_lines }}"
loop_control:
loop_var: node_name
when: overcloud_upgrade|bool

48
tasks/upgrade/undercloud_ssl_camap.yaml Normal file
View File

@ -0,0 +1,48 @@
- name: register undercloud public endpoint
shell: |
source {{ undercloud_rc }}
openstack catalog list | grep -Po 'https.*13000'
register: keystone_endpoint
- name: register first controller ip address
shell: |
source {{ undercloud_rc }}
openstack server list -f json | jq -r -c '.[] | select(.Name | contains("controller","ctrl")) | .Networks' | grep -oP '[0-9.]+' | head -1
register: ctrl_ip
- name: test undercloud keystone reachability
shell: |
ssh -q -o StrictHostKeyChecking=no heat-admin@{{ ctrl_ip.stdout }} curl --silent {{ keystone_endpoint.stdout }}
register: uc_keystone_conn
ignore_errors: True
- block:
- name: register ssl certificate location
shell: |
grep 13000 /etc/haproxy/haproxy.cfg | awk {'print $6'}
become: true
become_user: root
register: undercloudcert
- name: make a local copy of the certificate
copy:
src: "{{ undercloudcert.stdout }}"
dest: "{{ working_dir }}/undercloud.pem"
owner: stack
remote_src: true
become: true
become_user: root
- name: register overcloud nodes ip address
shell: |
source {{ undercloud_rc }}
openstack server list -f json | jq -r -c '.[] | .Networks' | grep -oP '[0-9.]+'
register: node_ip
- name: copy certificate to the overcloud nodes and update the trusted store
shell: |
scp -q -o StrictHostKeyChecking=no {{ working_dir }}/undercloud.pem heat-admin@{{ item }}:
ssh -q -o StrictHostKeyChecking=no heat-admin@{{ item }} 'sudo cp undercloud.pem /etc/pki/ca-trust/source/anchors/; sudo update-ca-trust extract'
with_items:
- "{{ node_ip.stdout_lines }}"
when: uc_keystone_conn|failed

64
tasks/upgrade/undercloud_validate_upgrade.yaml Normal file
View File

@ -0,0 +1,64 @@
---
- name: register latest installed kernel version
shell: |
rpm -qa | grep ^kernel-[0-9] | sort | tail -1 | awk -F 'kernel-' {'print $2'}
register: installed_kernel
- name: register loaded kernel
command: uname -r
register: loaded_kernel
- name: register installed openvswitch package version
shell: |
rpm --queryformat %{VERSION} -q openvswitch | awk -F "." '{print $1"."$2}'
register: installed_ovs
- name: register loaded openvswitch version
shell: |
ovs-vsctl show | grep ovs_version | awk -F \" {'print $2'} | awk -F "." '{print $1"."$2}'
become: true
become_user: root
register: loaded_ovs
- block:
- name: reboot the undercloud
shell: "sleep 2 && shutdown -r now"
async: 1
poll: 0
ignore_errors: true
become: true
become_user: root
- name: waiting for the undercloud to be available
become: no
wait_for:
port: 22
host: "{{ ansible_ssh_host }}"
search_regex: OpenSSH
delay: 10
delegate_to: localhost
when: "'hypervisor' not in groups and 'virthost' not in groups"
- name: waiting for the undercloud to be available
become: no
wait_for:
port: 22
host: "{{ ansible_ssh_host }}"
search_regex: OpenSSH
delay: 10
delegate_to: hypervisor
when: "'hypervisor' in groups"
- name: waiting for the undercloud to be available
become: no
wait_for:
port: 22
host: "{{ ansible_default_ipv4.address }}"
search_regex: OpenSSH
delay: 10
delegate_to: virthost
when: "'virthost' in groups"
when:
- not tripleo_ci and (undercloud_reboot or (installed_kernel.stdout != loaded_kernel.stdout) or (installed_ovs.stdout != loaded_ovs.stdout))
tags: undercloud_reboot

21
tasks/upgrade/use_oooq.yaml Normal file
View File

@ -0,0 +1,21 @@
---
- name: get the compute ip.
shell: ". {{working_dir}}/stackrc && nova list | awk '$1 !~ /^\\+/ && NR>3 && $0 ~ /compute/ {print $4}'"
register: compute
- name: create the compute group
add_host:
name: "{{item}}"
group: compute
ansible_fqdn: "{{item}}"
with_items:
- "{{ compute.stdout_lines|default([]) }}"
- name: create transformation script
template:
src: oooq_deploy_transformation.sh.j2
dest: "{{ working_dir }}/oooq_deploy_transformation.sh"
mode: 0775
- name: transform oooq script to compatible format
command: "{{ working_dir }}/oooq_deploy_transformation.sh"

6
templates/cephosd.yaml.j2 Normal file
View File

@ -0,0 +1,6 @@
parameter_defaults:
CephAnsibleDisksConfig:
devices:
{% for key, value in ceph_osds.iteritems()|sort %}
- '{{ key }}'
{% endfor %}

45
templates/check_service_galera.sh.j2 Normal file
View File

@ -0,0 +1,45 @@
source {{ undercloud_rc }}
NODE_IP=$(openstack server show {{ node_name | splitext | first }} -f json | jq -r .addresses | grep -oP '[0-9.]+')
## wait for galera resource to come back up
timeout_seconds={{ node_reboot_timeout }}
elapsed_seconds=0
while true; do
echo "Waiting for galera pcs resource to start"
GALERA_RES=$(ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo pcs status --full' | grep ocf::heartbeat:galera | grep -vi FAILED | grep -i master | wc -l)
if [[ $GALERA_RES = 1 ]] || [[ $GALERA_RES > 2 ]]; then
break
fi
sleep 3
(( elapsed_seconds += 3 ))
if [ $elapsed_seconds -ge $timeout_seconds ]; then
echo "WARNING: galera pcs resource didn't get started after reboot. Trying to workaround BZ#1499677"
GVWSTATE_SIZE=$(ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo touch /var/lib/mysql/gvwstate.dat; sudo wc -c /var/lib/mysql/gvwstate.dat' | awk {'print $1'})
if [ $GVWSTATE_SIZE -eq 0 ]; then
echo "Removing gvwstate.dat"
ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo rm -f /var/lib/mysql/gvwstate.dat'
echo "Cleanup galera resource"
ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo pcs resource cleanup galera'
timeout_seconds={{ node_reboot_timeout }}
elapsed_seconds=0
while true; do
echo "Waiting for galera pcs resource to start"
GALERA_RES=$(ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo pcs status --full' | grep ocf::heartbeat:galera | grep -i master | wc -l)
if [[ $GALERA_RES = 1 ]] || [[ $GALERA_RES > 2 ]]; then
break
fi
sleep 3
(( elapsed_seconds += 3 ))
if [ $elapsed_seconds -ge $timeout_seconds ]; then
echo "FAILURE: galera pcs resource didn't get started after reboot. Workaround for BZ#1499677 applied."
exit 1
fi
done
else
echo "FAILURE: galera pcs resource didn't get started after reboot"
exit 1
fi
fi
done

23
templates/check_service_haproxy.sh.j2 Normal file
View File

@ -0,0 +1,23 @@
source {{ undercloud_rc }}
NODE_IP=$(openstack server show {{ node_name | splitext | first }} -f json | jq -r .addresses | grep -oP '[0-9.]+')
## in case of external loadbalancer haproxy resource is not running on controller nodes
EXT_LB=$(ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo hiera -c /etc/puppet/hiera.yaml enable_load_balancer')
if [[ $EXT_LB != 'false' ]]; then
## wait for haproxy resource to come back up
timeout_seconds={{ node_reboot_timeout }}
elapsed_seconds=0
while true; do
echo "Waiting for haproxy pcs resource to start"
HAPROXY_RES=$(ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo pcs status --full' | grep haproxy-bundle | grep -i started | wc -l)
if [[ $HAPROXY_RES = 1 ]] || [[ $HAPROXY_RES > 2 ]]; then
break
fi
sleep 3
(( elapsed_seconds += 3 ))
if [ $elapsed_seconds -ge $timeout_seconds ]; then
echo "FAILURE: Haproxy pcs resource didn't get started after reboot"
exit 1
fi
done
fi

26
templates/check_service_haproxy_backend.sh.j2 Normal file
View File

@ -0,0 +1,26 @@
source {{ undercloud_rc }}
NODE_IP=$(openstack server show {{ node_name | splitext | first }} -f json | jq -r .addresses | grep -oP '[0-9.]+')
## in case of external loadbalancer haproxy resource is not running on controller nodes
EXT_LB=$(ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo hiera -c /etc/puppet/hiera.yaml enable_load_balancer')
if [[ $EXT_LB != 'false' ]]; then
timeout_seconds={{ node_reboot_timeout }}
elapsed_seconds=0
while true; do
rm -rf ~/haproxy.stats
echo "Waiting for haproxy backend services to come up"
ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP > ~/haproxy.stats <<-\SSH
sudo docker exec $(sudo docker ps | grep -oP haproxy-bundle.*) bash -c 'echo "show stat" | socat /var/lib/haproxy/stats stdio | grep -v redis'
SSH
grep DOWN ~/haproxy.stats > /dev/null
if [[ $? != 0 ]]; then
break
fi
sleep 3
(( elapsed_seconds += 3 ))
if [ $elapsed_seconds -ge $timeout_seconds ]; then
echo "FAILURE: $(grep DOWN haproxy.stats | awk -F ',' {'print $1'}) is down on $(grep DOWN haproxy.stats | awk -F ',' {'print $2'})"
exit 1
fi
done
fi

19
templates/check_service_rabbitmq.sh.j2 Normal file
View File

@ -0,0 +1,19 @@
source {{ undercloud_rc }}
NODE_IP=$(openstack server show {{ node_name | splitext | first }} -f json | jq -r .addresses | grep -oP '[0-9.]+')
## wait for rabbitmq resource to come back up
timeout_seconds={{ node_reboot_timeout }}
elapsed_seconds=0
while true; do
echo "Waiting for rabbitmq pcs resource to start"
RABBIT_RES=$(ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo pcs status --full' | grep ocf::heartbeat:rabbitmq-cluster | grep -vi FAILED | grep -i started | wc -l)
if [[ $RABBIT_RES = 1 ]] || [[ $RABBIT_RES > 2 ]]; then
break
fi
sleep 3
(( elapsed_seconds += 3 ))
if [ $elapsed_seconds -ge $timeout_seconds ]; then
echo "FAILURE: Rabbitmq pcs resource didn't get started after reboot"
exit 1
fi
done

51
templates/check_service_reboot.sh.j2 Normal file
View File

@ -0,0 +1,51 @@
source {{ undercloud_rc }}
NODE_IP=$(openstack server show {{ node_name | splitext | first }} -f json | jq -r .addresses | grep -oP '[0-9.]+')
{% if controller_reboot %}
OVS_RUNNING=$(ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo ovs-vsctl show' | grep ovs_version | awk -F \" {'print $2'} | awk -F "." '{print $1"."$2}')
OVS_INSTALLED=$(ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo rpm --queryformat %{VERSION} -q openvswitch' | awk -F "." '{print $1"."$2}')
if [[ $OVS_RUNNING != $OVS_INSTALLED ]]; then
echo "Upgraded OVS detected"
fi
echo "Rebooting {{ node_name | splitext | first }}"
{% if force_reboot %}
NOVA_ID=$(openstack server list | grep {{ node_name | splitext | first }} | awk {'print $2'})
IRONIC_ID=$(ironic node-list | grep $NOVA_ID | awk {'print $2'})
ironic node-set-power-state $IRONIC_ID reboot
{% else %}
ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo shutdown -r now'
{% endif %}
timeout_seconds={{ node_reboot_timeout }}
elapsed_seconds=0
while true; do
echo "Waiting for {{ node_name }} to go down ..."
NODE_DOWN=$(ping -c1 $NODE_IP)
if [ $? != 0 ]; then
break
fi
sleep 3
(( elapsed_seconds += 3 ))
if [ $elapsed_seconds -ge $timeout_seconds ]; then
echo "FAILURE: Node {{ node_name }} didn't reboot in time"
exit 1
fi
done
{% endif %}
## wait for node to get back online
timeout_seconds={{ node_reboot_timeout }}
elapsed_seconds=0
while true; do
echo "Waiting for {{ node_name }} to boot ..."
PCS_STATUS=$(ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo pcs status' | grep ^Online)
if [[ $PCS_STATUS == *{{ node_name }}* ]]; then
break
fi
sleep 3
(( elapsed_seconds += 3 ))
if [ $elapsed_seconds -ge $timeout_seconds ]; then
echo "FAILURE: {{ node_name }} didn't come back up as part of the cluster"
exit 1
fi
done

19
templates/check_service_redis.sh.j2 Normal file
View File

@ -0,0 +1,19 @@
source {{ undercloud_rc }}
NODE_IP=$(openstack server show {{ node_name | splitext | first }} -f json | jq -r .addresses | grep -oP '[0-9.]+')
## wait for redis resource to come back up
timeout_seconds={{ node_reboot_timeout }}
elapsed_seconds=0
while true; do
echo "Waiting for redis pcs resource to start"
REDIS_RES=$(ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo pcs status --full' | grep ocf::heartbeat:redis | grep -vi FAILED | grep -i master | wc -l)
if [[ $REDIS_RES = 1 ]]; then
break
fi
sleep 3
(( elapsed_seconds += 3 ))
if [ $elapsed_seconds -ge $timeout_seconds ]; then
echo "FAILURE: redis pcs resource didn't get started after reboot"
exit 1
fi
done

38
templates/create_registry_env.sh.j2
View File

@ -4,18 +4,52 @@
set -euo pipefail
{% if upstream_container_images %}
openstack overcloud container image prepare \
--tag latest \
--env-file {{ containers_default_parameters }} \
{% if use_local_docker_registry %}
--namespace={{ ansible_br_ctlplane.ipv4.address }}:8787/tripleoupstream
cat > {{ containers_default_parameters }} <<EOF
parameter_defaults:
DockerNamespace: {{ ansible_br_ctlplane.ipv4.address }}:8787/tripleoupstream
DockerNamespaceIsRegistry: true
EOF
{% else %}
{% if docker_registry_url != 'registry.example.local' %}
--namespace {{ docker_registry_url }}/tripleoupstream
{% else %}
--namespace tripleoupstream
{% endif %}
{% endif %}
{% else %}
REPO="$(find /etc/yum.repos.d/ -iname 'rhos-release-??.*')"
REPO_URL="$(grep -B2 enabled=1 $REPO | grep -m1 puddle | sed -E 's/.*(http.*[0-9]-RHEL-.\/).*/\1/')"
curl -L -o {{ containers_default_parameters }} $REPO_URL/latest_containers/docker-osp12.yaml
curl -L -o {{ working_dir }}/overcloud_container_image_prepare.yaml $REPO_URL/latest/overcloud_container_image_prepare.yaml
REGISTRY="$(grep -v ^# {{ working_dir }}/overcloud_container_image_prepare.yaml | grep ' namespace' | awk -F': ' {'print $2'} | awk -F'/' {'print $1'})"
sudo sed -i -E "s/(--insecure-registry.*)\"/\1\ --insecure-registry\ $REGISTRY\"/" /etc/sysconfig/docker
sudo systemctl restart docker
TAG="$(openstack overcloud container image tag discover --image $REGISTRY/rhosp12/openstack-base:latest --tag-from-label version-release)"
{% if docker_registry_url != 'registry.example.local' %}
REGISTRY='{{ docker_registry_url }}'
{% endif %}
openstack overcloud container image prepare \
--env-file={{ containers_default_parameters }} \
--prefix=openstack- \
--suffix=-docker \
--tag="$TAG" \
--set ceph_image=rhceph-2-rhel7 \
--set ceph_tag=latest \
{% if use_local_docker_registry %}
sed -i s/DockerNamespace:.*/DockerNamespace:\ {{ ansible_br_ctlplane.ipv4.address }}:8787/rhosp12/ {{ containers_default_parameters }}
{% for envs in services.stdout_lines|default([]) %}
--service-environment-file={{envs}} \
{% endfor %}
--namespace={{ ansible_br_ctlplane.ipv4.address }}:8787/rhosp12 \
--set ceph_namespace={{ ansible_br_ctlplane.ipv4.address }}:8787/ceph
{% else %}
{% for envs in services.stdout_lines|default([]) %}
--service-environment-file={{envs}} \
{% endfor %}
--namespace=$REGISTRY/rhosp12 \
--set ceph_namespace=$REGISTRY/ceph
{% endif %}
{% endif %}

29
templates/download_images.sh.j2
View File

@ -4,16 +4,37 @@
set -euo pipefail
{% if upstream_container_images %}
openstack overcloud container image prepare \
--namespace tripleoupstream \
--tag latest \
--push-destination {{ ansible_br_ctlplane.ipv4.address }}:8787 \
--images-file {{ container_images_location }}
source {{ undercloud_rc }}
openstack overcloud container image upload --verbose --config-file /usr/share/tripleo-common/container-images/overcloud_containers.yaml
openstack overcloud container image upload --verbose --config-file {{ container_images_location }}
{% else %}
REPO_FILE="$(find /etc/yum.repos.d/ -iname 'rhos-release-??.*')"
REPO_URL="$(grep -B2 enabled=1 $REPO_FILE | grep -m1 puddle | sed -E 's/.*(http.*[0-9]-RHEL-.\/).*/\1/')"
curl -L -o {{ container_images_location }} $REPO_URL/latest_containers/container_images.yaml
REGISTRY="$(grep pull_source {{ container_images_location }} | uniq | awk {'print $2'})"
curl -L -o {{ working_dir }}/overcloud_container_image_prepare.yaml $REPO_URL/latest/overcloud_container_image_prepare.yaml
REGISTRY="$(grep -v ^# {{ working_dir }}/overcloud_container_image_prepare.yaml | grep ' namespace' | awk -F': ' {'print $2'} | awk -F'/' {'print $1'})"
sudo sed -i -E "s/(--insecure-registry.*)\"/\1\ --insecure-registry\ $REGISTRY\"/" /etc/sysconfig/docker
sudo systemctl restart docker
TAG="$(openstack overcloud container image tag discover --image $REGISTRY/rhosp12/openstack-base:latest --tag-from-label version-release)"
{% if docker_registry_url != 'registry.example.local' %}
REGISTRY='{{ docker_registry_url }}'
{% endif %}
openstack overcloud container image prepare \
--prefix=openstack- \
--suffix=-docker \
--tag="$TAG" \
--set ceph_image=rhceph-2-rhel7 \
--set ceph_tag=latest \
--push-destination {{ ansible_br_ctlplane.ipv4.address }}:8787 \
{% for envs in services.stdout_lines|default([]) %}
--service-environment-file={{envs}} \
{% endfor %}
--namespace=$REGISTRY/rhosp12 \
--set ceph_namespace=$REGISTRY/ceph \
--output-images-file {{ container_images_location }}
source {{ undercloud_rc }}
openstack overcloud container image upload --verbose --config-file {{ container_images_location }}

11
templates/dpdk-upgrade-env.yaml.j2 Normal file
View File

@ -0,0 +1,11 @@
parameter_defaults:
# Run os-net-config only on DPDK compute nodes during the upgrade
# to ensure PCI address is updated to the existing DPDK port.
# NOTE: No changes in the nic-config templates.
ComputeNetworkDeploymentActions: ['CREATE', 'UPDATE']
resource_registry:
# With OSP12, the environment file will have a new service
# OS::TripleO::Services::ComputeNeutronOvsDpdk to support the addition
# of new role ComputeOvsDpdk. For upgrade, this need to be mapped externally.
OS::TripleO::Services::ComputeNeutronOvsAgent: /usr/share/openstack-tripleo-heat-templates/puppet/services/neutron-ovs-dpdk-agent.yaml

6
templates/l3_agent_start_ping.sh.j2 Normal file
View File

@ -0,0 +1,6 @@
#!/bin/bash
#
# Script which start an ICMP connectivity check on the first in use
# floating IP during upgrade
FIP=$(openstack floating ip list -f json | jq -r -c '.[] | select(.Port) | .["Floating IP Address"]' | head -1)
ping -D ${FIP} >> ~/ping_results_$(date +%Y%m%d%H%M) &

18
templates/l3_agent_stop_ping.sh.j2 Normal file
View File

@ -0,0 +1,18 @@
#!/bin/bash
#
# Script which stops the ICMP connectivity check on validates that
# there is no packet loss.
# kill the ping process
kill -s INT $(/usr/sbin/pidof ping)
# print the ping results
PING_RESULT_LOG=$(find ~ -iname 'ping_results*' | sort | tail -1)
tail -2 $PING_RESULT_LOG
# check results
PING_RESULT=$(tail -2 $PING_RESULT_LOG | head -1 | awk {'print $6'} | sed s/%//)
if [[ $PING_RESULT -gt 1 ]]; then
echo "Ping loss higher than 1% detected"
exit 1
fi

2
templates/node_upgrade.sh.j2
View File

@ -1,2 +1,2 @@
source {{ undercloud_rc }}
upgrade-non-controller.sh --upgrade {{ node_name }} &> {{ node_name }}-upgrade.log
upgrade-non-controller.sh --upgrade {{ node_name | splitext | first | splitext | first }} &> {{ node_name | splitext | first | splitext | first }}-upgrade.log

102
templates/node_upgrade_post.sh.j2
View File

@ -1,4 +1,104 @@
source {{ undercloud_rc }}
NODE_IP=$(openstack server show {{ node_name | splitext | first | splitext | first }} -f json | jq -r .addresses | grep -oP '[0-9.]+')
OVS_RUNNING=$(ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo ovs-vsctl show' | grep ovs_version | awk -F \" {'print $2'} | awk -F "." '{print $1"."$2}')
OVS_INSTALLED=$(ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo rpm --queryformat %{VERSION} -q openvswitch' | awk -F "." '{print $1"."$2}')
## change
if [[ $OVS_RUNNING != $OVS_INSTALLED ]]; then
echo "Upgraded OVS detected. Rebooting {{ node_name | splitext | first | splitext | first }}"
ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo reboot'
## wait for node to go down
timeout_seconds={{ node_reboot_timeout }}
elapsed_seconds=0
while true; do
echo "Waiting for {{ node_name | splitext | first | splitext | first }} to go down ..."
NODE_DOWN=$(ping -c1 $NODE_IP)
if [ $? != 0 ]; then
break
fi
sleep 3
(( elapsed_seconds += 3 ))
if [ $elapsed_seconds -ge $timeout_seconds ]; then
echo "FAILURE: Node {{ node_name | splitext | first | splitext | first }} didn't reboot"
exit 1
fi
done
{% if 'compute' in node_name %}
## wait for node to get back online
timeout_seconds={{ node_reboot_timeout }}
elapsed_seconds=0
while true; do
echo "Waiting for {{ node_name | splitext | first | splitext | first }} to boot ..."
NOVACOMPUTE_STATUS=$(ssh -q -o StrictHostKeyChecking=no heat-admin@$NODE_IP 'sudo docker inspect --format="{{ "{{" }} .State.Running {{ "}}" }}" nova_compute')
if [[ $NOVACOMPUTE_STATUS == 'true' ]]; then
break
fi
sleep 3
(( elapsed_seconds += 3 ))
if [ $elapsed_seconds -ge $timeout_seconds ]; then
echo "FAILURE: Nova compute container didn't start on {{ node_name | splitext | first | splitext | first }}"
exit 1
fi
done
## wait for nove compute service to be reported as up
source {{ overcloud_rc }}
timeout_seconds={{ node_reboot_timeout }}
elapsed_seconds=0
while true; do
echo "Waiting for nova-compute service on {{ node_name | splitext | first | splitext | first }} to go up ..."
NOVACOMPUTE_ENABLED=$(openstack compute service list --host {{ node_name }} -f json | jq -r -c '.[] | select(.Binary | contains("nova-compute")) | .State' | head -1)
if [[ $NOVACOMPUTE_ENABLED == 'up' ]]; then
break
fi
sleep 3
(( elapsed_seconds += 3 ))
if [ $elapsed_seconds -ge $timeout_seconds ]; then
echo "FAILURE: Nova compute service didn't come up on {{ node_name | splitext | first | splitext | first }}"
exit 1
fi
done
{% endif %}
fi
{% if compute_evacuate %}
{% if 'compute' in node_name %}
source {{ overcloud_rc }}
## Detect if ceph shared storage was used
STORAGE_BACKEND=$(openstack volume service list -f json | jq -r -c '.[] | select(.Binary | contains("cinder-volume")) | .Host' | sed s/hostgroup@tripleo_//)
{% for instance in node_instances.stdout_lines %}
nova live-migration {{ instance }} {{ hostvars[node_name].ansible_fqdn }}
if [ $STORAGE_BACKEND == 'ceph' ]; then
nova live-migration {{ instance }} {{ node_name }}
else
nova live-migration --block-migrate {{ instance }} {{ node_name }}
fi
timeout_seconds=120
elapsed_seconds=0
while true; do
if [ $STORAGE_BACKEND == 'ceph' ]; then
echo "Shared storage live migrating {{ instance }} back to {{ node_name }} ..."
## Live migration might not complete on the first run so we run it multiple times
## until the instance ends back on the originating host
nova live-migration {{ instance }} {{ node_name }}
else
echo "Block migrating {{ instance }} back to {{ node_name }} ..."
nova live-migration --block-migrate {{ instance }} {{ node_name }}
fi
INSTANCE_HOST=$(openstack server show {{ instance }} -f json | jq -r -c '. | .["OS-EXT-SRV-ATTR:host"]')
if [ $INSTANCE_HOST == '{{ node_name }}' ]; then
break
fi
sleep 3
(( elapsed_seconds += 3 ))
if [ $elapsed_seconds -ge $timeout_seconds ]; then
echo "FAILURE: Could not live migrate instance back to {{ node_name }}"
exit 1
fi
done
{% endfor %}
{% endif %}
{% endif %}

26
templates/node_upgrade_pre.sh.j2
View File

@ -1,2 +1,26 @@
{% if compute_evacuate %}
source {{ overcloud_rc }}
nova host-evacuate-live {{ hostvars[node_name].ansible_fqdn }}
## Detect if ceph shared storage was used
STORAGE_BACKEND=$(openstack volume service list -f json | jq -r -c '.[] | select(.Binary | contains("cinder-volume")) | .Host' | sed s/hostgroup@tripleo_//)
if [ $STORAGE_BACKEND == 'ceph' ]; then
nova host-evacuate-live {{ node_name }}
else
nova host-evacuate-live --block-migrate {{ node_name }}
fi
timeout_seconds=120
elapsed_seconds=0
while true; do
echo "Waiting for {{ node_name }} to get quiesced ..."
INSTANCE_COUNT=$(openstack server list --host {{ node_name }} -f json | jq -r -c '[.[] | select(.Status | contains("ACTIVE") or contains("PAUSED") or contains("MIGRATING"))] | length')
if [ $INSTANCE_COUNT == 0 ]; then
break
fi
sleep 3
(( elapsed_seconds += 3 ))
if [ $elapsed_seconds -ge $timeout_seconds ]; then
echo "FAILURE: Could not quiesce compute node {{ node_name }}"
exit 1
fi
done
{% endif %}

22
templates/oooq_deploy_transformation.sh.j2 Normal file
View File

@ -0,0 +1,22 @@
#!/usr/bin/bash
awk '/status_code/{inside=0}
/^openstack overcloud deploy/{inside=1}
{if (inside==1) {print}}' {{ working_dir }}/overcloud-deploy.sh > /tmp/overcloud-deploy-tr.1
sed -Ee 's/([a-zA-Z1-9]) +([-][^ ])/\1 \\\n\2/g' /tmp/overcloud-deploy-tr.1 > /tmp/overcloud-deploy-tr.2
# Check if tripleo-heat-templates directory used for deployment
# in deploy-overcloud.sh matches with one specified under {{ tht_directory }}
if [[ $(grep "\-\-templates" /tmp/overcloud-deploy-tr.2) ]]; then
templates_dir=$(grep "\-\-templates" /tmp/overcloud-deploy-tr.2 | awk '{ print $2}')
if [[ "$templates_dir" != "{{ tht_directory }}" ]]; then
# If different templates directory, substitute the old by
# the new tht directory in the same temporary file
sed -iEe "s|$templates_dir|{{ tht_directory }}|g" /tmp/overcloud-deploy-tr.2
fi
fi
sed -Ee '$s/\\ *//' -e '/^ *$/d' /tmp/overcloud-deploy-tr.2 > {{ overcloud_deploy_script }}

12
templates/overcloud_update.sh.j2 Normal file
View File

@ -0,0 +1,12 @@
#!/bin/env bash
#
# Run minor update on overcloud nodes
#
set -euo pipefail
{% if item|string == 'all' %}
echo "Running minor update of all overcloud nodes"
openstack overcloud update stack 2>&1
{% else %}
echo "Runing update of {{ item }}"
openstack overcloud update stack --nodes {{ item }} 2>&1
{% endif %}

9
templates/overcloud_update_setup.sh.j2 Normal file
View File

@ -0,0 +1,9 @@
#!/bin/env bash
#
# Setup HEAT's output
#
set -euo pipefail
echo "Setting up HEAT's output for minor update"
openstack overcloud update stack --init-minor-update \
--container-registry-file {{ working_dir}}/{{ container_registry_file }} 2>&1

5
templates/undercloud_update.sh.j2 Normal file
View File

@ -0,0 +1,5 @@
#!/bin/bash
sudo yum update -y python-tripleoclient
openstack undercloud upgrade 2>&1

6
templates/undercloud_upgrade.sh.j2
View File

@ -3,6 +3,8 @@
# Perform undercloud upgrade and related steps
set -euo pipefail
sudo systemctl stop 'openstack-*' 'neutron-*' httpd
sudo yum update -y instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient
# ceph-ansible needs to be installed manually
sudo yum install -y ceph-ansible
sudo yum update -y python-tripleoclient
openstack undercloud upgrade &> undercloud_upgrade.log

23
templates/update_workarounds.sh.j2 Normal file
View File

@ -0,0 +1,23 @@
#!/bin/env bash
#
# Apply workarounds for minor updates
#
set -euo pipefail
{% if item.workarounds|default([]) %}
{% for workaround in item.workarounds %}
{% if workaround.patch_id %}
curl -4 'https://review.openstack.org/changes/{{workaround.patch_id}}/revisions/current/patch?download' | \
base64 -d | \
sudo patch -d {{ workaround.directory }} -p1 {{ workaround.misc }}
{% endif %} {# end of item.patch_id #}
{% if workaround.command %}
{{ workaround.command }}
{% endif %} {# apply command for workaround #}
{% endfor %} {# end of workarounds iteration #}
{% else %} {# if workarounds not needed #}
echo "No Workarounds Needed"
exit 0
{% endif %} {# end if no workarounds #}

15
templates/upgrade_overcloud_workarounds.sh.j2
View File

@ -1,15 +0,0 @@
#!/bin/bash
#
# Apply upgrade workarounds for overcloud nodes
set -euo pipefail
{% for bugs in overcloud_workarounds %}
{% for key, value in bugs.items() %}
# {{ key }}
{% if value.patch %}
curl -4 https://review.openstack.org/changes/{{ value.id }}/revisions/current/patch?download | base64 -d | sudo patch -d /usr/share/openstack-tripleo-heat-templates/ -p1
{% else %}
{{ value.command }}
{% endif %}
{% endfor %}
{% endfor %}

15
templates/upgrade_undercloud_workarounds.sh.j2
View File

@ -1,15 +0,0 @@
#!/bin/bash
#
# Apply upgrade workarounds on undercloud node
set -euo pipefail
{% for bugs in undercloud_workarounds %}
{% for key, value in bugs.items() %}
# {{ key }}
{% if value.patch %}
curl -4 https://review.openstack.org/changes/{{ value.id }}/revisions/current/patch?download | base64 -d | sudo patch -d /usr/share/openstack-tripleo-heat-templates/ -p1
{% else %}
{{ value.command }}
{% endif %}
{% endfor %}
{% endfor %}

101
templates/workarounds.sh.j2 Normal file
View File

@ -0,0 +1,101 @@
#!/bin/bash
#
# Apply upgrade workarounds
set -euo pipefail
{% if 'pre_undercloud_upgrade_workarounds' in item %}
{% for bugs in pre_undercloud_upgrade_workarounds|default([]) %}
{% for key, value in bugs.items() %}
echo {{ key }}
{% if value.patch %}
curl -4 https://review.openstack.org/changes/{{ value.id }}/revisions/current/patch?download | \
base64 -d | \
sudo patch -d {{ value.basedir }} -p1
{% else %}
{{ value.command }}
{% endif %}
{% endfor %}
{% endfor %}
{% elif 'post_undercloud_upgrade_workarounds' in item %}
{% for bugs in post_undercloud_upgrade_workarounds|default([]) %}
{% for key, value in bugs.items() %}
echo {{ key }}
{% if value.patch %}
curl -4 https://review.openstack.org/changes/{{ value.id }}/revisions/current/patch?download | \
base64 -d | \
sudo patch -d {{ value.basedir }} -p1
{% else %}
{{ value.command }}
{% endif %}
{% endfor %}
{% endfor %}
{% elif 'pre_docker_composable_upgrade_workarounds' in item %}
{% for bugs in pre_docker_composable_upgrade_workarounds|default([]) %}
{% for key, value in bugs.items() %}
echo {{ key }}
{% if value.patch %}
curl -4 https://review.openstack.org/changes/{{ value.id }}/revisions/current/patch?download | \
base64 -d | \
sudo patch -d {{ value.basedir }} -p1
{% else %}
{{ value.command }}
{% endif %}
{% endfor %}
{% endfor %}
{% elif 'pre_nova_compute_upgrade_workarounds' in item %}
{% for bugs in pre_nova_compute_upgrade_workarounds|default([]) %}
{% for key, value in bugs.items() %}
echo {{ key }}
{% if value.patch %}
curl -4 https://review.openstack.org/changes/{{ value.id }}/revisions/current/patch?download | \
base64 -d | \
sudo patch -d {{ value.basedir }} -p1
{% else %}
{{ value.command }}
{% endif %}
{% endfor %}
{% endfor %}
{% elif 'pre_docker_converge_upgrade_workarounds' in item %}
{% for bugs in pre_docker_converge_upgrade_workarounds|default([]) %}
{% for key, value in bugs.items() %}
echo {{ key }}
{% if value.patch %}
curl -4 https://review.openstack.org/changes/{{ value.id }}/revisions/current/patch?download | \
base64 -d | \
sudo patch -d {{ value.basedir }} -p1
{% else %}
{{ value.command }}
{% endif %}
{% endfor %}
{% endfor %}
{% elif 'post_docker_converge_upgrade_workarounds' in item %}
{% for bugs in post_docker_converge_upgrade_workarounds|default([]) %}
{% for key, value in bugs.items() %}
echo {{ key }}
{% if value.patch %}
curl -4 https://review.openstack.org/changes/{{ value.id }}/revisions/current/patch?download | \
base64 -d | \
sudo patch -d {{ value.basedir }} -p1
{% else %}
{{ value.command }}
{% endif %}
{% endfor %}
{% endfor %}
{% endif %}

102
templates/workload_launch.sh.j2 Normal file
View File

@ -0,0 +1,102 @@
#!/bin/bash
#
# Script that spawns an instance
OVERCLOUD_RC=~/overcloudrc
IMAGE_URL='{{ workload_image_url }}'
IMAGE_NAME='upgrade_workload'
IMAGE_FILE=~/upgrade_workload.qcow2
KEYPAIR_NAME=userkey
FLAVOR_NAME='v1-1G-5G'
SECGROUP_NAME='allow-icmp-ssh'
TENANT_NET_NAME='internal_net'
EXTERNAL_NET_NAME='{{ external_network_name }}'
source ${OVERCLOUD_RC}
## create image
openstack image list | grep ${IMAGE_NAME}
if [ $? -ne 0 ]; then
curl --silent -L -4 -o ${IMAGE_FILE} ${IMAGE_URL}
openstack image create \
--file ${IMAGE_FILE} \
--disk-format qcow2 \
--container-format bare \
${IMAGE_NAME}
fi
## create user key
openstack keypair list | grep ${KEYPAIR_NAME}
if [ $? -ne 0 ]; then
openstack keypair create --public-key ~/.ssh/id_rsa.pub ${KEYPAIR_NAME}
fi
## create flavor
openstack flavor list | grep ${FLAVOR_NAME}
if [ $? -ne 0 ]; then
openstack flavor create --vcpus 1 --ram {{ workload_memory }} --disk 5 --swap 512 $FLAVOR_NAME
fi
## create networking
openstack network list | grep ${TENANT_NET_NAME}
if [ $? -ne 0 ]; then
NAMESERVER=$(grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' /etc/resolv.conf | head -1)
openstack router create ${TENANT_NET_NAME}_router
openstack network create ${TENANT_NET_NAME}
openstack subnet create \
--subnet-range 192.168.0.0/24 \
--allocation-pool start=192.168.0.10,end=192.168.0.100 \
--gateway 192.168.0.254 \
--dns-nameserver ${NAMESERVER} \
--network ${TENANT_NET_NAME} \
${TENANT_NET_NAME}_subnet
openstack router add subnet ${TENANT_NET_NAME}_router ${TENANT_NET_NAME}_subnet
openstack router set --external-gateway ${EXTERNAL_NET_NAME} ${TENANT_NET_NAME}_router
fi
## create security group
openstack security group list | grep ${SECGROUP_NAME}
if [ $? -ne 0 ]; then
openstack security group create ${SECGROUP_NAME}
openstack security group rule create --proto icmp ${SECGROUP_NAME}
openstack security group rule create --proto tcp --dst-port 22 ${SECGROUP_NAME}
fi
## create instance
INSTANCE_NAME="instance_$(openssl rand -hex 5)"
TENANT_NET_ID=$(openstack network list -f json | jq -r -c ".[] | select(.Name | contains(\"$TENANT_NET_NAME\")) | .ID")
openstack server create \
--image ${IMAGE_NAME} \
--flavor ${FLAVOR_NAME} \
--security-group ${SECGROUP_NAME} \
--key-name ${KEYPAIR_NAME} \
--nic net-id=${TENANT_NET_ID} \
$INSTANCE_NAME
timeout_seconds=120
elapsed_seconds=0
while true; do
INSTANCE_ACTIVE=$(openstack server show $INSTANCE_NAME -f json | jq -r .status)
if [ $INSTANCE_ACTIVE == 'ACTIVE' ]; then
break
fi
sleep 3
elapsed_seconds=$(expr $elapsed_seconds + 3)
if [ $elapsed_seconds -ge $timeout_seconds ]; then
echo "FAILURE: Instance failed to boot."
exit 1
fi
done
## assign floating ip
INSTANCE_FIP=$(openstack floating ip create ${EXTERNAL_NET_NAME} -f json | jq -r .id)
INSTANCE_IP=$(openstack server show $INSTANCE_NAME -f json | jq -r .addresses | grep -oP '[0-9.]+')
INSTANCE_PORT=$(openstack port list -f json | jq -r -c ".[] | select(.[\"Fixed IP Addresses\"] | contains(\"${INSTANCE_IP}\")) | .ID")
neutron floatingip-associate ${INSTANCE_FIP} ${INSTANCE_PORT}
## create and attach a volume
CINDER_VOL_ID=$(openstack volume create --size 1 vol_$(openssl rand -hex 5) -f json | jq -r .id)
openstack server add volume ${INSTANCE_NAME} ${CINDER_VOL_ID}
echo "floating-ip: $(openstack floating ip show ${INSTANCE_FIP} -f json | jq -r .floating_ip_address)" > ~/${INSTANCE_NAME}

8
tests/oooq-test.yaml Normal file
View File

@ -0,0 +1,8 @@
---
- hosts: undercloud
gather_facts: true
become: yes
become_method: sudo
become_user: stack
roles:
- { role: tripleo-upgrade, use_oooq: 'true' }

5
tests/test.yml
View File

@ -1,9 +1,6 @@
---
- hosts: all
gather_facts: true
- hosts: undercloud
gather_facts: false
gather_facts: true
become: yes
become_method: sudo
become_user: stack