tripleo-upgrade/templates/node_upgrade_pre.sh.j2
Sofer Athlan-Guyot 4572055ffb Merge tripleo-upgrade repo from redhat-openstack namespace
Make sure scripts are created with the executable bit set.

Change-Id: I731902411e987b4ea7c2aa84fef869fe5e1c25ae

Add a oooq comptatibility layer, documentation and example.

Change-Id: I30fe6359f1c0098ff9bcdd5939724491d94ef199

Add support for applying w/a before and after upgrade

This change adds the ability to apply workarounds before and after
the overcloud upgrade process has finished. This allows the user
to workaround particular issues that show up after the upgrade
process has finished.

Change-Id: I21a7e885bcc466af6bf80410ba2cc8d03865cb33

Fix missing quotes

This changes adds missing quotes to the node_upgrade_script.yml
templates. Currently the task is failing because of the missing quotes.

Change-Id: Ied9217df374d09bf90e6878a7c79e22042f44d99

Add support for applying workarounds post undercloud upgrade

This change adds support for applying workarounds after the undercloud
upgrade process has finished as part of the undercloud upgrade.

Change-Id: If85900969c0d591cf6024408d958a82fa8c8a534

Adjust overcloud_converge_upgrade_script script

This change adjusts the overcloud_converge_upgrade_script to allow
running the upgrade converge stage. In addition it adjust the ssh
config file to skip host key check so the non-controller script does
not get stuck waiting for user input.

Change-Id: Ic38f325c61e90165a5322ef754f7e5514ed8e687

Append working_dir to logs generated

Change-Id: I6bc9f0c58ad8684ed03dee042e9cfb2bdc6835f6

Install ceph-ansible during undercloud upgrade

ceph-ansible is required to be installed manually for deployments
with ceph nodes. This change installs the ceph-ansible package
before the undercloud upgrade.

Change-Id: If8918a38250a10681d965d0715ebc17078166336

Use openstack overcloud container image prepare command

This change adds the use of openstack overcloud container image prepare
command for generating the environment file containing the container
image names and local registry address.

Change-Id: I174f7e3aae415d51224cf73da83a859e90eed095

Do not rely on ansible inventory for upgrading non controller nodes

Currently we are relying on the ansible inventory to provide groups
containing compute nodes and their facts when creating the upgrade
scripts. In order to remove this requirement and provide easier
integration this change discovers the compute and swift storage nodes
from the undercloud. In addition it adds a wait loop for instance live
migration to complete before and after upgrading compute nodes and adds
support for swift storage nodes upgrade.

Change-Id: Ia4b2e81845c3fec9036c4695f0dd1746d4c5c6b8

Add silent and nobuffer options to curl command

Change-Id: I0f4bd71b67d4827717d9f7d3fc075fc396eb6363

Adjust tht environment files and custom roles data during upgrade

This change switches the environment files used in the overcloud deploy
command to their variants used for deploying containers. In addition, if
using a custom roles data file for composable roles deployments, this
change adjusts the local roles data copy with the changes introduced in
Pike.

Change-Id: Icd3c3b67342c0bd10fc3d28ed89a94fc9f714db4

Fix overcloud_deploy_script var location in oooq_test playbook

Change-Id: I01f18a5413ff223cbca900521037e739ebc43d5d

Disconnect ssh session before uploading images to local registry

This change disconnects the running ssh session in order to allow
the stack user to connect to the docker daemon. This acts as a
workaround for https://bugs.launchpad.net/tripleo/+bug/1699477

Change-Id: Ia06fe8581ba17525cbcb2d955d7947b7c546811d

Add undercloud_reboot var and reboot only on ovs or kernel updates

This change adds the undercloud_reboot var to manipulate if undercloud
reboots should be made or not and does the reboot only when kernel or
ovs updates occur.

Change-Id: Ic89291c5791bbd098204b2c85793335e0faa8d94

Add InfraRed docs

This change adds the steps required to run the tripleo-upgrade role
as an InfraRed plugin.

Change-Id: I08fde6c8954ec4eeedc47971607bec592fc801a1

Add docs for running the role manually from the undercloud

Change-Id: I6d5bb6479b6e1fb45e34aa776d3c315cfae98ae4

Do not stop services before undercloud upgrade

According to the docs stopping services should be done by the undercloud
upgrade process so we do not need to manually do it.

Change-Id: I7f14257b4e90d3a0610c0f90e856096643325861

Specifically become_user: root when running ovs command

Currently become_user is set to the undercloud user, usually
stack, when invoking the undercloud_validate_upgrade.yaml playbook.
This change overrides become_user and sets it to root when running
ovs-vsctl command as it requires privilege escalation.

Change-Id: Ia93245f4a2d09d73849c03401d38ffb25e7be802

Use ip address instead of name when rebooting

OOOQ doesn't set a name for the undercloud in hosts so in case the
reboot is triggered the virthost cannot reach the undercloud. This
change switches the wait condition and calls the ip address instead
of the name.

Change-Id: I2121e2dabe9794d31fe70bbfa0ff8e53da6b3b1b

Add reload ssh tag to the Kill SSH task

Add a tag to the Kill SSH task so it can be avoided when
running the role manually from the undercloud.

Change-Id: Iec088cc174d7e8270fbea0698ee76227b45842f7

Add option to create script with --setup-heat-output option

According to BZ#1477962 in order to be able to run non controller
upgrade scripts after major upgrade composable steps we need to run
the deploy command with the --setup-heat-outputs option.

Change-Id: I8137ff18047a130c5ea8dca2ce11378eabc30329

Add deprecated params to custom roles data file

In Pike there were additional flags assigned to the default roles
data file. This change adds these flags during the upgrade process
to the roles defined in the custom roles data file.

Change-Id: I58c2f30ff74d2302027d7488dc03c5146c371649

Use a default value for the HOME env var

Change-Id: I40e2f46f1311dc4b797977c8136ec4037505ef05

Update python-openstackclient before undercloud upgrade

Updating python-openstackclient before undercloud upgrade is a
requirement to get undercloud upgrade passing.

Change-Id: I4ab67f9af68036a29e11bb2457d70535bd94b7b0

Create environment file for injecting undercloud certificate

When undercloud is SSL enabled the overcloud nodes need to be aware
of the undercloud CA cert. This change creates this file during upgrade
and adds it to the overcloud deploy commands when the overcloud nodes
are not able to reach the undercloud ssl enabled public endpoints.

Change-Id: I79a03299bc28d0ca2dbd83c28087a4c56f6b2271

Convert puppet ceph parameters to ceph ansible during upgrade

This is a workaround for BZ#1488855.

Change-Id: I58ac44b2166abddf56a327a4ee09457139c831da

Remove setup heat outputs workaround

https://review.openstack.org/#/c/502470/ allows us to obtain the
RoleConfig output so there's no need to run the setup heat outputs
step anylonger. This change remove this step.

Change-Id: I7235b8625eea4f77a055d0ab1862d0318f3a776f

Echo bug number in workarounds script

This change replaces the bug number comments with an echo statement.
This provides easier way to debug what workarounds failed to apply
when the workarounds script is run.

Change-Id: I02c5534427ee5cf7b7af618f36577c1008d50992

Fix uc_keystone_conn condition

This change adds an addition condition to inject the undercloud
certificate in the CAMap only when  undercloud ssl is enabled.

Change-Id: Iae023922bf1ed0bb22e86710b122c65a8f1568a3

Gather facts only from undercloud node

Facts are only required for the undercloud node. This change adjusts
the existing playbooks to gather facts only from undercloud node in
order to save some time and not rely on nodes which are not required
to be reachable.

Change-Id: Idceec3d10d84ef112da558109d9904a1d8c6ed93

Do not include docker.yaml and docker-ha.yaml environments

As described in BZ#1466744 the docker.yaml and docker-ha.yaml
environments are currently included by default so we do not need
to specifically include them.

Change-Id: I44a72ddd65cf816003ceca21ef33470a3ab125a7

Reboot compute nodes post upgrade

As a post upgrade requirement we need to reboot the nodes in case
of an OVS upgrade. This change runs a post upgrade check for non
controller nodes and reboots the nodes if an ovs upgrade has been
detected. It also adds additional validation for compute nodes to
make sure that the nova-compute service is enabled after reboot.

Change-Id: I583e589118aabae84f8e1dc9ec2c4b43ca17a250

Add L3 agent connectivity check during upgrade

This change adds a check which validates that ICMP
connectivity with a floating IP is not interrupted
during the major upgrade composable step.

Change-Id: Iee55af85b9a2c3ece86731e043130d191ff6a821

Run pre docker composable upgrade workarounds at correct position

This change moves the pre docker composable upgrade workarounds to
be run right before the docker composable upgrade step.

Change-Id: I604ea2eb6202d48b0f771ea80e5e731df687600e

Use bool with ansible booleans

use bool filter when using ansible booleans.

Change-Id: Ibeb59772e935cc28a661ccddcaa4773388ce296d

Add option for creating workloads before upgrade

This change adds the option to launch an instance before
starting upgrade. This operation is useful when doing
tests such as instance live migration during upgrade or
floating ip connectivity testing during upgrade. The
script requires a network defined in the external_network_name
var which provides external connectivity to exist beforehand.

Change-Id: Ib39e41b36fac7794ea515c8a9d56141866dcfeed

Fix pre compute upgrade check

This change adds the MIGRATING state to be checked before the
compute nodes upgrade.

Change-Id: I0073a7e69a71a044882d4760dbb49cd4f455dd89

Fix workload_launch position

Change-Id: I6193ac6a60165bb20ece6277067c05696ed6d3b1

Run non controller node pre upgrade script

This change runs the non controller node pre upgrade script.
In addition it exposes the option to run instances migration
between compute nodes during upgrade.

Change-Id: Ief55eecdc85bb620f637c4ed4d9b5bc3243b37d1

Update roles_data adjustments to latest changes

This change updates the roles_data file adjustments to the latest
changes.

Change-Id: Ic787b135cdf96b33829e05140e069b398df7196f

Use docker and docker-ha environment files for upstream deployments

Change-Id: I70bb9767e97f616729adff983fff065858a6dcdc

Convert services environments to services-docker only for upstream

Per BZ#116463 in downstream the environments used for extra service
enablement now point to docker resources.

Change-Id: I379622ec2749ac8b485aec79a7500308ef74214e

Echo debug message to differentiate live migration from block

Change-Id: I69e7f381543d84fcd308acbd3a90f5d0ac23ae1b

Accept <= 5% ICMP packet loss during upgrade connectivity check

Change-Id: I34d1de225c0e391035e22e18f63356e04afbbfd5

Reorganize playbooks to separate upgrade/update

This change adds separate directories for upgrade/update which
provides a better separation between updates and upgrades.

Change-Id: Icf1a09514fb0e6236535ae32265bbd3805918478

Run block migrate multiple times

Block migrate doesn't work seem to work if triggered once but it
does if the command is run for a second time. This change runs the
block migrate command multiple times to make sure that the instance
gets migrated.

Change-Id: I8b9a9ecae21f7ce49a03945afec66b9e671622b7

Ensure files/ are part of the setup.cfg files to copy

When installing tripleo-upgrade into a .quickstart
environment, the files/ folder wasn't getting copied, which is necessary
at least for "adjust ssh config to skip host key check" in
create-upgrade-scripts.yaml.

Change-Id: I7d862ec5c13ba719923c90cc40790b842b582999

Add tasks for undercloud minor update

Start adding the minor undercloud update tasks

Change-Id: I33705b270e2d5e6a28f1cad8179e1f4b3e4ea975

Remove timeouts from upgrade scripts

Depending on the number of deployed nodes upgrade could take longer
and we want to rely on the heat stack timeout. This change clears
any manual timeouts set in the upgrade scripts.

Change-Id: I5d141e2cc13621d3be5fb0c27b0ac3c3fc30d424

Minor updates of RHOS 12.

Manage minor update workflow from within tripleo-upgrade repo.

Change-Id: I8c6771af4825ce166e8470413ca4687be0a58cb9

Reboot controller nodes post upgrade

This change adds the option to reboot controller nodes post upgrade
and performs basic verifications that the clustered services are
reported as up.

Change-Id: I370d421e5968ae50bd1ff140cdfcf98a4db03a5f

Don't force ssh_config on everybody.

This add an option to be able to not overwrite the ssh_config file.
As a side note the ssh_config is missing from the repo, so by default
this task is broken.

Change-Id: Idfb78e2b7226a7e6295acd3f250bbfb48d0a103d

Fix filter used in the node upgrade scripts

This change is fixes the current filter used for node upgrade
scripts so that deployment with $domain.tld format are supported.

Change-Id: I18f43c440bb93e0fcefb664c7d716ff9368673a4

Run live migration multiple times

Change-Id: I7c028defd3cb9080efa7bdbe9daa6ed201df8640

Manually inject undercloud SSL cert to overcloud nodes

Per BZ#1501779 the compute nodes do not get their trusted store
updated when using a CAMap and upgrade fails. This change updates
the overcloud nodes trusted store manually so the overcloud nodes
are prepared for update. This should translate in a documentation
step that before upgrade starts the user needs to make  sure the
overcloud nodes are able to reach the undercloud SSL public
endpoint.

Change-Id: Ib95a29c608803504a866ae71cbc0082faf3c194f

Replace puppet external ceph environment with ceph-ansible one

This change replaces during the upgrade the external ceph puppet
environment file with its ceph-ansible equivalent.

Change-Id: I9020e8f7c43f91259b551caa2e20f03be1424106

Append deprecated params only to predefined roles

We should append the deprecated params only to predefined roles
in order to avoid failures such as reported in BZ#1501237.

Change-Id: I3a7c332b35da9639fb6f8e5b38234dc0c55d8499

Split the post controller scripts into per services scripts

This change splits the post upgrade controller scripts into
per service checks and adds them to a common directory so
they can be shared between update and upgrade.

Change-Id: I8f2fb6162a5acb8a92057400a7b04e6e2388abaa

Add the ability to specify a remote docker registry

This change adds the ability to specify a remote docker registry
to be used for downloading the Docker images on the undercloud or
be used directly by the overcloud nodes during upgrade.

Change-Id: I132a8b94f9a101d1c9c624d202bb01527dc2b844

Fix BZ#1499677 workaround condition

In addition to empty gvwstate.dat file there might be situations
where the gvwstate.dat file is missing after reboot. This patch
addresses this condition for BZ#1499677 workaround.

Change-Id: I295b133248f48ab41b1748225cbe9359662b280d

Cleanup galera resource instead of rebooting node for BZ#1499677

Instead of rebooting the node while implementing the workaround for
BZ#1499677 we should simply cleanup the Galera resource. This way
we can save some time and potential issues caused by an additional
reboot.

Change-Id: I391daeae41321baec1cbd8c458132a3161cd96d5

[UPDATES] Introduce option for minor updates workarounds.

To speed up testing of minor updates it might be required
 to apply some patches before they are landed.
Hence we need a flag to differentiate if workarounds are required
 or not

Change-Id: I642e4ade204f5fd30ec9433f1d90a2d539287c5e

Do not pipe curl output in container images environment script

Curl can sometimes exit with exit code 23 when its output is piped
into another command. To avoid this errors we save the curl output
to a file.

Change-Id: I4123b6c66ae2873c11631f229cb8e3eec5a5a66b

Use service environment files when generating the images environment

In the last build openstack overcloud container image prepare only
generates the parameters for the services included by default. In
order to make it work when extra non-default services are enabled
we need to pass the environment files to the prepare command. This
change addresses this issue.

Change-Id: I86ab6faaffcd4c7cc1a07e9a6ed1e890cb5cf980

Place the oooq deploy command into overcloud_deploy_script var

This change places the openstack overcloud deploy command with its
arguments in the location defined by the overcloud_deploy_script
var. This way we don't require oooq users to specifically set the
overcloud_deploy_script to a hardcoded location and make it less
confusing.

Change-Id: Id2b14fcffbd169c342df4b5b9105dff81e18e3a0

Replace ceph radosgw environment during upgrade

Change-Id: I489211f39941bba5b1ca2ddf1b635c3bdb0151fe

Avoid losing undercloud connection in TripleO CI.

When running the role in the TripleO upstream CI
the connectivity to the undercloud gets lost when
rebooting the undercloud after upgrade or killing
the ssh service, this makes the playbook fail.

As a solution, a flag tripleo_ci has been added.
This flag will default to false, and when set to
true no undercloud reboot, nor ssh killing will
be executed.

Change-Id: If4a303fff49bbe55cdfb7142d8dd69264ab47ab4

Align deployment-files option with IR.

deployment-files option is not a list of choices in IR,
adjust it accordingly.

Change-Id: I6889e9b75f842cc466278fed5dbf85a80cb58ee0

Append docker-ha only when needed.

Before appending the docker-ha.yaml env
file, we need to check if the overcloud
was deployed with pacemaker. If so, then
we'll add the env file to the upgrade
script.

Change-Id: I9867d86b6d23385c576d2f8c5a25ab3333f7113d

Specify tht directory used in upgrade script.

When deploying with oooq, the generated script
overcloud-deploy.sh is reused in order to append
the cooresponding env files for the upgrade.

However, if the location of the tripleo-heat-templates
directory is different from the used when deploying
then the upgrade does not succeed.

This change modifies the tripleo-heat-templates
location used to upgrade when the directory found
in overcloud-deploy.sh is different. If it is the
same no change is done.

Change-Id: I55ada3e75b7463b1c14c8734410d2591cf162e67

Don't append DockerInsecureRegistryAddress

This is no longer required as the prepare command detects whether the
registry is secure and DockerInsecureRegistryAddress as necessary.

Depends on upstream https://review.openstack.org/#/c/514473/
Related-Bug: #1722632

Change-Id: Ia9d91f6280600c59d0079c5d1f26a00f04040426

Fix the controller regex during roles_data conversion

The Ocata roles_data controller role might include a comment for
the controller role name. This change adjusts the regexp used during
the roles data conversion to take into consideration that comment.

Change-Id: I7b43b1fcb9e477de8e1265ef6aa6ba5149e82d47

Use prepare --set for ceph image parameter

This is more maintainable, and consistent with other uses of prepare.

Change-Id: Ieec88e271973a248192c2b247cd2c5e0cccbfb7c

Add storage environment files to be used for containers prepare

This change adds the storage environments files to be matched when
creating the environment file containing the Docker images names
which gets created via container image prepare.

Change-Id: I34c3cdaab1b63ce3f43d748372d35143bc12b8b4

Add environment file containing required DPDK changes during upgrade

Change-Id: Iabaf16e18ee7546bd6275f8f84226892423a6c95

Create failed_upgrade log files.

Most of the stack failures in the TripleO
CI are registered inside two log files
failed_upgrade_list.log and failed_upgrade.log.

This patch adds the option of inject the
stack failures list command into these two
log files, as well as priting out the detailed
stack failure list (--long).

Change-Id: I4fad989818f67ad0a73e45b47f835750f18c3bb6

Replace storage-environemnt.yaml for upstream only

Per BZ#1502862 the Ceph environment files switch during upgrade is
only needed for upstream deployments. This change does the changes
to accommodate this.

Change-Id: Ia3adb120c9b524c66d069593b0779b3399295fd4

Do not update python-openstackclient before upgrade

BZ#1488471 was fixed so there's no need to update the python-
openstackclient package before upgrade.

Change-Id: I2aba7515ec43926ec4f8de5c701467dea31dba1b

Be more aggressive on accepted packet loss during upgrade

Tests have shown that the packet loss shouldn't eceed 1% during
the upgrade steps. This change adjusts the accepted level to this.

Change-Id: I9e47ea56a78e4e9eab40fca609cbedaecfcf1e14

Add more tags for the upgrade process.

This enable one to either do only a small part of the whole process.
This can be useful for debugging or development.

Change-Id: Ic6cc9a1e6aa2793fde65636d2ad92bc174173252

Use new method of discovering tag and adjust local registry upload

This change uses the new mechanism of retrieving the tag from the
latest image provided in the registry.

Change-Id: I7e063f13c7d4812e9986452774881235b620bd0e

Swap baremetal environment file for containerized.

In oooq, when upgrading from baremetal
to containerized overcloud two different
environment files are used. These env
files are located in [0].

When upgrading using tripleo-upgrade, we
need to convert that environment file name
to its corresponding containers version.

[0] https://github.com/openstack/tripleo-heat-templates/tree/master/ci/environments

Change-Id: I6c9fad2f402a162cf663c5089e79c2e10f3d0928

Add condition to create local docker registry.

The only way to not execute the docker
registry environment file creation task
is via tags, which is not easy to handle
in TripleO CI.

As tripleo-quickstart already prepares
the local docker registry file, there
is no need to execute it. So a new
parameter 'create_docker_registry' is
been added.

Also, the 'force' option is being added
to avoid overwritting a provided script
with the role template.

Change-Id: I800d6696b8dbb83f05f3d9381c6e5689558f4b77

Prepare workloads before update/upgrade.

Prepare scripts for managing workload on oc before running
 update/upgrade operation.

Allow to run ping test during minor update.

Change-Id: I1d5754f36f53588c97c646aa4e1380e9ca5938bc

Remove tag parsing from the image prepare command

Parsing the tag is not needed anymore and the one returned by the
container image prepare command can be used.

Change-Id: If782fc655da7b22e4d4a803509e9cc8c49774368

[UPDATES] Run minor update per role.

With recent changes it's advised to perform minor update in batches:
role-by-role.
This change limits the scope of update with '--nodes <Role>' option.

Change-Id: I0bc03873b749dc9c15b13cacbfff78cead4360d8
2017-12-08 19:01:00 -08:00

27 lines
968 B
Django/Jinja

{% if compute_evacuate %}
source {{ overcloud_rc }}
## Detect if ceph shared storage was used
STORAGE_BACKEND=$(openstack volume service list -f json | jq -r -c '.[] | select(.Binary | contains("cinder-volume")) | .Host' | sed s/hostgroup@tripleo_//)
if [ $STORAGE_BACKEND == 'ceph' ]; then
nova host-evacuate-live {{ node_name }}
else
nova host-evacuate-live --block-migrate {{ node_name }}
fi
timeout_seconds=120
elapsed_seconds=0
while true; do
echo "Waiting for {{ node_name }} to get quiesced ..."
INSTANCE_COUNT=$(openstack server list --host {{ node_name }} -f json | jq -r -c '[.[] | select(.Status | contains("ACTIVE") or contains("PAUSED") or contains("MIGRATING"))] | length')
if [ $INSTANCE_COUNT == 0 ]; then
break
fi
sleep 3
(( elapsed_seconds += 3 ))
if [ $elapsed_seconds -ge $timeout_seconds ]; then
echo "FAILURE: Could not quiesce compute node {{ node_name }}"
exit 1
fi
done
{% endif %}