4572055ffb
Make sure scripts are created with the executable bit set. Change-Id: I731902411e987b4ea7c2aa84fef869fe5e1c25ae Add a oooq comptatibility layer, documentation and example. Change-Id: I30fe6359f1c0098ff9bcdd5939724491d94ef199 Add support for applying w/a before and after upgrade This change adds the ability to apply workarounds before and after the overcloud upgrade process has finished. This allows the user to workaround particular issues that show up after the upgrade process has finished. Change-Id: I21a7e885bcc466af6bf80410ba2cc8d03865cb33 Fix missing quotes This changes adds missing quotes to the node_upgrade_script.yml templates. Currently the task is failing because of the missing quotes. Change-Id: Ied9217df374d09bf90e6878a7c79e22042f44d99 Add support for applying workarounds post undercloud upgrade This change adds support for applying workarounds after the undercloud upgrade process has finished as part of the undercloud upgrade. Change-Id: If85900969c0d591cf6024408d958a82fa8c8a534 Adjust overcloud_converge_upgrade_script script This change adjusts the overcloud_converge_upgrade_script to allow running the upgrade converge stage. In addition it adjust the ssh config file to skip host key check so the non-controller script does not get stuck waiting for user input. Change-Id: Ic38f325c61e90165a5322ef754f7e5514ed8e687 Append working_dir to logs generated Change-Id: I6bc9f0c58ad8684ed03dee042e9cfb2bdc6835f6 Install ceph-ansible during undercloud upgrade ceph-ansible is required to be installed manually for deployments with ceph nodes. This change installs the ceph-ansible package before the undercloud upgrade. Change-Id: If8918a38250a10681d965d0715ebc17078166336 Use openstack overcloud container image prepare command This change adds the use of openstack overcloud container image prepare command for generating the environment file containing the container image names and local registry address. Change-Id: I174f7e3aae415d51224cf73da83a859e90eed095 Do not rely on ansible inventory for upgrading non controller nodes Currently we are relying on the ansible inventory to provide groups containing compute nodes and their facts when creating the upgrade scripts. In order to remove this requirement and provide easier integration this change discovers the compute and swift storage nodes from the undercloud. In addition it adds a wait loop for instance live migration to complete before and after upgrading compute nodes and adds support for swift storage nodes upgrade. Change-Id: Ia4b2e81845c3fec9036c4695f0dd1746d4c5c6b8 Add silent and nobuffer options to curl command Change-Id: I0f4bd71b67d4827717d9f7d3fc075fc396eb6363 Adjust tht environment files and custom roles data during upgrade This change switches the environment files used in the overcloud deploy command to their variants used for deploying containers. In addition, if using a custom roles data file for composable roles deployments, this change adjusts the local roles data copy with the changes introduced in Pike. Change-Id: Icd3c3b67342c0bd10fc3d28ed89a94fc9f714db4 Fix overcloud_deploy_script var location in oooq_test playbook Change-Id: I01f18a5413ff223cbca900521037e739ebc43d5d Disconnect ssh session before uploading images to local registry This change disconnects the running ssh session in order to allow the stack user to connect to the docker daemon. This acts as a workaround for https://bugs.launchpad.net/tripleo/+bug/1699477 Change-Id: Ia06fe8581ba17525cbcb2d955d7947b7c546811d Add undercloud_reboot var and reboot only on ovs or kernel updates This change adds the undercloud_reboot var to manipulate if undercloud reboots should be made or not and does the reboot only when kernel or ovs updates occur. Change-Id: Ic89291c5791bbd098204b2c85793335e0faa8d94 Add InfraRed docs This change adds the steps required to run the tripleo-upgrade role as an InfraRed plugin. Change-Id: I08fde6c8954ec4eeedc47971607bec592fc801a1 Add docs for running the role manually from the undercloud Change-Id: I6d5bb6479b6e1fb45e34aa776d3c315cfae98ae4 Do not stop services before undercloud upgrade According to the docs stopping services should be done by the undercloud upgrade process so we do not need to manually do it. Change-Id: I7f14257b4e90d3a0610c0f90e856096643325861 Specifically become_user: root when running ovs command Currently become_user is set to the undercloud user, usually stack, when invoking the undercloud_validate_upgrade.yaml playbook. This change overrides become_user and sets it to root when running ovs-vsctl command as it requires privilege escalation. Change-Id: Ia93245f4a2d09d73849c03401d38ffb25e7be802 Use ip address instead of name when rebooting OOOQ doesn't set a name for the undercloud in hosts so in case the reboot is triggered the virthost cannot reach the undercloud. This change switches the wait condition and calls the ip address instead of the name. Change-Id: I2121e2dabe9794d31fe70bbfa0ff8e53da6b3b1b Add reload ssh tag to the Kill SSH task Add a tag to the Kill SSH task so it can be avoided when running the role manually from the undercloud. Change-Id: Iec088cc174d7e8270fbea0698ee76227b45842f7 Add option to create script with --setup-heat-output option According to BZ#1477962 in order to be able to run non controller upgrade scripts after major upgrade composable steps we need to run the deploy command with the --setup-heat-outputs option. Change-Id: I8137ff18047a130c5ea8dca2ce11378eabc30329 Add deprecated params to custom roles data file In Pike there were additional flags assigned to the default roles data file. This change adds these flags during the upgrade process to the roles defined in the custom roles data file. Change-Id: I58c2f30ff74d2302027d7488dc03c5146c371649 Use a default value for the HOME env var Change-Id: I40e2f46f1311dc4b797977c8136ec4037505ef05 Update python-openstackclient before undercloud upgrade Updating python-openstackclient before undercloud upgrade is a requirement to get undercloud upgrade passing. Change-Id: I4ab67f9af68036a29e11bb2457d70535bd94b7b0 Create environment file for injecting undercloud certificate When undercloud is SSL enabled the overcloud nodes need to be aware of the undercloud CA cert. This change creates this file during upgrade and adds it to the overcloud deploy commands when the overcloud nodes are not able to reach the undercloud ssl enabled public endpoints. Change-Id: I79a03299bc28d0ca2dbd83c28087a4c56f6b2271 Convert puppet ceph parameters to ceph ansible during upgrade This is a workaround for BZ#1488855. Change-Id: I58ac44b2166abddf56a327a4ee09457139c831da Remove setup heat outputs workaround https://review.openstack.org/#/c/502470/ allows us to obtain the RoleConfig output so there's no need to run the setup heat outputs step anylonger. This change remove this step. Change-Id: I7235b8625eea4f77a055d0ab1862d0318f3a776f Echo bug number in workarounds script This change replaces the bug number comments with an echo statement. This provides easier way to debug what workarounds failed to apply when the workarounds script is run. Change-Id: I02c5534427ee5cf7b7af618f36577c1008d50992 Fix uc_keystone_conn condition This change adds an addition condition to inject the undercloud certificate in the CAMap only when undercloud ssl is enabled. Change-Id: Iae023922bf1ed0bb22e86710b122c65a8f1568a3 Gather facts only from undercloud node Facts are only required for the undercloud node. This change adjusts the existing playbooks to gather facts only from undercloud node in order to save some time and not rely on nodes which are not required to be reachable. Change-Id: Idceec3d10d84ef112da558109d9904a1d8c6ed93 Do not include docker.yaml and docker-ha.yaml environments As described in BZ#1466744 the docker.yaml and docker-ha.yaml environments are currently included by default so we do not need to specifically include them. Change-Id: I44a72ddd65cf816003ceca21ef33470a3ab125a7 Reboot compute nodes post upgrade As a post upgrade requirement we need to reboot the nodes in case of an OVS upgrade. This change runs a post upgrade check for non controller nodes and reboots the nodes if an ovs upgrade has been detected. It also adds additional validation for compute nodes to make sure that the nova-compute service is enabled after reboot. Change-Id: I583e589118aabae84f8e1dc9ec2c4b43ca17a250 Add L3 agent connectivity check during upgrade This change adds a check which validates that ICMP connectivity with a floating IP is not interrupted during the major upgrade composable step. Change-Id: Iee55af85b9a2c3ece86731e043130d191ff6a821 Run pre docker composable upgrade workarounds at correct position This change moves the pre docker composable upgrade workarounds to be run right before the docker composable upgrade step. Change-Id: I604ea2eb6202d48b0f771ea80e5e731df687600e Use bool with ansible booleans use bool filter when using ansible booleans. Change-Id: Ibeb59772e935cc28a661ccddcaa4773388ce296d Add option for creating workloads before upgrade This change adds the option to launch an instance before starting upgrade. This operation is useful when doing tests such as instance live migration during upgrade or floating ip connectivity testing during upgrade. The script requires a network defined in the external_network_name var which provides external connectivity to exist beforehand. Change-Id: Ib39e41b36fac7794ea515c8a9d56141866dcfeed Fix pre compute upgrade check This change adds the MIGRATING state to be checked before the compute nodes upgrade. Change-Id: I0073a7e69a71a044882d4760dbb49cd4f455dd89 Fix workload_launch position Change-Id: I6193ac6a60165bb20ece6277067c05696ed6d3b1 Run non controller node pre upgrade script This change runs the non controller node pre upgrade script. In addition it exposes the option to run instances migration between compute nodes during upgrade. Change-Id: Ief55eecdc85bb620f637c4ed4d9b5bc3243b37d1 Update roles_data adjustments to latest changes This change updates the roles_data file adjustments to the latest changes. Change-Id: Ic787b135cdf96b33829e05140e069b398df7196f Use docker and docker-ha environment files for upstream deployments Change-Id: I70bb9767e97f616729adff983fff065858a6dcdc Convert services environments to services-docker only for upstream Per BZ#116463 in downstream the environments used for extra service enablement now point to docker resources. Change-Id: I379622ec2749ac8b485aec79a7500308ef74214e Echo debug message to differentiate live migration from block Change-Id: I69e7f381543d84fcd308acbd3a90f5d0ac23ae1b Accept <= 5% ICMP packet loss during upgrade connectivity check Change-Id: I34d1de225c0e391035e22e18f63356e04afbbfd5 Reorganize playbooks to separate upgrade/update This change adds separate directories for upgrade/update which provides a better separation between updates and upgrades. Change-Id: Icf1a09514fb0e6236535ae32265bbd3805918478 Run block migrate multiple times Block migrate doesn't work seem to work if triggered once but it does if the command is run for a second time. This change runs the block migrate command multiple times to make sure that the instance gets migrated. Change-Id: I8b9a9ecae21f7ce49a03945afec66b9e671622b7 Ensure files/ are part of the setup.cfg files to copy When installing tripleo-upgrade into a .quickstart environment, the files/ folder wasn't getting copied, which is necessary at least for "adjust ssh config to skip host key check" in create-upgrade-scripts.yaml. Change-Id: I7d862ec5c13ba719923c90cc40790b842b582999 Add tasks for undercloud minor update Start adding the minor undercloud update tasks Change-Id: I33705b270e2d5e6a28f1cad8179e1f4b3e4ea975 Remove timeouts from upgrade scripts Depending on the number of deployed nodes upgrade could take longer and we want to rely on the heat stack timeout. This change clears any manual timeouts set in the upgrade scripts. Change-Id: I5d141e2cc13621d3be5fb0c27b0ac3c3fc30d424 Minor updates of RHOS 12. Manage minor update workflow from within tripleo-upgrade repo. Change-Id: I8c6771af4825ce166e8470413ca4687be0a58cb9 Reboot controller nodes post upgrade This change adds the option to reboot controller nodes post upgrade and performs basic verifications that the clustered services are reported as up. Change-Id: I370d421e5968ae50bd1ff140cdfcf98a4db03a5f Don't force ssh_config on everybody. This add an option to be able to not overwrite the ssh_config file. As a side note the ssh_config is missing from the repo, so by default this task is broken. Change-Id: Idfb78e2b7226a7e6295acd3f250bbfb48d0a103d Fix filter used in the node upgrade scripts This change is fixes the current filter used for node upgrade scripts so that deployment with $domain.tld format are supported. Change-Id: I18f43c440bb93e0fcefb664c7d716ff9368673a4 Run live migration multiple times Change-Id: I7c028defd3cb9080efa7bdbe9daa6ed201df8640 Manually inject undercloud SSL cert to overcloud nodes Per BZ#1501779 the compute nodes do not get their trusted store updated when using a CAMap and upgrade fails. This change updates the overcloud nodes trusted store manually so the overcloud nodes are prepared for update. This should translate in a documentation step that before upgrade starts the user needs to make sure the overcloud nodes are able to reach the undercloud SSL public endpoint. Change-Id: Ib95a29c608803504a866ae71cbc0082faf3c194f Replace puppet external ceph environment with ceph-ansible one This change replaces during the upgrade the external ceph puppet environment file with its ceph-ansible equivalent. Change-Id: I9020e8f7c43f91259b551caa2e20f03be1424106 Append deprecated params only to predefined roles We should append the deprecated params only to predefined roles in order to avoid failures such as reported in BZ#1501237. Change-Id: I3a7c332b35da9639fb6f8e5b38234dc0c55d8499 Split the post controller scripts into per services scripts This change splits the post upgrade controller scripts into per service checks and adds them to a common directory so they can be shared between update and upgrade. Change-Id: I8f2fb6162a5acb8a92057400a7b04e6e2388abaa Add the ability to specify a remote docker registry This change adds the ability to specify a remote docker registry to be used for downloading the Docker images on the undercloud or be used directly by the overcloud nodes during upgrade. Change-Id: I132a8b94f9a101d1c9c624d202bb01527dc2b844 Fix BZ#1499677 workaround condition In addition to empty gvwstate.dat file there might be situations where the gvwstate.dat file is missing after reboot. This patch addresses this condition for BZ#1499677 workaround. Change-Id: I295b133248f48ab41b1748225cbe9359662b280d Cleanup galera resource instead of rebooting node for BZ#1499677 Instead of rebooting the node while implementing the workaround for BZ#1499677 we should simply cleanup the Galera resource. This way we can save some time and potential issues caused by an additional reboot. Change-Id: I391daeae41321baec1cbd8c458132a3161cd96d5 [UPDATES] Introduce option for minor updates workarounds. To speed up testing of minor updates it might be required to apply some patches before they are landed. Hence we need a flag to differentiate if workarounds are required or not Change-Id: I642e4ade204f5fd30ec9433f1d90a2d539287c5e Do not pipe curl output in container images environment script Curl can sometimes exit with exit code 23 when its output is piped into another command. To avoid this errors we save the curl output to a file. Change-Id: I4123b6c66ae2873c11631f229cb8e3eec5a5a66b Use service environment files when generating the images environment In the last build openstack overcloud container image prepare only generates the parameters for the services included by default. In order to make it work when extra non-default services are enabled we need to pass the environment files to the prepare command. This change addresses this issue. Change-Id: I86ab6faaffcd4c7cc1a07e9a6ed1e890cb5cf980 Place the oooq deploy command into overcloud_deploy_script var This change places the openstack overcloud deploy command with its arguments in the location defined by the overcloud_deploy_script var. This way we don't require oooq users to specifically set the overcloud_deploy_script to a hardcoded location and make it less confusing. Change-Id: Id2b14fcffbd169c342df4b5b9105dff81e18e3a0 Replace ceph radosgw environment during upgrade Change-Id: I489211f39941bba5b1ca2ddf1b635c3bdb0151fe Avoid losing undercloud connection in TripleO CI. When running the role in the TripleO upstream CI the connectivity to the undercloud gets lost when rebooting the undercloud after upgrade or killing the ssh service, this makes the playbook fail. As a solution, a flag tripleo_ci has been added. This flag will default to false, and when set to true no undercloud reboot, nor ssh killing will be executed. Change-Id: If4a303fff49bbe55cdfb7142d8dd69264ab47ab4 Align deployment-files option with IR. deployment-files option is not a list of choices in IR, adjust it accordingly. Change-Id: I6889e9b75f842cc466278fed5dbf85a80cb58ee0 Append docker-ha only when needed. Before appending the docker-ha.yaml env file, we need to check if the overcloud was deployed with pacemaker. If so, then we'll add the env file to the upgrade script. Change-Id: I9867d86b6d23385c576d2f8c5a25ab3333f7113d Specify tht directory used in upgrade script. When deploying with oooq, the generated script overcloud-deploy.sh is reused in order to append the cooresponding env files for the upgrade. However, if the location of the tripleo-heat-templates directory is different from the used when deploying then the upgrade does not succeed. This change modifies the tripleo-heat-templates location used to upgrade when the directory found in overcloud-deploy.sh is different. If it is the same no change is done. Change-Id: I55ada3e75b7463b1c14c8734410d2591cf162e67 Don't append DockerInsecureRegistryAddress This is no longer required as the prepare command detects whether the registry is secure and DockerInsecureRegistryAddress as necessary. Depends on upstream https://review.openstack.org/#/c/514473/ Related-Bug: #1722632 Change-Id: Ia9d91f6280600c59d0079c5d1f26a00f04040426 Fix the controller regex during roles_data conversion The Ocata roles_data controller role might include a comment for the controller role name. This change adjusts the regexp used during the roles data conversion to take into consideration that comment. Change-Id: I7b43b1fcb9e477de8e1265ef6aa6ba5149e82d47 Use prepare --set for ceph image parameter This is more maintainable, and consistent with other uses of prepare. Change-Id: Ieec88e271973a248192c2b247cd2c5e0cccbfb7c Add storage environment files to be used for containers prepare This change adds the storage environments files to be matched when creating the environment file containing the Docker images names which gets created via container image prepare. Change-Id: I34c3cdaab1b63ce3f43d748372d35143bc12b8b4 Add environment file containing required DPDK changes during upgrade Change-Id: Iabaf16e18ee7546bd6275f8f84226892423a6c95 Create failed_upgrade log files. Most of the stack failures in the TripleO CI are registered inside two log files failed_upgrade_list.log and failed_upgrade.log. This patch adds the option of inject the stack failures list command into these two log files, as well as priting out the detailed stack failure list (--long). Change-Id: I4fad989818f67ad0a73e45b47f835750f18c3bb6 Replace storage-environemnt.yaml for upstream only Per BZ#1502862 the Ceph environment files switch during upgrade is only needed for upstream deployments. This change does the changes to accommodate this. Change-Id: Ia3adb120c9b524c66d069593b0779b3399295fd4 Do not update python-openstackclient before upgrade BZ#1488471 was fixed so there's no need to update the python- openstackclient package before upgrade. Change-Id: I2aba7515ec43926ec4f8de5c701467dea31dba1b Be more aggressive on accepted packet loss during upgrade Tests have shown that the packet loss shouldn't eceed 1% during the upgrade steps. This change adjusts the accepted level to this. Change-Id: I9e47ea56a78e4e9eab40fca609cbedaecfcf1e14 Add more tags for the upgrade process. This enable one to either do only a small part of the whole process. This can be useful for debugging or development. Change-Id: Ic6cc9a1e6aa2793fde65636d2ad92bc174173252 Use new method of discovering tag and adjust local registry upload This change uses the new mechanism of retrieving the tag from the latest image provided in the registry. Change-Id: I7e063f13c7d4812e9986452774881235b620bd0e Swap baremetal environment file for containerized. In oooq, when upgrading from baremetal to containerized overcloud two different environment files are used. These env files are located in [0]. When upgrading using tripleo-upgrade, we need to convert that environment file name to its corresponding containers version. [0] https://github.com/openstack/tripleo-heat-templates/tree/master/ci/environments Change-Id: I6c9fad2f402a162cf663c5089e79c2e10f3d0928 Add condition to create local docker registry. The only way to not execute the docker registry environment file creation task is via tags, which is not easy to handle in TripleO CI. As tripleo-quickstart already prepares the local docker registry file, there is no need to execute it. So a new parameter 'create_docker_registry' is been added. Also, the 'force' option is being added to avoid overwritting a provided script with the role template. Change-Id: I800d6696b8dbb83f05f3d9381c6e5689558f4b77 Prepare workloads before update/upgrade. Prepare scripts for managing workload on oc before running update/upgrade operation. Allow to run ping test during minor update. Change-Id: I1d5754f36f53588c97c646aa4e1380e9ca5938bc Remove tag parsing from the image prepare command Parsing the tag is not needed anymore and the one returned by the container image prepare command can be used. Change-Id: If782fc655da7b22e4d4a803509e9cc8c49774368 [UPDATES] Run minor update per role. With recent changes it's advised to perform minor update in batches: role-by-role. This change limits the scope of update with '--nodes <Role>' option. Change-Id: I0bc03873b749dc9c15b13cacbfff78cead4360d8
27 lines
968 B
Django/Jinja
27 lines
968 B
Django/Jinja
{% if compute_evacuate %}
|
|
source {{ overcloud_rc }}
|
|
## Detect if ceph shared storage was used
|
|
STORAGE_BACKEND=$(openstack volume service list -f json | jq -r -c '.[] | select(.Binary | contains("cinder-volume")) | .Host' | sed s/hostgroup@tripleo_//)
|
|
if [ $STORAGE_BACKEND == 'ceph' ]; then
|
|
nova host-evacuate-live {{ node_name }}
|
|
else
|
|
nova host-evacuate-live --block-migrate {{ node_name }}
|
|
fi
|
|
|
|
timeout_seconds=120
|
|
elapsed_seconds=0
|
|
while true; do
|
|
echo "Waiting for {{ node_name }} to get quiesced ..."
|
|
INSTANCE_COUNT=$(openstack server list --host {{ node_name }} -f json | jq -r -c '[.[] | select(.Status | contains("ACTIVE") or contains("PAUSED") or contains("MIGRATING"))] | length')
|
|
if [ $INSTANCE_COUNT == 0 ]; then
|
|
break
|
|
fi
|
|
sleep 3
|
|
(( elapsed_seconds += 3 ))
|
|
if [ $elapsed_seconds -ge $timeout_seconds ]; then
|
|
echo "FAILURE: Could not quiesce compute node {{ node_name }}"
|
|
exit 1
|
|
fi
|
|
done
|
|
{% endif %}
|