cf465d87b1
Currently when we call the major-upgrade step we do the following:
"""
...
if [[ -n $(is_bootstrap_node) ]]; then
check_clean_cluster
fi
...
if [[ -n $(is_bootstrap_node) ]]; then
migrate_full_to_ng_ha
fi
...
for service in $(services_to_migrate); do
manage_systemd_service stop "${service%%-clone}"
...
done
"""
The problem with the above code is that it is open to the following race
condition:
1. Code gets run first on a non-bootstrap controller node so we start
stopping a bunch of services
2. Pacemaker notices will notice that services are down and will mark
the service as stopped
3. Code gets run on the bootstrap node (controller-0) and the
check_clean_cluster function will fail and exit
4. Eventually also the script on the non-bootstrap controller node will
timeout and exit because the cluster never shut down (it never actually
started the shutdown because we failed at 3)
Let's make sure we first only call the HA NG migration step as a
separate heat step. Only afterwards we start shutting down the systemd
services on all nodes.
We also need to move the STONITH_STATE variable into a file because it
is being used across two different scripts (1 and 2) and we need to
store that state.
Co-Authored-By: Athlan-Guyot Sofer <sathlang@redhat.com>
Closes-Bug: #1640407
Change-Id: Ifb9b9e633fcc77604cca2590071656f4b2275c60
(cherry picked from commit dde12b075f)
47 lines
1.8 KiB
Bash
Executable File
47 lines
1.8 KiB
Bash
Executable File
#!/bin/bash
|
|
|
|
set -eu
|
|
|
|
check_cluster
|
|
check_pcsd
|
|
if [[ -n $(is_bootstrap_node) ]]; then
|
|
check_clean_cluster
|
|
fi
|
|
check_python_rpm
|
|
check_galera_root_password
|
|
check_disk_for_mysql_dump
|
|
|
|
# M/N Upgrade only: By default RHEL/Centos has an /etc/sysconfig/iptables file which
|
|
# allows ssh and icmp only (INPUT table). During the install of OSP9/Mitaka
|
|
# usually the live iptables rules are not the ones in /etc/sysconfig/iptables but
|
|
# they are completely open (ACCEPT)
|
|
# Now when we run the convergence step while migrating to Newton we enable the firewall
|
|
# by default and this will actually first load the rules from /etc/sysconfig/iptables
|
|
# and only afterwards, it will start adding all the rules permitting openstack traffic.
|
|
# This causes an outage of roughly 1 minute in our env, which disrupts the cluster.
|
|
# Let's simply move the existing file out of the way, it will be recreated by
|
|
# puppet in newton with the proper firewall rules anyway
|
|
if [ ! -f /etc/sysconfig/iptables.m-n-upgrade ]; then
|
|
mv /etc/sysconfig/iptables /etc/sysconfig/iptables.m-n-upgrade || /bin/true
|
|
fi
|
|
|
|
# We want to disable fencing during the cluster --stop as it might fence
|
|
# nodes where a service fails to stop, which could be fatal during an upgrade
|
|
# procedure. So we remember the stonith state. If it was enabled we reenable it
|
|
# at the end of this script
|
|
if [[ -n $(is_bootstrap_node) ]]; then
|
|
STONITH_STATE=$(pcs property show stonith-enabled | grep "stonith-enabled" | awk '{ print $2 }')
|
|
# We create this empty file if stonith was set to true so we can reenable stonith in step2
|
|
rm -f /var/tmp/stonith-true
|
|
if [ $STONITH_STATE == "true" ]; then
|
|
touch /var/tmp/stonith-true
|
|
fi
|
|
pcs property set stonith-enabled=false
|
|
fi
|
|
|
|
# Migrate to HA NG
|
|
if [[ -n $(is_bootstrap_node) ]]; then
|
|
migrate_full_to_ng_ha
|
|
fi
|
|
|