openstack/openstack-ansible
Code Issues Proposed changes

Skip haproxy with setup-infrastructure for upgrades

Browse Source 
As we've enabled /healtcheck URIs for services [1], that we want to
verify, we need to apply haproxy backend changes only when all services
are re-configured, otherwise haproxy will fail to poll them and
mark all backends as down, which would mean continious downtime until
corresponsive roles will run to fix service api-paste configuration.

[1] https://review.opendev.org/q/topic:osa/healthcheck
Closes-Bug: #2002645

Change-Id: I004f1a86b4dba3c6e356ac14bb70a43abc17f538
This commit is contained in:
Dmitriy Rabotyagov
2023-01-12 13:52:52 +01:00
parent 31c4a07b97
commit befd8424e2
2 changed files with 16 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
doc/source/admin/upgrades/major-upgrades.rst
View File

@@ -200,10 +200,13 @@ Upgrade infrastructure
We can now go ahead with the upgrade of all the infrastructure components. To
ensure that rabbitmq and mariadb are upgraded, we pass the appropriate flags.
We also skip upgrading haproxy re-configuration at this step, since some
services will become unavailable right after running haproxy role, which we
want to avoid
.. code-block:: console
# openstack-ansible setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true' -e package_state=latest
# openstack-ansible setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true' -e 'package_state=latest' --skip-tags haproxy-config
With this complete, we can now restart the mariadb containers one at a time,
ensuring that each is started, responding, and synchronized with the other
@@ -224,6 +227,16 @@ We can now go ahead with the upgrade of all the OpenStack components.
# openstack-ansible setup-openstack.yml -e package_state=latest
Run haproxy re-configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Now, once all services are being re-configured, we can run haproxy
role to apply new configuration for the backends
.. code-block:: console
# openstack-ansible haproxy-install.yml -e package_state=latest
Upgrade Ceph
~~~~~~~~~~~~

3
scripts/run-upgrade.sh
View File

@@ -184,11 +184,12 @@ function main {
openstack-ansible setup-hosts.yml -e 'lxc_container_allow_restarts=false' --limit 'galera_all:rabbitmq_all' || \
openstack-ansible setup-hosts.yml -e package_state=latest")
# upgrade infrastructure
RUN_TASKS+=("setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true' -e package_state=latest")
RUN_TASKS+=("setup-infrastructure.yml -e 'galera_upgrade=true' -e 'rabbitmq_upgrade=true' -e 'package_state=latest' --skip-tags haproxy-config")
# explicitly perform controlled galera cluster restart with new lxc config
RUN_TASKS+=("${SCRIPTS_PATH}/upgrade-utilities/galera-cluster-rolling-restart.yml")
# upgrade openstack
RUN_TASKS+=("setup-openstack.yml -e package_state=latest")
RUN_TASKS+=("haproxy-install.yml -e package_state=latest")
# Run the tasks in order
for item in ${!RUN_TASKS[@]}; do
echo "### NOW RUNNING: ${RUN_TASKS[$item]}"