openstack-ansible/doc/source/upgrade-guide/manual-upgrade.rst

Manual upgrades

Deployers can run the upgrade steps manually. Manual execution is useful for scoping the changes in the upgrade process (For example, in very large deployments with strict SLA requirements), or for inclusion into other orchestration for upgrade automation beyond what OpenStack-Ansible provides.

The steps detailed here match those performed by the run-upgrade.sh script. Any of these steps can safely be run multiple times.

Check out the release

Ensure your OpenStack-Ansible code is on the latest tagged release.

# git checkout

Prepare the shell variables

Note

This step is optional, since these environment variables are simply shortcuts. Files can be referenced directly.

From the openstack-ansible root directory, run the following commands:

# export MAIN_PATH="$(pwd)"
# export SCRIPTS_PATH="${MAIN_PATH}/scripts"
# export UPGRADE_PLAYBOOKS="${SCRIPTS_PATH}/upgrade-utilities/playbooks"

These variables reduce typing when running the remaining upgrade tasks.

Re-bootstrap Ansible for

Bootstrapping Ansible again ensures that all OpenStack-Ansible role dependencies are in place before running playbooks from the release.

# ${SCRIPTS_PATH}/bootstrap-ansible.sh

Change to playbooks directory

Change to the playbooks directory so that the OpenStack-Ansible dynamic inventory is found automatically.

# cd playbooks

Pre-flight checks

Before starting with the new version, you should do pre-flight checks to ensure everything is fine. If any of those check fail, the upgrade should stop to let the deployer chose what to do.

Cleanup old facts

Some configuration changed, and old facts should be purged before the upgrade.

See fact-cleanup-playbook for more details.

# openstack-ansible "${UPGRADE_PLAYBOOKS}/ansible_fact_cleanup.yml"

Update configuration and environment files

The user configuration files in /etc/openstack_deploy/ and the environment layout in /etc/openstack_deploy/env.d have new name values added in .

See config-change-playbook for more details.

# openstack-ansible "${UPGRADE_PLAYBOOKS}/deploy-config-changes.yml"

Update user secrets file

introduces new user secrets to the stack. These are populated automatically with the following playbook.

See user-secrets-playbook for more details.

# openstack-ansible "${UPGRADE_PLAYBOOKS}/user-secrets-adjustment.yml"

Cleanup pip.conf file

The presence of pip.conf file can cause build failures when upgrading to . This play removes the pip.conf file on all the physical servers and on the repo containers.

See pip-conf-removal for more details.

# openstack-ansible "${UPGRADE_PLAYBOOKS}/pip-conf-removal.yml"

Upgrade hosts

Before installing the infrastructure and OpenStack, update the host machines.

# openstack-ansible setup-hosts.yml --limit '!galera_all'

This command is the same as doing host setups on a new install. The galera_all host group is excluded to prevent reconfiguration and restarting of any Galera containers.

Update Galera LXC container configuration

Update the Galera container configuration independently.

# openstack-ansible lxc-containers-create.yml -e \
'lxc_container_allow_restarts=false' --limit galera_all

This command is a subset of the host setup playbook, limited to the galera_all host group. The configuration of those containers will be updated but a restart for any changes to take effect will be deferred to a later playbook.

Perform a controlled rolling restart of the galera containers

Restart containers one at a time, ensuring that each is up, responding, and synced with the other nodes in the cluster, before moving on to the next. This step allows the lxc container config applied earlier to take effect, ensuring that the containers are restarted in a controlled fashion.

# openstack-ansible "${UPGRADE_PLAYBOOKS}/galera-cluster-rolling-restart.yml"

Update HAProxy configuration

Install and update any new or changed HAProxy service configurations.

# openstack-ansible haproxy-install.yml

Update repo servers

Update the configuration of the repo servers and build new packages required by the release.

# openstack-ansible repo-install.yml

Perform a mariadb version upgrade

Update mariadb to the most 10.0 minor release across the cluster.

# openstack-ansible galera-install.yml -e 'galera_upgrade=true'

Upgrade infrastructure

The following commands perform all steps from the setup-infrastructure playbook, except for repo-install.yml, haproxyinstall.yml, and `galera-install.yml` which we ran earlier. Running these playbook applies the relevant settings and packages.

For certain versions of , you must upgrade the RabbitMQ service.

See setup-infra-playbook for details.

# openstack-ansible unbound-install.yml
# openstack-ansible memcached-install.yml
# openstack-ansible rabbitmq-install.yml -e 'rabbitmq_upgrade=true'
# openstack-ansible etcd-install.yml
# openstack-ansible utility-install.yml
# openstack-ansible rsyslog-install.yml

Flush Memcached cache

See memcached-flush for details.

# openstack-ansible "${UPGRADE_PLAYBOOKS}/memcached-flush.yml"

Upgrade OpenStack

Upgrade the OpenStack components with the same installation playbook, without any additional options.

# openstack-ansible setup-openstack.yml