OpenStack Compute (Nova)
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

5.9 KiB

Testing Zero Downtime Upgrade Process

Zero Downtime upgrade eliminates any disruption to nova API service during upgrade.

Nova API services are upgraded at the end. The basic idea of the zero downtime upgrade process is to have the connections drain from the old API before being upgraded. In this process, new connections go to the new API nodes while old connections slowly drain from the old nodes. This ensures that the user sees the max_supported API version as a monotonically increasing number. There might be some performance degradation during the process due to slow HTTP responses and delayed request handling, but there is no API downtime.

This page describes how to test the zero downtime upgrade process.


  • Multinode devstack environment with 2 nodes:
    • controller - All services (N release)
    • compute-api - Only n-cpu and n-api services (N release)
  • Highly available load balancer (HAProxy) on top of the n-api services. This is required for zero downtime upgrade as it allows one n-api service to run while we upgrade the other. See instructions to setup HAProxy below.

Instructions to setup HAProxy

Install HAProxy and Keepalived on both nodes.

Let the kernel know that we intend to bind additional IP addresses that won't be defined in the interfaces file. To do this, edit /etc/sysctl.conf and add the following line:

Make this take effect without rebooting.

Configure HAProxy to add backend servers and assign virtual IP to the frontend. On both nodes add the below HAProxy config:


Just change the IP for log in the global section on each node.

On both nodes add keepalived.conf:


Change priority on node2 to 100 ( or vice-versa). Add HAProxy virtual IP.

Restart keepalived service.

Add ENABLED=1 in /etc/default/haproxy and then restart HAProxy service.

When both the services have restarted, node with the highest priority for keepalived claims the virtual IP. You can check which node claimed the virtual IP using:

Zero Downtime upgrade process

General rolling upgrade process: :ref:minimal_downtime_upgrade.

Before Upgrade

  • Change nova-api endpoint in keystone to point to the HAProxy virtual IP.
  • Run tempest tests
  • Check if n-api services on both nodes are serving the requests.

Before maintenance window

  • Start the upgrade process with controller node.
  • Follow the steps from the general rolling upgrade process to install new code and sync the db for schema changes.

During maintenance window

After maintenance window

  • Follow the steps from general rolling upgrade process to clear any cached service version data and complete all online data migrations.