Upgrade testing of OpenStack projects built on top of DevStack.
Go to file
melanie witt f1980a682b Set NOVA_VNC_ENABLED: true for subnode in the grenade job
Currently, because we don't run the n-novnc (console proxy) service on
the subnode, devstack won't enable VNC for guests hosted on the

We see occasional failures in the gate when VNC tempest tests attempt
to obtain a VNC console for a guest that happened to get scheduled to
the subnode with the following error:

  Details: {'code': 400, 'message': 'Unavailable console type novnc.'}

This sets NOVA_VNC_ENABLED = true for the subnode only, with the
thinking that the enabled n-novnc service is sufficient for the
controller node.

Jobs that derive from the "devstack" job [1] are already setting
NOVA_VNC_ENABLED: true for both the controller node and subnode.

[1] 3ffcc89d5d/.zuul.yaml (L535)

Change-Id: I1acbdb605e0e0d9e5b4604fcd21af850b06823b9
2023-11-06 23:44:17 +00:00
data Generate plugin list for registry document 2018-03-01 17:18:23 +00:00
doc Update IRC network to OFTC 2021-05-31 15:05:17 -05:00
inc Add GRENADE_USE_EXTERNAL_DEVSTACK to skip part of the setup 2020-03-02 18:20:36 +01:00
playbooks Use default neutron backend in the base grenade job 2022-12-09 11:48:00 +00:00
projects Increase cinder verify sync timeout 2023-10-19 11:41:31 -07:00
roles Update grenade setting on current master 2023-10-05 05:44:21 +00:00
tools References to review.openstack.org are replaced with review.opendev.org 2021-04-26 19:58:41 +00:00
.gitignore Generate plugin list for registry document 2018-03-01 17:18:23 +00:00
.gitreview OpenDev Migration Patch 2019-04-19 19:27:25 +00:00
.zuul.yaml Set NOVA_VNC_ENABLED: true for subnode in the grenade job 2023-11-06 23:44:17 +00:00
CONTRIBUTING.rst [ussuri][goal] Update contributor documentation 2020-04-22 14:07:41 +00:00
HACKING.rst [ussuri][goal] Update contributor documentation 2020-04-22 14:07:41 +00:00
LICENSE Initial repo setup 2012-08-03 17:07:12 -05:00
PLUGINS.rst Setup native grenade Zuul v3 jobs 2020-04-26 10:59:04 +02:00
README.rst docs: fix typo in README 2022-04-25 13:43:33 -04:00
TODO.rst Remove exercises and run tempest smoke tests instead 2014-02-04 15:52:06 +00:00
cache_git.sh Switch to opendev URLs 2020-02-27 09:59:40 +01:00
clean.sh Always source lib/apache when lib/tls is used 2019-02-18 12:10:25 +01:00
devstack.local.conf.base Update devstack keys from config files 2017-01-26 19:07:44 +00:00
devstack.local.conf.target Ensure we use uwsgi for everything 2017-09-05 17:58:25 -04:00
devstack.localrc.base Always assume glance is uwsgi mode for grenade 2020-07-16 10:49:00 -07:00
devstack.localrc.example Updating devstack.localrc.example for accuracy 2017-01-25 10:37:09 -08:00
devstack.localrc.target Always assume glance is uwsgi mode for grenade 2020-07-16 10:49:00 -07:00
functions Fix tempest venv constraints for target branch tempest run 2023-03-30 20:58:04 -05:00
grenade.sh Fix tempest venv constraints for target branch tempest run 2023-03-30 20:58:04 -05:00
grenaderc Enable nova workaround for upgrade bug 2022-02-21 14:14:46 -08:00
prep-target Add WSGI_MODE setting for target 2017-04-17 10:06:43 -04:00
run_resource.sh add convenience tools 2015-04-24 11:58:57 -04:00
save-state Switch the grenade job to focal 2021-03-12 15:13:48 +01:00
setup-grenade Switch to opendev URLs 2020-02-27 09:59:40 +01:00
setup.cfg setup.cfg: Replace dashes with underscores 2021-05-14 15:55:45 +08:00
setup.py Disable auto discovery 2022-03-28 12:39:49 +02:00
test-requirements.txt Cleanup py27 support 2020-05-22 16:37:26 +02:00
tox.ini Changed minversion in tox to 3.18.0 2021-05-06 20:02:50 +05:30
upgrade-tempest Remove lib/neutron-legacy leftovers 2022-12-20 11:15:47 +01:00



Grenade is an OpenStack test harness to exercise the upgrade process between releases. It uses DevStack to perform an initial OpenStack install and as a reference for the final configuration. Currently Grenade can upgrade Keystone, Glance, Nova, Neutron, Cinder, Swift, and Ceilometer in their default DevStack configurations.


Grenade has the following goals:

  • Block unintentional project changes that would break the Theory of Upgrade. Most Grenade fails that people hit are of this nature.
  • Ensure that upgrading a cloud doesn't do something dumb like delete and recreate all your servers/volumes/networks.
  • Be able to grow to support additional upgrade scenarios (like sideways migrations from one configuration to another equivalent configuration)

Theory of Upgrade

Grenade works under the following theory of upgrade.

  • New code should work with old configs

    The upgrade process should not require a config change to run a new release. All config behavior is supposed to be deprecated over a release cycle, so that upon release new code works with the last releases configs. Those configs may create deprecation warnings which need to be addressed before the next release, but they should still work and largely have the same behavior.

  • New code should need nothing more than 'db migrations'

    Clearly the release of new code may include new database models. Standard upgrade procedure is to turn off all services that touch the database, run the db migration script, and start with new code.

  • Resources created by services before upgrade, should still be there after the system is upgraded

    When upgrading Nova you expect all your VMs to still function during the entire upgrade (whether or not Nova services are up). Taking down the control plane should not take down your VMs.

  • Any other required changes on upgrade are an exception and must be called out in the release notes.

    Grenade supports per release specific upgrade scripts (from-juno, from-kilo). These are designed to support upgrades where additional manual steps are needed for a specific upgrade (i.e. from juno to kilo). These should be used sparingly.

    The Grenade core team requires the following before landing these kinds of changes:

    • The Release Notes for the release where this will be required clearly specify these manual upgrade steps.
    • The PTL for the project in question has signed off on this change.


    While we expect the various deployment projects within the OpenStack ecosystem, for example TripleO, Kolla, etc, to read the release notes of each project, it is good practice to communicate any exceptional upgrade changes made to Grenade to those teams directly or at least via the openstack-discuss mailing list.


Grenade is now running on every patch for projects that support upgrade. Gating Grenade configurations exist for the following in OpenStack's CI system (this is not an exhaustive list):

  • A cloud with neutron upgraded between releases
  • A cloud with neutron that upgrades all services except nova-compute, thus testing RPC backwards compatibility for rolling upgrades.

Basic Flow

The grenade.sh script attempts to be reasonably readable, so it's worth looking there to see what's really going on. This is the super high level version of what that does.

  • get 2 devstacks (base & target)
  • install base devstack
  • perform some sanity checking (currently tempest smoke) to ensure this is right
  • allow projects to create resources that should survive upgrade
    • see projects/*/resources.sh
  • shut down all services
  • verify resources are still working during shutdown
  • upgrade and restart all services
  • verify resources are still working after upgrade
  • perform some sanity checking (currently tempest smoke) to ensure everything seems good.

The script skips the first two steps (which take care of setting up the 2 devstack environments and installing the base one) when the value of GRENADE_USE_EXTERNAL_DEVSTACK is set to True.


Grenade has two DevStack installs present and distinguished between then as 'base' and 'target'.

  • Base: The initial install that will be upgraded.
  • Target: The reference install of target OpenStack (maybe just DevStack)

Directory Structure

Grenade creates a set of directories for both the base and target OpenStack installation sources and DevStack:

 |- logs                # Grenade logs
 |- save                # Grenade state logs
 |- <base>
 |   |- data            # base data
 |   |- logs            # base DevStack logs
 |   |- devstack
 |   |- images          # cache of downloaded images
 |   |- cinder
 |   |- ...
 |   |- swift
 |- <target>
 |   |- data            # target data
 |   |- logs            # target DevStack logs
 |   |- devstack
 |   |- cinder
 |   |- ...
 |   |- swift


This is a non-exhaustive list of dependencies:

  • git
  • tox

Install Grenade

Get Grenade from GitHub in the usual way:

git clone https://opendev.org/openstack/grenade

Optional: running grenade against a remote target

There is an optional setup-grenade script that is useful if you are running Grenade against a remote VM from a local laptop.

Grenade knows how to install the current master branch using the included setup-grenade script. The arguments are the hostname of the target system that will run the upgrade testing and the user for the target system:

./setup-grenade [testbox [testuser]]

If you are running Grenade on the same machine you cloned to, you do not need to do this.


The Grenade repo and branch used can be changed by adding something like this to localrc:


If you need to configure your local devstacks for your specific environment you can do that by creating devstack.localrc. This will get appended to the stub devstack configs for BASE and TARGET.

For instance, specifying interfaces for Nova is a common use of devstack.localrc:


Run the Upgrade Testing


Read grenade.sh for more details of the steps that happen from here.