update README.rst to reflect current reality
This attempts to bring README.rst into current reality. A few things are simplified along the way. It also explains our theory of upgrade. Change-Id: I2fbb457246a2faca8ac2405104e86885b93e8c48
This commit is contained in:
Sean Dague
parent
9bf1482680
commit
1a17e0c14d
210
README.rst
210
README.rst
@ -1,47 +1,102 @@
|
|||||||
Grenade
|
=========
|
||||||
=======
|
Grenade
|
||||||
|
=========
|
||||||
|
|
||||||
Grenade is an OpenStack test harness to exercise the upgrade process
|
Grenade is an OpenStack test harness to exercise the upgrade process
|
||||||
between releases. It uses DevStack to perform an initial OpenStack
|
between releases. It uses DevStack to perform an initial OpenStack
|
||||||
install and as a reference for the final configuration. Currently
|
install and as a reference for the final configuration. Currently
|
||||||
Grenade upgrades Keystone, Glance, Nova, Neutron, Cinder and Swift in
|
Grenade can upgrade Keystone, Glance, Nova, Neutron, Cinder, Swift,
|
||||||
their default DevStack configurations.
|
Ceilometer, and Ironic in their default DevStack configurations.
|
||||||
|
|
||||||
The master branch tests the upgrade path from the previous release
|
|
||||||
(aka 'base') to the current trunk (aka 'target'). Stable branches
|
|
||||||
of Grenade will be created soon after an OpenStack release and after
|
|
||||||
a corresponding DevStack stable branch is available.
|
|
||||||
|
|
||||||
For example, following the release of Grizzly and the creation of
|
|
||||||
DevStack's stable/grizzly branch a similar stable/grizzly branch
|
|
||||||
of Grenade will be created. At that time master will be re-worked
|
|
||||||
to base on Grizzly and the cycle will continue.
|
|
||||||
|
|
||||||
|
|
||||||
Goals
|
Goals
|
||||||
-----
|
=====
|
||||||
|
|
||||||
Continually test the upgrade process between OpenStack releases to
|
Grenade has the following goals:
|
||||||
find issues as they are introduced so they can be fixed immediately.
|
|
||||||
|
|
||||||
|
- Block unintentional project changes that would break the ``Theory of
|
||||||
|
Upgrade``. Most Grenade fails that people hit are of this nature.
|
||||||
|
- Ensure that upgrading a cloud doesn't do something dumb like delete
|
||||||
|
and recreate all your servers/volumes/networks.
|
||||||
|
- Be able to grow to support additional upgrade scenarios (like
|
||||||
|
sideways migrations from one configuration to another equivalent
|
||||||
|
configuration)
|
||||||
|
|
||||||
|
Theory of Upgrade
|
||||||
|
=================
|
||||||
|
|
||||||
|
Grenade works under the following theory of upgrade.
|
||||||
|
|
||||||
|
- New code should work with old configs
|
||||||
|
|
||||||
|
The upgrade process should not require a config change to run a new
|
||||||
|
release. All config behavior is supposed to be deprecated over a
|
||||||
|
release cycle, so that upon release new code works with the last
|
||||||
|
releases configs. Those configs may create deprecation warnings which
|
||||||
|
need to be addressed before the next release, but they should still
|
||||||
|
work and largely have the same behavior.
|
||||||
|
|
||||||
|
- New code should need nothing more than 'db migrations'
|
||||||
|
|
||||||
|
Clearly the release of new code may include new database
|
||||||
|
models. Standard upgrade procedure is to turn off all services that
|
||||||
|
touch the database, run the db migration script, and start with new
|
||||||
|
code.
|
||||||
|
|
||||||
|
- Resources created by services before upgrade, should still be there
|
||||||
|
after the system is upgraded
|
||||||
|
|
||||||
|
When upgrading Nova you expect all your VMs to still function during
|
||||||
|
the entire upgrade (whether or not Nova services are up). Taking down
|
||||||
|
the control plane should not take down your VMs.
|
||||||
|
|
||||||
|
- Any other required changes on upgrade are an *exception* and must be
|
||||||
|
called out in the release notes.
|
||||||
|
|
||||||
|
Grenade supports per release specific upgrade scripts (from-juno,
|
||||||
|
from-kilo). These are designed to support upgrades where additional
|
||||||
|
manual steps are needed for a specific upgrade (i.e. from juno to
|
||||||
|
kilo). These should be used sparingly.
|
||||||
|
|
||||||
|
The Grenade core team requires the following before landing these
|
||||||
|
kinds of changes:
|
||||||
|
|
||||||
|
- The Release Notes for the release where this will be required
|
||||||
|
clearly specify these manual upgrade steps.
|
||||||
|
|
||||||
|
- The PTL for the project in question has signed off on this change.
|
||||||
|
|
||||||
Status
|
Status
|
||||||
------
|
======
|
||||||
|
|
||||||
Preparations are ongoing to add Grenade as a non-voting job in the
|
Grenade is now running on every patch for projects that support
|
||||||
OpenStack CI Jenkins.
|
upgrade. Gating Grenade configurations exist for the following in
|
||||||
|
OpenStack's CI system.
|
||||||
|
|
||||||
* Testing of the 'javelin' project artifacts is incomplete
|
- A cloud with nova-network upgraded between releases
|
||||||
|
- A cloud with neutron upgraded between releases
|
||||||
|
- A cloud with nova-network that upgrades all services except
|
||||||
|
nova-compute, thus testing RPC backwards compatibility for rolling
|
||||||
|
upgrades.
|
||||||
|
|
||||||
Process
|
Basic Flow
|
||||||
-------
|
==========
|
||||||
|
|
||||||
* Install base OpenStack using current stable/<base-release> DevStack
|
The grenade.sh script attempts to be reasonably readable, so it's
|
||||||
* Perform basic testing (tempest's smoke and scenarios tests)
|
worth looking there to see what's really going on. This is the super
|
||||||
* Create some artifacts in a new project ('javelin') for comparison
|
high level version of what that does.
|
||||||
after the upgrade process.
|
|
||||||
* Install current target DevStack to support the upgrades
|
- get 2 devstacks (base & target)
|
||||||
* Run upgrade scripts preserving (running?) instances and data
|
- install base devstack
|
||||||
|
- perform some sanity checking (currently tempest smoke) to ensure
|
||||||
|
this is right
|
||||||
|
- allow projects to create resources that should survive upgrade
|
||||||
|
- see projects/*/resources.sh
|
||||||
|
- shut down all services
|
||||||
|
- verify resources are still working during shutdown
|
||||||
|
- upgrade and restart all services
|
||||||
|
- verify resources are still working after upgrade
|
||||||
|
- perform some sanity checking (currently tempest smoke) to ensure
|
||||||
|
everything seems good.
|
||||||
|
|
||||||
|
|
||||||
Terminology
|
Terminology
|
||||||
@ -55,13 +110,14 @@ as 'base' and 'target'.
|
|||||||
|
|
||||||
|
|
||||||
Directory Structure
|
Directory Structure
|
||||||
-------------------
|
===================
|
||||||
|
|
||||||
Grenade creates a set of directories for both the base and target
|
Grenade creates a set of directories for both the base and target
|
||||||
OpenStack installation sources and DevStack.
|
OpenStack installation sources and DevStack.
|
||||||
|
|
||||||
$STACK_ROOT
|
$STACK_ROOT
|
||||||
|- logs # Grenade logs
|
|- logs # Grenade logs
|
||||||
|
|- save # Grenade state logs
|
||||||
|- <base>
|
|- <base>
|
||||||
| |- data # base data
|
| |- data # base data
|
||||||
| |- logs # base DevStack logs
|
| |- logs # base DevStack logs
|
||||||
@ -79,20 +135,26 @@ $STACK_ROOT
|
|||||||
| |- swift
|
| |- swift
|
||||||
|
|
||||||
Dependencies
|
Dependencies
|
||||||
------------
|
============
|
||||||
|
|
||||||
This is a non-exhaustive list of dependencies:
|
This is a non-exhaustive list of dependencies:
|
||||||
|
|
||||||
* git
|
* git
|
||||||
* tox<1.7
|
* tox
|
||||||
|
|
||||||
Install Grenade
|
Install Grenade
|
||||||
---------------
|
===============
|
||||||
|
|
||||||
Get Grenade from GitHub in the usual way::
|
Get Grenade from GitHub in the usual way::
|
||||||
|
|
||||||
git clone git://git.openstack.org/openstack-dev/grenade
|
git clone git://git.openstack.org/openstack-dev/grenade
|
||||||
|
|
||||||
|
Optional: running grenade against a remote target
|
||||||
|
-------------------------------------------------
|
||||||
|
|
||||||
|
There is an *optional* setup-grenade script that is useful if you are
|
||||||
|
running Grenade against a remote VM from a local laptop.
|
||||||
|
|
||||||
Grenade knows how to install the current master branch using the included
|
Grenade knows how to install the current master branch using the included
|
||||||
``setup-grenade`` script. The only argument is the hostname of the target
|
``setup-grenade`` script. The only argument is the hostname of the target
|
||||||
system that will run the upgrade testing.
|
system that will run the upgrade testing.
|
||||||
@ -101,73 +163,35 @@ system that will run the upgrade testing.
|
|||||||
|
|
||||||
./setup-grenade testbox
|
./setup-grenade testbox
|
||||||
|
|
||||||
|
If you are running Grenade on the same maching you cloned to, you **do
|
||||||
|
not** need to do this.
|
||||||
|
|
||||||
|
Configuration
|
||||||
|
-------------
|
||||||
|
|
||||||
The Grenade repo and branch used can be changed by adding something like
|
The Grenade repo and branch used can be changed by adding something like
|
||||||
this to ``localrc``::
|
this to ``localrc``::
|
||||||
|
|
||||||
GRENADE_REPO=git@github.com:dtroyer/grenade.git
|
GRENADE_REPO=git@github.com:dtroyer/grenade.git
|
||||||
GRENADE_BRANCH=dt-test
|
GRENADE_BRANCH=dt-test
|
||||||
|
|
||||||
Grenade includes ``devstack.localrc.base`` and ``devstack.localrc.target``
|
If you need to configure your local devstacks for your specific
|
||||||
for DevStack that are used to customize its behaviour for use with Grenade.
|
environment you can do that by creating ``devstack.localrc``. This
|
||||||
If ``$BASE_DEVSTACK_DIR/localrc`` does not exist the following is
|
will get appended to the stub devstack configs for BASE and TARGET.
|
||||||
performed by ``prep-base``:
|
|
||||||
|
|
||||||
* ``devstack.localrc.base`` is copied to to ``$BASE_DEVSTACK_DIR/localrc``
|
For instance, specifying interfaces for Nova is a common use of
|
||||||
* if ``devstack.localrc`` exists it is appended ``$BASE_DEVSTACK_DIR/localrc``
|
``devstack.localrc``::
|
||||||
|
|
||||||
Similar steps are performed by ``prep-target`` for ``$TARGET_DEVSTACK_DIR``.
|
FLAT_INTERFACE=eth1
|
||||||
|
VLAN_INTERFACE=eth1
|
||||||
``devstack.localrc`` will be appended to both DevStack ``localrc`` files if it
|
|
||||||
exists. ``devstack.localrc`` is not included in Grenade and will not be
|
|
||||||
overwritten it if it exists.
|
|
||||||
|
|
||||||
|
|
||||||
Prepare For An Upgrade Test
|
Run the Upgrade Testing
|
||||||
---------------------------
|
-----------------------
|
||||||
|
|
||||||
::
|
::
|
||||||
|
|
||||||
./grenade.sh
|
./grenade.sh
|
||||||
|
|
||||||
``grenade.sh`` installs DevStack for the **Base** release and
|
Read ``grenade.sh`` for more details of the steps that happen from
|
||||||
runs its ``stack.sh``. Then it creates a 'javelin' project containing
|
here.
|
||||||
some non-default configuration.
|
|
||||||
|
|
||||||
This is roughly the equivalent to::
|
|
||||||
|
|
||||||
grenade/prep-base
|
|
||||||
(cd /opt/stack/grizzly/devstack
|
|
||||||
./stack.sh)
|
|
||||||
grenade/setup-javelin
|
|
||||||
(cd /opt/stack/grizzly/devstack
|
|
||||||
./unstack.sh)
|
|
||||||
# dump databases to $STACK_ROOT/save
|
|
||||||
grenade/prep-target
|
|
||||||
grenade/upgrade-devstack
|
|
||||||
grenade/upgrade-keystone
|
|
||||||
grenade/upgrade-glance
|
|
||||||
grenade/upgrade-nova
|
|
||||||
grenade/upgrade-neutron
|
|
||||||
grenade/upgrade-cinder
|
|
||||||
grenade/upgrade-swift
|
|
||||||
|
|
||||||
The **Target** release of DevStack is installed in a different
|
|
||||||
directory from the **Base** release.
|
|
||||||
|
|
||||||
While the **Base** release is running an imaginary **Javelin** tenant
|
|
||||||
is configured to populate the databases with some non-default content::
|
|
||||||
|
|
||||||
grenade/setup-javelin
|
|
||||||
|
|
||||||
Set up the **javelin** credentials with ``javelinrc``.
|
|
||||||
|
|
||||||
|
|
||||||
Testing Upgrades
|
|
||||||
----------------
|
|
||||||
|
|
||||||
The ``upgrade-*`` scripts are the individual components of the
|
|
||||||
DevStack/Grenade upgrade process. They typically stop any running
|
|
||||||
processes, checkout updated sources, migrate the database, any other
|
|
||||||
tasks that need to be done then start the processes in ``screen``.
|
|
||||||
|
|
||||||
These scripts are written to be idempotent.
|
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user