265 lines
8.5 KiB
ReStructuredText
265 lines
8.5 KiB
ReStructuredText
==============================
|
|
Modular Grenade Architecture
|
|
==============================
|
|
|
|
Grenade was originally created to demonstrate some level of upgrade
|
|
capacity for OpenStack projects. Originally this just included a small
|
|
number of services.
|
|
|
|
Proposed new basic flow:
|
|
|
|
- setup_grenade
|
|
- all the magic setup involved around err traps and filehandle redirects
|
|
- setup devstack trees
|
|
- setup_base
|
|
- run stack.sh to build the correct base environment
|
|
- verify_base
|
|
- for project in projects; do verify_project; done
|
|
- resources.sh create
|
|
- resources.sh verify pre-upgrade
|
|
- shutdown
|
|
- for project in projects; do shutdown; done
|
|
- snapshot.sh pre_upgrade (NOT YET IMPLEMENTED)
|
|
- resources.sh verify_noapi pre-upgrade
|
|
- upgrade ...
|
|
- resources.sh verify post-upgrade
|
|
- verify_target
|
|
- resources.sh destroy
|
|
|
|
|
|
|
|
Modular Components
|
|
==================
|
|
|
|
Assuming the following tree in target projects::
|
|
|
|
devstack/ - devstack plugin directory
|
|
upgrade/ - upgrade scripts
|
|
settings - adds settings for the upgrade path
|
|
upgrade.sh
|
|
snapshot.sh - snapshots the state of the service, typically a
|
|
database dump (NOT YET IMPLEMENTED)
|
|
from-juno/ - per release
|
|
within-juno/
|
|
from-kilo/
|
|
within-kilo/
|
|
resources.sh
|
|
|
|
|
|
This same modular structure exists in the grenade tree with::
|
|
grenade/
|
|
projects/
|
|
10_ceilometer/
|
|
settings
|
|
upgrade.sh
|
|
|
|
resources.sh
|
|
=================
|
|
|
|
resources.sh is a per-service resource create / verify / destroy
|
|
interface. What a service does inside a script is up to them.
|
|
|
|
You can assume your resource script will only be called if your
|
|
service is running in an upgrade environment. The script should return
|
|
zero on success for actions, and nonzero on failure.
|
|
|
|
Calling Interface
|
|
-----------------
|
|
|
|
The following is the supported calling interface
|
|
|
|
- resources.sh early_create
|
|
|
|
creates a set of sample resources that should survive very early in
|
|
the upgrade process. This should only be used for horizontal
|
|
resources that impact other services, that *have* to be available
|
|
before they do any of their setup. For instance setup of neutron
|
|
networks.
|
|
|
|
Do not use the phase unless you really know why ``create`` will not
|
|
work for you.
|
|
|
|
- resources.sh create
|
|
|
|
creates a set of sample resources that should survive
|
|
upgrade. Script should exit with a nonzero exit code if any
|
|
resources could not be created.
|
|
|
|
Example: create an instance in nova or a volume in cinder
|
|
|
|
- resources.sh verify (pre-upgrade|post-upgrade)
|
|
|
|
verify that the resources were created. Services are running at this
|
|
point, and the APIs may be expected to work. The second argument
|
|
indicates whether we are pre-upgrade or post-upgrade.
|
|
|
|
Example: use the nova command to verify that the test instance is
|
|
still ACTIVE, or the cinder command to verify that the volume is
|
|
still available.
|
|
|
|
- resources.sh verify_noapi
|
|
|
|
verify that the resources are still present. This is called in the
|
|
phase where services are stopped, and APIs are expected to not be
|
|
accessible. Resource verification at this phase my require probing
|
|
underlying components to make sure nothing has gone awry during
|
|
service shutdown. The second argument indicates whether we are
|
|
pre-upgrade or post-upgrade.
|
|
|
|
Example: check with libvirt to make sure the instance is actually
|
|
created and running. Bonus points for being able to ping the
|
|
instance, or otherwise check its live-ness. With cinder, checking
|
|
that the LVM volume exists and looks reasonable.
|
|
|
|
- resources.sh destroy
|
|
|
|
Resource scripts should be responsible and cleanup all their
|
|
resources when asked to destroy.
|
|
|
|
Calling Sequence
|
|
----------------
|
|
|
|
The calling sequence during a grenade run looks as follows:
|
|
|
|
- # start old side
|
|
- create (create will be called during the working old side)
|
|
- verify pre-upgrade
|
|
- # shutdown all services
|
|
- verify_noapi pre-upgrade
|
|
- # upgrade and start all services
|
|
- verify post-upgrade
|
|
- destroy
|
|
|
|
The important thing to remember is verify/verify_noapi will be called
|
|
multiple times, with multiple different versions of OpenStack. Those
|
|
phases of the script must not be rerunnable multiple times.
|
|
|
|
While create / destroy are only going to be called once in the current
|
|
interface, bonus points for also making those idempotent for
|
|
resiliancy in testing.
|
|
|
|
**Per-release upgrade scripts**
|
|
|
|
There are times when :ref:`exceptional <upgrade-exceptions>` manual upgrade
|
|
steps must be performed to get from one release to the next, or even within
|
|
the same release. Grenade supports this with per-release scripts found in
|
|
each project, e.g.::
|
|
|
|
projects/
|
|
60_nova/
|
|
from-ocata/
|
|
upgrade-nova
|
|
|
|
Regarding the sequence of when these per-release scripts are called, any
|
|
``within-$base`` script should be run *before* installing new code, and any
|
|
``from-$base`` script should be run *after* installing new code but before
|
|
starting the services with the new code. This is because configuration or
|
|
database changes may be needed before the upgraded code is started.
|
|
|
|
Supporting Methods
|
|
------------------
|
|
|
|
In order to assist with the checks listed the following functions
|
|
exist::
|
|
|
|
resource_save project key value
|
|
resource_get project key
|
|
|
|
This allow resource scripts to have memory, and keep track of things
|
|
like the allocated IP addresses, IDs, and other non deterministic data
|
|
that is returned from OpenStack API calls.
|
|
|
|
Environment
|
|
-----------
|
|
|
|
Resource scripts get called in a specific environment already set:
|
|
|
|
- TOP_DIR - will be set to the root of the devstack directory for the
|
|
BASE version of devstack incase this is needed to find files like a
|
|
working ``openrc``
|
|
|
|
- GRENADE_DIR - the root directory of the grenade directory.
|
|
|
|
The following snippet will give you access to both the grenade and
|
|
TARGET devstack functions::
|
|
|
|
source $GRENADE_DIR/grenaderc
|
|
source $GRENADE_DIR/functions
|
|
|
|
|
|
Best Practices
|
|
--------------
|
|
|
|
Do as many actions as non admin as possible. As early as you can in
|
|
your resource script it's worth allocating a user/project for the
|
|
script to run as. This ensures isolation against other scripts, and
|
|
ensures that actions don't only work because admin gets to bypass
|
|
safeties.
|
|
|
|
Test side effects, not just API actions. The point of these resource
|
|
survival scripts is to test that things created beyond the API / DB
|
|
interaction still work later. Just testing that data can be stored /
|
|
retrieved from the database isn't very interesting, and should be
|
|
covered other places. The value in the resource scripts is these side
|
|
effects. Actual VMs running, actual iscsi targets running, etc. And
|
|
ensuring these things are not disrupted when the control plane is
|
|
shifted out from under them.
|
|
|
|
Out of Tree Plugins
|
|
===================
|
|
|
|
A grenade plugin can be hosted out of tree in a project tree, similar
|
|
to external devstack plugins. There are a few subtle differences when
|
|
this happens.
|
|
|
|
The plugin structure will live under ``$project/devstack/upgrade/``
|
|
directory.
|
|
|
|
The plugin is enabled by adding::
|
|
|
|
enable_grenade_plugin <$project> <giturl> [branch]
|
|
|
|
To ``pluginrc`` in the ``GRENADE_DIR``. An additional rc file was
|
|
required due to sequencing of when plugin functions become available.
|
|
|
|
Note: when running a job based on the ``grenade-base`` job,
|
|
for each devstack plugin defined using the ``devstack_plugins``,
|
|
the corresponding grenade plugin is enabled automatically.
|
|
|
|
|
|
Changing Devstack Localrc
|
|
-------------------------
|
|
|
|
There is also a mechanism that allows a ``settings`` file change the
|
|
devstack localrc files with the ``devstack_localrc`` function.
|
|
|
|
::
|
|
devstack_localrc <base|target> arbitrary stuff to add
|
|
|
|
Which will take all the rest of the stuff on that line and add it to
|
|
the localrc for either the base or target devstack.
|
|
|
|
Please note that ``devstack_localrc`` only works when grenade
|
|
performs the configuration of the devstack settings and runs devstack
|
|
against the base target. When GRENADE_USE_EXTERNAL_DEVSTACK is set
|
|
to True, as it happens on the Zuul grenade jobs where devstack is
|
|
configured and executed before grenade, the function has no effect.
|
|
|
|
Example settings
|
|
----------------
|
|
|
|
The following is a reasonable example ``settings`` for out of tree
|
|
plugin::
|
|
|
|
register_project_for_upgrade heat
|
|
register_db_to_save heat
|
|
devstack_localrc base enable_service h-api h-api-cfn h-api-cw h-eng heat
|
|
devstack_localrc target enable_service h-api h-api-cfn h-api-cw h-eng heat
|
|
|
|
This registers the project for upgrade, symbolicly enables the heat
|
|
database for dump during upgrade, and adds the heat services into the
|
|
service list for base and target.
|
|
|
|
It's expected that most ``settings`` files for out of tree plugins
|
|
will need equivalent lines.
|