Merge "Rolling upgrade procedure documentation"
This commit is contained in:
Jenkins
Gerrit Code Review
commit
62fd4d1a6e
@ -7,37 +7,270 @@ Bare Metal Service Upgrade Guide
|
|||||||
This document outlines various steps and notes for operators to consider when
|
This document outlines various steps and notes for operators to consider when
|
||||||
upgrading their ironic-driven clouds from previous versions of OpenStack.
|
upgrading their ironic-driven clouds from previous versions of OpenStack.
|
||||||
|
|
||||||
The ironic service is tightly coupled with the ironic driver that is shipped
|
The Bare Metal (ironic) service is tightly coupled with the ironic driver that
|
||||||
with nova. Some special considerations must be taken into account
|
is shipped with the Compute (nova) service. Some special considerations must be
|
||||||
when upgrading your cloud.
|
taken into account when upgrading your cloud.
|
||||||
|
|
||||||
Plan your Upgrade
|
Both offline and rolling upgrades are supported.
|
||||||
|
|
||||||
|
Plan your upgrade
|
||||||
=================
|
=================
|
||||||
|
|
||||||
|
* Rolling upgrades are available starting with the Pike release; that is, when
|
||||||
|
upgrading from Ocata. This means that it is possible to do an upgrade with
|
||||||
|
minimal to no downtime of the Bare Metal API.
|
||||||
|
|
||||||
|
* Upgrades are only supported between two consecutive named releases.
|
||||||
|
This means that you cannot upgrade Ocata directly into Queens; you need to
|
||||||
|
upgrade into Pike first.
|
||||||
|
|
||||||
* The `release notes <http://docs.openstack.org/releasenotes/ironic/>`_
|
* The `release notes <http://docs.openstack.org/releasenotes/ironic/>`_
|
||||||
should always be read carefully when upgrading the ironic service. Starting
|
should always be read carefully when upgrading the Bare Metal service.
|
||||||
with the Mitaka release, specific upgrade steps and considerations are
|
Specific upgrade steps and considerations are documented there.
|
||||||
well-documented in the release notes.
|
|
||||||
|
|
||||||
* Upgrades are only supported one series at a time, or within a series.
|
* The Bare Metal service should always be upgraded before the Compute service.
|
||||||
|
|
||||||
* Starting with the Liberty release, the ironic service should always be
|
|
||||||
upgraded before the nova service.
|
|
||||||
|
|
||||||
.. note::
|
.. note::
|
||||||
The ironic virt driver in nova always uses a specific version of the
|
The ironic virt driver in nova always uses a specific version of the
|
||||||
ironic REST API. This API version may be one that was introduced in the
|
ironic REST API. This API version may be one that was introduced in the
|
||||||
same development cycle, so upgrading nova first may result in nova being
|
same development cycle, so upgrading nova first may result in nova being
|
||||||
unable to use ironic's API.
|
unable to use the Bare Metal API.
|
||||||
|
|
||||||
* When upgrading ironic, the following steps should always be taken:
|
* Make a backup of your database. Ironic does not support downgrading of the
|
||||||
|
database. Hence, in case of upgrade failure, restoring the database from
|
||||||
|
a backup is the only choice.
|
||||||
|
|
||||||
#. Update ironic code, without restarting services.
|
|
||||||
|
|
||||||
#. Run database migrations.
|
Offline upgrades
|
||||||
|
================
|
||||||
|
|
||||||
#. Restart ironic-conductor and ironic-api services.
|
In an offline (or cold) upgrade, the Bare Metal service is not available
|
||||||
|
during the upgrade, because all the services have to be taken down.
|
||||||
|
|
||||||
|
When upgrading the Bare Metal service, the following steps should always be
|
||||||
|
taken in this order:
|
||||||
|
|
||||||
|
#. upgrade the ironic-python-agent image
|
||||||
|
|
||||||
|
#. update ironic code, without restarting services
|
||||||
|
|
||||||
|
#. run database schema migrations via ``ironic-dbsync upgrade``
|
||||||
|
|
||||||
|
#. restart ironic-conductor and ironic-api services
|
||||||
|
|
||||||
|
Once the above is done, do the following:
|
||||||
|
|
||||||
|
* update any applicable configuration options to stop using any deprecated
|
||||||
|
features or options, and perform any required work to transition to
|
||||||
|
alternatives. All the deprecated features and options will be supported for
|
||||||
|
one release cycle, so should be removed before your next upgrade is
|
||||||
|
performed.
|
||||||
|
|
||||||
|
* upgrade python-ironicclient along with any other services connecting
|
||||||
|
to the Bare Metal service as a client, such as nova-compute
|
||||||
|
|
||||||
|
* run the ``ironic-dbsync online_data_migrations`` command to make sure
|
||||||
|
that data migrations are applied. The command lets you limit
|
||||||
|
the impact of the data migrations with the ``--max-count`` option, which
|
||||||
|
limits the number of migrations executed in one run. You should complete
|
||||||
|
all of the migrations as soon as possible after the upgrade.
|
||||||
|
|
||||||
|
.. warning:: You will not be able to start an upgrade to the next release
|
||||||
|
after this one, until this has been completed for the current
|
||||||
|
release. For example, as part of upgrading from Ocata to Pike,
|
||||||
|
you need to complete Pike's data migrations. If this not done,
|
||||||
|
you will not be able to upgrade to Queens -- it will not be
|
||||||
|
possible to execute Queens' database schema updates.
|
||||||
|
|
||||||
|
|
||||||
|
Rolling upgrades
|
||||||
|
================
|
||||||
|
|
||||||
|
Rolling upgrades are available starting with the Pike release; that is, when
|
||||||
|
upgrading from Ocata. This means that it is possible to do an upgrade with
|
||||||
|
minimal to no downtime of the Bare Metal API.
|
||||||
|
|
||||||
|
Concepts
|
||||||
|
--------
|
||||||
|
|
||||||
|
There are four aspects of the rolling upgrade process to keep in mind:
|
||||||
|
|
||||||
|
* RPC version pinning and versioned object backports
|
||||||
|
* online data migrations
|
||||||
|
* graceful service shutdown
|
||||||
|
* API load balancer draining
|
||||||
|
|
||||||
|
RPC version pinning and versioned object backports
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Through careful RPC versioning, newer services are able to talk to older
|
||||||
|
services (and vice-versa). The ``[DEFAULT]/pin_release_version`` configuration
|
||||||
|
option is used for this. It should be set (pinned) to the release version
|
||||||
|
that the older services are using. The newer services will backport RPC calls
|
||||||
|
and objects to their appropriate versions from the pinned release. If the
|
||||||
|
``IncompatibleObjectVersion`` exception occurs, it is most likely due to an
|
||||||
|
incorrect or unspecified ``[DEFAULT]/pin_release_version`` configuration value.
|
||||||
|
For example, when it is not set to the older release version, no conversion
|
||||||
|
will happen during the upgrade.
|
||||||
|
|
||||||
|
Online data migrations
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
To make database schema migrations less painful to execute, all data migrations
|
||||||
|
are banned from schema migration scripts. The schema migration scripts only
|
||||||
|
update the database schema. Data migrations must be done at the end of the
|
||||||
|
rolling upgrade process, after the schema migration and after the services
|
||||||
|
have been upgraded to the latest release. The data migration is performed
|
||||||
|
using the ``ironic-dbsync online_data_migrations`` command. It can be run in
|
||||||
|
a background process so that it does not interrupt running services.
|
||||||
|
(You would also execute the same command with services turned off if
|
||||||
|
you are doing a cold upgrade).
|
||||||
|
|
||||||
|
This data migration must be completed. If not, you will not be able to
|
||||||
|
upgrade to future releases. For example, if you had upgraded from Ocata to
|
||||||
|
Pike but did not do the data migrations, you will not be able to upgrade from
|
||||||
|
Pike to Queens. (More precisely, you will not be able to apply Queens' schema
|
||||||
|
migrations.)
|
||||||
|
|
||||||
|
Graceful service shutdown
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The ironic-conductor service is a Python process listening for messages on a
|
||||||
|
message queue. When the operator sends the SIGTERM signal to the process, the
|
||||||
|
service stops consuming messages from the queue, so that no additional work is
|
||||||
|
picked up. It completes any outstanding work and then terminates. During this
|
||||||
|
process, messages can be left on the queue and will be processed after the
|
||||||
|
Python process starts back up. This gives us a way to shutdown a service using
|
||||||
|
older code, and start up a service using newer code with minimal impact.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
This was tested with RabbitMQ messaging backend and may vary with other
|
||||||
|
backends.
|
||||||
|
|
||||||
|
API load balancer draining
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
If you are using a load balancer for the ironic-api services, we recommend that
|
||||||
|
you redirect requests to the new API services and drain off of the ironic-api
|
||||||
|
services that have not yet been upgraded.
|
||||||
|
|
||||||
|
Rolling upgrade process
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
To reduce downtime, the services can be upgraded in a rolling fashion. It means
|
||||||
|
upgrading one or a few services at a time. To minimise downtime you need to
|
||||||
|
have HA ironic deployment (at least two ironic-api and two ironic-conductor
|
||||||
|
services) so that when a service instance is being upgraded, the other
|
||||||
|
instances are still running.
|
||||||
|
|
||||||
|
**New features should not be used until after the upgrade has been completed.**
|
||||||
|
|
||||||
|
Before maintenance window
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
* Upgrade the ironic-python-agent image
|
||||||
|
|
||||||
|
* Using the new release (ironic code), execute the required database schema
|
||||||
|
updates by running the database upgrade command: ``ironic-dbsync upgrade``.
|
||||||
|
These schema change operations should have minimal or no effect on
|
||||||
|
performance, and should not cause any operations to fail (but please check
|
||||||
|
the release notes). You can:
|
||||||
|
|
||||||
|
* install the new release on an existing system
|
||||||
|
* install the new release in a new virtualenv or a container
|
||||||
|
|
||||||
|
At this point, new columns and tables may exist in the database. These
|
||||||
|
database schema changes are done in a way that both the old and new (N and
|
||||||
|
N+1) releases can perform operations against the same schema.
|
||||||
|
|
||||||
|
.. note:: Ironic bases its RPC and object storage format versions on the
|
||||||
|
``[DEFAULT]/pin_release_version`` configuration option. It is
|
||||||
|
advisable to automate the deployment of changes in configuration
|
||||||
|
files to make the process less error prone and repeatable.
|
||||||
|
|
||||||
|
During maintenance window
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
#. ironic-conductor services should be upgraded first. Ensure that at least
|
||||||
|
one ironic-conductor service is running at all times. For every
|
||||||
|
ironic-conductor, either one by one or a few at a time:
|
||||||
|
|
||||||
|
* shut down the service. Conductors are load-balanced by the message queue,
|
||||||
|
so the only thing you need to worry about is to shut the service down
|
||||||
|
gracefully (using ``SIGTERM`` signal) to make sure it will finish all the
|
||||||
|
requests being processed before shutting down
|
||||||
|
* upgrade the code and dependencies
|
||||||
|
* set the ``[DEFAULT]/pin_release_version`` configuration option value to
|
||||||
|
the version you are upgrading from (that is, the old version). Based on
|
||||||
|
this setting, the new ironic-conductor services will downgrade any
|
||||||
|
RPC communication and data objects to conform to the old service.
|
||||||
|
For example, if you are upgrading from Ocata to Pike, set this value to
|
||||||
|
``ocata``.
|
||||||
|
* start the service
|
||||||
|
|
||||||
|
#. The next service to upgrade is ironic-api. Ensure that at least one
|
||||||
|
ironic-api service is running at all times. You may want to start another
|
||||||
|
instance of the older ironic-api to handle the load while you are upgrading
|
||||||
|
the original ironic-api services. For every ironic-api service, either one
|
||||||
|
by one or a few at a time:
|
||||||
|
|
||||||
|
* in HA deployment you are typically running them behind a load balancer
|
||||||
|
(for example HAProxy), so you need to take the service instance out of the
|
||||||
|
balancer
|
||||||
|
* shut it down
|
||||||
|
* upgrade the code and dependencies
|
||||||
|
* set the ``[DEFAULT]/pin_release_version`` configuration option value to
|
||||||
|
the version you are upgrading from (that is, the old version). Based on
|
||||||
|
this setting, the new ironic-api services will downgrade any RPC
|
||||||
|
communication and data objects to conform to the old service.
|
||||||
|
For example, if you are upgrading from Ocata to Pike, set this value to
|
||||||
|
``ocata``.
|
||||||
|
* restart the service
|
||||||
|
* add it back into the load balancer
|
||||||
|
|
||||||
|
After upgrading all the ironic-api services, the Bare Metal service is
|
||||||
|
running in the new version but with downgraded RPC communication and
|
||||||
|
database object storage formats. New features can fail when objects are in
|
||||||
|
the downgraded object formats and some internal RPC API functions may still
|
||||||
|
not be available.
|
||||||
|
|
||||||
|
#. For all the ironic-conductor services, one at a time:
|
||||||
|
|
||||||
|
* remove the ``[DEFAULT]/pin_release_version`` configuration option setting
|
||||||
|
* restart the ironic-conductor service
|
||||||
|
|
||||||
|
#. For all the ironic-api services, one at a time:
|
||||||
|
|
||||||
|
* remove the ``[DEFAULT]/pin_release_version`` configuration option setting
|
||||||
|
* restart the ironic-api service
|
||||||
|
|
||||||
|
After maintenance window
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Now that all the services are upgraded, the system is able to use the latest
|
||||||
|
version of the RPC protocol and able to access all the features of the new
|
||||||
|
release.
|
||||||
|
|
||||||
|
* Update any applicable configuration options to stop using any deprecated
|
||||||
|
features or options, and perform any required work to transition to
|
||||||
|
alternatives. All the deprecated features and options will be supported for
|
||||||
|
one release cycle, so should be removed before your next upgrade is
|
||||||
|
performed.
|
||||||
|
|
||||||
|
* Upgrade ``python-ironicclient`` along with other services connecting
|
||||||
|
to the Bare Metal service as a client, such as nova-compute.
|
||||||
|
|
||||||
|
* Run the ``ironic-dbsync online_data_migrations`` command to make sure
|
||||||
|
that data migrations are applied. The command lets you limit
|
||||||
|
the impact of the data migrations with the ``--max-count`` option, which
|
||||||
|
limits the number of migrations executed in one run. You should complete
|
||||||
|
all of the migrations as soon as possible after the upgrade.
|
||||||
|
|
||||||
|
Note that you will not be able to start an upgrade to the next release after
|
||||||
|
this one, until this has been completed for the current release. For example,
|
||||||
|
as part of upgrading from Ocata to Pike, you need to complete Pike's data
|
||||||
|
migrations. If this not done, you will not be able to upgrade to Queens --
|
||||||
|
it will not be possible to execute Queens' database schema updates.
|
||||||
|
|
||||||
Upgrading from Ocata to Pike
|
Upgrading from Ocata to Pike
|
||||||
============================
|
============================
|
||||||
|
|||||||
@ -0,0 +1,4 @@
|
|||||||
|
---
|
||||||
|
features:
|
||||||
|
- Adds support for rolling upgrades, starting from upgrading Ocata to Pike.
|
||||||
|
For details, see http://docs.openstack.org/ironic/admin/upgrade-guide.html.
|
||||||
Loading…
Reference in New Issue
Block a user