openstack/glance
Code Issues Proposed changes

Update developer docs for rolling upgrades

Browse Source 
Add more detailed information about Alembic migrations and the
EXPERIMENTAL zero-downtime database upgrade process introduced
in Ocata plus initial rolling upgrade operator docs.

Change-Id: Ic30cd5872962df0fdfa9064391d1f7d04104bc49
This commit is contained in:
Brian Rosmaita 2017-02-04 22:24:15 -05:00
parent 2a764c9d5e
commit 5393599e0b
4 changed files with 351 additions and 39 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

65
doc/source/db-sqlalchemy-migrate.rst Normal file
View File

@ -0,0 +1,65 @@
..
Copyright 2012 OpenStack Foundation
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Legacy Database Management
==========================
.. note::
This page applies only to Glance releases prior to Ocata. From Ocata
onward, please see :doc:`db`.
The default metadata driver for Glance uses sqlalchemy, which implies there
exists a backend database which must be managed. The ``glance-manage`` binary
provides a set of commands for making this easier.
The commands should be executed as a subcommand of 'db':
glance-manage db <cmd> <args>
Sync the Database
-----------------
glance-manage db sync <version> <current_version>
Place a database under migration control and upgrade, creating it first if necessary.
Determining the Database Version
--------------------------------
glance-manage db version
This will print the current migration level of a Glance database.
Upgrading an Existing Database
------------------------------
glance-manage db upgrade <VERSION>
This will take an existing database and upgrade it to the specified VERSION.
Downgrading an Existing Database
--------------------------------
Upgrades involve complex operations and can fail. Before attempting any
upgrade, you should make a full database backup of your production data. As of
Kilo, database downgrades are not supported, and the only method available to
get back to a prior database version is to restore from backup[1].
[1]: http://docs.openstack.org/ops-guide/ops-upgrades.html#perform-a-backup

214
doc/source/db.rst
View File

@ -1,7 +1,4 @@
.. ..
Copyright 2012 OpenStack Foundation
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain not use this file except in compliance with the License. You may obtain
a copy of the License at a copy of the License at
@ -17,7 +14,7 @@
Database Management Database Management
=================== ===================
The default metadata driver for Glance uses sqlalchemy, which implies there The default metadata driver for Glance uses `SQLAlchemy`_, which implies there
exists a backend database which must be managed. The ``glance-manage`` binary exists a backend database which must be managed. The ``glance-manage`` binary
provides a set of commands for making this easier. provides a set of commands for making this easier.
@ -25,6 +22,56 @@ The commands should be executed as a subcommand of 'db':
glance-manage db <cmd> <args> glance-manage db <cmd> <args>
.. note::
In the Ocata release (14.0.0), the database migration engine was changed
from *SQLAlchemy Migrate* to *Alembic*. This necessitated some changes in
the ``glance-manage`` tool. While the user interface has been kept as
similar as possible, the ``glance-manage`` tool included with the Ocata and
more recent releases is incompatible with the "legacy" tool. If you are
consulting these documents for information about the ``glance-manage`` tool
in the Newton or earlier releases, please see the
:doc:`db-sqlalchemy-migrate` page.
.. _`SQLAlchemy`: http://www.sqlalchemy.org/
Migration Scripts
-----------------
The migration scripts are stored in the directory:
``glance/db/sqlalchemy/alembic_migrations/versions``
As mentioned above, these scripts utilize the Alembic migration engine, which
was first introduced in the Ocata release. All database migrations up through
the Liberty release are consolidated into one Alembic migration script named
``liberty_initial``. Mitaka migrations are retained, but have been rewritten
for Alembic and named using the new naming convention.
A fresh Glance installation will apply the following
migrations:
* ``liberty-initial``
* ``mitaka01``
* ``mitaka02``
* ``ocata01``
.. note::
The "old-style" migration scripts have been retained in their `current
directory`_ in the Ocata release so that interested operators can correlate
them with the new migrations. This directory will be removed in future
releases.
In particular, the "old-style" script for the Ocata migration,
`045_add_visibility.py`_ is retained for operators who are conversant in
SQLAlchemy Migrate and are interested in comparing it with a "new-style"
Alembic migration script. The Alembic script, which is the one actually
used to do the upgrade to Ocata, is
`ocata01_add_visibility_remove_is_public.py`_.
.. _`current directory`: http://git.openstack.org/cgit/openstack/glance/tree/glance/db/sqlalchemy/migrate_repo/versions?h=stable/ocata
.. _`045_add_visibility.py`: http://git.openstack.org/cgit/openstack/glance/tree/glance/db/sqlalchemy/migrate_repo/versions/045_add_visibility.py?h=stable/ocata
.. _`ocata01_add_visibility_remove_is_public.py`: http://git.openstack.org/cgit/openstack/glance/tree/glance/db/sqlalchemy/alembic_migrations/versions/ocata01_add_visibility_remove_is_public.py?h=stable/ocata
Sync the Database Sync the Database
----------------- -----------------
@ -34,26 +81,20 @@ Sync the Database
Place an existing database under migration control and upgrade it to the Place an existing database under migration control and upgrade it to the
specified VERSION or to the latest migration level if VERSION is not specified. specified VERSION or to the latest migration level if VERSION is not specified.
.. note:: Prior to Ocata release the database version was a numeric value. .. note::
For example: for the Newton release, the latest migration level was ``44``.
Starting with Ocata, database version will be a revision name Prior to Ocata release the database version was a numeric value. For
corresponding to the latest migration included in the release. For the example: for the Newton release, the latest migration level was ``44``.
Ocata release, there is only one database migration and it is identified Starting with Ocata, database version is a revision name corresponding to
by revision ``ocata01``. So, the database version for Ocata release would the latest migration included in the release. For the Ocata release, there
be ``ocata01``. is only one database migration and it is identified by revision
``ocata01``. So, the database version for Ocata release is ``ocata01``.
This naming convention will change slightly with the introduction of
zero-downtime upgrades, which is EXPERIMENTAL in Ocata, but is projected to
be the official upgrade method beginning with the Pike release. See
:ref:`zero-downtime` below for more information.
However, with the introduction of zero-downtime upgrades, database version
will be a composite version including both expand and contract revisions.
To achieve zero-downtime upgrades, we split the ``ocata01`` migration into
``ocata_expand01`` and ``ocata_contract01``. During the upgrade process,
the database would initially be marked with ``ocata_expand01`` and
eventually after completing the full upgrade process, the database will be
marked with ``ocata_contract01``. So, instead of one database version, an
operator will see a composite database version that will have both expand
and contract versions. A database will be considered at Ocata version only
when both expand and contract revisions are at the latest revisions
possible. For a successful Ocata rolling upgrade, the database should be
marked with both ``ocata_expand01``, ``ocata_contract01``.
Determining the Database Version Determining the Database Version
-------------------------------- --------------------------------
@ -70,15 +111,120 @@ Upgrading an Existing Database
This will take an existing database and upgrade it to the specified VERSION. This will take an existing database and upgrade it to the specified VERSION.
.. _downgrades:
Downgrading an Existing Database
--------------------------------
Upgrades involve complex operations and can fail. Before attempting any
upgrade, you should make a full database backup of your production data. As of
Kilo, database downgrades are not supported, and the only method available to
get back to a prior database version is to restore from backup [1].
[1]: http://docs.openstack.org/ops-guide/ops-upgrades.html#perform-a-backup
.. _zero-downtime:
Zero-Downtime Database Upgrades
===============================
.. warning::
This feature is EXPERIMENTAL in the Ocata release. We encourage operators
to try it out, but its use in production environments is currently NOT
SUPPORTED.
A zero-downtime database upgrade enables true rolling upgrades of the Glance
nodes in your cloud's control plane. At the appropriate point in the upgrade,
you can have a mixed deployment of release *n* (for example, Ocata) and release
*n-1* (for example, Newton) Glance nodes, take the *n-1* release nodes out of
rotation, allow them to drain, and then take them out of service permanently,
leaving all Glance nodes in your cloud at release *n*.
That's a rough sketch of how a rolling upgrade would work. For full details,
see :doc:`rollingupgrades`.
.. note::
Downgrading a database is not supported. See :ref:`downgrades` for more
information.
The Expand-Migrate-Contract Cycle
---------------------------------
For Glance, a zero-downtime database upgrade has three phases:
1. **Expand**: in this phase, new columns, tables, indexes, or triggers are
added to the database.
2. **Migrate**: in this phase, data is migrated to the new columns or tables.
3. **Contract**: in this phase, the "old" tables or columns (and any database
triggers used during the migration), which are no longer in use, are removed
from the database.
The above phases are abbreviated as an **E-M-C** database upgrade.
New Database Version Identifiers
--------------------------------
In order to perform zero-downtime upgrades, the version identifier of a
database becomes more complicated since it must reflect knowledge of what point
in the E-M-C cycle the upgrade has reached. To make this evident, the
identifier explicitly contains 'expand' or 'contract' as part of its name.
Thus the ``ocata01`` migration (that is, the migration that's currently used in
the fully supported upgrade path) has two identifiers associated with it for
zero-downtime upgrades: ``ocata_expand01`` and ``ocata_contract01``.
During the upgrade process, the database is initially marked with
``ocata_expand01``. Eventually, after completing the full upgrade process, the
database will be marked with ``ocata_contract01``. So, instead of one database
version, an operator will see a composite database version that will have both
expand and contract versions. A database will be considered at Ocata version
only when both expand and contract revisions are at the latest revisions. For
a successful Ocata zero-downtime upgrade, for example, the database will be
marked with both ``ocata_expand01``, ``ocata_contract01``.
In the case in which there are multiple changes in a cycle, the database
version record would go through the following progression:
+-------+--------------------------------------+-------------------------+
| stage | database identifier | comment |
+=======+======================================+=========================+
| E | ``bexar_expand01`` | upgrade begins |
+-------+--------------------------------------+-------------------------+
| E | ``bexar_expand02`` | |
+-------+--------------------------------------+-------------------------+
| E | ``bexar_expand03`` | |
+-------+--------------------------------------+-------------------------+
| M | ``bexar_expand03`` | bexar_migrate01 occurs |
+-------+--------------------------------------+-------------------------+
| M | ``bexar_expand03`` | bexar_migrate02 occurs |
+-------+--------------------------------------+-------------------------+
| M | ``bexar_expand03`` | bexar_migrate03 occurs |
+-------+--------------------------------------+-------------------------+
| C | ``bexar_expand03, bexar_contract01`` | |
+-------+--------------------------------------+-------------------------+
| C | ``bexar_expand03, bexar_contract02`` | |
+-------+--------------------------------------+-------------------------+
| C | ``bexar_expand03, bexar_contract03`` | upgrade completed |
+-------+--------------------------------------+-------------------------+
Database Upgrade
----------------
In order to enable the E-M-C database upgrade cycle, and to enable Glance
rolling upgrades, the ``glance-manage`` tool has been augmented to include the
following operations.
Expanding the Database Expanding the Database
---------------------- ----------------------
glance-manage db expand glance-manage db expand
This will run the expansion phase of a rolling upgrade process. This will run the expansion phase of a rolling upgrade process. Database
Database expansion should be run as the first step in the rolling upgrade expansion should be run as the first step in the rolling upgrade process before
process before any new services are started. any new services are started.
Migrating the Data Migrating the Data
@ -86,9 +232,9 @@ Migrating the Data
glance-manage db migrate glance-manage db migrate
This will run the data migrate phase of a rolling upgrade process. This will run the data migrate phase of a rolling upgrade process. Database
Database migration should be run after database expansion and before migration should be run after database expansion but before any new services
database contraction has been performed. are started.
Contracting the Database Contracting the Database
@ -99,13 +245,3 @@ Contracting the Database
This will run the contraction phase of a rolling upgrade process. This will run the contraction phase of a rolling upgrade process.
Database contraction should be run as the last step of the rolling upgrade Database contraction should be run as the last step of the rolling upgrade
process after all old services are upgraded to new ones. process after all old services are upgraded to new ones.
Downgrading an Existing Database
--------------------------------
Upgrades involve complex operations and can fail. Before attempting any
upgrade, you should make a full database backup of your production data. As of
Kilo, database downgrades are not supported, and the only method available to
get back to a prior database version is to restore from backup[1].
[1]: http://docs.openstack.org/ops-guide/ops-upgrades.html#perform-a-backup

1
doc/source/index.rst
View File

@ -89,6 +89,7 @@ Operating Glance
controllingservers controllingservers
db db
rollingupgrades
cache cache
notifications notifications

110
doc/source/rollingupgrades.rst Normal file
View File

@ -0,0 +1,110 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Rolling Upgrades
================
.. note:: The Rolling Upgrades feature is EXPERIMENTAL and its use in
production systems is currently **not supported**.
Scope of this document
----------------------
This page describes one way to perform a rolling upgrade from Newton to Ocata
for a particular configuration of Glance services. There may be other ways to
perform a rolling upgrade from Newton to Ocata for other configurations of
Glance services, but those are beyond the scope of this document. For the
experimental rollout of rolling upgrades, we describe only the following
simple case.
Prerequisites
-------------
* MySQL/MariaDB 5.5 or later
* Glance running Images API v2 only
* Glance not using the Glance Registry
* Multiple Glance nodes
* A load balancer or some other type of redirection device is being used
in front of the Glance nodes in such a way that a node can be dropped
out of rotation, that is, that Glance node continues running the Glance
service but is no longer having requests routed to it
Procedure
---------
Following is the process to upgrade Glance with zero downtime:
1. Backup the Glance database.
2. Choose an arbitrary Glance node or provision a new node to install the new
release. If an existing Glance node is chosen, gracefully stop the Glance
services. In what follows, this node will be referred to as the NEW NODE.
.. _Stop the Glance processes gracefully:
.. note::
**Gracefully stopping services**
Before stopping the Glance processes on a node, one may choose to wait until
all the existing connections drain out. This could be achieved by taking the
node out of rotation, that is, by ensuring that requests are no longer
routed to that node. This way all the requests that are currently being
processed will get a chance to finish processing. However, some Glance
requests like uploading and downloading images may last a long time. This
increases the wait time to drain out all connections and consequently the
time to upgrade Glance completely. On the other hand, stopping the Glance
services before the connections drain out will present the user with errors.
While arguably this is not downtime given that Images API requests are
continually being serviced by other nodes, this is nonetheless an unpleasant
user experience for the user whose in-flight request has terminated in an
error. Hence, an operator must be judicious when stopping the services.
3. Upgrade the NEW NODE with new release and update the configuration
accordingly. **DO NOT** start the Glance services on the NEW NODE at
this time.
4. Using the NEW NODE, expand the database using the command
``glance-manage db expand``.
5. Then, also on the NEW NODE, perform the data migrations using the command
``glance-manage db migrate``.
*The data migrations must be completed before you proceed to the next step.*
6. Start the Glance processes on the NEW NODE. It is now ready to receive
traffic from the load balancer.
7. Taking one node at a time from the remaining nodes, for each node:
a. `Stop the Glance processes gracefully`_ as described in Step 2, above.
*Do not proceed until the "old" Glance services on the node have been
completely shut down.*
b. Upgrade the node to the new release (and corresponding configuration).
c. Start the updated Glance processes on the upgraded node.
8. After **ALL** of the nodes have been upgraded to run the new Glance
services, and there are **NO** nodes running any old Glance services,
contract the database by running the command
``glance manage db contract``
from any one of the upgraded nodes.