openstack/glance
Code Issues Proposed changes

Revise database rolling upgrade documentation

Browse Source 
- mark zero-downtime-db-upgrade as EXPERIMENTAL for queens
- clarify the relation between the E-M-C strategy and
  zero-downtime db upgrades
- add note that for MySQL, using the glance-manage expand or
  glance-manage contract command requires that the glance
  is granted SUPER privileges
- add note to contributor docs about checking the trigger
  flag in expand and contract scripts

Co-authored-by: Abhishek Kekane <akekane@redhat.com>
Co-authored-by: Brian Rosmaita <rosmaita.fossdev@gmail.com>

Change-Id: I5af4a1428b89ecb05a1be9c420c5f0afc05b9a95
Closes-Bug: #1750555
This commit is contained in:
Abhishek Kekane 2018-02-20 14:47:43 +00:00 committed by Brian Rosmaita
parent f2f8637d9c
commit d500b3f883
3 changed files with 101 additions and 22 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
doc/source/admin/rollingupgrades.rst
View File

@ -19,6 +19,23 @@ Rolling Upgrades
.. note:: The Rolling Upgrades feature is EXPERIMENTAL and its use in
production systems is currently **not supported**.
This statement remains true for the Queens release of Glance. What
is the holdup, you ask? Before asserting that the feature is fully
supported, the Glance team needs to have automated tests that perform
rolling upgrades in the OpenStack Continuous Integration gates. The
Glance project team has not had sufficient testing and development
resources in recent cycles to prioritize this work.
The Glance project team is committed to the stability of Glance. As
part of OpenStack, we are committed to `The Four Opens`_. If the
ability to perform rolling upgrades in production systems is
important to you, feel free to participate in the Glance community to
help coordinate and drive such an effort. (We gently remind you that
"participation" includes providing testing and development
resources.)
.. _`The Four Opens`: https://governance.openstack.org/tc/reference/opens.html
Scope of this document
----------------------
@ -83,6 +100,12 @@ Following is the process to upgrade Glance with zero downtime:
glance-manage db expand
.. warning::
For MySQL, using the ``glance-manage db_expand`` command requires that
you either grant your glance user ``SUPER`` privileges, or run
``set global log_bin_trust_function_creators=1;`` in mysql beforehand.
5. Then, also on the NEW NODE, perform the data migrations using the command::
glance-manage db migrate

81
doc/source/admin/zero-downtime-db-upgrade.rst
View File

@ -17,11 +17,11 @@ Zero-Downtime Database Upgrades
===============================
.. warning::
This feature is EXPERIMENTAL in the Ocata and Pike releases.
This feature is EXPERIMENTAL in the Ocata, Pike and Queens releases.
We encourage operators to try it out, but its use in production
environments is currently NOT SUPPORTED.
A zero-downtime database upgrade enables true rolling upgrades of the Glance
A *zero-downtime database upgrade* enables true rolling upgrades of the Glance
nodes in your cloud's control plane. At the appropriate point in the upgrade,
you can have a mixed deployment of release *n* (for example, Ocata) and release
*n-1* (for example, Newton) Glance nodes, take the *n-1* release nodes out of
@ -31,6 +31,13 @@ leaving all Glance nodes in your cloud at release *n*.
That's a rough sketch of how a rolling upgrade would work. For full details,
see :ref:`rolling-upgrades`.
.. note::
When we speak of a "database upgrade", we are simply talking about changing
the database schema and its data from the version used in OpenStack release
*n* (say, Pike) to the version used in OpenStack release *n+1* (say,
Queens). We are **not** talking about upgrading the database management
software.
.. note::
Downgrading a database is not supported. See :ref:`downgrades` for more
information.
@ -38,18 +45,45 @@ see :ref:`rolling-upgrades`.
The Expand-Migrate-Contract Cycle
---------------------------------
For Glance, a zero-downtime database upgrade has three phases:
It's possible to characterize three phases of a database upgrade:
1. **Expand**: in this phase, new columns, tables, indexes, or triggers are
added to the database.
1. **Expand**: in this phase, new columns, tables, indexes, are added to the
database.
2. **Migrate**: in this phase, data is migrated to the new columns or tables.
3. **Contract**: in this phase, the "old" tables or columns (and any database
triggers used during the migration), which are no longer in use, are removed
from the database.
3. **Contract**: in this phase, the "old" tables or columns (which are no
longer in use) are removed from the database.
The above phases are abbreviated as an **E-M-C** database upgrade.
The "legacy" Glance database migrations performed these phases as part of a
single monolithic upgrade script. Currently, the Glance project creates a
separate script for each the three parts of the cycle. We call such an
upgrade an **E-M-C** database migration.
Zero-Downtime Database Upgrade
------------------------------
The E-M-C strategy can be performed offline when Glance is not using the
database. With some adjustments, however, the E-M-C strategy can be applied
online when the database is in use, making true rolling upgrades possible.
.. note::
Don't forget that zero-downtime database upgrades are currently considered
experimental and their use in production environments is NOT SUPPORTED.
A zero-downtime database upgrade takes place as part of a :ref:`rolling upgrade
strategy <rolling-upgrades>` for upgrading your entire Glance installation. In
such a situation, you want to upgrade to release *n* of Glance (say, Queens)
while your release *n-1* API nodes are still running Pike. To make this
possible, in the **Expand** phase, database triggers can be added to the
database to keep the data in "old" and "new" columns synchronized. Likewise,
after all data has been migrated and all Glance nodes have been updated to
release *n* code, these triggers are deleted in the **Contract** phase.
.. note::
Unlike the E-M-C scripts, database triggers are particular to each database
technology. That's why the Glance project currently provides experimental
support only for MySQL.
New Database Version Identifiers
--------------------------------
@ -59,9 +93,8 @@ database becomes more complicated since it must reflect knowledge of what point
in the E-M-C cycle the upgrade has reached. To make this evident, the
identifier explicitly contains 'expand' or 'contract' as part of its name.
Thus the ``ocata01`` migration (that is, the migration that's currently used in
the fully supported upgrade path) has two identifiers associated with it for
zero-downtime upgrades: ``ocata_expand01`` and ``ocata_contract01``.
Thus the Ocata cycle migration has two identifiers associated with it:
``ocata_expand01`` and ``ocata_contract01``.
During the upgrade process, the database is initially marked with
``ocata_expand01``. Eventually, after completing the full upgrade process, the
@ -100,12 +133,24 @@ version record would go through the following progression:
Database Upgrade
----------------
In order to enable the E-M-C database upgrade cycle, and to enable Glance
rolling upgrades, the ``glance-manage`` tool has been augmented to include the
following operations.
For offline database upgrades, the ``glance-manage`` tool still has the
``glance-manage db sync`` command. This command will execute the expand,
migrate, and contract scripts for you, just as if they were contained in
a single script.
In order to enable zero-downtime database upgrades, the ``glance-manage`` tool
has been augmented to include the following operations so that you can
explicitly manage the upgrade.
.. warning::
For MySQL, using the ``glance-manage db expand`` or
``glance-manage db contract`` command requires that
you either grant your glance user ``SUPER`` privileges, or run
``set global log_bin_trust_function_creators=1;`` in mysql beforehand.
Expanding the Database
----------------------
~~~~~~~~~~~~~~~~~~~~~~
::
glance-manage db expand
@ -116,7 +161,7 @@ any new services are started.
Migrating the Data
------------------
~~~~~~~~~~~~~~~~~~
::
glance-manage db migrate
@ -127,7 +172,7 @@ are started.
Contracting the Database
------------------------
~~~~~~~~~~~~~~~~~~~~~~~~
::
glance-manage db contract

19
doc/source/contributor/database_migrations.rst
View File

@ -282,10 +282,21 @@ Data Migrations
NOTES
-----
* Starting in Ocata, Glance needs every database migration to include both
monolithic and Expand-Migrate-Contract (E-M-C) style migrations. At some
point in Pike, E-M-C migrations will be made default. At that point, it
would be no longer required to include monolithic migration script.
* In Ocata and Pike, Glance required every database migration to include
both monolithic and Expand-Migrate-Contract (E-M-C) style migrations. In
Queens, E-M-C migrations became the default and a monolithic migration
script is no longer required.
In Queens, the glance-manage tool was refactored so that the ``glance-manage
db sync`` command runs the expand, migrate, and contract scripts "under
the hood". From the viewpoint of the operator, there is no difference
between having a single monolithic script and having three scripts.
Since we are using the same scripts for offline and online (zero-downtime)
database upgrades, as a developer you have to pay attention in your scripts
to determine whether you need to add/remove triggers in the expand/contract
scripts. See the changes to the ocata scripts in
https://review.openstack.org/#/c/544792/ for an example of how to do this.
* Alembic is a database migration engine written for SQLAlchemy. So, any
migration script written for SQLAlchemy Migrate should work with Alembic as