docs: Add documentation on database migrations
Alembic does lots of new things. Provide docs for how to use this. We also improve upgrade docs slightly, removing references to ancient reviews that are no longer really helpful as well as calling out our N -> N+1 constraint. Change-Id: I3760b82ce3bd71aa0a760d7137d69dfa3f29dc1d Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
This commit is contained in:
Stephen Finucane
parent
1ba2c1c55d
commit
a7584ec1a5
|
|
@ -62,12 +62,13 @@ redirectmatch 301 ^/nova/([^/]+)/testing/libvirt-numa.html$ /nova/$1/contributor
|
|||
redirectmatch 301 ^/nova/([^/]+)/testing/serial-console.html$ /nova/$1/contributor/testing/serial-console.html
|
||||
redirectmatch 301 ^/nova/([^/]+)/testing/zero-downtime-upgrade.html$ /nova/$1/contributor/testing/zero-downtime-upgrade.html
|
||||
redirectmatch 301 ^/nova/([^/]+)/threading.html$ /nova/$1/reference/threading.html
|
||||
redirectmatch 301 ^/nova/([^/]+)/upgrade.html$ /nova/$1/user/upgrade.html
|
||||
redirectmatch 301 ^/nova/([^/]+)/upgrade.html$ /nova/$1/admin/upgrades.html
|
||||
redirectmatch 301 ^/nova/([^/]+)/user/aggregates.html$ /nova/$1/admin/aggregates.html
|
||||
redirectmatch 301 ^/nova/([^/]+)/user/cellsv2_layout.html$ /nova/$1/user/cellsv2-layout.html
|
||||
redirectmatch 301 ^/nova/([^/]+)/user/config-drive.html$ /nova/$1/user/metadata.html
|
||||
redirectmatch 301 ^/nova/([^/]+)/user/metadata-service.html$ /nova/$1/user/metadata.html
|
||||
redirectmatch 301 ^/nova/([^/]+)/user/placement.html$ /placement/$1/
|
||||
redirectmatch 301 ^/nova/([^/]+)/user/upgrade.html$ /nova/$1/admin/upgrades.html
|
||||
redirectmatch 301 ^/nova/([^/]+)/user/user-data.html$ /nova/$1/user/metadata.html
|
||||
redirectmatch 301 ^/nova/([^/]+)/user/vendordata.html$ /nova/$1/user/metadata.html
|
||||
redirectmatch 301 ^/nova/([^/]+)/vendordata.html$ /nova/$1/user/metadata.html
|
||||
|
|
|
|||
|
|
@ -77,6 +77,7 @@ responsibilities of services and drivers are:
|
|||
remote-console-access
|
||||
service-groups
|
||||
node-down
|
||||
upgrades
|
||||
|
||||
|
||||
Advanced configuration
|
||||
|
|
|
|||
|
|
@ -1,19 +1,4 @@
|
|||
..
|
||||
Copyright 2014 Rackspace
|
||||
All Rights Reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
not use this file except in compliance with the License. You may obtain
|
||||
a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
License for the specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
========
|
||||
Upgrades
|
||||
========
|
||||
|
||||
|
|
@ -28,14 +13,14 @@ This document is trying to describe how we can achieve that.
|
|||
Once we have introduced the key concepts relating to upgrade, we will
|
||||
introduce the process needed for a no downtime upgrade of nova.
|
||||
|
||||
|
||||
.. _minimal_downtime_upgrade:
|
||||
|
||||
Minimal Downtime Upgrade Process
|
||||
--------------------------------
|
||||
|
||||
|
||||
Plan your upgrade
|
||||
'''''''''''''''''
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
* Read and ensure you understand the release notes for the next release.
|
||||
|
||||
|
|
@ -50,18 +35,28 @@ Plan your upgrade
|
|||
You may find you need to add extra nova-conductor workers to deal with the
|
||||
additional upgrade related load.
|
||||
|
||||
|
||||
Rolling upgrade process
|
||||
'''''''''''''''''''''''
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To reduce downtime, the compute services can be upgraded in a rolling fashion. It
|
||||
means upgrading a few services at a time. This results in a condition where
|
||||
To reduce downtime, the compute services can be upgraded in a rolling fashion.
|
||||
It means upgrading a few services at a time. This results in a condition where
|
||||
both old (N) and new (N+1) nova-compute services co-exist for a certain time
|
||||
period. Note that, there is no upgrade of the hypervisor here, this is just
|
||||
upgrading the nova services. If reduced downtime is not a concern (or lower
|
||||
complexity is desired), all services may be taken down and restarted at the
|
||||
same time.
|
||||
|
||||
.. important::
|
||||
|
||||
Nova does not currently support the coexistence of N and N+2 or greater
|
||||
:program:`nova-compute` or :program:`nova-conductor` services in the same
|
||||
deployment. The `nova-conductor`` service will fail to start when a
|
||||
``nova-compute`` service that is older than the previous release (N-2 or
|
||||
greater) is detected. Similarly, in a :doc:`deployment with multiple cells
|
||||
</user/cellsv2-layout>`, neither the super conductor service nor any
|
||||
per-cell conductor service will start if any other conductor service in the
|
||||
deployment is older than the previous release.
|
||||
|
||||
#. Before maintenance window:
|
||||
|
||||
* Start the process with the controller node. Install the code for the next
|
||||
|
|
@ -158,49 +153,54 @@ same time.
|
|||
|
||||
* At this point, you must also ensure you update the configuration, to stop
|
||||
using any deprecated features or options, and perform any required work
|
||||
to transition to alternative features. All the deprecated options should
|
||||
be supported for one cycle, but should be removed before your next
|
||||
to transition to alternative features. All deprecated options are
|
||||
supported for at least one cycle, but should be removed before your next
|
||||
upgrade is performed.
|
||||
|
||||
|
||||
Current Database Upgrade Types
|
||||
------------------------------
|
||||
|
||||
Currently Nova has 2 types of database upgrades that are in use.
|
||||
Currently Nova has two types of database upgrades that are in use.
|
||||
|
||||
#. Schema Migrations
|
||||
#. Data Migrations
|
||||
- Schema Migrations
|
||||
- Data Migrations
|
||||
|
||||
Nova does not support database downgrades.
|
||||
|
||||
.. _schema-migrations:
|
||||
|
||||
Schema Migrations
|
||||
''''''''''''''''''
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
Schema migrations are defined in
|
||||
``nova/db/main/legacy_migrations/versions`` and in
|
||||
``nova/db/api/legacy_migrations/versions``. They are
|
||||
the routines that transform our database structure, which should be
|
||||
additive and able to be applied to a running system before service
|
||||
code has been upgraded.
|
||||
Schema migrations are defined in ``nova/db/main/migrations/versions`` and
|
||||
``nova/db/api/migrations/versions``. They are the routines that transform our
|
||||
database structure, which should be additive and able to be applied to a
|
||||
running system before service code has been upgraded.
|
||||
|
||||
For information on developing your own schema migrations as part of a feature
|
||||
or bugfix, refer to :doc:`/reference/database-migrations`.
|
||||
|
||||
.. note::
|
||||
|
||||
The API database migrations should be assumed to run before the
|
||||
migrations for the main/cell databases. This is because the former
|
||||
contains information about how to find and connect to the latter.
|
||||
Some management commands that operate on multiple cells will attempt
|
||||
to list and iterate over cell mapping records, which require a
|
||||
functioning API database schema.
|
||||
The API database migrations should be assumed to run before the
|
||||
migrations for the main/cell databases. This is because the former
|
||||
contains information about how to find and connect to the latter.
|
||||
Some management commands that operate on multiple cells will attempt
|
||||
to list and iterate over cell mapping records, which require a
|
||||
functioning API database schema.
|
||||
|
||||
.. _data-migrations:
|
||||
|
||||
Data Migrations
|
||||
'''''''''''''''''
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
Online data migrations occur in two places:
|
||||
|
||||
#. Inline migrations that occur as part of normal run-time
|
||||
activity as data is read in the old format and written in the
|
||||
new format
|
||||
|
||||
#. Background online migrations that are performed using
|
||||
``nova-manage`` to complete transformations that will not occur
|
||||
incidentally due to normal runtime activity.
|
||||
|
|
@ -209,12 +209,11 @@ An example of online data migrations are the flavor migrations done as part
|
|||
of Nova object version 1.18. This included a transient migration of flavor
|
||||
storage from one database location to another.
|
||||
|
||||
.. note::
|
||||
For information on developing your own schema migrations as part of a feature
|
||||
or bugfix, refer to :doc:`/reference/database-migrations`.
|
||||
|
||||
Database downgrades are not supported.
|
||||
|
||||
Migration policy:
|
||||
'''''''''''''''''
|
||||
Migration policy
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
The following guidelines for schema and data migrations are followed in order
|
||||
to ease upgrades:
|
||||
|
|
@ -232,17 +231,11 @@ to ease upgrades:
|
|||
#. Data migration, by the objects layer, must completely migrate data from
|
||||
the old version of the schema to the new version.
|
||||
|
||||
* `Data migration example
|
||||
<http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/flavor-from-sysmeta-to-blob.html>`_
|
||||
* `Data migration enforcement example
|
||||
<https://review.opendev.org/#/c/174480/15/nova/db/sqlalchemy/migrate_repo/versions/291_enforce_flavors_migrated.py>`_
|
||||
(for sqlalchemy migrate/deprecated scripts):
|
||||
|
||||
#. The column can then be removed with a migration at the start of N+2.
|
||||
|
||||
* All schema migrations should be idempotent. (For example, a migration
|
||||
* All schema migrations should be idempotent. For example, a migration
|
||||
should check if an element exists in the schema before attempting to add
|
||||
it.) This logic comes for free in the autogenerated workflow of
|
||||
it. This logic comes for free in the autogenerated workflow of
|
||||
the online migrations.
|
||||
|
||||
* Constraints - When adding a foreign or unique key constraint, the schema
|
||||
|
|
@ -256,16 +249,6 @@ to ease upgrades:
|
|||
of data migration performed, there should exist a nova-manage option for an
|
||||
operator to manually request that rows be migrated.
|
||||
|
||||
* See `flavor migration spec
|
||||
<http://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/flavor-from-sysmeta-to-blob.html>`_
|
||||
for an example of data migrations in the object layer.
|
||||
|
||||
*Future* work -
|
||||
#. Adding plumbing to enforce that relevant data migrations are completed
|
||||
before running `contract` in the expand/migrate/contract schema migration
|
||||
workflow. A potential solution would be for `contract` to run a gating
|
||||
test for each specific subtract operation to determine if the operation
|
||||
can be completed.
|
||||
|
||||
Concepts
|
||||
--------
|
||||
|
|
@ -308,8 +291,8 @@ Graceful service shutdown
|
|||
|
||||
.. note::
|
||||
|
||||
While this is true for the RabbitMQ RPC backend, we need to confirm
|
||||
what happens for other RPC backends.
|
||||
While this is true for the RabbitMQ RPC backend, we need to confirm
|
||||
what happens for other RPC backends.
|
||||
|
||||
API load balancer draining
|
||||
When upgrading API nodes, you can make your load balancer only send new
|
||||
|
|
@ -327,7 +310,7 @@ Expand/Contract DB Migrations
|
|||
migration without affecting runtime code.
|
||||
|
||||
Online Data Migrations using objects
|
||||
In Kilo we are moving all data migration into the DB objects code.
|
||||
Since Kilo, we have moved all data migration into the DB objects code.
|
||||
When trying to migrate data in the database from the old format to the
|
||||
new format, this is done in the object code when reading or saving things
|
||||
that are in the old format. For records that are not updated, you need to
|
||||
|
|
@ -353,10 +336,6 @@ nova-conductor object backports
|
|||
Testing
|
||||
-------
|
||||
|
||||
Once we have all the pieces in place, we hope to move the Grenade testing
|
||||
to follow this new pattern.
|
||||
|
||||
The current tests only cover the existing upgrade process where:
|
||||
|
||||
* old computes can run with new control plane
|
||||
* but control plane is turned off for DB migrations
|
||||
We use the "grenade" jobs to test upgrades. The current tests only cover the
|
||||
existing upgrade process where old computes can run with new control plane but
|
||||
control plane is turned off for DB migrations.
|
||||
|
|
@ -177,7 +177,7 @@ Once you are running nova, the following information is extremely useful.
|
|||
* :doc:`Admin Guide </admin/index>`: A collection of guides for administrating
|
||||
nova.
|
||||
* :doc:`Flavors </user/flavors>`: What flavors are and why they are used.
|
||||
* :doc:`Upgrades </user/upgrade>`: How nova is designed to be upgraded for minimal
|
||||
* :doc:`Upgrades </admin/upgrades>`: How nova is designed to be upgraded for minimal
|
||||
service impact, and the order you should do them in.
|
||||
* :doc:`Quotas </user/quotas>`: Managing project quotas in nova.
|
||||
* :doc:`Aggregates </admin/aggregates>`: Aggregates are a useful way of grouping
|
||||
|
|
@ -198,7 +198,7 @@ Once you are running nova, the following information is extremely useful.
|
|||
|
||||
admin/index
|
||||
user/flavors
|
||||
user/upgrade
|
||||
admin/upgrades
|
||||
user/quotas
|
||||
user/filter-scheduler
|
||||
admin/vendordata
|
||||
|
|
|
|||
|
|
@ -0,0 +1,187 @@
|
|||
===================
|
||||
Database migrations
|
||||
===================
|
||||
|
||||
.. note::
|
||||
|
||||
This document details how to generate database migrations as part of a new
|
||||
feature or bugfix. For info on how to apply existing database migrations,
|
||||
refer to the documentation for the :program:`nova-manage db sync` and
|
||||
:program:`nova-manage api_db sync` commands in :doc:`/cli/nova-manage`.
|
||||
For info on the general upgrade process for a nova deployment, refer to
|
||||
:doc:`/admin/upgrades`.
|
||||
|
||||
A typical nova deployments consists of an "API" database and one or more
|
||||
cell-specific "main" databases. Occasionally these databases will require
|
||||
schema or data migrations.
|
||||
|
||||
|
||||
Schema migrations
|
||||
-----------------
|
||||
|
||||
.. versionchanged:: 24.0.0 (Xena)
|
||||
|
||||
The database migration engine was changed from ``sqlalchemy-migrate`` to
|
||||
``alembic``.
|
||||
|
||||
The `alembic`__ database migration tool is used to manage schema migrations in
|
||||
nova. The migration files and related metadata can be found in
|
||||
``nova/db/api/migrations`` (for the API database) and
|
||||
``nova/db/main/migrations`` (for the main database(s)). As discussed in
|
||||
:doc:`/admin/upgrades`, these can be run by end users using the
|
||||
:program:`nova-manage api_db sync` and :program:`nova-manage db sync` commands,
|
||||
respectively.
|
||||
|
||||
.. __: https://alembic.sqlalchemy.org/en/latest/
|
||||
|
||||
.. note::
|
||||
|
||||
There are also legacy migrations provided in the ``legacy_migrations``
|
||||
subdirectory for both the API and main databases. These are provided to
|
||||
facilitate upgrades from pre-Xena (24.0.0) deployments and will be removed
|
||||
in a future release. They should not be modified or extended.
|
||||
|
||||
The best reference for alembic is the `alembic documentation`__, but a small
|
||||
example is provided here. You can create the migration either manually or
|
||||
automatically. Manual generation might be necessary for some corner cases such
|
||||
as renamed tables but auto-generation will typically handle your issues.
|
||||
Examples of both are provided below. In both examples, we're going to
|
||||
demonstrate how you could add a new model, ``Foo``, to the main database.
|
||||
|
||||
.. __: https://alembic.sqlalchemy.org/en/latest/
|
||||
|
||||
.. code-block:: diff
|
||||
|
||||
diff --git nova/db/main/models.py nova/db/main/models.py
|
||||
index 7eab643e14..8f70bcdaca 100644
|
||||
--- nova/db/main/models.py
|
||||
+++ nova/db/main/models.py
|
||||
@@ -73,6 +73,16 @@ def MediumText():
|
||||
sqlalchemy.dialects.mysql.MEDIUMTEXT(), 'mysql')
|
||||
|
||||
|
||||
+class Foo(BASE, models.SoftDeleteMixin):
|
||||
+ """A test-only model."""
|
||||
+
|
||||
+ __tablename__ = 'foo'
|
||||
+
|
||||
+ id = sa.Column(sa.Integer, primary_key=True)
|
||||
+ uuid = sa.Column(sa.String(36), nullable=True)
|
||||
+ bar = sa.Column(sa.String(255))
|
||||
+
|
||||
+
|
||||
class Service(BASE, models.SoftDeleteMixin):
|
||||
"""Represents a running service on a host."""
|
||||
|
||||
(you might not be able to apply the diff above cleanly - this is just a demo).
|
||||
|
||||
.. rubric:: Auto-generating migration scripts
|
||||
|
||||
In order for alembic to compare the migrations with the underlying models, it
|
||||
require a database that it can inspect and compare the models against. As such,
|
||||
we first need to create a working database. We'll bypass ``nova-manage`` for
|
||||
this and go straight to the :program:`alembic` CLI. The ``alembic.ini`` file
|
||||
provided in the ``migrations`` directories for both databases is helpfully
|
||||
configured to use an SQLite database by default (``nova.db`` for the main
|
||||
database and ``nova_api.db`` for the API database). Create this database and
|
||||
apply the current schema, as dictated by the current migration scripts:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ tox -e venv -- alembic -c nova/db/main/alembic.ini \
|
||||
upgrade head
|
||||
|
||||
Once done, you should notice the new ``nova.db`` file in the root of the repo.
|
||||
Now, let's generate the new revision:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ tox -e venv -- alembic -c nova/db/main/alembic.ini \
|
||||
revision -m "Add foo model" --autogenerate
|
||||
|
||||
This will create a new file in ``nova/db/main/migrations`` with
|
||||
``add_foo_model`` in the name including (hopefully!) the necessary changes to
|
||||
add the new ``Foo`` model. You **must** inspect this file once created, since
|
||||
there's a chance you'll be missing imports or something else which will need to
|
||||
be manually corrected. Once you've inspected this file and made any required
|
||||
changes, you can apply the migration and make sure it works:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ tox -e venv -- alembic -c nova/db/main/alembic.ini \
|
||||
upgrade head
|
||||
|
||||
.. rubric:: Manually generating migration scripts
|
||||
|
||||
For trickier migrations or things that alembic doesn't understand, you may need
|
||||
to manually create a migration script. This is very similar to the
|
||||
auto-generation step, with the exception being that you don't need to have a
|
||||
database in place beforehand. As such, you can simply run:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ tox -e venv -- alembic -c nova/db/main/alembic.ini \
|
||||
revision -m "Add foo model"
|
||||
|
||||
As before, this will create a new file in ``nova/db/main/migrations`` with
|
||||
``add_foo_model`` in the name. You can simply modify this to make whatever
|
||||
changes are necessary. Once done, you can apply the migration and make sure it
|
||||
works:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ tox -e venv -- alembic -c nova/db/main/alembic.ini \
|
||||
upgrade head
|
||||
|
||||
|
||||
Data migrations
|
||||
---------------
|
||||
|
||||
As discussed in :doc:`/admin/upgrades`, online data migrations occur in two
|
||||
places:
|
||||
|
||||
- Inline migrations that occur as part of normal run-time activity as data is
|
||||
read in the old format and written in the new format.
|
||||
|
||||
- Background online migrations that are performed using ``nova-manage`` to
|
||||
complete transformations that will not occur incidentally due to normal
|
||||
runtime activity.
|
||||
|
||||
.. rubric:: Inline data migrations
|
||||
|
||||
Inline data migrations are arguably the easier of the two to implement. Almost
|
||||
all of nova's database models correspond to an oslo.versionedobject (o.vo) or
|
||||
part of one. These o.vos load their data from the underlying database by
|
||||
implementing the ``obj_load_attr`` method. By modifying this method, it's
|
||||
possible to detect missing changes to the data - for example, a missing field -
|
||||
modify the data, save it back to the database, and finally return an object
|
||||
with the newly updated data. Change I6cd206542fdd28f3ef551dcc727f4cb35a53f6a3
|
||||
provides a fully worked example of this approach.
|
||||
|
||||
The main advantage of these is that they are completely transparent to the
|
||||
operator who does not have to take any additional steps to upgrade their
|
||||
deployment: the database updates should happen at runtime as data is pulled
|
||||
from the database. The main disadvantage of this approach is that some
|
||||
records may not be frequently pulled from the database, meaning they never have
|
||||
a chance to get updated. This can prevent the eventual removal of the inline
|
||||
migration in a future release. To avoid this issue, you should inspect the
|
||||
object to see if it's something that will be loaded as part of a standard
|
||||
runtime operation - for example, on startup or as part of a background task -
|
||||
and if necessary add a blocking online migration in a later release to catch
|
||||
and migrate the laggards.
|
||||
|
||||
.. rubric:: Online data migrations
|
||||
|
||||
Unlike inline data migrations, online data migrations require operator
|
||||
involvement. They are run using the ``nova-manage db online_data_migrations``
|
||||
command which, as noted in :doc:`/cli/nova-manage`, this should be run straight
|
||||
after upgrading to a new release once the database schema migrations have been
|
||||
applied and the code updated. Online migrations can be blocking, in that it
|
||||
will be necessary to apply given migrations while running N code before
|
||||
upgrading to N+1. Change I44919422c48570f2647f2325ff895255fc2adf27 provides a
|
||||
fully worked example of this approach.
|
||||
|
||||
The advantages and disadvantages of this approach are the inverse of those of
|
||||
the inline data migrations approach. While they can be used to ensure an data
|
||||
migration is actually applied, they require operator involvement and can
|
||||
prevent upgrades until fully applied.
|
||||
|
|
@ -30,6 +30,8 @@ The following is a dive into some of the internals in nova.
|
|||
``ComputeDriver.update_provider_tree`` method.
|
||||
* :doc:`/reference/upgrade-checks`: A guide to writing automated upgrade
|
||||
checks.
|
||||
* :doc:`/reference/database-migrations`: A guide to writing database
|
||||
migrations, be they online or offline.
|
||||
* :doc:`/reference/conductor`
|
||||
|
||||
.. todo:: Need something about versioned objects and how they fit in with
|
||||
|
|
@ -56,6 +58,7 @@ The following is a dive into some of the internals in nova.
|
|||
vm-states
|
||||
threading
|
||||
notifications
|
||||
database-migrations
|
||||
update-provider-tree
|
||||
upgrade-checks
|
||||
conductor
|
||||
|
|
|
|||
|
|
@ -2,6 +2,14 @@
|
|||
Upgrade checks
|
||||
==============
|
||||
|
||||
.. note::
|
||||
|
||||
This document details how to generate upgrade checks as part of a new
|
||||
feature or bugfix. For info on how to apply existing upgrade checks, refer
|
||||
to the documentation for the :program:`nova-status` command in
|
||||
:doc:`/cli/nova-status`. For info on the general upgrade process for a nova
|
||||
deployment, refer to :doc:`/admin/upgrades`.
|
||||
|
||||
Nova provides automated :ref:`upgrade check tooling <nova-status-checks>` to
|
||||
assist deployment tools in verifying critical parts of the deployment,
|
||||
especially when it comes to major changes during upgrades that require operator
|
||||
|
|
@ -10,15 +18,16 @@ intervention.
|
|||
This guide covers the background on nova's upgrade check tooling, how it is
|
||||
used, and what to look for in writing new checks.
|
||||
|
||||
|
||||
Background
|
||||
==========
|
||||
----------
|
||||
|
||||
Nova has historically supported offline database schema migrations
|
||||
(``nova-manage db sync``) and :ref:`online data migrations <data-migrations>`
|
||||
during upgrades.
|
||||
|
||||
The ``nova-status upgrade check`` command was introduced in the 15.0.0 Ocata
|
||||
release to aid in the verification of two major required changes in that
|
||||
(:program:`nova-manage db sync` and :program:`nova-manage api_db sync`) and
|
||||
online data migrations (:program:`nova-manage db online_data_migrations`)
|
||||
during upgrades, as discussed in :doc:`/reference/database-migrations`.
|
||||
The :program:`nova-status upgrade check` command was introduced in the 15.0.0
|
||||
(Ocata) release to aid in the verification of two major required changes in that
|
||||
release, namely Placement and Cells v2.
|
||||
|
||||
Integration with the Placement service and deploying Cells v2 was optional
|
||||
|
|
@ -36,8 +45,9 @@ Reference the `Ocata changes`_ for implementation details.
|
|||
|
||||
.. _Ocata changes: https://review.opendev.org/#/q/topic:bp/resource-providers-scheduler-db-filters+status:merged+file:%255Enova/cmd/status.py
|
||||
|
||||
|
||||
Guidelines
|
||||
==========
|
||||
----------
|
||||
|
||||
* The checks should be able to run within a virtual environment or container.
|
||||
All that is required is a full configuration file, similar to running other
|
||||
|
|
@ -58,9 +68,9 @@ Guidelines
|
|||
host.
|
||||
|
||||
* Checks are typically meant to be run before re-starting and upgrading to new
|
||||
service code, which is how `grenade uses them`_, but they can also be run
|
||||
service code, which is how `grenade uses them`__, but they can also be run
|
||||
as a :ref:`post-install verify step <verify-install-nova-status>` which is
|
||||
how `openstack-ansible`_ also uses them. The high-level set of upgrade steps
|
||||
how `openstack-ansible`__ also uses them. The high-level set of upgrade steps
|
||||
for upgrading nova in grenade is:
|
||||
|
||||
* Install new code
|
||||
|
|
@ -70,6 +80,9 @@ Guidelines
|
|||
* Run the upgrade check (``nova-status upgrade check``)
|
||||
* Restart services with new code
|
||||
|
||||
.. __: https://github.com/openstack-dev/grenade/blob/dc7f4a4ba/projects/60_nova/upgrade.sh#L96
|
||||
.. __: https://review.opendev.org/#/c/575125/
|
||||
|
||||
* Checks must be idempotent so they can be run repeatedly and the results are
|
||||
always based on the latest data. This allows an operator to run the checks,
|
||||
fix any issues reported, and then iterate until the status check no longer
|
||||
|
|
@ -81,26 +94,27 @@ Guidelines
|
|||
should be a corresponding ``request_specs`` table entry in the ``nova_api``
|
||||
database. A ``nova-manage db online_data_migrations`` routine was added in
|
||||
the Newton release to back-fill request specs for existing instances, and
|
||||
`in Rocky`_ an upgrade check was added to make sure all non-deleted instances
|
||||
have a request spec so compatibility code can be removed in Stein. In older
|
||||
releases of nova we would have added a `blocker migration`_ as part of the
|
||||
database schema migrations to make sure the online data migrations had been
|
||||
completed before the upgrade could proceed.
|
||||
`in Rocky`__ an upgrade check was added to make sure all non-deleted
|
||||
instances have a request spec so compatibility code can be removed in Stein.
|
||||
In older releases of nova we would have added a `blocker migration`__ as part
|
||||
of the database schema migrations to make sure the online data migrations had
|
||||
been completed before the upgrade could proceed.
|
||||
|
||||
.. note:: Usage of ``nova-status upgrade check`` does not preclude the need
|
||||
for blocker migrations within a given database, but in the case of
|
||||
request specs the check spans multiple databases and was a better
|
||||
fit for the nova-status tooling.
|
||||
.. note::
|
||||
|
||||
Usage of ``nova-status upgrade check`` does not preclude the need
|
||||
for blocker migrations within a given database, but in the case of
|
||||
request specs the check spans multiple databases and was a better
|
||||
fit for the ``nova-status`` tooling.
|
||||
|
||||
.. __: https://review.opendev.org/#/c/581813/
|
||||
.. __: https://review.opendev.org/#/c/289450/
|
||||
|
||||
* All checks should have an accompanying upgrade release note.
|
||||
|
||||
.. _grenade uses them: https://github.com/openstack-dev/grenade/blob/dc7f4a4ba/projects/60_nova/upgrade.sh#L96
|
||||
.. _openstack-ansible: https://review.opendev.org/#/c/575125/
|
||||
.. _in Rocky: https://review.opendev.org/#/c/581813/
|
||||
.. _blocker migration: https://review.opendev.org/#/c/289450/
|
||||
|
||||
Structure
|
||||
=========
|
||||
---------
|
||||
|
||||
There is no graph logic for checks, meaning each check is meant to be run
|
||||
independently of other checks in the same set. For example, a project could
|
||||
|
|
@ -167,94 +181,86 @@ The results feed into a standard output for the checks:
|
|||
| service catalog. |
|
||||
+----------------------------------------------------+
|
||||
|
||||
.. note:: Long-term the framework for upgrade checks will come from the
|
||||
`oslo.upgradecheck library`_.
|
||||
|
||||
.. _initial change: https://review.opendev.org/#/c/411517/
|
||||
.. _cells v2 check: https://review.opendev.org/#/c/411525/
|
||||
.. _placement check: https://review.opendev.org/#/c/413250/
|
||||
.. _oslo.upgradecheck library: http://opendev.org/openstack/oslo.upgradecheck/
|
||||
|
||||
Other
|
||||
=====
|
||||
|
||||
Documentation
|
||||
-------------
|
||||
|
||||
Each check should be documented in the
|
||||
:ref:`history section <nova-status-checks>` of the CLI guide and have a
|
||||
release note. This is important since the checks can be run in an isolated
|
||||
environment apart from the actual deployed version of the code and since the
|
||||
checks should be idempotent, the history / change log is good for knowing
|
||||
what is being validated.
|
||||
|
||||
Backports
|
||||
---------
|
||||
|
||||
Sometimes upgrade checks can be backported to aid in pre-empting bugs on
|
||||
stable branches. For example, a check was added for `bug 1759316`_ in Rocky
|
||||
which was also backported to stable/queens in case anyone upgrading from Pike
|
||||
to Queens would hit the same issue. Backportable checks are generally only
|
||||
made for latent bugs since someone who has already passed checks and upgraded
|
||||
to a given stable branch should not start failing after a patch release on that
|
||||
same branch. For this reason, any check being backported should have a release
|
||||
note with it.
|
||||
|
||||
.. _bug 1759316: https://bugs.launchpad.net/nova/+bug/1759316
|
||||
|
||||
Other projects
|
||||
--------------
|
||||
|
||||
A community-wide `goal for the Stein release`_ is adding the same type of
|
||||
``$PROJECT-status upgrade check`` tooling to other projects to ease in
|
||||
upgrading OpenStack across the board. So while the guidelines in this document
|
||||
are primarily specific to nova, they should apply generically to other projects
|
||||
wishing to incorporate the same tooling.
|
||||
|
||||
.. _goal for the Stein release: https://governance.openstack.org/tc/goals/stein/upgrade-checkers.html
|
||||
|
||||
FAQs
|
||||
----
|
||||
|
||||
#. How is the nova-status upgrade script packaged and deployed?
|
||||
- How is the ``nova-status`` upgrade script packaged and deployed?
|
||||
|
||||
There is a ``console_scripts`` entry for ``nova-status`` in the
|
||||
``setup.cfg`` file.
|
||||
There is a ``console_scripts`` entry for ``nova-status`` in the ``setup.cfg``
|
||||
file.
|
||||
|
||||
#. Why are there multiple parts to the command structure, i.e. "upgrade" and
|
||||
"check"?
|
||||
- Why are there multiple parts to the command structure, i.e. "upgrade" and
|
||||
"check"?
|
||||
|
||||
This is an artifact of how the ``nova-manage`` command is structured which
|
||||
has categories of sub-commands, like ``nova-manage db`` is a sub-category
|
||||
made up of other sub-commands like ``nova-manage db sync``. The
|
||||
``nova-status upgrade check`` command was written in the same way for
|
||||
consistency and extensibility if other sub-commands need to be added later.
|
||||
This is an artifact of how the ``nova-manage`` command is structured which
|
||||
has categories of sub-commands, like ``nova-manage db`` is a sub-category
|
||||
made up of other sub-commands like ``nova-manage db sync``. The
|
||||
``nova-status upgrade check`` command was written in the same way for
|
||||
consistency and extensibility if other sub-commands need to be added later.
|
||||
|
||||
#. Where should the documentation live for projects other than nova?
|
||||
- Why is the upgrade check command not part of the standard python-\*client
|
||||
CLIs?
|
||||
|
||||
As part of the standard OpenStack project `documentation guidelines`_ the
|
||||
command should be documented under ``doc/source/cli`` in each project repo.
|
||||
The ``nova-status`` command was modeled after the ``nova-manage`` command
|
||||
which is meant to be admin-only and has direct access to the database,
|
||||
unlike other CLI packages like python-novaclient which requires a token
|
||||
and communicates with nova over the REST API. Because of this, it is also
|
||||
possible to write commands in ``nova-manage`` and ``nova-status`` that can
|
||||
work while the API service is down for maintenance.
|
||||
|
||||
#. Why is the upgrade check command not part of the standard python-\*client
|
||||
CLIs?
|
||||
- How should the checks be documented?
|
||||
|
||||
The ``nova-status`` command was modeled after the ``nova-manage`` command
|
||||
which is meant to be admin-only and has direct access to the database,
|
||||
unlike other CLI packages like python-novaclient which requires a token
|
||||
and communicates with nova over the REST API. Because of this, it is also
|
||||
possible to write commands in ``nova-manage`` and ``nova-status`` that can
|
||||
work while the API service is down for maintenance.
|
||||
Each check should be documented in the :ref:`history section
|
||||
<nova-status-checks>` of the CLI guide and have a release note. This is
|
||||
important since the checks can be run in an isolated environment apart from
|
||||
the actual deployed version of the code and since the checks should be
|
||||
idempotent, the history / change log is good for knowing what is being
|
||||
validated.
|
||||
|
||||
#. Can upgrade checks only be for N-1 to N version upgrades?
|
||||
- Do other projects support upgrade checks?
|
||||
|
||||
No, not necessarily. The upgrade checks are also an essential part of
|
||||
`fast-forward upgrades`_ to make sure that as you roll through each release
|
||||
performing schema (data model) updates and data migrations that you are
|
||||
also completing all of the necessary changes. For example, if you are
|
||||
fast forward upgrading from Ocata to Rocky, something could have been
|
||||
added, deprecated or removed in Pike or Queens and a pre-upgrade check is
|
||||
a way to make sure the necessary steps were taking while upgrading through
|
||||
those releases before restarting the Rocky code at the end.
|
||||
A community-wide `goal for the Stein release`__ is adding the same type of
|
||||
``$PROJECT-status upgrade check`` tooling to other projects to ease in
|
||||
upgrading OpenStack across the board. So while the guidelines in this
|
||||
document are primarily specific to nova, they should apply generically to
|
||||
other projects wishing to incorporate the same tooling.
|
||||
|
||||
.. _documentation guidelines: https://docs.openstack.org/doc-contrib-guide/project-guides.html
|
||||
.. _fast-forward upgrades: https://wiki.openstack.org/wiki/Fast_forward_upgrades
|
||||
.. __: https://governance.openstack.org/tc/goals/stein/upgrade-checkers.html
|
||||
|
||||
- Where should the documentation live for projects other than nova?
|
||||
|
||||
As part of the standard OpenStack project `documentation guidelines`__ the
|
||||
command should be documented under ``doc/source/cli`` in each project repo.
|
||||
|
||||
.. __: https://docs.openstack.org/doc-contrib-guide/project-guides.html
|
||||
|
||||
- Can upgrade checks be backported?
|
||||
|
||||
Sometimes upgrade checks can be backported to aid in pre-empting bugs on
|
||||
stable branches. For example, a check was added for `bug 1759316`__ in Rocky
|
||||
which was also backported to stable/queens in case anyone upgrading from Pike
|
||||
to Queens would hit the same issue. Backportable checks are generally only
|
||||
made for latent bugs since someone who has already passed checks and upgraded
|
||||
to a given stable branch should not start failing after a patch release on
|
||||
that same branch. For this reason, any check being backported should have a
|
||||
release note with it.
|
||||
|
||||
.. __: https://bugs.launchpad.net/nova/+bug/1759316
|
||||
|
||||
- Can upgrade checks only be for N-1 to N version upgrades?
|
||||
|
||||
No, not necessarily. The upgrade checks are also an essential part of
|
||||
`fast-forward upgrades`__ to make sure that as you roll through each release
|
||||
performing schema (data model) updates and data migrations that you are
|
||||
also completing all of the necessary changes. For example, if you are
|
||||
fast forward upgrading from Ocata to Rocky, something could have been
|
||||
added, deprecated or removed in Pike or Queens and a pre-upgrade check is
|
||||
a way to make sure the necessary steps were taking while upgrading through
|
||||
those releases before restarting the Rocky code at the end.
|
||||
|
||||
.. __: https://wiki.openstack.org/wiki/Fast_forward_upgrades
|
||||
|
|
|
|||
|
|
@ -72,9 +72,6 @@ Once you are running nova, the following information is extremely useful.
|
|||
* :doc:`Admin Guide </admin/index>`: A collection of guides for administrating
|
||||
nova.
|
||||
|
||||
* :doc:`Upgrades </user/upgrade>`: How nova is designed to be upgraded for minimal
|
||||
service impact, and the order you should do them in.
|
||||
|
||||
* :doc:`Quotas </user/quotas>`: Managing project quotas in nova.
|
||||
|
||||
* :doc:`Availablity Zones </admin/availability-zones>`: Availability Zones are
|
||||
|
|
|
|||
|
|
@ -62,13 +62,14 @@
|
|||
/nova/latest/testing/serial-console.html 301 /nova/latest/contributor/testing/serial-console.html
|
||||
/nova/latest/testing/zero-downtime-upgrade.html 301 /nova/latest/contributor/testing/zero-downtime-upgrade.html
|
||||
/nova/latest/threading.html 301 /nova/latest/reference/threading.html
|
||||
/nova/latest/upgrade.html 301 /nova/latest/user/upgrade.html
|
||||
/nova/latest/upgrade.html 301 /nova/latest/admin/upgrades.html
|
||||
/nova/latest/user/aggregates.html 301 /nova/latest/admin/aggregates.html
|
||||
/nova/latest/user/cellsv2_layout.html 301 /nova/latest/user/cellsv2-layout.html
|
||||
/nova/latest/user/config-drive.html 301 /nova/latest/user/metadata.html
|
||||
/nova/latest/user/metadata-service.html 301 /nova/latest/user/metadata.html
|
||||
/nova/latest/user/placement.html 301 /placement/latest/
|
||||
/nova/latest/user/user-data.html 301 /nova/latest/user/metadata.html
|
||||
/nova/latest/user/upgrade.html 301 /nova/latest/admin/upgrades.html
|
||||
/nova/latest/user/vendordata.html 301 /nova/latest/user/metadata.html
|
||||
/nova/latest/vendordata.html 301 /nova/latest/user/metadata.html
|
||||
/nova/latest/vmstates.html 301 /nova/latest/reference/vm-states.html
|
||||
|
|
|
|||
Loading…
Reference in New Issue