795ec26417
Remove conditional migration, implement healing migration and include all tables in schema after Icehouse. Change-Id: Idb7f92be1520dfec0bb90e27f4e4a33ca6c0f399
258 lines
7.1 KiB
ReStructuredText
258 lines
7.1 KiB
ReStructuredText
..
|
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
|
License.
|
|
|
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
|
|
|
==========================================
|
|
Database Migration Refactoring
|
|
==========================================
|
|
|
|
Launchpad blueprint:
|
|
|
|
https://blueprints.launchpad.net/neutron/+spec/db-migration-refactor
|
|
|
|
The neutron database schema will be made consistent. It will be independent of
|
|
the configured core plugin and service plugins.
|
|
|
|
|
|
Problem description
|
|
===================
|
|
|
|
The database schema is different based on which core and service plugins are
|
|
configured. If a new or different plugin is configured after a deployment is
|
|
started, the earlier migrations that correspond with the new configuration will
|
|
be missing.
|
|
|
|
This means that the migration process is not idempotent, resulting in schema
|
|
versions which depend on the current configuration. In other words, when one
|
|
environment's schema is at a specific version it may not be the same as another
|
|
environment at the same version. This even happens in the same environment
|
|
during a downgrade if the configuration changes.
|
|
|
|
|
|
Proposed change
|
|
===============
|
|
|
|
Make all migrations unconditional. Make a single point in the migration
|
|
timeline where all tables are created and the database schema is the same no
|
|
matter which plugins are used.
|
|
|
|
This solution will introduce a "healing" migration that will call the DDLs
|
|
needed to complete the database schema so that it contains all tables for all
|
|
database models. This migration will occur in the timeline between the
|
|
**icehouse_release** version and the **juno_release** version.
|
|
|
|
The refactoring will comprise the following:
|
|
|
|
1) Investigation of models and current migrations to establish what is needed
|
|
to achieve the complete database schema.
|
|
|
|
2) A healing migration that ensures that the database schema is complete and
|
|
consistent. The version will be named **db_healing**.
|
|
|
|
3) All migrations after **db_healing** will be unconditional. This requires
|
|
updating the behavior of the --autogenerate for neutron-db-manage, and
|
|
documentation for developers.
|
|
|
|
The healing migration will work like any other alembic migration, i.e. it will
|
|
allow upgrade and downgrade. However, it will be different in the following
|
|
aspects:
|
|
|
|
* In the upgrade direction the healing migration will introspect the schema and
|
|
only add tables that are missing. Therefore it cannot be run in offline
|
|
mode. (See also :ref:`onlinedetails`.)
|
|
|
|
Upgrade Notes:
|
|
|
|
* When the healing migration starts, all table schema changes for existing
|
|
tables will already be in place (either directly from the model, or from
|
|
previous migrations). For each missing table, the healing migration will
|
|
add the table by creating its schema from the model.
|
|
|
|
* The healing migration may need to alter some existing tables to match its
|
|
model. We won't know for sure until the proposed implementation is tested
|
|
in more detail. These alterations, if needed, will be dealt with on a
|
|
case-by-case basis. For more details, see the :ref:`implementation`
|
|
section.
|
|
|
|
* In the downgrade direction the healing migration will make no schema
|
|
changes. This means it will not remove any tables.
|
|
|
|
Downgrade Notes:
|
|
|
|
* If a deployment is upgraded and then downgraded through the healing
|
|
migration, the schema will contain tables that were not present before. In
|
|
other words, if a deployment downgrades back to Icehouse or Havana, Neutron
|
|
will appear and behave as before, yet if you look at the DB it is not the
|
|
old Icehouse or Havana DB but a healed one.
|
|
|
|
* If during the upgrade a table was altered to match its model, then the
|
|
downgrade will reverse the alteration if needed.
|
|
|
|
|
|
Migration Timeline
|
|
------------------
|
|
|
|
.. blockdiag::
|
|
|
|
blockdiag timeline {
|
|
orientation = portrait;
|
|
default_group_color = lightgreen;
|
|
|
|
'...' -> icehouse_release -> '....' -> db_healing;
|
|
db_healing -> '.....' -> juno_release -> '......';
|
|
group {
|
|
db_healing;
|
|
}
|
|
}
|
|
|
|
|
|
.. _onlinedetails:
|
|
|
|
Online Requirement
|
|
------------------
|
|
|
|
More reasons why we cannot support offline mode for the healing migration:
|
|
|
|
1) We cannot use "create table if not exists" because it is only supported by
|
|
specific sql dialects.
|
|
|
|
2) Dependencies between tables (e.g. foreign keys) mean we would need to check
|
|
that all tables with dependencies on the current one are already created.
|
|
|
|
3) Alembic has no support for conditional DDL.
|
|
|
|
|
|
Alternatives
|
|
------------
|
|
|
|
Instead of a healing migration we could break the migration timeline after
|
|
Icehouse and create a new one beginning at Juno that includes all tables. A
|
|
manual script could be provided to convert the schema from the old timeline to
|
|
the new. This alternative approach has the following disadvantages:
|
|
|
|
* It would not allow a downgrade via migration.
|
|
|
|
* Switching from the old timeline to the new timeline is more a complex process
|
|
for the deployer and DB Administrator than a simple migration in one timeline.
|
|
|
|
* We would need to support two migration timelines in neutron until the
|
|
Icehouse release is deprecated.
|
|
|
|
|
|
Data model impact
|
|
-----------------
|
|
|
|
Some database models may need to be updated in order to have non-conflicting
|
|
models based on core and service plugins.
|
|
|
|
|
|
REST API impact
|
|
---------------
|
|
|
|
None
|
|
|
|
|
|
Security impact
|
|
---------------
|
|
|
|
None
|
|
|
|
|
|
Notifications impact
|
|
--------------------
|
|
|
|
None
|
|
|
|
|
|
Other end user impact
|
|
---------------------
|
|
|
|
End (non-admin) users should see no impact.
|
|
|
|
|
|
Performance Impact
|
|
------------------
|
|
|
|
None
|
|
|
|
Other deployer impact
|
|
---------------------
|
|
|
|
The healing migration is an online operation that must be run by the DB
|
|
Administrator. Thus the deployer must co-ordinate with the DBA when upgrading
|
|
or downgrading through Juno.
|
|
|
|
The healing migration will be run when migrating from Icehouse or earlier
|
|
to Juno. It behaves like a normal migration, but it does not support upgrade in
|
|
offline mode.
|
|
|
|
Downgrade of the healing migration does nothing. Thus all tables are present in
|
|
the schema if downgrading from **db_healing** to a previous version.
|
|
|
|
Note: Greenfield deployment of the Juno release or later will start at the new
|
|
migration timeline and therefore no healing will be involved.
|
|
|
|
|
|
Developer impact
|
|
----------------
|
|
|
|
None
|
|
|
|
|
|
.. _implementation:
|
|
|
|
Implementation
|
|
==============
|
|
|
|
Most of the work lies in developing a robust healing migration. Alembic will be
|
|
used where possible to maximize automation, but some healing may need to be
|
|
manually coded.
|
|
|
|
Examples of conflicts which may need manual coding to resolve:
|
|
|
|
* If two different migrations for two different plugins add an attribute with
|
|
the same name but different type.
|
|
|
|
* If an ENUM was modified in an earlier migration but its specification was not
|
|
updated for PostgreSQL.
|
|
|
|
|
|
Assignee(s)
|
|
-----------
|
|
|
|
* akamyshnikova
|
|
* libosvar
|
|
* rpodolyaka
|
|
|
|
|
|
Work Items
|
|
----------
|
|
|
|
|
|
Dependencies
|
|
============
|
|
|
|
|
|
Testing
|
|
=======
|
|
|
|
Unit (and functional?) testing of migrations shall be added. We plan to utilize
|
|
the unit test framework from the graduated oslo.db package.
|
|
|
|
|
|
Documentation Impact
|
|
====================
|
|
|
|
* Create Release Note for this change.
|
|
|
|
* Update Operators Guide for upgrading.
|
|
|
|
* Update Developer Documentation for creating migration scripts.
|
|
|
|
|
|
References
|
|
==========
|
|
|