DB migration refactoring

Remove conditional migration, implement healing migration and include
all tables in schema after Icehouse.

Change-Id: Idb7f92be1520dfec0bb90e27f4e4a33ca6c0f399
This commit is contained in:
Jakub Libosvar 2014-05-27 14:02:34 +02:00 committed by Henry Gessau
parent 33813b2c58
commit 795ec26417
1 changed files with 257 additions and 0 deletions

View File

@ -0,0 +1,257 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Database Migration Refactoring
==========================================
Launchpad blueprint:
https://blueprints.launchpad.net/neutron/+spec/db-migration-refactor
The neutron database schema will be made consistent. It will be independent of
the configured core plugin and service plugins.
Problem description
===================
The database schema is different based on which core and service plugins are
configured. If a new or different plugin is configured after a deployment is
started, the earlier migrations that correspond with the new configuration will
be missing.
This means that the migration process is not idempotent, resulting in schema
versions which depend on the current configuration. In other words, when one
environment's schema is at a specific version it may not be the same as another
environment at the same version. This even happens in the same environment
during a downgrade if the configuration changes.
Proposed change
===============
Make all migrations unconditional. Make a single point in the migration
timeline where all tables are created and the database schema is the same no
matter which plugins are used.
This solution will introduce a "healing" migration that will call the DDLs
needed to complete the database schema so that it contains all tables for all
database models. This migration will occur in the timeline between the
**icehouse_release** version and the **juno_release** version.
The refactoring will comprise the following:
1) Investigation of models and current migrations to establish what is needed
to achieve the complete database schema.
2) A healing migration that ensures that the database schema is complete and
consistent. The version will be named **db_healing**.
3) All migrations after **db_healing** will be unconditional. This requires
updating the behavior of the --autogenerate for neutron-db-manage, and
documentation for developers.
The healing migration will work like any other alembic migration, i.e. it will
allow upgrade and downgrade. However, it will be different in the following
aspects:
* In the upgrade direction the healing migration will introspect the schema and
only add tables that are missing. Therefore it cannot be run in offline
mode. (See also :ref:`onlinedetails`.)
Upgrade Notes:
* When the healing migration starts, all table schema changes for existing
tables will already be in place (either directly from the model, or from
previous migrations). For each missing table, the healing migration will
add the table by creating its schema from the model.
* The healing migration may need to alter some existing tables to match its
model. We won't know for sure until the proposed implementation is tested
in more detail. These alterations, if needed, will be dealt with on a
case-by-case basis. For more details, see the :ref:`implementation`
section.
* In the downgrade direction the healing migration will make no schema
changes. This means it will not remove any tables.
Downgrade Notes:
* If a deployment is upgraded and then downgraded through the healing
migration, the schema will contain tables that were not present before. In
other words, if a deployment downgrades back to Icehouse or Havana, Neutron
will appear and behave as before, yet if you look at the DB it is not the
old Icehouse or Havana DB but a healed one.
* If during the upgrade a table was altered to match its model, then the
downgrade will reverse the alteration if needed.
Migration Timeline
------------------
.. blockdiag::
blockdiag timeline {
orientation = portrait;
default_group_color = lightgreen;
'...' -> icehouse_release -> '....' -> db_healing;
db_healing -> '.....' -> juno_release -> '......';
group {
db_healing;
}
}
.. _onlinedetails:
Online Requirement
------------------
More reasons why we cannot support offline mode for the healing migration:
1) We cannot use "create table if not exists" because it is only supported by
specific sql dialects.
2) Dependencies between tables (e.g. foreign keys) mean we would need to check
that all tables with dependencies on the current one are already created.
3) Alembic has no support for conditional DDL.
Alternatives
------------
Instead of a healing migration we could break the migration timeline after
Icehouse and create a new one beginning at Juno that includes all tables. A
manual script could be provided to convert the schema from the old timeline to
the new. This alternative approach has the following disadvantages:
* It would not allow a downgrade via migration.
* Switching from the old timeline to the new timeline is more a complex process
for the deployer and DB Administrator than a simple migration in one timeline.
* We would need to support two migration timelines in neutron until the
Icehouse release is deprecated.
Data model impact
-----------------
Some database models may need to be updated in order to have non-conflicting
models based on core and service plugins.
REST API impact
---------------
None
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
End (non-admin) users should see no impact.
Performance Impact
------------------
None
Other deployer impact
---------------------
The healing migration is an online operation that must be run by the DB
Administrator. Thus the deployer must co-ordinate with the DBA when upgrading
or downgrading through Juno.
The healing migration will be run when migrating from Icehouse or earlier
to Juno. It behaves like a normal migration, but it does not support upgrade in
offline mode.
Downgrade of the healing migration does nothing. Thus all tables are present in
the schema if downgrading from **db_healing** to a previous version.
Note: Greenfield deployment of the Juno release or later will start at the new
migration timeline and therefore no healing will be involved.
Developer impact
----------------
None
.. _implementation:
Implementation
==============
Most of the work lies in developing a robust healing migration. Alembic will be
used where possible to maximize automation, but some healing may need to be
manually coded.
Examples of conflicts which may need manual coding to resolve:
* If two different migrations for two different plugins add an attribute with
the same name but different type.
* If an ENUM was modified in an earlier migration but its specification was not
updated for PostgreSQL.
Assignee(s)
-----------
* akamyshnikova
* libosvar
* rpodolyaka
Work Items
----------
Dependencies
============
Testing
=======
Unit (and functional?) testing of migrations shall be added. We plan to utilize
the unit test framework from the graduated oslo.db package.
Documentation Impact
====================
* Create Release Note for this change.
* Update Operators Guide for upgrading.
* Update Developer Documentation for creating migration scripts.
References
==========