Migrating to relational db from ES
Change-Id: I36878bf54debe905df1d825f67cab30d6148b8b4
This commit is contained in:
parent
dc0ffee9f7
commit
c67f1e159a
|
@ -0,0 +1,305 @@
|
||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
=================================
|
||||||
|
Relational DB Schema with OSLO.DB
|
||||||
|
=================================
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/freezer/+spec/oslo.db
|
||||||
|
|
||||||
|
Taking advantage of the oslo.db library to have a more uniform database
|
||||||
|
backend architecture to other OpenStack projects.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
Currently Freezer uses Elastic Search (ES) as a database backend, which
|
||||||
|
is a NoSQL database specialized for ranked query results. Elastic Search
|
||||||
|
adds additional complexity to an OpenStack system. Most of the
|
||||||
|
components use a relational database management system (DBMS like MySQL or
|
||||||
|
PostgreSQL) which are more common. It is more familiar how to
|
||||||
|
maintain, troubleshoot and develop on top of relational databases.
|
||||||
|
|
||||||
|
Since Freezer related data turned out to be relational, it would be more
|
||||||
|
convenient to use it trough the oslo.db pattern library. Using it, the
|
||||||
|
database mapping would be more uniform to other OpenStack projects.
|
||||||
|
It would be less challenging for new developers to contribute.
|
||||||
|
|
||||||
|
Use Cases
|
||||||
|
---------
|
||||||
|
|
||||||
|
* For new developers, already familiar with OpenStack, it should be less
|
||||||
|
challenging to get familiar with the backend code, since most of the
|
||||||
|
OpenStack projects use relational database backend trough the oslo.db
|
||||||
|
pattern library.
|
||||||
|
|
||||||
|
* For Deployers there would be no longer needed to set up a special
|
||||||
|
DBMS just for Freezer, since it could share the relational DBMS used
|
||||||
|
by the other (core) OpenStack projects, still well isolated in it's
|
||||||
|
own database.
|
||||||
|
|
||||||
|
* For End User it would be less difficult to maintain, since Freezer
|
||||||
|
would not add additional complexity with a less common component,
|
||||||
|
instead it can take advantage of the DBMS that is already deployed for
|
||||||
|
OpenStack.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
Implementing the entities using oslo.db and SQLAlchemy base classes.
|
||||||
|
And expose the new entities trough the REST API.
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
Oslo.db with SQLAlchemy is the de facto standard for OpenStack projects
|
||||||
|
to implement database backends with relational DBMS. It provides high
|
||||||
|
level ORM mapping and abstracts the different database backends.
|
||||||
|
Therefore we gain compability with multiple relational DBMS just like
|
||||||
|
any other OpenStack component using oslo.db.
|
||||||
|
|
||||||
|
Using other alternative would either not be more uniform to other
|
||||||
|
OpenStack project tooling, either we would have to implement low level,
|
||||||
|
directly to a specific database driver (just like now with ES).
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
Changes which require modifications to the data model often have a wider impact
|
||||||
|
on the system. The community often has strong opinions on how the data model
|
||||||
|
should be evolved, from both a functional and performance perspective. It is
|
||||||
|
therefore important to capture and gain agreement as early as possible on any
|
||||||
|
proposed changes to the data model.
|
||||||
|
|
||||||
|
Questions which need to be addressed by this section include:
|
||||||
|
|
||||||
|
* What new data objects and/or database schema changes is this going to
|
||||||
|
require?
|
||||||
|
|
||||||
|
* What database migrations will accompany this change.
|
||||||
|
|
||||||
|
* How will the initial set of new data objects be generated, for example if you
|
||||||
|
need to take into account existing backups/jobs/... , or modify other
|
||||||
|
existing data describe how that will work.
|
||||||
|
|
||||||
|
|
||||||
|
As there will be a brand new relational database schema [MIG1]_:
|
||||||
|
|
||||||
|
|
||||||
|
Clients
|
||||||
|
|
||||||
|
id (varchar) [p_key]
|
||||||
|
|
||||||
|
project_id (uuid)
|
||||||
|
|
||||||
|
config_id (varchar)
|
||||||
|
|
||||||
|
description (varchar)
|
||||||
|
|
||||||
|
uuid (uuid)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Actions
|
||||||
|
|
||||||
|
id (uuid) [p_key]
|
||||||
|
|
||||||
|
action (varchar)
|
||||||
|
|
||||||
|
project_id (uuid)
|
||||||
|
|
||||||
|
mode (varchar)
|
||||||
|
|
||||||
|
src_file (varchar)
|
||||||
|
|
||||||
|
backup_name (varchar)
|
||||||
|
|
||||||
|
container (varchar(
|
||||||
|
|
||||||
|
restore_abs_path (varchar)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Action_reports
|
||||||
|
|
||||||
|
id (uuid) [p_key]
|
||||||
|
|
||||||
|
action_id (uuid) [f_key]
|
||||||
|
|
||||||
|
action_attachment_id (uuid) [f_key]
|
||||||
|
|
||||||
|
project_id (uuid)
|
||||||
|
|
||||||
|
result (varchar)
|
||||||
|
|
||||||
|
time_elapsed (varchar)
|
||||||
|
|
||||||
|
metadata (JSON)
|
||||||
|
|
||||||
|
report_date (timestamp)
|
||||||
|
|
||||||
|
log (blob) < only on failure
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Jobs
|
||||||
|
|
||||||
|
id (uuid) [p_key]
|
||||||
|
|
||||||
|
project_id (uuid)
|
||||||
|
|
||||||
|
scheduling (JSON)
|
||||||
|
|
||||||
|
description (varchar)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Action_attachments
|
||||||
|
|
||||||
|
id (uuid) [p_key]
|
||||||
|
|
||||||
|
action_id (uuid) [f_key]
|
||||||
|
|
||||||
|
job_id (uuid) [f_key]
|
||||||
|
|
||||||
|
project_id (uuid)
|
||||||
|
|
||||||
|
priority (int)
|
||||||
|
|
||||||
|
retries (int)
|
||||||
|
|
||||||
|
retry_interval (int)
|
||||||
|
|
||||||
|
mandatory (bool)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Sessions
|
||||||
|
|
||||||
|
id (uuid) [p_key]
|
||||||
|
|
||||||
|
project_id (uuid)
|
||||||
|
|
||||||
|
scheduling (JSON)
|
||||||
|
|
||||||
|
policy (varchar)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Job_attachments
|
||||||
|
|
||||||
|
id (uuid) [p_key]
|
||||||
|
|
||||||
|
client_id (varchar) [f_key]
|
||||||
|
|
||||||
|
job_id (uuid) [f_key]
|
||||||
|
|
||||||
|
session_id (uuid) [f_key]
|
||||||
|
|
||||||
|
project_id (uuid)
|
||||||
|
|
||||||
|
event (varchar)
|
||||||
|
|
||||||
|
status (varchar)
|
||||||
|
|
||||||
|
current_pid (int)
|
||||||
|
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
There should be a new v2 API implemented. TBD.
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Notifications impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
TBD.
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
None. TBD.
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
* The Elastic Search configurations should be replaced with oslo.db
|
||||||
|
configurations
|
||||||
|
|
||||||
|
* When updating from a previous version there must be a data migration
|
||||||
|
from ES to oslo.db (this will be addressed by a nother spec - TBD).
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
There will be no longer needed to deploy ES.
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
neilus
|
||||||
|
|
||||||
|
Other contributors:
|
||||||
|
daemontool
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
Work items or tasks -- break the feature up into the things that need to be
|
||||||
|
done to implement it. Those parts might end up being done by different people,
|
||||||
|
but we're mostly trying to understand the timeline for implementation.
|
||||||
|
|
||||||
|
* implementing the database models
|
||||||
|
|
||||||
|
* create adapter for API v1(?) and v2
|
||||||
|
|
||||||
|
* implementing the CRUD API
|
||||||
|
|
||||||
|
* updating the devStack plugin
|
||||||
|
|
||||||
|
* updating documentation
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
* Implementing the database migration script (TBD), which migrates data
|
||||||
|
from ES to oslo.db backend DB.
|
||||||
|
|
||||||
|
* We will be using oslo.db library and SQLAlchemy for iplementation.
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
TBD.
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
TBD.
|
||||||
|
|
||||||
|
* Freezer API installation doc
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
.. [MIG1] https://etherpad.openstack.org/p/freezer_mysql_migration
|
||||||
|
|
||||||
|
.. https://etherpad.openstack.org/p/freezer_db_switch
|
||||||
|
|
Loading…
Reference in New Issue