Browse Source

Migrating to relational db from ES

Change-Id: I36878bf54debe905df1d825f67cab30d6148b8b4
Norbert Sana 2 years ago
parent
commit
c67f1e159a
1 changed files with 305 additions and 0 deletions
  1. 305
    0
      specs/ocata/approved/relational_db_schema_with_oslo_db.rst

+ 305
- 0
specs/ocata/approved/relational_db_schema_with_oslo_db.rst View File

@@ -0,0 +1,305 @@
1
+..
2
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
3
+ License.
4
+
5
+ http://creativecommons.org/licenses/by/3.0/legalcode
6
+
7
+=================================
8
+Relational DB Schema with OSLO.DB
9
+=================================
10
+
11
+https://blueprints.launchpad.net/freezer/+spec/oslo.db
12
+
13
+Taking advantage of the oslo.db library to have a more uniform database
14
+backend architecture to other OpenStack projects.
15
+
16
+Problem description
17
+===================
18
+
19
+Currently Freezer uses Elastic Search (ES) as a database backend, which
20
+is a NoSQL database specialized for ranked query results. Elastic Search
21
+adds additional complexity to an OpenStack system. Most of the
22
+components use a relational database management system (DBMS like MySQL or
23
+PostgreSQL) which are more common. It is more familiar how to
24
+maintain, troubleshoot and develop on top of relational databases.
25
+
26
+Since Freezer related data turned out to be relational, it would be more
27
+convenient to use it trough the oslo.db pattern library. Using it, the
28
+database mapping would be more uniform to other OpenStack projects.
29
+It would be less challenging for new developers to contribute.
30
+
31
+Use Cases
32
+---------
33
+
34
+* For new developers, already familiar with OpenStack, it should be less
35
+  challenging to get familiar with the backend code, since most of the
36
+  OpenStack projects use relational database backend trough the oslo.db
37
+  pattern library.
38
+
39
+* For Deployers there would be no longer needed to set up a special
40
+  DBMS just for Freezer, since it could share the relational DBMS used
41
+  by the other (core) OpenStack projects, still well isolated in it's
42
+  own database.
43
+
44
+* For End User it would be less difficult to maintain, since Freezer
45
+  would not add additional complexity with a less common component,
46
+  instead it can take advantage of the DBMS that is already deployed for
47
+  OpenStack.
48
+
49
+Proposed change
50
+===============
51
+
52
+Implementing the entities using oslo.db and SQLAlchemy base classes.
53
+And expose the new entities trough the REST API.
54
+
55
+Alternatives
56
+------------
57
+
58
+Oslo.db with SQLAlchemy is the de facto standard for OpenStack projects
59
+to implement database backends with relational DBMS. It provides high
60
+level ORM mapping and abstracts the different database backends.
61
+Therefore we gain compability with multiple relational DBMS just like
62
+any other OpenStack component using oslo.db.
63
+
64
+Using other alternative would either not be more uniform to other
65
+OpenStack project tooling, either we would have to implement low level,
66
+directly to a specific database driver (just like now with ES).
67
+
68
+Data model impact
69
+-----------------
70
+
71
+Changes which require modifications to the data model often have a wider impact
72
+on the system.  The community often has strong opinions on how the data model
73
+should be evolved, from both a functional and performance perspective. It is
74
+therefore important to capture and gain agreement as early as possible on any
75
+proposed changes to the data model.
76
+
77
+Questions which need to be addressed by this section include:
78
+
79
+* What new data objects and/or database schema changes is this going to
80
+  require?
81
+
82
+* What database migrations will accompany this change.
83
+
84
+* How will the initial set of new data objects be generated, for example if you
85
+  need to take into account existing backups/jobs/... , or modify other
86
+  existing data describe how that will work.
87
+
88
+
89
+As there will be a brand new relational database schema [MIG1]_:
90
+
91
+
92
+Clients
93
+
94
+    id (varchar) [p_key]
95
+
96
+    project_id (uuid)
97
+
98
+    config_id (varchar)
99
+
100
+    description (varchar)
101
+
102
+    uuid (uuid)
103
+
104
+
105
+
106
+Actions
107
+
108
+    id (uuid) [p_key]
109
+
110
+    action (varchar)
111
+
112
+    project_id (uuid)
113
+
114
+    mode (varchar)
115
+
116
+    src_file (varchar)
117
+
118
+    backup_name (varchar)
119
+
120
+    container (varchar(
121
+
122
+    restore_abs_path (varchar)
123
+
124
+
125
+
126
+Action_reports
127
+
128
+    id (uuid) [p_key]
129
+
130
+    action_id (uuid) [f_key]
131
+
132
+    action_attachment_id (uuid) [f_key]
133
+
134
+    project_id (uuid)
135
+
136
+    result (varchar)
137
+
138
+    time_elapsed (varchar)
139
+
140
+    metadata (JSON)
141
+
142
+    report_date (timestamp)
143
+
144
+    log (blob) < only on failure
145
+
146
+
147
+
148
+Jobs
149
+
150
+    id (uuid) [p_key]
151
+
152
+    project_id (uuid)
153
+
154
+    scheduling (JSON)
155
+
156
+    description (varchar)
157
+
158
+
159
+
160
+Action_attachments
161
+
162
+    id (uuid) [p_key]
163
+
164
+    action_id (uuid) [f_key]
165
+
166
+    job_id (uuid) [f_key]
167
+
168
+    project_id (uuid)
169
+
170
+    priority (int)
171
+
172
+    retries (int)
173
+
174
+    retry_interval (int)
175
+
176
+    mandatory (bool)
177
+
178
+
179
+
180
+Sessions
181
+
182
+    id (uuid) [p_key]
183
+
184
+    project_id (uuid)
185
+
186
+    scheduling (JSON)
187
+
188
+    policy (varchar)
189
+
190
+
191
+
192
+Job_attachments
193
+
194
+    id (uuid) [p_key]
195
+
196
+    client_id (varchar) [f_key]
197
+
198
+    job_id (uuid) [f_key]
199
+
200
+    session_id (uuid) [f_key]
201
+
202
+    project_id (uuid)
203
+
204
+    event (varchar)
205
+
206
+    status (varchar)
207
+
208
+    current_pid (int)
209
+
210
+
211
+REST API impact
212
+---------------
213
+
214
+There should be a new v2 API implemented. TBD.
215
+
216
+Security impact
217
+---------------
218
+
219
+None
220
+
221
+Notifications impact
222
+--------------------
223
+
224
+TBD.
225
+
226
+Other end user impact
227
+---------------------
228
+
229
+None. TBD.
230
+
231
+Performance Impact
232
+------------------
233
+
234
+None.
235
+
236
+Other deployer impact
237
+---------------------
238
+
239
+* The Elastic Search configurations should be replaced with oslo.db
240
+  configurations
241
+
242
+* When updating from a previous version there must be a data migration
243
+  from ES to oslo.db (this will be addressed by a nother spec - TBD).
244
+
245
+Developer impact
246
+----------------
247
+
248
+There will be no longer needed to deploy ES.
249
+
250
+Implementation
251
+==============
252
+
253
+Assignee(s)
254
+-----------
255
+
256
+Primary assignee:
257
+  neilus
258
+
259
+Other contributors:
260
+  daemontool
261
+
262
+Work Items
263
+----------
264
+
265
+Work items or tasks -- break the feature up into the things that need to be
266
+done to implement it. Those parts might end up being done by different people,
267
+but we're mostly trying to understand the timeline for implementation.
268
+
269
+* implementing the database models
270
+
271
+* create adapter for API v1(?) and v2
272
+
273
+* implementing the CRUD API
274
+
275
+* updating the devStack plugin
276
+
277
+* updating documentation
278
+
279
+Dependencies
280
+============
281
+
282
+* Implementing the database migration script (TBD), which migrates data
283
+  from ES to oslo.db backend DB.
284
+
285
+* We will be using oslo.db library and SQLAlchemy for iplementation.
286
+
287
+Testing
288
+=======
289
+
290
+TBD.
291
+
292
+Documentation Impact
293
+====================
294
+
295
+TBD.
296
+
297
+* Freezer API installation doc
298
+
299
+References
300
+==========
301
+
302
+.. [MIG1] https://etherpad.openstack.org/p/freezer_mysql_migration
303
+
304
+.. https://etherpad.openstack.org/p/freezer_db_switch
305
+

Loading…
Cancel
Save