Browse Source

Added Pike directory for pike specs

Created Pike directory and required files for Pike release.
Moved un-completed blueprints and specs to pike

Change-Id: I98506ce6f047accea7820f729f250431bceff298
Saad Zaher 2 years ago
parent
commit
d0f312f15d

+ 9
- 0
priorities/pike-priorities.rst View File

@@ -0,0 +1,9 @@
1
+.. _pike-priorities:
2
+
3
+=======================
4
+Pike Project Priorities
5
+=======================
6
+
7
+List of themes (in the form of use cases) the freezer development team will
8
+prioritize in Pike.
9
+

+ 365
- 0
specs/pike-template.rst View File

@@ -0,0 +1,365 @@
1
+..
2
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
3
+ License.
4
+
5
+ http://creativecommons.org/licenses/by/3.0/legalcode
6
+
7
+==========================================
8
+Example Spec - The title of your blueprint
9
+==========================================
10
+
11
+Include the URL of your launchpad blueprint:
12
+
13
+https://blueprints.launchpad.net/freezer/+spec/example
14
+
15
+Introduction paragraph -- why are we doing anything? A single paragraph of
16
+prose that operators can understand. The title and this first paragraph
17
+should be used as the subject line and body of the commit message
18
+respectively.
19
+
20
+Some notes about the freezer-spec and blueprint process:
21
+
22
+* Not all blueprints need a spec. For more information see
23
+  http://docs.openstack.org/developer/nova/devref/kilo.blueprints.html#when-is-a-blueprint-needed
24
+
25
+* The aim of this document is first to define the problem we need to solve,
26
+  and second agree the overall approach to solve that problem.
27
+
28
+* This is not intended to be extensive documentation for a new feature.
29
+  For example, there is no need to specify the exact configuration changes,
30
+
31
+  nor the exact details of any DB model changes. But you should still define
32
+  that such changes are required, and be clear on how that will affect
33
+  upgrades.
34
+
35
+* You should aim to get your spec approved before writing your code.
36
+  While you are free to write prototypes and code before getting your spec
37
+  approved, its possible that the outcome of the spec review process leads
38
+  you towards a fundamentally different solution than you first envisaged.
39
+
40
+* But, API changes are held to a much higher level of scrutiny.
41
+  As soon as an API change merges, we must assume it could be in production
42
+  somewhere, and as such, we then need to support that API change forever.
43
+  To avoid getting that wrong, we do want lots of details about API changes
44
+  upfront.
45
+
46
+Some notes about using this template:
47
+
48
+* Your spec should be in ReSTructured text, like this template.
49
+
50
+* Please wrap text at 79 columns.
51
+
52
+* The filename in the git repository should match the launchpad URL, for
53
+  example a URL of: https://blueprints.launchpad.net/freezer/+spec/awesome-thing
54
+  should be named awesome-thing.rst
55
+
56
+* Please do not delete any of the sections in this template.  If you have
57
+  nothing to say for a whole section, just write: None
58
+
59
+* For help with syntax, see http://sphinx-doc.org/rest.html
60
+
61
+* To test out your formatting, build the docs using tox and see the generated
62
+  HTML file in doc/build/html/specs/<path_of_your_file>
63
+
64
+* If you would like to provide a diagram with your spec, ascii diagrams are
65
+  required.  http://asciiflow.com/ is a very nice tool to assist with making
66
+  ascii diagrams.  The reason for this is that the tool used to review specs is
67
+  based purely on plain text.  Plain text will allow review to proceed without
68
+  having to look at additional files which can not be viewed in gerrit.  It
69
+  will also allow inline feedback on the diagram itself.
70
+
71
+
72
+Problem description
73
+===================
74
+
75
+A detailed description of the problem. What problem is this blueprint
76
+addressing?
77
+
78
+Use Cases
79
+---------
80
+
81
+What use cases does this address? What impact on actors does this change have?
82
+Ensure you are clear about the actors in each use case: Developer, End User,
83
+Deployer etc.
84
+
85
+Proposed change
86
+===============
87
+
88
+Here is where you cover the change you propose to make in detail. How do you
89
+propose to solve this problem?
90
+
91
+If this is one part of a larger effort make it clear where this piece ends. In
92
+other words, what's the scope of this effort?
93
+
94
+At this point, if you would like to just get feedback on if the problem and
95
+proposed change fit in freezer, you can stop here and post this for review to get
96
+preliminary feedback. If so please say:
97
+Posting to get preliminary feedback on the scope of this spec.
98
+
99
+Alternatives
100
+------------
101
+
102
+What other ways could we do this thing? Why aren't we using those? This doesn't
103
+have to be a full literature review, but it should demonstrate that thought has
104
+been put into why the proposed solution is an appropriate one.
105
+
106
+Data model impact
107
+-----------------
108
+
109
+Changes which require modifications to the data model often have a wider impact
110
+on the system.  The community often has strong opinions on how the data model
111
+should be evolved, from both a functional and performance perspective. It is
112
+therefore important to capture and gain agreement as early as possible on any
113
+proposed changes to the data model.
114
+
115
+Questions which need to be addressed by this section include:
116
+
117
+* What new data objects and/or database schema changes is this going to
118
+  require?
119
+
120
+* What database migrations will accompany this change.
121
+
122
+* How will the initial set of new data objects be generated, for example if you
123
+  need to take into account existing backups/jobs/... , or modify other
124
+  existing data describe how that will work.
125
+
126
+REST API impact
127
+---------------
128
+
129
+Each API method which is either added or changed should have the following
130
+
131
+* Specification for the method
132
+
133
+  * A description of what the method does suitable for use in
134
+    user documentation
135
+
136
+  * Method type (POST/PUT/GET/DELETE)
137
+
138
+  * Normal http response code(s)
139
+
140
+  * Expected error http response code(s)
141
+
142
+    * A description for each possible error code should be included
143
+      describing semantic errors which can cause it such as
144
+      inconsistent parameters supplied to the method, or when an
145
+      instance is not in an appropriate state for the request to
146
+      succeed. Errors caused by syntactic problems covered by the JSON
147
+      schema definition do not need to be included.
148
+
149
+  * URL for the resource
150
+
151
+    * URL should not include underscores, and use hyphens instead.
152
+
153
+  * Parameters which can be passed via the url
154
+
155
+  * JSON schema definition for the request body data if allowed
156
+
157
+    * Field names should use snake_case style, not CamelCase or MixedCase
158
+      style.
159
+
160
+  * JSON schema definition for the response body data if any
161
+
162
+    * Field names should use snake_case style, not CamelCase or MixedCase
163
+      style.
164
+
165
+* Example use case including typical API samples for both data supplied
166
+  by the caller and the response
167
+
168
+* Discuss any policy changes, and discuss what things a deployer needs to
169
+  think about when defining their policy.
170
+
171
+Reuse of existing predefined parameter types such as regexps for
172
+passwords and user defined names is highly encouraged.
173
+
174
+Security impact
175
+---------------
176
+
177
+Describe any potential security impact on the system.  Some of the items to
178
+consider include:
179
+
180
+* Does this change touch sensitive data such as tokens, keys, or user data?
181
+
182
+* Does this change alter the API in a way that may impact security, such as
183
+  a new way to access sensitive information or a new way to login?
184
+
185
+* Does this change involve cryptography or hashing?
186
+
187
+* Does this change require the use of sudo or any elevated privileges?
188
+
189
+* Does this change involve using or parsing user-provided data? This could
190
+  be directly at the API level or indirectly such as changes to a cache layer.
191
+
192
+* Can this change enable a resource exhaustion attack, such as allowing a
193
+  single API interaction to consume significant server resources? Some examples
194
+  of this include launching subprocesses for each connection, or entity
195
+  expansion attacks in XML.
196
+
197
+For more detailed guidance, please see the OpenStack Security Guidelines as
198
+a reference (https://wiki.openstack.org/wiki/Security/Guidelines).  These
199
+guidelines are a work in progress and are designed to help you identify
200
+security best practices.  For further information, feel free to reach out
201
+to the OpenStack Security Group at openstack-security@lists.openstack.org.
202
+
203
+Notifications impact
204
+--------------------
205
+
206
+Please specify any changes to notifications. Be that an extra notification,
207
+changes to an existing notification, or removing a notification.
208
+
209
+Other end user impact
210
+---------------------
211
+
212
+Aside from the API, are there other ways a user will interact with this
213
+feature?
214
+
215
+* Does this change have an impact on python-freezerclient? What does the user
216
+  interface there look like?
217
+
218
+* Does this change have an impact on freezer-web-ui? What does the user
219
+  interface there look like?
220
+
221
+* Does this change have an impact on freezeri-dr?
222
+
223
+Performance Impact
224
+------------------
225
+
226
+Describe any potential performance impact on the system, for example
227
+how often will new code be called, and is there a major change to the calling
228
+pattern of existing code.
229
+
230
+Examples of things to consider here include:
231
+
232
+* A small change in a utility function or a commonly used decorator can have a
233
+  large impacts on performance.
234
+
235
+* Calls which result in a database queries can have a profound impact on
236
+  performance when called in critical sections of the code.
237
+
238
+* Will the change include any locking, and if so what considerations are there
239
+  on holding the lock?
240
+
241
+Other deployer impact
242
+---------------------
243
+
244
+Discuss things that will affect how you deploy and configure OpenStack
245
+that have not already been mentioned, such as:
246
+
247
+* What config options are being added? Should they be more generic than
248
+  proposed? Are the default values ones which will work well in real
249
+  deployments?
250
+
251
+* Is this a change that takes immediate effect after its merged, or is it
252
+  something that has to be explicitly enabled?
253
+
254
+* If this change is a new binary, how would it be deployed?
255
+
256
+* Please state anything that those doing continuous deployment, or those
257
+  upgrading from the previous release, need to be aware of. Also describe
258
+  any plans to deprecate configuration values or features.  For example, if we
259
+  change the directory name that instances are stored in, how do we handle
260
+  instance directories created before the change landed?  Do we move them?  Do
261
+  we have a special case in the code? Do we assume that the operator will
262
+  recreate all the instances in their cloud?
263
+
264
+Developer impact
265
+----------------
266
+
267
+Discuss things that will affect other developers working on OpenStack.
268
+
269
+Implementation
270
+==============
271
+
272
+Assignee(s)
273
+-----------
274
+
275
+Who is leading the writing of the code? Or is this a blueprint where you're
276
+throwing it out there to see who picks it up?
277
+
278
+If more than one person is working on the implementation, please designate the
279
+primary author and contact.
280
+
281
+Primary assignee:
282
+  <launchpad-id or None>
283
+
284
+Other contributors:
285
+  <launchpad-id or None>
286
+
287
+Work Items
288
+----------
289
+
290
+Work items or tasks -- break the feature up into the things that need to be
291
+done to implement it. Those parts might end up being done by different people,
292
+but we're mostly trying to understand the timeline for implementation.
293
+
294
+
295
+Dependencies
296
+============
297
+
298
+* Include specific references to specs and/or blueprints in freezer, or in
299
+  other projects, that this one either depends on or is related to.
300
+
301
+* If this requires functionality of another project that is not currently used
302
+  by Freezer (such as the glance v2 API when we previously only required v1),
303
+  document that fact.
304
+
305
+* Does this feature require any new library dependencies or code otherwise not
306
+  included in OpenStack? Or does it depend on a specific version of library?
307
+
308
+
309
+Testing
310
+=======
311
+
312
+Please discuss the important scenarios needed to test here, as well as
313
+specific edge cases we should be ensuring work correctly.
314
+
315
+Please discuss how the change will be tested. We especially want to know what
316
+tempest tests will be added. It is assumed that unit test coverage will be
317
+added so that doesn't need to be mentioned explicitly, but discussion of why
318
+you think unit tests are sufficient and we don't need to add more tempest
319
+tests would need to be included.
320
+
321
+Is this untestable in gate given current limitations (specific hardware /
322
+software configurations available)? If so, are there mitigation plans (3rd
323
+party testing, gate enhancements, etc).
324
+
325
+
326
+Documentation Impact
327
+====================
328
+
329
+Please describe how and where this change will be documented.
330
+
331
+
332
+References
333
+==========
334
+
335
+Please add any useful references here. You are not required to have any
336
+reference. Moreover, this specification should still make sense when your
337
+references are unavailable. Examples of what you could include are:
338
+
339
+* Links to mailing list or IRC discussions
340
+
341
+* Links to notes from a summit session
342
+
343
+* Links to relevant research, if appropriate
344
+
345
+* Related specifications as appropriate (e.g.  if it's an EC2 thing, link the
346
+  EC2 docs)
347
+
348
+* Anything else you feel it is worthwhile to refer to
349
+
350
+
351
+History
352
+=======
353
+
354
+Optional section for Pike intended to be used each time the spec
355
+is updated to describe new design, API or any database schema
356
+updated. Useful to let reader understand what's happened along the
357
+time.
358
+
359
+.. list-table:: Revisions
360
+   :header-rows: 1
361
+
362
+   * - Release Name
363
+     - Description
364
+   * - Pike
365
+     - Introduced

+ 229
- 0
specs/pike/approved/block_based_backup_support.rst View File

@@ -0,0 +1,229 @@
1
+..
2
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
3
+ License.
4
+
5
+ http://creativecommons.org/licenses/by/3.0/legalcode
6
+
7
+==================================
8
+Block based backup support (rsync)
9
+==================================
10
+
11
+https://blueprints.launchpad.net/freezer/+spec/rsync
12
+
13
+Taking advantage of the rsync to provide a possibility to create
14
+space/bandwidth efficient backups.
15
+
16
+Problem description
17
+===================
18
+
19
+Currently Freezer checks only ctime and mtime inode information
20
+to verify if files are changed or not (tar functionality). While
21
+this approach gives speed (time efficient), it is not bandwidth
22
+and storage efficient. Freezer needs to support both rsync and tar
23
+approach to execute incremental backups and restore.
24
+
25
+Since Freezer will provide two options for incremental backups, it
26
+would be more convenient to choose the best approach to backup data
27
+in accordance with each particular case (more speed or storage/bandwidth
28
+efficient).
29
+
30
+Use Cases
31
+---------
32
+
33
+* For developers, this change will not create negative impacts because
34
+  this code will be gracefully bundled in Freezer engine API and
35
+  will not cause any major changes in Freezer architecture.
36
+
37
+* For Deployers there is no need to install any additional components,
38
+  Freezer will use it's own implementation of rsync algorithm
39
+  (written in Python).
40
+
41
+* For End User it would be less difficult to select more efficient
42
+  option for create backups based on dataset (e.g. few big files or a lot of
43
+  small files) for backup and speed/storage/bandwidth requirements,
44
+  since Freezer would support both rsync and tar approaches.
45
+
46
+Proposed change
47
+===============
48
+
49
+Implementing the new engine classes for rsync (as well as for tar).
50
+Providing new engine (-e) choice in config.
51
+
52
+For this type of backup will be created following metadata structure:
53
+
54
+files_meta = {
55
+            'files': {},
56
+            'directories': {},
57
+            'meta': {
58
+                'broken_links_tot': '',
59
+                'total_files': '',
60
+                'total_directories': '',
61
+                'backup_size_on_disk': '',
62
+                'backup_size_uncompressed': '',
63
+                'backup_size_compressed': '',
64
+                'platform': sys.platform
65
+            },
66
+            'abs_backup_path': os.getcwd(),
67
+            'broken_links': [],
68
+            'rsync_struct_ver': RSYNC_DATA_STRUCT_VERSION,
69
+            'rsync_block_size': RSYNC_BLOCK_SIZE}
70
+
71
+
72
+file_meta = {'inode': {
73
+                      'inumber': os_stat.st_ino,
74
+                      'nlink': os_stat.st_nlink,
75
+                      'mode': file_mode,
76
+                      'uid': os_stat.st_uid,
77
+                      'gid': os_stat.st_gid,
78
+                      'size': os_stat.st_size,
79
+                      'devmajor': os.major(dev),
80
+                      'devminor': os.minor(dev),
81
+                      'mtime': mtime,
82
+                      'ctime': ctime,
83
+                      'uname': uname,
84
+                      'gname': gname,
85
+                      'ftype': file_type,
86
+                      'lname': lname,
87
+                      'rsync_block_size': rsync_block_size,
88
+                      'file_status: status
89
+                      }
90
+            }
91
+
92
+Current version of implementation you always can find here [1].
93
+
94
+Alternatives
95
+------------
96
+
97
+Because of the flexibility, speed, and scriptability of rsync, it has
98
+become a standard Linux utility, included in all popular Linux distributions.
99
+It has been ported to Windows (via Cygwin, Grsync, or SFU), FreeBSD, NetBSD,
100
+OpenBSD, and Mac OS. De facto, rsync is the default fallback for most data
101
+transfers. It has a clear algorithm written for 20 years ago and different
102
+variations (e.g. acrosync, zsync, etc). librsync is used by Dropbox.
103
+
104
+Using other alternative (like bbcp or lftp) would not be more effective
105
+or portable solution.
106
+
107
+Data model impact
108
+-----------------
109
+
110
+Changes in data model has already described in oslo.db migration document.
111
+Actions entity should contain 'engine' field for performing appropriate action
112
+using particular type of engine (tar, rsync or openstack).
113
+
114
+From new relational database schema:
115
+
116
+Actions
117
+    action_id (uuid) [p_key]
118
+    resource (varchar)
119
+    type (varchar)
120
+    name (varchar)
121
+    application (varchar)
122
+    engine (varchar)    <-- Require this
123
+    snapshot (varchar)
124
+    storage (varchar)
125
+    global_options (JSON)
126
+    application_options (JSON)
127
+    storage_options (JSON)
128
+    snapshot_options (JSON)
129
+    engine_options (JSON)
130
+
131
+
132
+REST API impact
133
+---------------
134
+
135
+None.
136
+
137
+Security impact
138
+---------------
139
+
140
+None.
141
+
142
+Notifications impact
143
+--------------------
144
+
145
+There are no special logs will be added, just some info messages about
146
+start/stop backup process, backup metrics, etc.
147
+
148
+Other end user impact
149
+---------------------
150
+
151
+* There are no additional changes to python-freezerclient CLI. To choice
152
+  appropriate engine for action, end user should specify 'engine' field
153
+  in provided JSON configuration in case of creating or updating action.
154
+
155
+* freezer-web-ui should provide additional 'engine' field in 'Action
156
+  Configuration' window. It has to be drop-down list with values 'tar',
157
+  'rsync' or 'openstack'.
158
+
159
+Performance Impact
160
+------------------
161
+
162
+None.
163
+
164
+Other deployer impact
165
+---------------------
166
+
167
+Will be added new choice to freezer-agent -e (engine) option - 'rsync'.
168
+
169
+Developer impact
170
+----------------
171
+
172
+None.
173
+
174
+Implementation
175
+==============
176
+
177
+Assignee(s)
178
+-----------
179
+
180
+Primary assignee:
181
+  Ruslan Aliev (raliev) <raliev@mirantis.com>
182
+
183
+Other contributors:
184
+  Fausto Marzi (daemontool) <fausto.marzi@ericsson.com>
185
+
186
+Work Items
187
+----------
188
+
189
+* implementing the new engine (rsync)
190
+
191
+* bundling this engine to freezer code (API calls) and mechanism
192
+  for using this engine ('-e rsync' option)
193
+
194
+* implementing the new database schema for actions (oslo.db migration)
195
+
196
+* updating freezer-web-ui 'Action Configuration' window
197
+
198
+* updating documentation
199
+
200
+Dependencies
201
+============
202
+
203
+* This spec depends on Freezer oslo.db migration [2].
204
+
205
+* Pluggable engines described here [3].
206
+
207
+* There are no additional library dependencies.
208
+
209
+Testing
210
+=======
211
+
212
+There is a question - do we actually need separate tempest test
213
+for this change or we can be satisfied with existing one?
214
+
215
+Documentation Impact
216
+====================
217
+
218
+* freezer README doc
219
+
220
+* freezer-api README doc
221
+
222
+* freezer-web-ui README doc
223
+
224
+References
225
+==========
226
+
227
+.. [1] https://review.openstack.org/#/c/409796/
228
+.. [2] https://etherpad.openstack.org/p/freezer_mysql_migration
229
+.. [3] https://etherpad.openstack.org/p/freezer_new_archi

+ 284
- 0
specs/pike/approved/cinder_os_brick.rst View File

@@ -0,0 +1,284 @@
1
+..
2
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
3
+ License.
4
+
5
+ http://creativecommons.org/licenses/by/3.0/legalcode
6
+
7
+=============================================
8
+Freezer Cinder Volumes backup using OS Bricks
9
+=============================================
10
+
11
+* https://blueprints.launchpad.net/freezer/+spec/cinder-osbrick
12
+
13
+Provide efficient way to backup Cinder Volumes leveraging os_bricks
14
+
15
+Problem description
16
+===================
17
+
18
+Currently Freezer provides basic features to execute Cinder volumes backup.
19
+The current approach present significant challenges,
20
+due mainly to the difficulty of downloading Cinder Volumes without passing
21
+through Glance. This can be an issue for time and scalability reasons,
22
+(i.e. volumes of few hundreds GB size, potential error probability increase,
23
+as more services are part of the process, unailability of cinder-backup)
24
+
25
+Use Cases
26
+---------
27
+
28
+* Users that want to backup cinder volumes.
29
+
30
+* Store backed up volumes in a different storage media than Swift.
31
+  This is important for disaster recovery purpose, as it should be
32
+  possible to restore the volume even if the swift or other services are
33
+  down in the original OpenStack deployment.
34
+
35
+* OpenStack distributions deployed without cinder-backup module.
36
+
37
+* Provide a more efficient way of executing incrementals backup.
38
+
39
+* Avoid uploading volumes image to Glance to be processed.
40
+
41
+* Volumes can be backed up while attached or detached (hot and cold).
42
+
43
+- Hot backup will be provide a crach consistent backup and the data  present
44
+  in the volumes can be accesses at all times during backup
45
+- If Cold backup is executed, the Volume is detached first,
46
+  then the backup is executed.
47
+
48
+
49
+Proposed change
50
+===============
51
+
52
+Implement in the freezer-agent a new engine called cinder-osbrick.
53
+
54
+The new engine cinder-osbrick execute backup and restore related
55
+operations direclty on the Volumes, without passing through Glance API.
56
+
57
+The freezer-agent needs to back up a single volume, all the volumes
58
+owned by the tenant or all volumes from all tenants (admin).
59
+Volumes backup and restore can happen in parallel (i.e. 10 Volumes
60
+simultaneously can be backup or restore)
61
+
62
+
63
+Technical details
64
+-----------------
65
+
66
+Openstack provide the os_brick library to attach volumes:
67
+
68
+* https://github.com/openstack/os-brick
69
+
70
+It mainly provides the following features:
71
+
72
+* Volumes discovery
73
+* Volumes attach
74
+* Volumes removal
75
+
76
+Related docs:
77
+
78
+* http://docs.openstack.org/developer/os-brick/api/index.html
79
+* http://docs.openstack.org/developer/os-brick/tutorial.html
80
+
81
+The python client module that could be used is brick-cinderclient-ex:
82
+
83
+* https://github.com/openstack/python-brick-cinderclient-ext
84
+
85
+It is preferrable to implement the Volumes related operations from cinder
86
+in python, rather wrapping around any possible related os-brick command.
87
+
88
+* The freezer-scheduler and the freezer-agent needs to support,
89
+  in the json and ini config file respectively, engine specific settings.
90
+
91
+
92
+Backup workflow with osbrick:
93
+-----------------------------
94
+
95
+* freezer-agent workflow:
96
+
97
+(Common steps)
98
+
99
+single-vol-backup:
100
+ - Backup any available metadata of the Volume
101
+ - A Snapshot is execute on the volume (--force if volume is attached)
102
+ - Snapshot is converted to Volume, in oder to be mounted using osbrick
103
+ - The new Volume is attached using os_brick. The Volume can be attached
104
+   using iSCSI, Local or FC according the information provided by
105
+   os_brick about the volume.
106
+ - The new Volume is mounted on the node where the freezer-agent is executing
107
+ - The freezer-agent will execute a backup of the volume content, starting
108
+   from the volume mount (i.e. volume root /)
109
+ - Every single file in the volume is backed up.
110
+ - If the execution is part of an incremental backup, each file/block is
111
+   compared against the previous execution.
112
+ - Data can be storage on each supported freezer storage backend
113
+ - When finished, the new Volume is detached
114
+ - Once detached, the new volume is removed
115
+
116
+Backup of a Single Volume:
117
+ 1) The freezer-agent take the volume id as input param (either from ini
118
+    file or json file provided to the scheduler):
119
+ 2) single-vol-backup from Common steps
120
+
121
+Backup of all Volumes owned by a tenant:
122
+ 1) freezer-agent discover all the volumes owned by the tenant from Cinder API
123
+ 2) Iterate over each Volume
124
+ 3) single-vol-backup from Common steps
125
+
126
+Backup of all Volumes (admin):
127
+ 1) freezer-agent get the list of all Volumes available from Cinder API
128
+ 2) Iterate over each volume
129
+ 3) single-vol-backup from Common steps
130
+
131
+Backup of all volumes part of a Consistency Group
132
+ 1) get list of all volumes from the Consistency Group. It can be provided
133
+    as a single element id or a list of elements comma separated:
134
+ 2) freezer-agent get the list of all Volumes available from Cinder API
135
+ 3) Iterate over each volume
136
+ 4) single-vol-backup from Common steps
137
+
138
+
139
+Restore workflow with osbrick
140
+-----------------------------
141
+
142
+* freezer-agent workflow:
143
+
144
+(Common steps)
145
+
146
+single-vol-restore:
147
+  - Get the original Volume metadata
148
+  - Check if the volume id exists
149
+  - If the same volume id exists
150
+
151
+    + snapshot the volume
152
+    + convert from snap to volume
153
+    + attach the volume
154
+    + mount the volume
155
+    + restore the backup data in the volume filesystem
156
+    + if meta-override option is provided, the volume metadata from backup
157
+      is applied to the current Volume meta
158
+  - If the volume id does not exist
159
+    + Create a new Volume with the same metadata from backup
160
+    + attach the volume with os-brick
161
+    + mount the volume
162
+    + restore the backup data in the volume filesystem
163
+  - unmount
164
+  - deattach the volum
165
+  - if remove_old_vol is provided, any existing volume not matching with the
166
+    new ones will be removed (Dangerous Option)
167
+
168
+Restote of a single volume:
169
+ 1) The freezer-agent take the volume id as input param (either from ini
170
+    file or json file provided to the scheduler):
171
+ 2) single-vol-restore from Common steps
172
+
173
+Restore of all volumes owned by a tenant:
174
+ 1) freezer-agent discover all the volumes owned by the tenant from Cinder API
175
+ 2) Iterate over each volume
176
+ 3) single-vol-restore from Common steps
177
+
178
+Restore of all volumes from all tenants (admin):
179
+ 1) freezer-agent get the list of all Volumes available from Cinder API
180
+ 2) Iterate over each volume
181
+ 3) single-vol-restore from Common steps
182
+
183
+Restore of all volumes part of a Consistency Group
184
+ 1) get list of all volumes from the Consistency Group. It can be provided as
185
+    a single element id or a list of elements comma separated:
186
+ 2) Iterate over each volume
187
+ 3) single-vol-restore from Common steps
188
+
189
+Data model impact
190
+-----------------
191
+* new engine in the db
192
+* DB model for single, all tenant, tenant owned volumes and consistency groups
193
+
194
+New Options to be added:
195
+------------------------
196
+* engine-os-brick
197
+* recreate-vol-on-error
198
+* meta-override
199
+* consistency groups [id]
200
+* all_tenants
201
+* all_tenant_volumes
202
+* single_volume_id
203
+* remove_old_vol
204
+
205
+Alternatives
206
+------------
207
+
208
+
209
+Impacts
210
+-------
211
+* freezer-agent
212
+* freezer-api
213
+* freezer-web-ui
214
+
215
+REST API impact
216
+-----------------------
217
+
218
+* API needs to support this new engine
219
+
220
+Security impact
221
+---------------------
222
+
223
+None
224
+
225
+Notifications impact
226
+---------------------------
227
+
228
+TBD.
229
+
230
+Other end user impact
231
+------------------------------
232
+
233
+None. TBD.
234
+
235
+Performance Impact
236
+------------------
237
+
238
+None.
239
+
240
+Other deployer impact
241
+------------------------------
242
+
243
+
244
+Developer impact
245
+------------------------
246
+
247
+
248
+Implementation
249
+==============
250
+
251
+Assignee(s)
252
+-----------------
253
+
254
+Primary assignee:
255
+
256
+Other contributors:
257
+  daemontool
258
+
259
+Work Items
260
+----------
261
+
262
+
263
+Dependencies
264
+============
265
+
266
+
267
+Testing
268
+=======
269
+
270
+TBD.
271
+
272
+Documentation Impact
273
+====================
274
+
275
+
276
+* Freezer API installation doc
277
+* Freezer agent docu
278
+* Freezer web ui doc
279
+
280
+References
281
+==========
282
+
283
+.. https://etherpad.openstack.org/p/freezer_cinder-os-brick
284
+

+ 305
- 0
specs/pike/approved/relational_db_schema_with_oslo_db.rst View File

@@ -0,0 +1,305 @@
1
+..
2
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
3
+ License.
4
+
5
+ http://creativecommons.org/licenses/by/3.0/legalcode
6
+
7
+=================================
8
+Relational DB Schema with OSLO.DB
9
+=================================
10
+
11
+https://blueprints.launchpad.net/freezer/+spec/oslo.db
12
+
13
+Taking advantage of the oslo.db library to have a more uniform database
14
+backend architecture to other OpenStack projects.
15
+
16
+Problem description
17
+===================
18
+
19
+Currently Freezer uses Elastic Search (ES) as a database backend, which
20
+is a NoSQL database specialized for ranked query results. Elastic Search
21
+adds additional complexity to an OpenStack system. Most of the
22
+components use a relational database management system (DBMS like MySQL or
23
+PostgreSQL) which are more common. It is more familiar how to
24
+maintain, troubleshoot and develop on top of relational databases.
25
+
26
+Since Freezer related data turned out to be relational, it would be more
27
+convenient to use it trough the oslo.db pattern library. Using it, the
28
+database mapping would be more uniform to other OpenStack projects.
29
+It would be less challenging for new developers to contribute.
30
+
31
+Use Cases
32
+---------
33
+
34
+* For new developers, already familiar with OpenStack, it should be less
35
+  challenging to get familiar with the backend code, since most of the
36
+  OpenStack projects use relational database backend trough the oslo.db
37
+  pattern library.
38
+
39
+* For Deployers there would be no longer needed to set up a special
40
+  DBMS just for Freezer, since it could share the relational DBMS used
41
+  by the other (core) OpenStack projects, still well isolated in it's
42
+  own database.
43
+
44
+* For End User it would be less difficult to maintain, since Freezer
45
+  would not add additional complexity with a less common component,
46
+  instead it can take advantage of the DBMS that is already deployed for
47
+  OpenStack.
48
+
49
+Proposed change
50
+===============
51
+
52
+Implementing the entities using oslo.db and SQLAlchemy base classes.
53
+And expose the new entities trough the REST API.
54
+
55
+Alternatives
56
+------------
57
+
58
+Oslo.db with SQLAlchemy is the de facto standard for OpenStack projects
59
+to implement database backends with relational DBMS. It provides high
60
+level ORM mapping and abstracts the different database backends.
61
+Therefore we gain compability with multiple relational DBMS just like
62
+any other OpenStack component using oslo.db.
63
+
64
+Using other alternative would either not be more uniform to other
65
+OpenStack project tooling, either we would have to implement low level,
66
+directly to a specific database driver (just like now with ES).
67
+
68
+Data model impact
69
+-----------------
70
+
71
+Changes which require modifications to the data model often have a wider impact
72
+on the system.  The community often has strong opinions on how the data model
73
+should be evolved, from both a functional and performance perspective. It is
74
+therefore important to capture and gain agreement as early as possible on any
75
+proposed changes to the data model.
76
+
77
+Questions which need to be addressed by this section include:
78
+
79
+* What new data objects and/or database schema changes is this going to
80
+  require?
81
+
82
+* What database migrations will accompany this change.
83
+
84
+* How will the initial set of new data objects be generated, for example if you
85
+  need to take into account existing backups/jobs/... , or modify other
86
+  existing data describe how that will work.
87
+
88
+
89
+As there will be a brand new relational database schema [MIG1]_:
90
+
91
+
92
+Clients
93
+
94
+    id (varchar) [p_key]
95
+
96
+    project_id (uuid)
97
+
98
+    config_id (varchar)
99
+
100
+    description (varchar)
101
+
102
+    uuid (uuid)
103
+
104
+
105
+
106
+Actions
107
+
108
+    id (uuid) [p_key]
109
+
110
+    action (varchar)
111
+
112
+    project_id (uuid)
113
+
114
+    mode (varchar)
115
+
116
+    src_file (varchar)
117
+
118
+    backup_name (varchar)
119
+
120
+    container (varchar(
121
+
122
+    restore_abs_path (varchar)
123
+
124
+
125
+
126
+Action_reports
127
+
128
+    id (uuid) [p_key]
129
+
130
+    action_id (uuid) [f_key]
131
+
132
+    action_attachment_id (uuid) [f_key]
133
+
134
+    project_id (uuid)
135
+
136
+    result (varchar)
137
+
138
+    time_elapsed (varchar)
139
+
140
+    metadata (JSON)
141
+
142
+    report_date (timestamp)
143
+
144
+    log (blob) < only on failure
145
+
146
+
147
+
148
+Jobs
149
+
150
+    id (uuid) [p_key]
151
+
152
+    project_id (uuid)
153
+
154
+    scheduling (JSON)
155
+
156
+    description (varchar)
157
+
158
+
159
+
160
+Action_attachments
161
+
162
+    id (uuid) [p_key]
163
+
164
+    action_id (uuid) [f_key]
165
+
166
+    job_id (uuid) [f_key]
167
+
168
+    project_id (uuid)
169
+
170
+    priority (int)
171
+
172
+    retries (int)
173
+
174
+    retry_interval (int)
175
+
176
+    mandatory (bool)
177
+
178
+
179
+
180
+Sessions
181
+
182
+    id (uuid) [p_key]
183
+
184
+    project_id (uuid)
185
+
186
+    scheduling (JSON)
187
+
188
+    policy (varchar)
189
+
190
+
191
+
192
+Job_attachments
193
+
194
+    id (uuid) [p_key]
195
+
196
+    client_id (varchar) [f_key]
197
+
198
+    job_id (uuid) [f_key]
199
+
200
+    session_id (uuid) [f_key]
201
+
202
+    project_id (uuid)
203
+
204
+    event (varchar)
205
+
206
+    status (varchar)
207
+
208
+    current_pid (int)
209
+
210
+
211
+REST API impact
212
+---------------
213
+
214
+There should be a new v2 API implemented. TBD.
215
+
216
+Security impact
217
+---------------
218
+
219
+None
220
+
221
+Notifications impact
222
+--------------------
223
+
224
+TBD.
225
+
226
+Other end user impact
227
+---------------------
228
+
229
+None. TBD.
230
+
231
+Performance Impact
232
+------------------
233
+
234
+None.
235
+
236
+Other deployer impact
237
+---------------------
238
+
239
+* The Elastic Search configurations should be replaced with oslo.db
240
+  configurations
241
+
242
+* When updating from a previous version there must be a data migration
243
+  from ES to oslo.db (this will be addressed by a nother spec - TBD).
244
+
245
+Developer impact
246
+----------------
247
+
248
+There will be no longer needed to deploy ES.
249
+
250
+Implementation
251
+==============
252
+
253
+Assignee(s)
254
+-----------
255
+
256
+Primary assignee:
257
+  neilus
258
+
259
+Other contributors:
260
+  daemontool
261
+
262
+Work Items
263
+----------
264
+
265
+Work items or tasks -- break the feature up into the things that need to be
266
+done to implement it. Those parts might end up being done by different people,
267
+but we're mostly trying to understand the timeline for implementation.
268
+
269
+* implementing the database models
270
+
271
+* create adapter for API v1(?) and v2
272
+
273
+* implementing the CRUD API
274
+
275
+* updating the devStack plugin
276
+
277
+* updating documentation
278
+
279
+Dependencies
280
+============
281
+
282
+* Implementing the database migration script (TBD), which migrates data
283
+  from ES to oslo.db backend DB.
284
+
285
+* We will be using oslo.db library and SQLAlchemy for iplementation.
286
+
287
+Testing
288
+=======
289
+
290
+TBD.
291
+
292
+Documentation Impact
293
+====================
294
+
295
+TBD.
296
+
297
+* Freezer API installation doc
298
+
299
+References
300
+==========
301
+
302
+.. [MIG1] https://etherpad.openstack.org/p/freezer_mysql_migration
303
+
304
+.. https://etherpad.openstack.org/p/freezer_db_switch
305
+

+ 0
- 0
specs/pike/redirects View File


Loading…
Cancel
Save