Browse Source

Block based backup support (rsync)

This is a spec for block based support implementation.
Using rsync as approach.

Change-Id: I38e73ac970b35bceddfcc382b6e533b4629211b6
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
changes/38/413638/4
Ruslan Aliev 2 years ago
parent
commit
a853067a13
3 changed files with 235 additions and 5 deletions
  1. 1
    0
      README.rst
  2. 5
    5
      requirements.txt
  3. 229
    0
      specs/ocata/approved/block_based_backup_support.rst

+ 1
- 0
README.rst View File

@@ -4,6 +4,7 @@ Team and repository tags
4 4
 
5 5
 .. image:: http://governance.openstack.org/badges/freezer-specs.svg
6 6
     :target: http://governance.openstack.org/reference/tags/index.html
7
+    :remote:
7 8
 
8 9
 .. Change things from this point on
9 10
 

+ 5
- 5
requirements.txt View File

@@ -1,6 +1,6 @@
1
-oslosphinx
2
-pbr>=1.6  # Apache-2.0
3
-sphinx>=1.1.2,<1.2
4
-testrepository>=0.0.18
5
-testtools>=0.9.34
1
+oslosphinx>=4.7.0  # Apache-2.0
2
+pbr>=1.8  # Apache-2.0
3
+sphinx>=1.2.1,!=1.3b1,<1.4  # BSD
4
+testrepository>=0.0.18  # Apache-2.0/BSD
5
+testtools>=1.4.0  # MIT
6 6
 yasfb>=0.5.1

+ 229
- 0
specs/ocata/approved/block_based_backup_support.rst View File

@@ -0,0 +1,229 @@
1
+..
2
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
3
+ License.
4
+
5
+ http://creativecommons.org/licenses/by/3.0/legalcode
6
+
7
+==================================
8
+Block based backup support (rsync)
9
+==================================
10
+
11
+https://blueprints.launchpad.net/freezer/+spec/rsync
12
+
13
+Taking advantage of the rsync to provide a possibility to create
14
+space/bandwidth efficient backups.
15
+
16
+Problem description
17
+===================
18
+
19
+Currently Freezer checks only ctime and mtime inode information
20
+to verify if files are changed or not (tar functionality). While
21
+this approach gives speed (time efficient), it is not bandwidth
22
+and storage efficient. Freezer needs to support both rsync and tar
23
+approach to execute incremental backups and restore.
24
+
25
+Since Freezer will provide two options for incremental backups, it
26
+would be more convenient to choose the best approach to backup data
27
+in accordance with each particular case (more speed or storage/bandwidth
28
+efficient).
29
+
30
+Use Cases
31
+---------
32
+
33
+* For developers, this change will not create negative impacts because
34
+  this code will be gracefully bundled in Freezer engine API and
35
+  will not cause any major changes in Freezer architecture.
36
+
37
+* For Deployers there is no need to install any additional components,
38
+  Freezer will use it's own implementation of rsync algorithm
39
+  (written in Python).
40
+
41
+* For End User it would be less difficult to select more efficient
42
+  option for create backups based on dataset (e.g. few big files or a lot of
43
+  small files) for backup and speed/storage/bandwidth requirements,
44
+  since Freezer would support both rsync and tar approaches.
45
+
46
+Proposed change
47
+===============
48
+
49
+Implementing the new engine classes for rsync (as well as for tar).
50
+Providing new engine (-e) choice in config.
51
+
52
+For this type of backup will be created following metadata structure:
53
+
54
+files_meta = {
55
+            'files': {},
56
+            'directories': {},
57
+            'meta': {
58
+                'broken_links_tot': '',
59
+                'total_files': '',
60
+                'total_directories': '',
61
+                'backup_size_on_disk': '',
62
+                'backup_size_uncompressed': '',
63
+                'backup_size_compressed': '',
64
+                'platform': sys.platform
65
+            },
66
+            'abs_backup_path': os.getcwd(),
67
+            'broken_links': [],
68
+            'rsync_struct_ver': RSYNC_DATA_STRUCT_VERSION,
69
+            'rsync_block_size': RSYNC_BLOCK_SIZE}
70
+
71
+
72
+file_meta = {'inode': {
73
+                      'inumber': os_stat.st_ino,
74
+                      'nlink': os_stat.st_nlink,
75
+                      'mode': file_mode,
76
+                      'uid': os_stat.st_uid,
77
+                      'gid': os_stat.st_gid,
78
+                      'size': os_stat.st_size,
79
+                      'devmajor': os.major(dev),
80
+                      'devminor': os.minor(dev),
81
+                      'mtime': mtime,
82
+                      'ctime': ctime,
83
+                      'uname': uname,
84
+                      'gname': gname,
85
+                      'ftype': file_type,
86
+                      'lname': lname,
87
+                      'rsync_block_size': rsync_block_size,
88
+                      'file_status: status
89
+                      }
90
+            }
91
+
92
+Current version of implementation you always can find here [1].
93
+
94
+Alternatives
95
+------------
96
+
97
+Because of the flexibility, speed, and scriptability of rsync, it has
98
+become a standard Linux utility, included in all popular Linux distributions.
99
+It has been ported to Windows (via Cygwin, Grsync, or SFU), FreeBSD, NetBSD,
100
+OpenBSD, and Mac OS. De facto, rsync is the default fallback for most data
101
+transfers. It has a clear algorithm written for 20 years ago and different
102
+variations (e.g. acrosync, zsync, etc). librsync is used by Dropbox.
103
+
104
+Using other alternative (like bbcp or lftp) would not be more effective
105
+or portable solution.
106
+
107
+Data model impact
108
+-----------------
109
+
110
+Changes in data model has already described in oslo.db migration document.
111
+Actions entity should contain 'engine' field for performing appropriate action
112
+using particular type of engine (tar, rsync or openstack).
113
+
114
+From new relational database schema:
115
+
116
+Actions
117
+    action_id (uuid) [p_key]
118
+    resource (varchar)
119
+    type (varchar)
120
+    name (varchar)
121
+    application (varchar)
122
+    engine (varchar)    <-- Require this
123
+    snapshot (varchar)
124
+    storage (varchar)
125
+    global_options (JSON)
126
+    application_options (JSON)
127
+    storage_options (JSON)
128
+    snapshot_options (JSON)
129
+    engine_options (JSON)
130
+
131
+
132
+REST API impact
133
+---------------
134
+
135
+None.
136
+
137
+Security impact
138
+---------------
139
+
140
+None.
141
+
142
+Notifications impact
143
+--------------------
144
+
145
+There are no special logs will be added, just some info messages about
146
+start/stop backup process, backup metrics, etc.
147
+
148
+Other end user impact
149
+---------------------
150
+
151
+* There are no additional changes to python-freezerclient CLI. To choice
152
+  appropriate engine for action, end user should specify 'engine' field
153
+  in provided JSON configuration in case of creating or updating action.
154
+
155
+* freezer-web-ui should provide additional 'engine' field in 'Action
156
+  Configuration' window. It has to be drop-down list with values 'tar',
157
+  'rsync' or 'openstack'.
158
+
159
+Performance Impact
160
+------------------
161
+
162
+None.
163
+
164
+Other deployer impact
165
+---------------------
166
+
167
+Will be added new choice to freezer-agent -e (engine) option - 'rsync'.
168
+
169
+Developer impact
170
+----------------
171
+
172
+None.
173
+
174
+Implementation
175
+==============
176
+
177
+Assignee(s)
178
+-----------
179
+
180
+Primary assignee:
181
+  Ruslan Aliev (raliev) <raliev@mirantis.com>
182
+
183
+Other contributors:
184
+  Fausto Marzi (daemontool) <fausto.marzi@ericsson.com>
185
+
186
+Work Items
187
+----------
188
+
189
+* implementing the new engine (rsync)
190
+
191
+* bundling this engine to freezer code (API calls) and mechanism
192
+  for using this engine ('-e rsync' option)
193
+
194
+* implementing the new database schema for actions (oslo.db migration)
195
+
196
+* updating freezer-web-ui 'Action Configuration' window
197
+
198
+* updating documentation
199
+
200
+Dependencies
201
+============
202
+
203
+* This spec depends on Freezer oslo.db migration [2].
204
+
205
+* Pluggable engines described here [3].
206
+
207
+* There are no additional library dependencies.
208
+
209
+Testing
210
+=======
211
+
212
+There is a question - do we actually need separate tempest test
213
+for this change or we can be satisfied with existing one?
214
+
215
+Documentation Impact
216
+====================
217
+
218
+* freezer README doc
219
+
220
+* freezer-api README doc
221
+
222
+* freezer-web-ui README doc
223
+
224
+References
225
+==========
226
+
227
+.. [1] https://review.openstack.org/#/c/409796/
228
+.. [2] https://etherpad.openstack.org/p/freezer_mysql_migration
229
+.. [3] https://etherpad.openstack.org/p/freezer_new_archi

Loading…
Cancel
Save