Block based backup support (rsync)

This is a spec for block based support implementation.
Using rsync as approach.

Change-Id: I38e73ac970b35bceddfcc382b6e533b4629211b6
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
This commit is contained in:
Ruslan Aliev 2016-12-21 17:54:12 +04:00
parent 20256d7277
commit a853067a13
3 changed files with 235 additions and 5 deletions

View File

@ -4,6 +4,7 @@ Team and repository tags
.. image:: http://governance.openstack.org/badges/freezer-specs.svg .. image:: http://governance.openstack.org/badges/freezer-specs.svg
:target: http://governance.openstack.org/reference/tags/index.html :target: http://governance.openstack.org/reference/tags/index.html
:remote:
.. Change things from this point on .. Change things from this point on

View File

@ -1,6 +1,6 @@
oslosphinx oslosphinx>=4.7.0 # Apache-2.0
pbr>=1.6 # Apache-2.0 pbr>=1.8 # Apache-2.0
sphinx>=1.1.2,<1.2 sphinx>=1.2.1,!=1.3b1,<1.4 # BSD
testrepository>=0.0.18 testrepository>=0.0.18 # Apache-2.0/BSD
testtools>=0.9.34 testtools>=1.4.0 # MIT
yasfb>=0.5.1 yasfb>=0.5.1

View File

@ -0,0 +1,229 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==================================
Block based backup support (rsync)
==================================
https://blueprints.launchpad.net/freezer/+spec/rsync
Taking advantage of the rsync to provide a possibility to create
space/bandwidth efficient backups.
Problem description
===================
Currently Freezer checks only ctime and mtime inode information
to verify if files are changed or not (tar functionality). While
this approach gives speed (time efficient), it is not bandwidth
and storage efficient. Freezer needs to support both rsync and tar
approach to execute incremental backups and restore.
Since Freezer will provide two options for incremental backups, it
would be more convenient to choose the best approach to backup data
in accordance with each particular case (more speed or storage/bandwidth
efficient).
Use Cases
---------
* For developers, this change will not create negative impacts because
this code will be gracefully bundled in Freezer engine API and
will not cause any major changes in Freezer architecture.
* For Deployers there is no need to install any additional components,
Freezer will use it's own implementation of rsync algorithm
(written in Python).
* For End User it would be less difficult to select more efficient
option for create backups based on dataset (e.g. few big files or a lot of
small files) for backup and speed/storage/bandwidth requirements,
since Freezer would support both rsync and tar approaches.
Proposed change
===============
Implementing the new engine classes for rsync (as well as for tar).
Providing new engine (-e) choice in config.
For this type of backup will be created following metadata structure:
files_meta = {
'files': {},
'directories': {},
'meta': {
'broken_links_tot': '',
'total_files': '',
'total_directories': '',
'backup_size_on_disk': '',
'backup_size_uncompressed': '',
'backup_size_compressed': '',
'platform': sys.platform
},
'abs_backup_path': os.getcwd(),
'broken_links': [],
'rsync_struct_ver': RSYNC_DATA_STRUCT_VERSION,
'rsync_block_size': RSYNC_BLOCK_SIZE}
file_meta = {'inode': {
'inumber': os_stat.st_ino,
'nlink': os_stat.st_nlink,
'mode': file_mode,
'uid': os_stat.st_uid,
'gid': os_stat.st_gid,
'size': os_stat.st_size,
'devmajor': os.major(dev),
'devminor': os.minor(dev),
'mtime': mtime,
'ctime': ctime,
'uname': uname,
'gname': gname,
'ftype': file_type,
'lname': lname,
'rsync_block_size': rsync_block_size,
'file_status: status
}
}
Current version of implementation you always can find here [1].
Alternatives
------------
Because of the flexibility, speed, and scriptability of rsync, it has
become a standard Linux utility, included in all popular Linux distributions.
It has been ported to Windows (via Cygwin, Grsync, or SFU), FreeBSD, NetBSD,
OpenBSD, and Mac OS. De facto, rsync is the default fallback for most data
transfers. It has a clear algorithm written for 20 years ago and different
variations (e.g. acrosync, zsync, etc). librsync is used by Dropbox.
Using other alternative (like bbcp or lftp) would not be more effective
or portable solution.
Data model impact
-----------------
Changes in data model has already described in oslo.db migration document.
Actions entity should contain 'engine' field for performing appropriate action
using particular type of engine (tar, rsync or openstack).
From new relational database schema:
Actions
action_id (uuid) [p_key]
resource (varchar)
type (varchar)
name (varchar)
application (varchar)
engine (varchar) <-- Require this
snapshot (varchar)
storage (varchar)
global_options (JSON)
application_options (JSON)
storage_options (JSON)
snapshot_options (JSON)
engine_options (JSON)
REST API impact
---------------
None.
Security impact
---------------
None.
Notifications impact
--------------------
There are no special logs will be added, just some info messages about
start/stop backup process, backup metrics, etc.
Other end user impact
---------------------
* There are no additional changes to python-freezerclient CLI. To choice
appropriate engine for action, end user should specify 'engine' field
in provided JSON configuration in case of creating or updating action.
* freezer-web-ui should provide additional 'engine' field in 'Action
Configuration' window. It has to be drop-down list with values 'tar',
'rsync' or 'openstack'.
Performance Impact
------------------
None.
Other deployer impact
---------------------
Will be added new choice to freezer-agent -e (engine) option - 'rsync'.
Developer impact
----------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Ruslan Aliev (raliev) <raliev@mirantis.com>
Other contributors:
Fausto Marzi (daemontool) <fausto.marzi@ericsson.com>
Work Items
----------
* implementing the new engine (rsync)
* bundling this engine to freezer code (API calls) and mechanism
for using this engine ('-e rsync' option)
* implementing the new database schema for actions (oslo.db migration)
* updating freezer-web-ui 'Action Configuration' window
* updating documentation
Dependencies
============
* This spec depends on Freezer oslo.db migration [2].
* Pluggable engines described here [3].
* There are no additional library dependencies.
Testing
=======
There is a question - do we actually need separate tempest test
for this change or we can be satisfied with existing one?
Documentation Impact
====================
* freezer README doc
* freezer-api README doc
* freezer-web-ui README doc
References
==========
.. [1] https://review.openstack.org/#/c/409796/
.. [2] https://etherpad.openstack.org/p/freezer_mysql_migration
.. [3] https://etherpad.openstack.org/p/freezer_new_archi