openstack
/
freezer
Code Issues Proposed changes

Removed remnant of git conflict in README.rst 

Change-Id: I5eb26a17c480c9a68041b0341eb7687a268b5013
Closes-Bug: #1600183
This commit is contained in:
Domhnall Walsh 2016-07-08 12:26:00 +01:00
parent d5cf47c4eb
commit f967745fdf
1 changed files with 80 additions and 59 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

139
README.rst
View File

@ -37,7 +37,7 @@ The following features are available:
- Backup file system using point-in-time snapshot - Backup file system using point-in-time snapshot
- Strong encryption supported: AES-256-CFB - Strong encryption supported: AES-256-CFB
- Backup file system tree directly (without volume snapshot) - Backup file system tree directly (without volume snapshot)
- Backup journaled MongoDB directory tree using lvm snapshot to Swift - Backup journalled MongoDB directory tree using lvm snapshot to Swift
- Backup MySQL with lvm snapshot - Backup MySQL with lvm snapshot
- Restore data from a specific date automatically to file system - Restore data from a specific date automatically to file system
- Low storage consumption as the backup are uploaded as a stream - Low storage consumption as the backup are uploaded as a stream
@ -60,7 +60,7 @@ Freezer Components
| Component | Description | | Component | Description |
+===================+================================================================================================================================================+ +===================+================================================================================================================================================+
| Freezer Web UI | Web interface that interacts with the Freezer API to configure and change settings. | | Freezer Web UI | Web interface that interacts with the Freezer API to configure and change settings. |
| | It provides most of the features from the Freezer Agent CLI, advanced scheduler settings such as multi-node backup synchronization, | | | It provides most of the features from the Freezer Agent CLI, advanced scheduler settings such as multi-node backup synchronization, |
| | metrics, and reporting. | | | metrics, and reporting. |
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+ +-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Freezer Scheduler | A client side component, running on the node where the data backup is to be executed. | | Freezer Scheduler | A client side component, running on the node where the data backup is to be executed. |
@ -75,7 +75,7 @@ Freezer Components
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+ +-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Freezer Agent | Multiprocessing Python software that runs on the client side, where the data backup is to be executed. | | Freezer Agent | Multiprocessing Python software that runs on the client side, where the data backup is to be executed. |
| | It can be executed standalone or by the Freezer Scheduler. | | | It can be executed standalone or by the Freezer Scheduler. |
| | The Freezer Agent provides a flexible way to execute backup, restore and other actions on a running system. | | | The Freezer Agent provides a flexible way to execute backup, restore and other actions on a running system. |
| | In order to provide flexibility in terms of data integrity, speed, performance, resources usage, etc the freezer agent offers a | | | In order to provide flexibility in terms of data integrity, speed, performance, resources usage, etc the freezer agent offers a |
| | wide range of options to execute optimized backup according the available resources as: | | | wide range of options to execute optimized backup according the available resources as: |
| | | | | |
@ -88,7 +88,7 @@ Freezer Components
| | - Parallel upload to pluggable storage media (i.e., upload backup to swift and to a remote node by ssh, | | | - Parallel upload to pluggable storage media (i.e., upload backup to swift and to a remote node by ssh, |
| | or upload to two or more independent swift instances with different credentials, etc) | | | or upload to two or more independent swift instances with different credentials, etc) |
| | - Execute file based incremental (like tar), block based incremental (like rsync algorithm) and differential based backup and restore | | | - Execute file based incremental (like tar), block based incremental (like rsync algorithm) and differential based backup and restore |
| | - Multiplatform as it can be run on Linux, Windows, \*BSD and OSX | | | - Multi-platform as it can be run on Linux, Windows, \*BSD and OSX |
| | - Automatic removal of old backups | | | - Automatic removal of old backups |
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+ +-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Freezer API | The API is used to store and provide metadata to the Freezer Web UI and to the Freezer Scheduler. | | Freezer API | The API is used to store and provide metadata to the Freezer Web UI and to the Freezer Scheduler. |
@ -337,6 +337,7 @@ Execute a MySQL backup using lvm snapshot::
freezer_mysql-backup-prod --mode mysql --backup-name mysql-ops002 freezer_mysql-backup-prod --mode mysql --backup-name mysql-ops002
Cinder backups Cinder backups
==============
To make a cinder backup you should provide cinder-vol-id or cindernative-vol-id To make a cinder backup you should provide cinder-vol-id or cindernative-vol-id
parameters in command line arguments. Freezer doesn't do any additional checks parameters in command line arguments. Freezer doesn't do any additional checks
@ -355,6 +356,7 @@ Execute a MySQL backup with Cinder::
--cinder-vol-id 3ad7a62f-217a-48cd-a861-43ec0a04a78b --cinder-vol-id 3ad7a62f-217a-48cd-a861-43ec0a04a78b
Nova backups Nova backups
============
To make a Nova backup you should provide a Nova parameter in the arguments. To make a Nova backup you should provide a Nova parameter in the arguments.
Freezer doesn't do any additional checks and assumes that making a backup Freezer doesn't do any additional checks and assumes that making a backup
@ -375,69 +377,78 @@ All the freezer-agent activities are logged into /var/log/freezer.log.
Swift, Local and SSH Storage Swift, Local and SSH Storage
----------------------------- ----------------------------
Freezer can use: Freezer can use:
local storage - a folder that is available in the same OS (may be mounted) * local storage - a folder that is available in the same OS (may be mounted)
* Swift storage - OS object storage
* SSH storage - a folder on a remote machine
To use local storage specify "--storage local" Local Storage
And use "--container <path-to-folder-with-backups>" =============
Backup example::
To use local storage specify "--storage local"
And use "--container <path-to-folder-with-backups>"
Backup example::
$ sudo freezer-agent --path-to-backup /data/dir/to/backup $ sudo freezer-agent --path-to-backup /data/dir/to/backup
--container /tmp/my_backup_path/ --backup-name my-backup-name --container /tmp/my_backup_path/ --backup-name my-backup-name
--storage local --storage local
Restore example:: Restore example::
$ sudo freezer-agent --action restore --restore-abs-path /data/dir/to/backup $ sudo freezer-agent --action restore --restore-abs-path /data/dir/to/backup
--container /tmp/my_backup_path/ --backup-name my-backup-name --container /tmp/my_backup_path/ --backup-name my-backup-name
--storage local --storage local
Swift storage - OS object storage
To use swift storage specify "--storage swift" or omit "--storage" parameter Swift storage
altogether (Swift storage is the default). =============
And use "--container <swift-container-name>"
Backup example:: To use swift storage specify "--storage swift" or omit "--storage" parameter
altogether (Swift storage is the default).
And use "--container <swift-container-name>"
Backup example::
$ sudo freezer-agent --path-to-backup /data/dir/to/backup $ sudo freezer-agent --path-to-backup /data/dir/to/backup
--container freezer-container --backup-name my-backup-name --container freezer-container --backup-name my-backup-name
--storage swift --storage swift
Restore example:: Restore example::
$ sudo freezer-agent --action restore --restore-abs-path /data/dir/to/backup $ sudo freezer-agent --action restore --restore-abs-path /data/dir/to/backup
--container freezer-container --backup-name my-backup-name --container freezer-container --backup-name my-backup-name
--storage swift --storage swift
SSH storage - a folder on a remote machine SSH storage
===========
To use ssh storage specify "--storage ssh" To use ssh storage specify "--storage ssh"
And use "--container <path-to-folder-with-backups-on-remote-machine>" And use "--container <path-to-folder-with-backups-on-remote-machine>"
Also you should specify ssh-username, ssh-key and ssh-host parameters. Also you should specify ssh-username, ssh-key and ssh-host parameters.
ssh-port is optional parameter, default is 22. ssh-port is optional parameter, default is 22.
ssh-username for user ubuntu should be "--ssh-username ubuntu" ssh-username for user ubuntu should be "--ssh-username ubuntu"
ssh-key should be path to your secret ssh key "--ssh-key <path-to-secret-key>" ssh-key should be path to your secret ssh key "--ssh-key <path-to-secret-key>"
ssh-host can be ip of remote machine or resolvable dns name "--ssh-host 8.8.8.8" ssh-host can be ip of remote machine or resolvable dns name "--ssh-host 8.8.8.8"
Backup example:: Backup example::
$ sudo freezer-agent --path-to-backup /data/dir/to/backup $ sudo freezer-agent --path-to-backup /data/dir/to/backup
--container /remote-machine-path/ --backup-name my-backup-name --container /remote-machine-path/ --backup-name my-backup-name
--storage ssh --ssh-username ubuntu --ssh-key ~/.ssh/id_rsa --storage ssh --ssh-username ubuntu --ssh-key ~/.ssh/id_rsa
--ssh-host 8.8.8.8 --ssh-host 8.8.8.8
Restore example:: Restore example::
$ sudo freezer-agent --action restore --restore-abs-pat /data/dir/to/backup $ sudo freezer-agent --action restore --restore-abs-pat /data/dir/to/backup
--container /remote-machine-path/ --backup-name my-backup-name --container /remote-machine-path/ --backup-name my-backup-name
--storage ssh --ssh-username ubuntu --ssh-key ~/.ssh/id_rsa --storage ssh --ssh-username ubuntu --ssh-key ~/.ssh/id_rsa
--ssh-host 8.8.8.8 --ssh-host 8.8.8.8
Restore Restore
------- -------
@ -497,10 +508,10 @@ Remove backups older then 1 day::
$ freezer-agent --action admin --container freezer_dev-test --remove-older-then 1 --backup-name dev-test-01 $ freezer-agent --action admin --container freezer_dev-test --remove-older-then 1 --backup-name dev-test-01
Cinder restore currently creates a volume with the contents of the saved one, but Cinder restore currently creates a volume with the contents of the saved one,
doesn't implement deattach of existing volume and attach of the new one to the but doesn't implement detachment of existing volume and attachment of the new
vm. You should implement these steps manually. To create a new volume from one to the vm. You should implement these steps manually. To create a new volume
existing content run the next command: from existing content run the next command:
Execute a cinder restore:: Execute a cinder restore::
@ -574,18 +585,14 @@ The Freezer architecture is composed of the following components:
| | - Execute file based incremental (like tar), block based incremental (like rsync algorithm) and differential based backup and restore | | | - Execute file based incremental (like tar), block based incremental (like rsync algorithm) and differential based backup and restore |
| | - Multi-platform as it can be run on Linux, Windows, \*BSD and OSX | | | - Multi-platform as it can be run on Linux, Windows, \*BSD and OSX |
| | - Automatic removal of old backups | | | - Automatic removal of old backups |
<<<<<<< HEAD
| | | | | |
| | freezerc binary is deprecated and won't be available in next releases. Use freezer-agent instead. | | | "freezerc" binary is deprecated and won't be available in next releases. Use freezer-agent instead. |
=======
| | "freezerc" binary is deprecated and won't be available in next releases. Use freezer-agent instead . |
>>>>>>> New Better & Well Formatted Docs with Sphinx
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+ +-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Freezer API | The API is used to store and provide metadata to the Freezer Web UI and to the Freezer Scheduler. | | Freezer API | The API is used to store and provide metadata to the Freezer Web UI and to the Freezer Scheduler. |
| | Also the API is used to store session information for multi node backup synchronization. No workload data is stored in the API. | | | Also the API is used to store session information for multi node backup synchronization. No workload data is stored in the API. |
| | For more information on the API please refer to: freezer_api/README.rst | | | For more information on the API please refer to: freezer_api/README.rst |
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+ +-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| DB Elasticsearch | Backend used by the API to store and retrieve metrics, metadata sessions information, job status, etc. | | DB ElasticSearch | Backend used by the API to store and retrieve metrics, metadata sessions information, job status, etc. |
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+ +-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
Freezer currently uses GNU Tar under the hood to execute incremental backup and Freezer currently uses GNU Tar under the hood to execute incremental backup and
@ -678,7 +685,8 @@ Nova and Cinder Backups
If our data is stored on cinder volume or nova instance disk, we can implement If our data is stored on cinder volume or nova instance disk, we can implement
file backup using nova snapshots or volume backups. file backup using nova snapshots or volume backups.
Nova backups: Nova backups
============
If you provide nova arguments in the parameters, freezer assumes that all If you provide nova arguments in the parameters, freezer assumes that all
necessary data is located on instance disk and it can be successfully stored necessary data is located on instance disk and it can be successfully stored
@ -698,7 +706,8 @@ this snapshot. Instance will have different id and old instance should be
terminated manually. terminated manually.
Cinder backups: Cinder backups
==============
Cinder has its own mechanism for backups, and freezer supports it. But it also Cinder has its own mechanism for backups, and freezer supports it. But it also
allows creating a glance image from volume and uploading to swift. allows creating a glance image from volume and uploading to swift.
@ -768,7 +777,7 @@ positional arguments::
freezer-scheduler start|stop freezer-scheduler start|stop
It can be also be started as a foreground process using the --no-daemon flag:: It can be also be started as a foreground process using the ``--no-daemon`` flag::
freezer-scheduler --no-daemon start freezer-scheduler --no-daemon start
@ -778,8 +787,9 @@ or by a babysitting process such as systemd.
The cli-tool version is used to manage the jobs in the API. The cli-tool version is used to manage the jobs in the API.
A "job" is basically a container; a document which contains one A "job" is basically a container; a document which contains one
or more "actions". or more "actions".
An action contains the instructions for the freezer-agent. They are the same parameters
that would be passed to the agent on the command line. For example: An action contains the instructions for the freezer-agent. They are the same
parameters that would be passed to the agent on the command line. For example:
"backup_name", "path_to_backup", "max_level" "backup_name", "path_to_backup", "max_level"
To sum it up, a job is a sequence of parameters that the scheduler pulls To sum it up, a job is a sequence of parameters that the scheduler pulls
@ -790,13 +800,13 @@ The scheduler understands the "scheduling" part of the job document,
which it uses to actually schedule the job, while the rest of the parameters which it uses to actually schedule the job, while the rest of the parameters
are substantially opaque. are substantially opaque.
It may also be useful to use the "-c" parameter to specify the client-id that It may also be useful to use the ``-c`` parameter to specify the client-id that
the scheduler will use when interacting with the API. the scheduler will use when interacting with the API.
The purpose of the *client-id* is to associate a job with the The purpose of the *client-id* is to associate a job with the
scheduler instance which is supposed to execute that job. scheduler instance which is supposed to execute that job.
A single openstack user could manage different resources on different nodes A single OpenStack user could manage different resources on different nodes
(and actually may even have different freezer-scheduler instances running (and actually may even have different freezer-scheduler instances running
on the same node with different local privileges, for example), on the same node with different local privileges, for example),
and the client-id allows him to associate the specific scheduler instance and the client-id allows him to associate the specific scheduler instance
@ -864,15 +874,14 @@ Misc
Dependencies notes Dependencies notes
------------------ ------------------
In stable/kilo and stable/liberty the module peppep3134daemon is imported In stable/kilo and stable/liberty the module ``pep3143daemon`` is imported
from local path from local path rather than pip. This generated many issues as the package
rather than pip. This generated many issues is not in the global-requirements.txt of kilo and liberty.
as the package is not in the global-requirements.txt
of kilo and liberty. Also pbr in the kilo release
does not support env markers which further complicated
the installation.
Please check the FAQ to: FAQ.rst Also pbr in the kilo release does not support env markers which further
complicated the installation.
Please check the `FAQ <FAQ.rst>`_ too.
Available options:: Available options::
@ -1196,9 +1205,16 @@ this will print all options to the screen you direct the output to a file if you
Bandwidth limitation (Trickle) Bandwidth limitation (Trickle)
------------------------------ ------------------------------
Trickle for bandwidth limiting ( How it works ): Trickle for bandwidth limiting (How it works ):
We have 3 cases to handle
1- User used --upload-limit or --download-limit from the cli We have 3 cases to handle:
1. User used ``--upload-limit`` or ``--download-limit`` from the CLI
2. User used configuration files to execute an action
3. A combination of both of these options.
User used --upload-limit or -download-limit from the CLI
========================================================
We need to remove these limits from the cli arguments and then run trickle We need to remove these limits from the cli arguments and then run trickle
using subprocess using subprocess
@ -1210,7 +1226,8 @@ this will be translated to::
# trickle -u 1024 -d -1 freezer-agent --action backup -F /etc/ -C freezer # trickle -u 1024 -d -1 freezer-agent --action backup -F /etc/ -C freezer
2- User used config files to execute an action User used config files to execute an action
===========================================
We need to create a new config file without the limits So we will get the all We need to create a new config file without the limits So we will get the all
the arguments provided and remove limits then run trickle using subprocess the arguments provided and remove limits then run trickle using subprocess
@ -1249,10 +1266,14 @@ The new config file has the following arguments::
backup_name = freezer_jobs backup_name = freezer_jobs
path_to_backup = /etc path_to_backup = /etc
3- Hybrid using config file and cli options Hybrid approach using config file and CLI options
we will use a mix of both procedures: =================================================
- remove limits (cli or config )
- reproduce the same command again with trickle We will use a mix of both procedures:
* remove limits (cli or config )
* reproduce the same command again with trickle
EX:: EX::
$ freezer-agent --config /home/user/job2.ini --upload-limit 1k $ freezer-agent --config /home/user/job2.ini --upload-limit 1k