openstack/freezer
Code Issues Proposed changes

Freezer initial commit

Browse Source
This commit is contained in:
Fausto Marzi 2014-07-22 22:46:31 +01:00
commit ef5695c36f
28 changed files with 3133 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored Normal file
View File

@ -0,0 +1,2 @@
*.pyc
__pycache__

4
.gitreview Normal file
View File

@ -0,0 +1,4 @@
[gerrit]
host=gerrit.hpcloud.net
port=29418
project=automation/automation-backup.git

5
CHANGES.rst Normal file
View File

@ -0,0 +1,5 @@
v1.0.5, 2014-05-16 - Freezer initial release. v1.0.6, 2014-05-24: -
Fixed error restore date is not provided. - Changed the datetime format
for --restore-from-date to "yyyy-mm-ddThh:mm:ss" - Created a FAQ.txt
file - Extended use case example in README.txt and Hacking.txt v1.0.7,
2014-06-05: - added multiprocessing support for backup and restore

34
CREDITS.rst Normal file
View File

@ -0,0 +1,34 @@
Authors
=======
- Fausto Marzi
- Ryszard Chojnacki
- Emil Dimitrov
Maintainers
===========
- Fausto Marzi
- Ryszard Chojnacki
- Emil Dimitrov
Contributors
============
- Duncan Thomas
- Coleman Corrigan
Credits
=======
- Davide Guerri
- Jim West
- Lars Noldan
- Stephen Pearson
- Sneha Mini
- Chris Delaney
- James Bishop
- Matt Joyce
- Anupriya Ramraj
- HP OpsAuto Team

66
FAQ.rst Normal file
View File

@ -0,0 +1,66 @@
FAQ
===
1) What is freezer? Is a tool to automate data backup and restore
process using OpenStack Swift.
2) Does freezer support incremental backup? Yes. Incremental backup are
done using GNU tar incremental features
3) Does freezer check the file contents to establish if a file was
modified or not? No. Freezer check for changes at mtime and ctime in
every file inode to evaluate if a file changed or not.
4) Why GNU tar rather then rsync? Both approaches are good. Rsync check
the file content, while tar check the file inode. In our
environment, we need to backup directories with size > 300GB and
tens of thousands of files. Rsync approach is effective but slow.
Tar is fast as it needs to check only the file inodes, rather then
the full file content.
5) Does feezer support encrypted backup? Yes. Freezer encrypt data
using OpenSSL (AES-256-CFB).
6) Does freezer execute point-in-time backup? Yes. For point in time
backup LVM snapshot feature used.
7) Can I use freezer on OSX or other OS where GNU Tar is not installed
by default? Yes. For OSX and \*BSD, just install gtar and freezer
automatically will use gtar to execute backup. OS other then Linux,
OSX and \*BSD are currently not supported.
8) What Application backup does freezer support currently? MongoDB and
File system.
9) How does the MongoDB backup happens? Freezer required journal
enabled in mongo and lvm volume to execute backup. It checks if the
mongo instance is the master and create lvm snapshot to have
consistent data.
10) Does freezer manage sparse files efficiently? Yes. Zeroed/null data
is not backed up. So less space and bandwidth will be used.
11) Does freezer remove automatically backup after some time? Yes. From
command line the --remove-older-then option (days) can be used to
remove objects older then (days).
12) Does freezer support MySQL Backup? MySQL and MariaDB support soon
will be included.
13) Is there any other storage media supported? Not currently. There's a
plan to add:
- Amazon S3
- Store files on a remote host file system
- MongoDB object storage.
- Other directory in the local host (NFS mounted volumes)
14) Does freezer has any UI or API? Not currently. The UI in OpenStack
Horizon is being currently developed as REST API too.
15) Tar is not capable to detect deleted files between different backup
levels. Is freezer capable of doing that? Not currently. We are
writing a set of tar extensions in python to overcome tar
limitations like this and others.

344
HACKING.rst Normal file
View File

@ -0,0 +1,344 @@
======== Freezer ========
Freezer is a Python tool that helps you to automate the data backup and
restore process.
The following features are avaialble:
- Backup your filesystem using snapshot to swift
- Strong encryption supported: AES-256-CFB
- Backup your file system tree directly (without volume snapshot)
- Backup your journaled MongoDB directory tree using lvm snap to swift
- Backup MySQL DB with lvm snapshot
- Restore your data automatically from Swift to your file system
- Low storage consumption as the backup are uploaded as a stream
- Flexible Incremental backup policy
- Data is archived in GNU Tar format
- Data compression with gzip
- Remove old backup automatically according the provided parameters
Requirements
============
- OpenStack Swift Account (Auth V2 used)
- python >= 2.6 (2.7 advised)
- GNU Tar >= 1.26
- gzip
- OpenSSL
- python-swiftclient >= 2.0.3
- python-keystoneclient >= 0.8.0
- pymongo >= 2.6.2 (if MongoDB backups will be executed)
- At least 128 MB of memory reserved for freezer
Installation & Env Setup
========================
Install required packages
-------------------------
Ubuntu / Debian
---------------
Swift client and Keystone client:: $ sudo apt-get install -y
python-swiftclient python-keystoneclient
MongoDB backup:: $ sudo apt-get install -y python-pymongo
MySQL backup:: $ sudo apt-get install -y python-mysqldb
Freezer installation from Python package repo:: $ sudo pip install
freezer OR $ sudo easy\_install freezer
The basic Swift account configuration is needed to use freezer. Make
sure python-swiftclient is installed.
Also the following ENV var are needed you can put them in ~/.bashrc::
::
export OS_REGION_NAME=region-a.geo-1
export OS_TENANT_ID=<account tenant>
export OS_PASSWORD=<account password>
export OS_AUTH_URL=https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0
export OS_USERNAME=automationbackup
export OS_TENANT_NAME=automationbackup
$ source ~/.barshrc
Let's say you have a container called foobar-contaienr, by executing
"swift list" you should see something like::
::
$ swift list
foobar-container-2
$
These are just use case example using Swift in the HP Cloud.
*Is strongly advised to use execute a backup using LVM snapshot, so
freezer will execute a backup on point-in-time data. This avoid risks of
data inconsistencies and corruption.*
Usage Example
=============
Backup
------
The most simple backup execution is a direct file system backup::
::
$ sudo freezerc --file-to-backup /data/dir/to/backup --container new-data-backup \
--backup-name my-backup-name
By default --mode fs is set. The command would generate a compressed tar
gzip file of the directory /data/dir/to/backup. The generated file will
be segmented in stream and uploaded in the swift container called
new-data-backup, with backup name my-backup-name
Now check if your backup is executing correctly looking at
/var/log/freezer.log
Execute a MongoDB backup using lvm snapshot::
We need to check before on which volume group and logical volume our
mongo data is. These information can be obtained as per following::
::
$ mount
[...]
Once we know the volume where our mongo data is mounted on, we can get
the volume group and logical volume info::
::
$ sudo vgdisplay
[...]
$ sudo lvdisplay
[...]
We assume our mongo volume is "/dev/mongo/mongolv" and the volume group
is "mongo"::
::
$ sudo freezerc --lvm-srcvol /dev/mongo/mongolv --lvm-dirmount /var/lib/snapshot-backup \
--lvm-volgroup mongo --file-to-backup /var/lib/snapshot-backup/mongod_ops2 \
--container mongodb-backup-prod --exclude "*.lock" --mode mongo --backup-name mongod-ops2
Now freezerc create a lvm snapshot of the volume /dev/mongo/mongolv. If
no options are provided, default snapshot name is freezer\_backup\_snap.
The snap vol will be mounted automatically on /var/lib/snapshot-backup
and the backup meta and segments will be upload in the container
mongodb-backup-prod with the namemongod-ops2.
Execute a file system backup using lvm snapshot:: $ sudo freezerc
--lvm-srcvol /dev/jenkins/jenkins-home --lvm-dirmount
/var/snapshot-backup
--lvm-volgroup jenkins --file-to-backup /var/snapshot-backup
--container jenkins-backup-prod --exclude "\*.lock" --mode fs
--backup-name jenkins-ops2
MySQL backup require a basic configuration file. The following is an
example of the config:: $ sudo cat /root/.freezer/db.conf host =
your.mysql.host.ip user = backup password =
Every listed option is mandatory. There's no need to stop the mysql
service before the backup execution.
Execute a MySQL backup using lvm snapshot:: $ sudo freezerc --lvm-srcvol
/dev/mysqlvg/mysqlvol --lvm-dirmount /var/snapshot-backup
--lvm-volgroup mysqlvg --file-to-backup /var/snapshot-backup
--mysql-conf /root/.freezer/freezer-mysql.conf--container
mysql-backup-prod
--mode mysql --backup-name mysql-ops002
All the freezerc activities are logged into /var/log/freezer.log.
Restore
-------
As a general rule, when you execute a restore, the application that
write or read data should be stopped.
There are 3 main options that need to be set for data restore
File System Restore:: Execute a file system restore of the backup name
adminui.git:: $ sudo freezerc --container foobar-container-2
--backup-name adminui.git
--restore-from-host git-HP-DL380-host-001 --restore-abs-path
/home/git/repositories/adminui.git/
--restore-from-date "23-05-2014T23:23:23"
MySQL restore:: Execute a MySQL restore of the backup name holly-mysql.
Let's stop mysql service first:: $ sudo service mysql stop
Execute Restore:: $ sudo freezerc --container foobar-container-2
--backup-name mysq-prod
--restore-from-host db-HP-DL380-host-001 --restore-abs-path
/var/lib/mysql
--restore-from-date "23-05-2014T23:23:23"
And finally restart mysql:: $ sudo service mysql start
Execute a MongoDB restore of the backup name mongobigdata:: $ sudo
freezerc --container foobar-container-2 --backup-name mongobigdata
--restore-from-host db-HP-DL380-host-001 --restore-abs-path
/var/lib/mongo
--restore-from-date "23-05-2014T23:23:23"
Architecture
============
Freezer architecture is simple. The components are:
- OpenStack Swift (the storage)
- freezer client running on the node you want to execute the backups or
restore
Frezeer use GNU Tar under the hood to execute incremental backup and
restore. When a key is provided, it uses OpenSSL to encrypt data
(AES-256-CFB)
Low resources requirement
-------------------------
Freezer is designed to reduce at the minimum I/O, CPU and Memory Usage.
This is achieved by generating a data stream from tar (for archiving)
and gzip (for compressing). Freezer segment the stream in a configurable
chunk size (with the option --max-seg-size). The default segment size is
128MB, so it can be safely stored in memory, encrypted if the key is
provided, and uploaded to Swift as segment.
Multiple segments are sequentially uploaded using the Swift Manifest.
All the segments are uploaded first, and then the Manifest file is
uploaded too, so the data segments cannot be accessed directly. This
ensue data consistency.
By keeping small segments in memory, I/O usage is reduced. Also as
there's no need to store locally the final compressed archive
(tar-gziped), no additional or dedicated storage is required for the
backup execution. The only additional storage needed is the LVM snapshot
size (set by default at 5GB). The lvm snapshot size can be set with the
option --lvm-snapsize. It is important to not specify a too small snap
size, because in case a quantity of data is being wrote to the source
volume and consequently the lvm snapshot is filled up, then the data is
corrupted.
If the more memory is available for the backup process, the maximum
segment size can be increased, this will speed up the process. Please
note, the segments must be smaller then 5GB, is that is the maximum
object size in the Swift server.
Au contraire, if a server have small memory availability, the
--max-seg-size option can be set to lower values. The unit of this
option is in bytes.
How the incremental works
-------------------------
The incremental backups is one of the most crucial feature. The
following basic logic happens when Freezer execute:
1) Freezer start the execution and check if the provided backup name for
the current node already exist in Swift
2) If the backup exists, the Manifest file is retrieved. This is
important as the Manifest file contains the information of the
previous Freezer execution.
The following is what the Swift Manifest looks like::
::
{
'X-Object-Meta-Encrypt-Data': 'Yes',
'X-Object-Meta-Segments-Size-Bytes': '134217728',
'X-Object-Meta-Backup-Created-Timestamp': '1395734461',
'X-Object-Meta-Remove-Backup-Older-Than-Days': '',
'X-Object-Meta-Src-File-To-Backup': '/var/lib/snapshot-backup/mongod_dev-mongo-s1',
'X-Object-Meta-Maximum-Backup-level': '0',
'X-Object-Meta-Always-Backup-Level': '',
'X-Object-Manifest': u'socorro-backup-dev_segments/dev-mongo-s1-r1_mongod_dev-mongo-s1_1395734461_0',
'X-Object-Meta-Providers-List': 'HP',
'X-Object-Meta-Backup-Current-Level': '0',
'X-Object-Meta-Abs-File-Path': '',
'X-Object-Meta-Backup-Name': 'mongod_dev-mongo-s1',
'X-Object-Meta-Tar-Meta-Obj-Name': 'tar_metadata_dev-mongo-s1-r1_mongod_dev-mongo-s1_1395734461_0',
'X-Object-Meta-Hostname': 'dev-mongo-s1-r1',
'X-Object-Meta-Container-Segments': 'socorro-backup-dev_segments'
}
3) The most relevant data taken in consideration for incremental are:
- 'X-Object-Meta-Maximum-Backup-level': '7'
Value set by the option: --max-level int
Assuming we are executing the backup daily, let's say managed from the
crontab, the first backup will start from Level 0, that is, a full
backup. At every daily execution, the current backup level will be
incremented by 1. Then current backup level is equal to the maximum
backup level, then the backup restart to level 0. That is, every week a
full backup will be executed.
- 'X-Object-Meta-Always-Backup-Level': ''
Value set by the option: --always-level int
When current level is equal to 'Always-Backup-Level', every next backup
will be executed to the specified level. Let's say --always-level is set
to 1, the first backup will be a level 0 (complete backup) and every
next execution will backup the data exactly from the where the level 0
ended. The main difference between Always-Backup-Level and
Maximum-Backup-level is that the counter level doesn't restart from
level 0
- 'X-Object-Manifest':
u'socorro-backup-dev/dev-mongo-s1-r1\_mongod\_dev-mongo-s1\_1395734461\_0'
Through this meta data, we can identify the exact Manifest name of the
provided backup name. The syntax is:
container\_name/hostname\_backup\_name\_timestamp\_initiallevel
- 'X-Object-Meta-Providers-List': 'HP'
This option is NOT implemented yet The idea of Freezer is to support
every Cloud provider that provide Object Storage service using OpenStack
Swift. The meta data allows you to specify multiple provider and
therefore store your data in different Geographic location.
- 'X-Object-Meta-Backup-Current-Level': '0'
Record the current backup level. This is important as the value is
incremented by 1 in the next freezer execution.
- 'X-Object-Meta-Backup-Name': 'mongod\_dev-mongo-s1'
Value set by the option: -N BACKUP\_NAME, --backup-name BACKUP\_NAME The
option is used to identify the backup. It is a mandatory option and
fundamental to execute incremental backup. 'Meta-Backup-Name' and
'Meta-Hostname' are used to uniquely identify the current and next
incremental backups
- 'X-Object-Meta-Tar-Meta-Obj-Name':
'tar\_metadata\_dev-mongo-s1-r1\_mongod\_dev-mongo-s1\_1395734461\_0'
Freezer use tar to execute incremental backup. What tar do is to store
in a meta data file the inode information of every file archived. Thus,
on the next Freezer execution, the tar meta data file is retrieved and
download from swift and it is used to generate the next backup level.
After the next level backup execution is terminated, the file update tar
meta data file will be uploaded and recorded in the Manifest file. The
naming convention used for this file is:
tar\_metadata\_backupname\_hostname\_timestamp\_backuplevel
- 'X-Object-Meta-Hostname': 'dev-mongo-s1-r1'
The hostname of the node where the Freezer perform the backup. This meta
data is important to identify a backup with a specific node, thus avoid
possible confusion and associate backup to the wrong node.

16
INSTALL.rst Normal file
View File

@ -0,0 +1,16 @@
Install instructions
====================
Install from sources::
----------------------
You have now the freezerc tool installed in /usr/local/bin/freezerc
Please execute the following command to all the available options:
$ freezerc --help [...]
Please read README.txt or HACKING.txt to see the requirement and more
technical details about how to run freezer
Thanks, The Freezer Team.

19
LICENSE.rst Normal file
View File

@ -0,0 +1,19 @@
Copyright 2014 Hewlett-Packard
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
::
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This product includes cryptographic software written by Eric Young
(eay@cryptsoft.com). This product includes software written by Tim
Hudson (tjh@cryptsoft.com).

18
MANIFEST Normal file
View File

@ -0,0 +1,18 @@
# file GENERATED by distutils, do NOT edit
CHANGES.txt
CREDITS.txt
HACKING.txt
LICENSE.txt
README.txt
TODO.txt
setup.py
bin/freezerc
freezer/__init__.py
freezer/arguments.py
freezer/backup.py
freezer/lvm.py
freezer/main.py
freezer/restore.py
freezer/swift.py
freezer/tar.py
freezer/utils.py

5
MANIFEST.in Normal file
View File

@ -0,0 +1,5 @@
include *.rst
include *.txt
include bin/freezerc
recursive-include docs *.rst
recursive-include freezer *.py

357
README.rst Normal file
View File

@ -0,0 +1,357 @@
=======
Freezer
=======
Freezer is a Python tool that helps you to automate the data backup and
restore process.
The following features are avaialble:
- Backup your filesystem using snapshot to swift
- Strong encryption supported: AES-256-CFB
- Backup your file system tree directly (without volume snapshot)
- Backup your journaled MongoDB directory tree using lvm snap to swift
- Backup MySQL DB with lvm snapshot
- Restore your data automatically from Swift to your file system
- Low storage consumption as the backup are uploaded as a stream
- Flexible Incremental backup policy
- Data is archived in GNU Tar format
- Data compression with gzip
- Remove old backup automatically according the provided parameters
Requirements
============
- OpenStack Swift Account (Auth V2 used)
- python >= 2.6 (2.7 advised)
- GNU Tar >= 1.26
- gzip
- OpenSSL
- python-swiftclient >= 2.0.3
- python-keystoneclient >= 0.8.0
- pymongo >= 2.6.2 (if MongoDB backups will be executed)
- At least 128 MB of memory reserved for freezer
Installation & Env Setup
========================
Install required packages
-------------------------
Ubuntu / Debian
---------------
Swift client and Keystone client::
$ sudo apt-get install -y python-swiftclient python-keystoneclient
MongoDB backup::
$ sudo apt-get install -y python-pymongo
MySQL backup::
$ sudo apt-get install -y python-mysqldb
Freezer installation from Python package repo::
$ sudo pip install freezer
OR::
$ sudo easy\_install freezer
The basic Swift account configuration is needed to use freezer. Make
sure python-swiftclient is installed.
Also the following ENV var are needed you can put them in ~/.bashrc::
export OS_REGION_NAME=region-a.geo-1
export OS_TENANT_ID=<account tenant>
export OS_PASSWORD=<account password>
export OS_AUTH_URL=https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0
export OS_USERNAME=automationbackup
export OS_TENANT_NAME=automationbackup
$ source ~/.barshrc
Let's say you have a container called foobar-contaienr, by executing
"swift list" you should see something like::
$ swift list
foobar-container-2
$
These are just use case example using Swift in the HP Cloud.
*Is strongly advised to use execute a backup using LVM snapshot, so
freezer will execute a backup on point-in-time data. This avoid risks of
data inconsistencies and corruption.*
Usage Example
=============
Backup
------
The most simple backup execution is a direct file system backup::
$ sudo freezerc --file-to-backup /data/dir/to/backup
--container new-data-backup --backup-name my-backup-name
By default --mode fs is set. The command would generate a compressed tar
gzip file of the directory /data/dir/to/backup. The generated file will
be segmented in stream and uploaded in the swift container called
new-data-backup, with backup name my-backup-name
Now check if your backup is executing correctly looking at
/var/log/freezer.log
Execute a MongoDB backup using lvm snapshot:
We need to check before on which volume group and logical volume our
mongo data is. These information can be obtained as per following::
$ mount
[...]
Once we know the volume where our mongo data is mounted on, we can get
the volume group and logical volume info::
$ sudo vgdisplay
[...]
$ sudo lvdisplay
[...]
We assume our mongo volume is "/dev/mongo/mongolv" and the volume group
is "mongo"::
$ sudo freezerc --lvm-srcvol /dev/mongo/mongolv --lvm-dirmount /var/lib/snapshot-backup
--lvm-volgroup mongo --file-to-backup /var/lib/snapshot-backup/mongod_ops2
--container mongodb-backup-prod --exclude "*.lock" --mode mongo --backup-name mongod-ops2
Now freezerc create a lvm snapshot of the volume /dev/mongo/mongolv. If
no options are provided, default snapshot name is freezer\_backup\_snap.
The snap vol will be mounted automatically on /var/lib/snapshot-backup
and the backup meta and segments will be upload in the container
mongodb-backup-prod with the namemongod-ops2.
Execute a file system backup using lvm snapshot::
$ sudo freezerc --lvm-srcvol /dev/jenkins/jenkins-home --lvm-dirmount
/var/snapshot-backup --lvm-volgroup jenkins
--file-to-backup /var/snapshot-backup --container jenkins-backup-prod
--exclude "\*.lock" --mode fs --backup-name jenkins-ops2
MySQL backup require a basic configuration file. The following is an
example of the config::
$ sudo cat /root/.freezer/db.conf
host = your.mysql.host.ip
user = backup
password = userpassword
Every listed option is mandatory. There's no need to stop the mysql
service before the backup execution.
Execute a MySQL backup using lvm snapshot::
$ sudo freezerc --lvm-srcvol /dev/mysqlvg/mysqlvol
--lvm-dirmount /var/snapshot-backup
--lvm-volgroup mysqlvg --file-to-backup /var/snapshot-backup
--mysql-conf /root/.freezer/freezer-mysql.conf--container
mysql-backup-prod --mode mysql --backup-name mysql-ops002
All the freezerc activities are logged into /var/log/freezer.log.
Restore
-------
As a general rule, when you execute a restore, the application that
write or read data should be stopped.
There are 3 main options that need to be set for data restore
File System Restore:
Execute a file system restore of the backup name
adminui.git::
$ sudo freezerc --container foobar-container-2
--backup-name adminui.git
--restore-from-host git-HP-DL380-host-001 --restore-abs-path
/home/git/repositories/adminui.git/
--restore-from-date "23-05-2014T23:23:23"
MySQL restore:
Execute a MySQL restore of the backup name holly-mysql.
Let's stop mysql service first::
$ sudo service mysql stop
Execute Restore::
$ sudo freezerc --container foobar-container-2
--backup-name mysq-prod --restore-from-host db-HP-DL380-host-001
--restore-abs-path /var/lib/mysql --restore-from-date "23-05-2014T23:23:23"
And finally restart mysql::
$ sudo service mysql start
Execute a MongoDB restore of the backup name mongobigdata::
$ sudo freezerc --container foobar-container-2 --backup-name mongobigdata
--restore-from-host db-HP-DL380-host-001 --restore-abs-path
/var/lib/mongo --restore-from-date "23-05-2014T23:23:23"
Architecture
============
Freezer architecture is simple. The components are:
- OpenStack Swift (the storage)
- freezer client running on the node you want to execute the backups or
restore
Frezeer use GNU Tar under the hood to execute incremental backup and
restore. When a key is provided, it uses OpenSSL to encrypt data
(AES-256-CFB)
Low resources requirement
-------------------------
Freezer is designed to reduce at the minimum I/O, CPU and Memory Usage.
This is achieved by generating a data stream from tar (for archiving)
and gzip (for compressing). Freezer segment the stream in a configurable
chunk size (with the option --max-seg-size). The default segment size is
128MB, so it can be safely stored in memory, encrypted if the key is
provided, and uploaded to Swift as segment.
Multiple segments are sequentially uploaded using the Swift Manifest.
All the segments are uploaded first, and then the Manifest file is
uploaded too, so the data segments cannot be accessed directly. This
ensue data consistency.
By keeping small segments in memory, I/O usage is reduced. Also as
there's no need to store locally the final compressed archive
(tar-gziped), no additional or dedicated storage is required for the
backup execution. The only additional storage needed is the LVM snapshot
size (set by default at 5GB). The lvm snapshot size can be set with the
option --lvm-snapsize. It is important to not specify a too small snap
size, because in case a quantity of data is being wrote to the source
volume and consequently the lvm snapshot is filled up, then the data is
corrupted.
If the more memory is available for the backup process, the maximum
segment size can be increased, this will speed up the process. Please
note, the segments must be smaller then 5GB, is that is the maximum
object size in the Swift server.
Au contraire, if a server have small memory availability, the
--max-seg-size option can be set to lower values. The unit of this
option is in bytes.
How the incremental works
-------------------------
The incremental backups is one of the most crucial feature. The
following basic logic happens when Freezer execute:
1) Freezer start the execution and check if the provided backup name for
the current node already exist in Swift
2) If the backup exists, the Manifest file is retrieved. This is
important as the Manifest file contains the information of the
previous Freezer execution.
The following is what the Swift Manifest looks like::
{
'X-Object-Meta-Encrypt-Data': 'Yes',
'X-Object-Meta-Segments-Size-Bytes': '134217728',
'X-Object-Meta-Backup-Created-Timestamp': '1395734461',
'X-Object-Meta-Remove-Backup-Older-Than-Days': '',
'X-Object-Meta-Src-File-To-Backup': '/var/lib/snapshot-backup/mongod_dev-mongo-s1',
'X-Object-Meta-Maximum-Backup-level': '0',
'X-Object-Meta-Always-Backup-Level': '',
'X-Object-Manifest': u'socorro-backup-dev_segments/dev-mongo-s1-r1_mongod_dev-mongo-s1_1395734461_0',
'X-Object-Meta-Providers-List': 'HP',
'X-Object-Meta-Backup-Current-Level': '0',
'X-Object-Meta-Abs-File-Path': '',
'X-Object-Meta-Backup-Name': 'mongod_dev-mongo-s1',
'X-Object-Meta-Tar-Meta-Obj-Name': 'tar_metadata_dev-mongo-s1-r1_mongod_dev-mongo-s1_1395734461_0',
'X-Object-Meta-Hostname': 'dev-mongo-s1-r1',
'X-Object-Meta-Container-Segments': 'socorro-backup-dev_segments'
}
3) The most relevant data taken in consideration for incremental are:
- 'X-Object-Meta-Maximum-Backup-level': '7'
Value set by the option: --max-level int
Assuming we are executing the backup daily, let's say managed from the
crontab, the first backup will start from Level 0, that is, a full
backup. At every daily execution, the current backup level will be
incremented by 1. Then current backup level is equal to the maximum
backup level, then the backup restart to level 0. That is, every week a
full backup will be executed.
- 'X-Object-Meta-Always-Backup-Level': ''
Value set by the option: --always-level int
When current level is equal to 'Always-Backup-Level', every next backup
will be executed to the specified level. Let's say --always-level is set
to 1, the first backup will be a level 0 (complete backup) and every
next execution will backup the data exactly from the where the level 0
ended. The main difference between Always-Backup-Level and
Maximum-Backup-level is that the counter level doesn't restart from
level 0
- 'X-Object-Manifest':
u'socorro-backup-dev/dev-mongo-s1-r1\_mongod\_dev-mongo-s1\_1395734461\_0'
Through this meta data, we can identify the exact Manifest name of the
provided backup name. The syntax is:
container\_name/hostname\_backup\_name\_timestamp\_initiallevel
- 'X-Object-Meta-Providers-List': 'HP'
This option is NOT implemented yet The idea of Freezer is to support
every Cloud provider that provide Object Storage service using OpenStack
Swift. The meta data allows you to specify multiple provider and
therefore store your data in different Geographic location.
- 'X-Object-Meta-Backup-Current-Level': '0'
Record the current backup level. This is important as the value is
incremented by 1 in the next freezer execution.
- 'X-Object-Meta-Backup-Name': 'mongod\_dev-mongo-s1'
Value set by the option: -N BACKUP\_NAME, --backup-name BACKUP\_NAME The
option is used to identify the backup. It is a mandatory option and
fundamental to execute incremental backup. 'Meta-Backup-Name' and
'Meta-Hostname' are used to uniquely identify the current and next
incremental backups
- 'X-Object-Meta-Tar-Meta-Obj-Name':
'tar\_metadata\_dev-mongo-s1-r1\_mongod\_dev-mongo-s1\_1395734461\_0'
Freezer use tar to execute incremental backup. What tar do is to store
in a meta data file the inode information of every file archived. Thus,
on the next Freezer execution, the tar meta data file is retrieved and
download from swift and it is used to generate the next backup level.
After the next level backup execution is terminated, the file update tar
meta data file will be uploaded and recorded in the Manifest file. The
naming convention used for this file is:
tar\_metadata\_backupname\_hostname\_timestamp\_backuplevel
- 'X-Object-Meta-Hostname': 'dev-mongo-s1-r1'
The hostname of the node where the Freezer perform the backup. This meta
data is important to identify a backup with a specific node, thus avoid
possible confusion and associate backup to the wrong node.

1
TODO.rst Normal file
View File

@ -0,0 +1 @@

82
bin/freezerc Executable file
View File

@ -0,0 +1,82 @@
#!/usr/bin/env python
'''
Copyright 2014 Hewlett-Packard
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This product includes cryptographic software written by Eric Young
(eay@cryptsoft.com). This product includes software written by Tim
Hudson (tjh@cryptsoft.com).
========================================================================
Freezer offer the following features:
[*] Backup your filesystem using lvm snapshot to swift
[*] Data Encryption (AES-256-CFB)
[*] Backup your file system tree directly (without volume snapshot)
[*] Backup your journaled mongodb directory tree using lvm snap to swift
[*] Backup MySQL DB with lvm snapshot
[*] Restore automatically your data from swift to your filesystems
[*] Low storage consumption as the backup are uploaded as a stream
[*] Flexible Incremental backup policy
'''
from freezer.main import freezer_main
from freezer.arguments import backup_arguments
import os
import subprocess
import logging
import sys
# Initialize backup options
(backup_args, arg_parse) = backup_arguments()
# Configure logging
logging.basicConfig(
filename=backup_args.log_file,
level=logging.INFO,
format='%(asctime)s %(name)s %(levelname)s %(message)s')
# Try to execute freezer with highest priority execution if
# backup_args.max_priority is True. New priority will be set for child
# processes too, as niceness is inherited from father
if backup_args.max_priority:
try:
logging.warning(
'[*] Setting freezer execution with high CPU and I/O priority')
PID = os.getpid()
# Set cpu priority
os.nice(-19)
# Set I/O Priority to Real Time class with level 0
subprocess.call(
[u'{0}'.format(backup_args.ionice),
u'-c', u'1', u'-n', u'0', u'-t', u'-p', u'{0}'.format(PID)])
except Exception as priority_error:
logging.warning('[*] Priority: {0}'.format(priority_error))
if backup_args.version:
print "freezer version {0}".format(backup_args.__version__)
sys.exit(1)
if len(sys.argv) < 2:
arg_parse.print_help()
sys.exit(1)
if __name__ == '__main__':
try:
freezer_main(backup_args)
except ValueError as err:
logging.critical('[*] ValueError: {0}'.format(err))
except Exception as err:
logging.critical('[*] Error: {0}'.format(err))

20
freezer/__init__.py Normal file
View File

@ -0,0 +1,20 @@
'''
Copyright 2014 Hewlett-Packard
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This product includes cryptographic software written by Eric Young
(eay@cryptsoft.com). This product includes software written by Tim
Hudson (tjh@cryptsoft.com).
========================================================================
'''

244
freezer/arguments.py Normal file
View File

@ -0,0 +1,244 @@
'''
Copyright 2014 Hewlett-Packard
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This product includes cryptographic software written by Eric Young
(eay@cryptsoft.com). This product includes software written by Tim
Hudson (tjh@cryptsoft.com).
========================================================================
Arguments and general parameters definitions
'''
import sys
import argparse
import distutils.spawn as distspawn
import os
import logging
def backup_arguments():
'''
Default arguments and command line options interface. The function return
a name space called backup_args.
'''
arg_parser = argparse.ArgumentParser(prog='freezerc')
arg_parser.add_argument(
'-F', '--file-to-backup', action='store',
help="The file or directory you want to back up to Swift",
dest='src_file', default=False)
arg_parser.add_argument(
'-N', '--backup-name', action='store',
help="The backup name you want to use to identify your backup \
on Swift", dest='backup_name', default=False)
arg_parser.add_argument(
'-m', '--mode', action='store',
help="Set the technology to back from. Options are, fs (filesystem),\
mongo (MongoDB), mysql (MySQL). Default set to fs", dest='mode',
default='fs')
arg_parser.add_argument(
'-C', '--container', action='store',
help="The Swift container used to upload files to",
dest='container', default=False)
arg_parser.add_argument(
'-L', '--list-containers', action='store_true',
help='''List the Swift containers on remote Object Storage Server''',
dest='list_container', default=False)
arg_parser.add_argument(
'-l', '--list-objects', action='store_true',
help='''List the Swift objects stored in a container on remote Object\
Storage Server.''', dest='list_objects', default=False)
arg_parser.add_argument(
'-o', '--get-object', action='store',
help="The Object Name you want to download. This options is mandatory \
when --restore is used", dest='object', default=False)
arg_parser.add_argument(
'-d', '--dst-file', action='store',
help="The file name used to save the object on your local disk and\
upload file in swift", dest='dst_file', default=False)
arg_parser.add_argument(
'--lvm-srcvol', action='store',
help="Set the lvm volume you want to take a snaphost from. Default\
no volume", dest='lvm_srcvol', default=False)
arg_parser.add_argument(
'--lvm-snapname', action='store',
help="Set the lvm snapshot name to use. If the snapshot name already\
exists, the old one will be used a no new one will be created. Default\
freezer_backup_snap.", dest='lvm_snapname', default=False)
arg_parser.add_argument(
'--lvm-snapsize', action='store',
help="Set the lvm snapshot size when creating a new snapshot.\
Please add G for Gigabytes or M for Megabytes, i.e. 500M or 8G.\
Default 5G.", dest='lvm_snapsize', default=False)
arg_parser.add_argument(
'--lvm-dirmount', action='store',
help="Set the directory you want to mount the lvm snapshot to.\
Default not set", dest='lvm_dirmount', default=False)
arg_parser.add_argument(
'--lvm-volgroup', action='store',
help="Specify the volume group of your logical volume.\
This is important to mount your snapshot volume. Default not set",
dest='lvm_volgroup', default=False)
arg_parser.add_argument(
'--max-level', action='store',
help="Set the backup level used with tar to implement incremental \
backup. If a level 1 is specified but no level 0 is already \
available, a level 0 will be done and subesequently backs to level 1.\
Default 0 (No Incremental)", dest='max_backup_level',
type=int, default=False)
arg_parser.add_argument(
'--always-level', action='store', help="Set backup\
maximum level used with tar to implement incremental backup. If a \
level 3 is specified, the backup will be executed from level 0 to \
level 3 and to that point always a backup level 3 will be executed. \
It will not restart from level 0. This option has precedence over \
--max-backup-level. Default False (Disabled)",
dest='always_backup_level', type=int, default=False)
arg_parser.add_argument(
'--restart-always-level', action='store', help="Restart the backup \
from level 0 after n days. Valid only if --always-level option \
if set. If --always-level is used together with --remove-older-then, \
there might be the chance where the initial level 0 will be removed \
Default False (Disabled)",
dest='restart_always_backup', type=float, default=False)
arg_parser.add_argument(
'-R', '--remove-older-then', action='store',
help="Checks in the specified container for object older then the \
specified days. If i.e. 30 is specified, it will remove the remote \
object older than 30 days. Default False (Disabled)",
dest='remove_older_than', type=float, default=False)
arg_parser.add_argument(
'--no-incremental', action='store_true',
help='''Disable incremantal feature. By default freezer build the
meta data even for level 0 backup. By setting this option incremental
meta data is not created at all. Default disabled''',
dest='no_incremental', default=False)
arg_parser.add_argument(
'--hostname', action='store',
help='''Set hostname to execute actions. If you are executing freezer
from one host but you want to delete objects belonging to another
host then you can set this option that hostname and execute appropriate
actions. Default current node hostname.''',
dest='hostname', default=False)
arg_parser.add_argument(
'--mysql-conf', action='store',
help='''Set the MySQL configuration file where freezer retrieve
important information as db_name, user, password, host.
Following is an example of config file:
# cat ~/.freezer/backup_mysql_conf
host = <db-host>
user = <mysqluser>
password = <mysqlpass>''',
dest='mysql_conf_file', default=False)
arg_parser.add_argument(
'--log-file', action='store',
help='Set log file. By default logs to /var/log/freezer.log',
dest='log_file', default='/var/log/freezer.log')
arg_parser.add_argument(
'--exclude', action='store', help="Exclude files,\
given as a PATTERN.Ex: --exclude '*.log' will exclude any file with \
name ending with .log. Default no exclude", dest='exclude',
default=False)
arg_parser.add_argument(
'-U', '--upload', action='store_true',
help="Upload to Swift the destination file passed to the -d option.\
Default upload the data", dest='upload', default=True)
arg_parser.add_argument(
'--encrypt-pass-file', action='store',
help="Passing a private key to this option, allow you to encrypt the \
files before to be uploaded in Swift. Default do not encrypt.",
dest='encrypt_pass_file', default=False)
arg_parser.add_argument(
'-M', '--max-segment-size', action='store',
help="Set the maximum file chunk size in bytes to upload to swift\
Default 134217728 bytes (128MB)",
dest='max_seg_size', type=int, default=134217728)
arg_parser.add_argument(
'--restore-abs-path', action='store',
help='Set the absolute path where you want your data restored. \
option --restore need to be set along with --get-object in order \
to execute data restore. Default False.',
dest='restore_abs_path', default=False)
arg_parser.add_argument(
'--restore-from-host', action='store',
help='''Set the hostname used to identify the data you want to restore
from. If you want to restore data in the same host where the backup
was executed just type from your shell: "$ hostname" and the output is
the value that needs to be passed to this option. Mandatory with
Restore Default False.''', dest='restore_from_host', default=False)
arg_parser.add_argument(
'--restore-from-date', action='store',
help='''Set the absolute path where you want your data restored.
Please provide datime in forma "YYYY-MM-DDThh:mm:ss"
i.e. "1979-10-03T23:23:23". Make sure the "T" is between date and time
Default False.''', dest='restore_from_date', default=False)
arg_parser.add_argument(
'--max-priority', action='store_true',
help='''Set the cpu process to the highest priority (i.e. -20 on Linux)
and real-time for I/O. The process priority will be set only if nice
and ionice are installed Default disabled. Use with caution.''',
dest='max_priority', default=False)
arg_parser.add_argument(
'-V', '--version', action='store_true',
help='''Print the release version and exit''',
dest='version', default=False)
backup_args = arg_parser.parse_args()
# Set additional namespace attributes
backup_args.__dict__['remote_match_backup'] = []
backup_args.__dict__['remote_objects'] = []
backup_args.__dict__['remote_obj_list'] = []
backup_args.__dict__['remote_newest_backup'] = u''
# Set default workdir to ~/.freezer
backup_args.__dict__['workdir'] = os.path.expanduser(u'~/.freezer')
# Create a new namespace attribute for container_segments
backup_args.__dict__['container_segments'] = u'{0}_segments'.format(
backup_args.container)
# If hostname is not set, hostname of the current node will be used
if not backup_args.hostname:
backup_args.__dict__['hostname'] = os.uname()[1]
backup_args.__dict__['manifest_meta_dict'] = {}
backup_args.__dict__['curr_backup_level'] = ''
backup_args.__dict__['manifest_meta_dict'] = ''
backup_args.__dict__['tar_path'] = distspawn.find_executable('tar')
# If freezer is being used under OSX, please install gnutar and
# rename the executable as gnutar
if 'darwin' in sys.platform or 'bsd' in sys.platform:
if distspawn.find_executable('gtar'):
backup_args.__dict__['tar_path'] = \
distspawn.find_executable('gtar')
else:
logging.critical('[*] Please install gnu tar (gtar) as it is a \
mandatory requirement to use freezer.')
raise Exception
# Get absolute path of other commands used by freezer
backup_args.__dict__['lvcreate_path'] = distspawn.find_executable(
'lvcreate')
backup_args.__dict__['lvremove_path'] = distspawn.find_executable(
'lvremove')
backup_args.__dict__['bash_path'] = distspawn.find_executable('bash')
backup_args.__dict__['openssl_path'] = distspawn.find_executable('openssl')
backup_args.__dict__['file_path'] = distspawn.find_executable('file')
backup_args.__dict__['mount_path'] = distspawn.find_executable('mount')
backup_args.__dict__['umount_path'] = distspawn.find_executable('umount')
backup_args.__dict__['ionice'] = distspawn.find_executable('ionice')
# MySQLdb object
backup_args.__dict__['mysql_db_inst'] = ''
# Freezer version
backup_args.__dict__['__version__'] = '1.0.8'
return backup_args, arg_parser

179
freezer/backup.py Normal file
View File

@ -0,0 +1,179 @@
'''
Copyright 2014 Hewlett-Packard
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This product includes cryptographic software written by Eric Young
(eay@cryptsoft.com). This product includes software written by Tim
Hudson (tjh@cryptsoft.com).
========================================================================
Freezer Backup modes related functions
'''
from freezer.lvm import lvm_snap, lvm_snap_remove
from freezer.tar import tar_backup, gen_tar_command
from freezer.swift import add_object, manifest_upload
from freezer.utils import gen_manifest_meta, add_host_name_ts_level
from multiprocessing import Process, Queue
import logging
import os
def backup_mode_mysql(backup_opt_dict, time_stamp, manifest_meta_dict):
'''
Execute a MySQL DB backup. currently only backup with lvm snapshots
are supported. This mean, just before the lvm snap vol is created,
the db tables will be flushed and locked for read, then the lvm create
command will be executed and after that, the table will be unlocked and
the backup will be executed. It is important to have the available in
backup_args.mysql_conf_file the file where the database host, name, user,
and password are set.
'''
try:
import MySQLdb
except ImportError as error:
logging.critical('[*] Error: please install MySQLdb module')
raise ImportError('[*] Error: please install MySQLdb module')
if not backup_opt_dict.mysql_conf_file:
logging.critical(
'[*] MySQL Error: please provide a valid config file')
raise ValueError
# Open the file provided in backup_args.mysql_conf_file and extract the
# db host, name, user and password.
db_user = db_host = db_pass = False
with open(backup_opt_dict.mysql_conf_file, 'r') as mysql_file_fd:
for line in mysql_file_fd:
if 'host' in line:
db_host = line.split('=')[1].strip()
continue
elif 'user' in line:
db_user = line.split('=')[1].strip()
continue
elif 'password' in line:
db_pass = line.split('=')[1].strip()
continue
# Initialize the DB object and connect to the db according to
# the db mysql backup file config
try:
backup_opt_dict.mysql_db_inst = MySQLdb.connect(
host=db_host, user=db_user, passwd=db_pass)
except Exception as error:
logging.critical('[*] MySQL Error: {0}'.format(error))
raise Exception
# Execute LVM backup
backup_mode_fs(backup_opt_dict, time_stamp, manifest_meta_dict)
def backup_mode_mongo(backup_opt_dict, time_stamp, manifest_meta_dict):
'''
Execute the necessary tasks for file system backup mode
'''
try:
from pymongo import MongoClient
except ImportError:
logging.critical('[*] Error: please install pymongo module')
raise ImportError('[*] Error: please install pymongo module')
logging.info('[*] MongoDB backup is being executed...')
logging.info('[*] Checking is the localhost is Master/Primary...')
mongodb_port = '27017'
local_hostname = backup_opt_dict.hostname
db_host_port = '{0}:{1}'.format(local_hostname, mongodb_port)
mongo_client = MongoClient(db_host_port)
master_dict = dict(mongo_client.admin.command("isMaster"))
mongo_me = master_dict['me']
mongo_primary = master_dict['primary']
if mongo_me == mongo_primary:
backup_mode_fs(backup_opt_dict, time_stamp, manifest_meta_dict)
else:
logging.warning('[*] localhost {0} is not Master/Primary,\
exiting...'.format(local_hostname))
return True
def backup_mode_fs(backup_opt_dict, time_stamp, manifest_meta_dict):
'''
Execute the necessary tasks for file system backup mode
'''
logging.info('[*] File System backup is being executed...')
lvm_snap(backup_opt_dict)
# Extract some values from arguments that will be used later on
# Initialize swift client object, generate container segments name
# and extract backup name
sw_connector = backup_opt_dict.sw_connector
# Execute a tar gzip of the specified directory and return
# small chunks (default 128MB), timestamp, backup, filename,
# file chunk index and the tar meta-data file
# Generate a string hostname, backup name, timestamp and backup level
file_name = add_host_name_ts_level(backup_opt_dict, time_stamp)
meta_data_backup_file = u'tar_metadata_{0}'.format(file_name)
(backup_opt_dict, tar_command, manifest_meta_dict) = gen_tar_command(
opt_dict=backup_opt_dict, time_stamp=time_stamp,
remote_manifest_meta=manifest_meta_dict)
# Initialize a Queue for a maximum of 2 items
tar_backup_queue = Queue(maxsize=2)
tar_backup_stream = Process(
target=tar_backup, args=(
backup_opt_dict, tar_command, tar_backup_queue,))
tar_backup_stream.daemon = True
tar_backup_stream.start()
add_object_stream = Process(
target=add_object, args=(
backup_opt_dict, tar_backup_queue, file_name, time_stamp))
add_object_stream.daemon = True
add_object_stream.start()
tar_backup_stream.join()
tar_backup_queue.put(
({False : False}))
tar_backup_queue.close()
add_object_stream.join()
(backup_opt_dict, manifest_meta_dict, tar_meta_to_upload,
tar_meta_prev) = gen_manifest_meta(
backup_opt_dict, manifest_meta_dict, meta_data_backup_file)
manifest_file = u''
meta_data_abs_path = '{0}/{1}'.format(
backup_opt_dict.workdir, tar_meta_prev)
# Upload swift manifest for segments
if backup_opt_dict.upload:
if not backup_opt_dict.no_incremental:
# Upload tar incremental meta data file and remove it
logging.info('[*] Uploading tar meta data file: {0}'.format(
tar_meta_to_upload))
with open(meta_data_abs_path, 'r') as meta_fd:
sw_connector.put_object(
backup_opt_dict.container, tar_meta_to_upload, meta_fd)
# Removing tar meta data file, so we have only one authoritative
# version on swift
logging.info('[*] Removing tar meta data file: {0}'.format(
meta_data_abs_path))
os.remove(meta_data_abs_path)
# Upload manifest to swift
manifest_upload(
manifest_file, backup_opt_dict, file_name, manifest_meta_dict)
# Unmount and remove lvm snapshot volume
lvm_snap_remove(backup_opt_dict)

206
freezer/lvm.py Normal file
View File

@ -0,0 +1,206 @@
'''
Copyright 2014 Hewlett-Packard
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This product includes cryptographic software written by Eric Young
(eay@cryptsoft.com). This product includes software written by Tim
Hudson (tjh@cryptsoft.com).
========================================================================
Freezer LVM related functions
'''
from freezer.utils import (
create_dir, get_vol_fs_type, validate_all_args)
import re
import os
import subprocess
import logging
def lvm_eval(backup_opt_dict):
'''
Evaluate if the backup must be executed using lvm snapshot
or just directly on the plain filesystem. If no lvm options are specified
the backup will be executed directly on the file system and without
use lvm snapshot. If one of the lvm options are set, then the lvm snap
will be used to execute backup. This mean all the required options
must be set accordingly
'''
required_list = [
backup_opt_dict.lvm_volgroup,
backup_opt_dict.lvm_srcvol,
backup_opt_dict.lvm_dirmount]
if not validate_all_args(required_list):
logging.warning('[*] Required lvm options not set. The backup will \
execute without lvm snapshot.')
return False
# Create lvm_dirmount dir if it doesn't exists and write action in logs
create_dir(backup_opt_dict.lvm_dirmount)
return True
def lvm_snap_remove(backup_opt_dict):
'''
Remove the specified lvm_snapshot. If the volume is mounted
it will unmount it and then removed
'''
if not lvm_eval(backup_opt_dict):
return True
vol_group = backup_opt_dict.lvm_volgroup.replace('-', '--')
snap_name = backup_opt_dict.lvm_snapname.replace('-', '--')
mapper_snap_vol = '/dev/mapper/{0}-{1}'.format(vol_group, snap_name)
with open('/proc/mounts', 'r') as proc_mount_fd:
for mount_line in proc_mount_fd:
if mapper_snap_vol.lower() in mount_line.lower():
mount_list = mount_line.split(' ')
(dev_vol, mount_point) = mount_list[0], mount_list[1]
logging.warning('[*] Found lvm snapshot {0} mounted on {1}\
'.format(dev_vol, mount_point))
umount_proc = subprocess.Popen('{0} -l -f {1}'.format(
backup_opt_dict.umount_path, mount_point),
stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
shell=True, executable=backup_opt_dict.bash_path)
(umount_out, mount_err) = umount_proc.communicate()
if re.search(r'\S+', umount_out):
logging.critical('[*] Error: impossible to umount {0} {1}\
'.format(mount_point, mount_err))
raise Exception
else:
# Change working directory to be able to unmount
os.chdir(backup_opt_dict.workdir)
logging.info('[*] Volume {0} unmounted'.format(
mapper_snap_vol))
snap_rm_proc = subprocess.Popen(
'{0} -f {1}'.format(
backup_opt_dict.lvremove_path, mapper_snap_vol),
stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
shell=True, executable=backup_opt_dict.bash_path)
(lvm_rm_out, lvm_rm_err) = snap_rm_proc.communicate()
if 'successfully removed' in lvm_rm_out:
logging.info('[*] {0}'.format(lvm_rm_out))
return True
else:
logging.critical(
'[*] Error: lvm_snap_rm {0}'.format(lvm_rm_err))
raise Exception
raise Exception
def lvm_snap(backup_opt_dict):
'''
Implement checks on lvm volumes availability. According to these checks
we might create an lvm snapshot and mount it or use an existing one
'''
if lvm_eval(backup_opt_dict) is not True:
return True
# Setting lvm snapsize to 5G is not set
if backup_opt_dict.lvm_snapsize is False:
backup_opt_dict.lvm_snapsize = '5G'
logging.warning('[*] lvm_snapsize not configured. Setting the \
lvm snapshot size to 5G')
# Setting lvm snapshot name to freezer_backup_snap it not set
if backup_opt_dict.lvm_snapname is False:
backup_opt_dict.lvm_snapname = 'freezer_backup_snap'
logging.warning('[*] lvm_snapname not configured. Setting default \
name "freezer_backup_snap" for the lvm backup snap session')
logging.info('[*] Source LVM Volume: {0}'.format(
backup_opt_dict.lvm_srcvol))
logging.info('[*] LVM Volume Group: {0}'.format(
backup_opt_dict.lvm_volgroup))
logging.info('[*] Snapshot name: {0}'.format(
backup_opt_dict.lvm_snapname))
logging.info('[*] Snapshot size: {0}'.format(
backup_opt_dict.lvm_snapsize))
logging.info('[*] Directory where the lvm snaphost will be mounted on:\
{0}'.format(backup_opt_dict.lvm_dirmount.strip()))
# Create the snapshot according the values passed from command line
lvm_create_snap = '{0} --size {1} --snapshot --name {2} {3}\
'.format(
backup_opt_dict.lvcreate_path,
backup_opt_dict.lvm_snapsize,
backup_opt_dict.lvm_snapname,
backup_opt_dict.lvm_srcvol)
# If backup mode is mysql, then the db will be flushed and read locked
# before the creation of the lvm snap
if backup_opt_dict.mode == 'mysql':
cursor = backup_opt_dict.mysql_db_inst.cursor()
cursor.execute('FLUSH TABLES WITH READ LOCK')
backup_opt_dict.mysql_db_inst.commit()
lvm_process = subprocess.Popen(
lvm_create_snap, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True,
executable=backup_opt_dict.bash_path)
(lvm_out, lvm_err) = lvm_process.communicate()
if lvm_err is False:
logging.critical('[*] lvm snapshot creation error: {0}\
'.format(lvm_err))
raise Exception
else:
logging.warning('[*] {0}'.format(lvm_out))
# Unlock MySQL Tables if backup is == mysql
if backup_opt_dict.mode == 'mysql':
cursor.execute('UNLOCK TABLES')
backup_opt_dict.mysql_db_inst.commit()
cursor.close()
backup_opt_dict.mysql_db_inst.close()
# Guess the file system of the provided source volume and st mount
# options accordingly
filesys_type = get_vol_fs_type(backup_opt_dict)
mount_options = ' '
if 'xfs' == filesys_type:
mount_options = ' -onouuid '
# Mount the newly created snapshot to dir_mount
abs_snap_name = '/dev/{0}/{1}'.format(
backup_opt_dict.lvm_volgroup,
backup_opt_dict.lvm_snapname)
mount_snap = '{0} {1} {2} {3}'.format(
backup_opt_dict.mount_path,
mount_options,
abs_snap_name,
backup_opt_dict.lvm_dirmount)
mount_process = subprocess.Popen(
mount_snap, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True,
executable=backup_opt_dict.bash_path)
mount_err = mount_process.communicate()[1]
if 'already mounted' in mount_err:
logging.warning('[*] Volume {0} already mounted on {1}\
'.format(abs_snap_name, backup_opt_dict.lvm_dirmount))
return True
if mount_err:
logging.critical('[*] lvm snapshot mounting error: {0}'.format(
mount_err))
raise Exception
else:
logging.warning('[*] Volume {0} succesfully mounted on {1}\
'.format(abs_snap_name, backup_opt_dict.lvm_dirmount))
return True

107
freezer/main.py Normal file
View File

@ -0,0 +1,107 @@
'''
Copyright 2014 Hewlett-Packard
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This product includes cryptographic software written by Eric Young
(eay@cryptsoft.com). This product includes software written by Tim
Hudson (tjh@cryptsoft.com).
========================================================================
Freezer main execution function
'''
from freezer.utils import (
start_time, elapsed_time, set_backup_level, validate_any_args,
check_backup_existance)
from freezer.swift import (
get_client, get_containers_list, show_containers,
check_container_existance, get_container_content, remove_obj_older_than,
show_objects)
from freezer.backup import (
backup_mode_fs, backup_mode_mongo, backup_mode_mysql)
from freezer.restore import restore_fs
import logging
def freezer_main(backup_args):
'''
Program Main Execution. This main function is a wrapper for most
of the other functions. By calling main() the program execution start
and the respective actions are taken. If you want only use the single
function is probably better to not import main()
'''
# Computing execution start datetime and Timestamp
(time_stamp, today_start) = start_time()
# Add timestamp to the arguments namespace
backup_args.__dict__['time_stamp'] = time_stamp
# Initialize the swift connector and store it in the same dict passed
# as argument under the dict.sw_connector namespace. This is helpful
# so the swift client object doesn't need to be initialized every time
backup_args = get_client(backup_args)
# Get the list of the containers
backup_args = get_containers_list(backup_args)
if show_containers(backup_args):
elapsed_time(today_start)
return True
# Check if the provided container already exists in swift.
# If it doesn't exist a new one will be created along with the segments
# container as container_segments
backup_args = check_container_existance(backup_args)
# Get the object list of the remote containers and store id in the
# same dict passes as argument under the dict.remote_obj_list namespace
backup_args = get_container_content(backup_args)
if show_objects(backup_args):
elapsed_time(today_start)
return True
# Check if a backup exist in swift with same name. If not, set
# backup level to 0
manifest_meta_dict = check_backup_existance(backup_args)
# Set the right backup level for incremental backup
(backup_args, manifest_meta_dict) = set_backup_level(
backup_args, manifest_meta_dict)
backup_args.manifest_meta_dict = manifest_meta_dict
# File system backup mode selected
if backup_args.mode == 'fs':
# If any of the restore options was specified, then a data restore
# will be executed
if validate_any_args([
backup_args.restore_from_date, backup_args.restore_from_host,
backup_args.restore_abs_path]):
logging.info('[*] Executing FS restore...')
restore_fs(backup_args)
else:
backup_mode_fs(backup_args, time_stamp, manifest_meta_dict)
elif backup_args.mode == 'mongo':
backup_mode_mongo(backup_args, time_stamp, manifest_meta_dict)
elif backup_args.mode == 'mysql':
backup_mode_mysql(backup_args, time_stamp, manifest_meta_dict)
else:
logging.critical('[*] Error: Please provide a valid backup mode')
raise ValueError
remove_obj_older_than(backup_args)
# Elapsed time:
elapsed_time(today_start)

164
freezer/restore.py Normal file
View File

@ -0,0 +1,164 @@
'''
Copyright 2014 Hewlett-Packard
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This product includes cryptographic software written by Eric Young
(eay@cryptsoft.com). This product includes software written by Tim
Hudson (tjh@cryptsoft.com).
========================================================================
Freezer restore modes related functions
'''
from freezer.tar import tar_restore
from freezer.swift import object_to_stream
from freezer.utils import (
validate_all_args, get_match_backup, sort_backup_list)
from multiprocessing import Process, Pipe
import os
import logging
import re
import datetime
import time
def restore_fs(backup_opt_dict):
'''
Restore data from swift server to your local node. Data will be restored
in the directory specified in backup_opt_dict.restore_abs_path. The
object specified with the --get-object option will be downloaded from
the Swift server and will be downloaded inside the parent directory of
backup_opt_dict.restore_abs_path. If the object was compressed during
backup time, then it is decrypted, decompressed and de-archived to
backup_opt_dict.restore_abs_path. Before download the file, the size of
the local volume/disk/partition will be computed. If there is enough space
the full restore will be executed. Please remember to stop any service
that require access to the data before to start the restore execution
and to start the service at the end of the restore execution
'''
# List of mandatory values
required_list = [
os.path.exists(backup_opt_dict.restore_abs_path),
backup_opt_dict.remote_obj_list,
backup_opt_dict.container,
backup_opt_dict.backup_name
]
# Arugment validation. Raise ValueError is all the arguments are not True
if not validate_all_args(required_list):
logging.critical("[*] Error: please provide ALL the following \
arguments: {0}".format(' '.join(required_list)))
raise ValueError
if not backup_opt_dict.restore_from_date:
logging.warning('[*] Restore date time not available. Setting to \
current datetime')
backup_opt_dict.restore_from_date = \
re.sub(
r'^(\S+?) (.+?:\d{,2})\.\d+?$', r'\1T\2',
str(datetime.datetime.now()))
# If restore_from_host is set to local hostname is not set in
# backup_opt_dict.restore_from_host
if backup_opt_dict.restore_from_host:
backup_opt_dict.hostname = backup_opt_dict.restore_from_host
# Check if there's a backup matching. If not raise Exception
backup_opt_dict = get_match_backup(backup_opt_dict)
if not backup_opt_dict.remote_match_backup:
logging.critical(
'[*] Not backup found matching with name: {0},\
hostname: {1}'.format(
backup_opt_dict.backup_name, backup_opt_dict.hostname))
raise ValueError
restore_fs_sort_obj(backup_opt_dict)
def restore_fs_sort_obj(backup_opt_dict):
'''
Take options dict as argument and sort/remove duplicate elements from
backup_opt_dict.remote_match_backup and find the closes backup to the
provided from backup_opt_dict.restore_from_date. Once the objects are
looped backwards and the level 0 backup is found, along with the other
level 1,2,n, is download the object from swift and untar them locally
starting from level 0 to level N.
'''
# Convert backup_opt_dict.restore_from_date to timestamp
fmt = '%Y-%m-%dT%H:%M:%S'
opt_backup_date = datetime.datetime.strptime(
backup_opt_dict.restore_from_date, fmt)
opt_backup_timestamp = int(time.mktime(opt_backup_date .timetuple()))
# Sort remote backup list using timestamp in reverse order,
# that is from the newest to the oldest executed backup
sorted_backups_list = sort_backup_list(backup_opt_dict)
# Get the closest earlier backup to date set in
# backup_opt_dict.restore_from_date
closest_backup_list = []
for backup_obj in sorted_backups_list:
if backup_obj.startswith('tar_metadata'):
continue
obj_name_match = re.search(
r'\S+?_{0}_(\d+)_(\d+?)$'.format(backup_opt_dict.backup_name),
backup_obj, re.I)
if not obj_name_match:
continue
# Ensure provided timestamp is bigger then object timestamp
if opt_backup_timestamp >= int(obj_name_match.group(1)):
closest_backup_list.append(backup_obj)
# If level 0 is reached, break the loop as level 0 is the first
# backup we want to restore
if int(obj_name_match.group(2)) == 0:
break
if not closest_backup_list:
logging.info('[*] No matching backup name {0} found in \
container {1} for hostname {2}'.format(
backup_opt_dict.backup_name, backup_opt_dict.container,
backup_opt_dict.hostname))
raise ValueError
# Backups are looped from the last element of the list going
# backwards, as we want to restore starting from the oldest object
for backup in closest_backup_list[::-1]:
write_pipe, read_pipe = Pipe()
process_stream = Process(
target=object_to_stream, args=(
backup_opt_dict, write_pipe, backup,))
process_stream.daemon = True
process_stream.start()
tar_stream = Process(
target=tar_restore, args=(backup_opt_dict, read_pipe,))
tar_stream.daemon = True
tar_stream.start()
process_stream.join()
tar_stream.join()
logging.info(
'[*] Restore execution successfully finished for backup name {0}, \
from container {1}, into directory {2}'.format(
backup_opt_dict.backup_name, backup_opt_dict.container,
backup_opt_dict.restore_abs_path))
logging.info(
'[*] Restore execution successfully executed for backup name {0}, \
from container {1}, into directory {2}'.format(
backup_opt_dict.backup_name, backup_opt_dict.container,
backup_opt_dict.restore_abs_path))

398
freezer/swift.py Normal file
View File

@ -0,0 +1,398 @@
'''
Copyright 2014 Hewlett-Packard
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This product includes cryptographic software written by Eric Young
(eay@cryptsoft.com). This product includes software written by Tim
Hudson (tjh@cryptsoft.com).
========================================================================
Freezer functions to interact with OpenStack Swift client and server
'''
from freezer.utils import (
validate_all_args, get_match_backup,
sort_backup_list)
import os
import swiftclient
import json
import re
from copy import deepcopy
import time
import logging
def show_containers(backup_opt_dict):
'''
Print remote containers in sorted order
'''
if not backup_opt_dict.list_container:
return False
ordered_container = {}
for container in backup_opt_dict.containers_list:
ordered_container['container_name'] = container['name']
size = '{0}'.format((int(container['bytes']) / 1024) / 1024)
if size == '0':
size = '1'
ordered_container['size'] = '{0}MB'.format(size)
ordered_container['objects_count'] = container['count']
print json.dumps(
ordered_container, indent=4,
separators=(',', ': '), sort_keys=True)
return True
def show_objects(backup_opt_dict):
'''
Retreive the list of backups from backup_opt_dict for the specified \
container and print them nicely to std out.
'''
if not backup_opt_dict.list_objects:
return False
required_list = [
backup_opt_dict.remote_obj_list]
if not validate_all_args(required_list):
logging.critical('[*] Error: Remote Object list not avaiblale')
raise Exception
ordered_objects = {}
remote_obj = backup_opt_dict.remote_obj_list
for obj in remote_obj:
ordered_objects['object_name'] = obj['name']
ordered_objects['upload_date'] = obj['last_modified']
print json.dumps(
ordered_objects, indent=4,
separators=(',', ': '), sort_keys=True)
return True
def remove_obj_older_than(backup_opt_dict):
'''
Remove object in remote swift server older more tqhen days
'''
if not backup_opt_dict.remote_obj_list \
or backup_opt_dict.remove_older_than is False:
logging.warning('[*] No remote objects will be removed')
return False
backup_opt_dict.remove_older_than = int(
float(backup_opt_dict.remove_older_than))
logging.info('[*] Removing object older {0} day(s)'.format(
backup_opt_dict.remove_older_than))
# Compute the amount of seconds from days to compare with
# the remote backup timestamp
max_time = backup_opt_dict.remove_older_than * 86400
current_timestamp = backup_opt_dict.time_stamp
backup_name = backup_opt_dict.backup_name
hostname = backup_opt_dict.hostname
backup_opt_dict = get_match_backup(backup_opt_dict)
sorted_remote_list = sort_backup_list(backup_opt_dict)
sw_connector = backup_opt_dict.sw_connector
for match_object in sorted_remote_list:
obj_name_match = re.search(r'{0}_({1})_(\d+)_\d+?$'.format(
hostname, backup_name), match_object, re.I)
if not obj_name_match:
continue
remote_obj_timestamp = int(obj_name_match.group(2))
time_delta = current_timestamp - remote_obj_timestamp
if time_delta > max_time:
logging.info('[*] Removing backup object: {0}'.format(
match_object))
sw_connector.delete_object(
backup_opt_dict.container, match_object)
# Try to remove also the corresponding tar_meta
# NEED TO BE IMPROVED!
try:
tar_match_object = 'tar_metadata_{0}'.format(match_object)
sw_connector.delete_object(
backup_opt_dict.container, tar_match_object)
logging.info(
'[*] Object tar meta data removed: {0}'.format(
tar_match_object))
except Exception:
pass
def get_container_content(backup_opt_dict):
'''
Download the list of object of the provided container
and print them out as container meta-data and container object list
'''
if not backup_opt_dict.container:
print '[*] Error: please provide a valid container name'
logging.critical(
'[*] Error: please provide a valid container name')
raise Exception
sw_connector = backup_opt_dict.sw_connector
try:
backup_opt_dict.remote_obj_list = \
sw_connector.get_container(backup_opt_dict.container)[1]
return backup_opt_dict
except Exception as error:
logging.critical('[*] Error: get_object_list: {0}'.format(error))
raise Exception
def check_container_existance(backup_opt_dict):
'''
Check if the provided container is already available on Swift.
The verification is done by exact matching between the provided container
name and the whole list of container available for the swift account.
If the container is not found, it will be automatically create and used
to execute the backup
'''
required_list = [
backup_opt_dict.container_segments,
backup_opt_dict.container]
if not validate_all_args(required_list):
logging.critical("[*] Error: please provide ALL the following args \
{0}".format(','.join(required_list)))
raise Exception
logging.info(
"[*] Retrieving container {0}".format(backup_opt_dict.container))
sw_connector = backup_opt_dict.sw_connector
containers_list = sw_connector.get_account()[1]
match_container = None
match_container_seg = None
match_container = [
container_object['name'] for container_object in containers_list
if container_object['name'] == backup_opt_dict.container]
match_container_seg = [
container_object['name'] for container_object in containers_list
if container_object['name'] == backup_opt_dict.container_segments]
# If no container is available, create it and write to logs
if not match_container:
logging.warning("[*] No such container {0} available... ".format(
backup_opt_dict.container))
logging.warning(
"[*] Creating container {0}".format(backup_opt_dict.container))
sw_connector.put_container(backup_opt_dict.container)
else:
logging.info(
"[*] Container {0} found!".format(backup_opt_dict.container))
if not match_container_seg:
logging.warning("[*] Creating segments container {0}".format(
backup_opt_dict.container_segments))
sw_connector.put_container(backup_opt_dict.container_segments)
else:
logging.info("[*] Container Segments {0} found!".format(
backup_opt_dict.container_segments))
return backup_opt_dict
# This function is useless? Remove is and use the single env accordingly
def get_swift_os_env():
'''
Get the swift related environment variable
'''
environ_dict = os.environ
return environ_dict['OS_REGION_NAME'], environ_dict['OS_TENANT_ID'], \
environ_dict['OS_PASSWORD'], environ_dict['OS_AUTH_URL'], \
environ_dict['OS_USERNAME'], environ_dict['OS_TENANT_NAME']
def get_client(backup_opt_dict):
'''
Initialize a swift client object and return it in
backup_opt_dict
'''
sw_client = swiftclient.client
options = {}
(options['region_name'], options['tenant_id'], options['password'],
options['auth_url'], options['username'],
options['tenant_name']) = get_swift_os_env()
backup_opt_dict.sw_connector = sw_client.Connection(
authurl=options['auth_url'],
user=options['username'], key=options['password'], os_options=options,
tenant_name=options['tenant_name'], auth_version='2', retries=6)
return backup_opt_dict
def manifest_upload(
manifest_file, backup_opt_dict, file_prefix, manifest_meta_dict):
'''
Upload Manifest to manage segments in Swift
'''
if not manifest_meta_dict:
logging.critical('[*] Error Manifest Meta dictionary not available')
raise Exception
sw_connector = backup_opt_dict.sw_connector
tmp_manifest_meta = dict()
for key, value in manifest_meta_dict.items():
if key.startswith('x-object-meta'):
tmp_manifest_meta[key] = value
manifest_meta_dict = deepcopy(tmp_manifest_meta)
header = manifest_meta_dict
manifest_meta_dict['x-object-manifest'] = u'{0}/{1}'.format(
backup_opt_dict.container_segments.strip(), file_prefix.strip())
logging.info('[*] Uploading Swift Manifest: {0}'.format(header))
sw_connector.put_object(
backup_opt_dict.container, file_prefix, manifest_file, headers=header)
logging.info('[*] Manifest successfully uploaded!')
def add_object(
backup_opt_dict, backup_queue, absolute_file_path=None,
time_stamp=None):
'''
Upload object on the remote swift server
'''
if not backup_opt_dict.container:
logging.critical('[*] Error: Please specify the container \
name with -C option')
raise Exception
if absolute_file_path is None and backup_queue is None:
logging.critical('[*] Error: Please specify the file you want to \
upload on swift with -d option')
raise Exception
max_segment_size = backup_opt_dict.max_seg_size
if not backup_opt_dict.max_seg_size:
max_segment_size = 134217728
sw_connector = backup_opt_dict.sw_connector
while True:
package_name = absolute_file_path.split('/')[-1]
file_chunk_index, file_chunk = backup_queue.get().popitem()
if file_chunk_index == False and file_chunk == False:
break
package_name = u'{0}/{1}/{2}/{3}'.format(
package_name, time_stamp, max_segment_size, file_chunk_index)
# If for some reason the swift client object is not available anymore
# an exception is generated and a new client object is initialized/
# If the exception happens for 10 consecutive times for a total of
# 1 hour, then the program will exit with an Exception.
count = 0
while True:
try:
logging.info(
'[*] Uploading file chunk index: {0}'.format(
package_name))
sw_connector.put_object(
backup_opt_dict.container_segments,
package_name, file_chunk)
logging.info('[*] Data successfully uploaded!')
break
except Exception as error:
time.sleep(60)
logging.info(
'[*] Retrying to upload file chunk index: {0}'.format(
package_name))
backup_opt_dict = get_client(backup_opt_dict)
count += 1
if count == 10:
logging.info(
'[*] Error: add_object: {0}'.format(error))
raise Exception
def get_containers_list(backup_opt_dict):
'''
Get a list and information of all the available containers
'''
try:
sw_connector = backup_opt_dict.sw_connector
backup_opt_dict.containers_list = sw_connector.get_account()[1]
return backup_opt_dict
except Exception as error:
logging.error('[*] Get containers list error: {0}').format(error)
raise Exception
def object_to_file(backup_opt_dict, file_name_abs_path):
'''
Take a payload downloaded from Swift
and save it to the disk as file_name
'''
required_list = [
backup_opt_dict.container,
file_name_abs_path]
if not validate_all_args(required_list):
logging.critical('[*] Error: Please provide ALL the following \
arguments: {0}'.format(','.join(required_list)))
raise ValueError
sw_connector = backup_opt_dict.sw_connector
file_name = file_name_abs_path.split('/')[-1]
logging.info('[*] Downloading object {0} on {1}'.format(
file_name, file_name_abs_path))
# As the file is download by chunks and each chunk will be appened
# to file_name_abs_path, we make sure file_name_abs_path does not
# exists by removing it before
if os.path.exists(file_name_abs_path):
os.remove(file_name_abs_path)
with open(file_name_abs_path, 'ab') as obj_fd:
for obj_chunk in sw_connector.get_object(
backup_opt_dict.container, file_name,
resp_chunk_size=16000000)[1]:
obj_fd.write(obj_chunk)
return True
def object_to_stream(backup_opt_dict, write_pipe, obj_name):
'''
Take a payload downloaded from Swift
and generate a stream to be consumed from other processes
'''
required_list = [
backup_opt_dict.container]
if not validate_all_args(required_list):
logging.critical('[*] Error: Please provide ALL the following \
arguments: {0}'.format(','.join(required_list)))
raise ValueError
sw_connector = backup_opt_dict.sw_connector
logging.info('[*] Downloading data stream...')
# As the file is download by chunks and each chunk will be appened
# to file_name_abs_path, we make sure file_name_abs_path does not
# exists by removing it before
#stream_write_pipe = os.fdopen(stream_write_pipe, 'w', 0)
for obj_chunk in sw_connector.get_object(
backup_opt_dict.container, obj_name,
resp_chunk_size=backup_opt_dict.max_seg_size)[1]:
write_pipe.send(obj_chunk)

225
freezer/tar.py Normal file
View File

@ -0,0 +1,225 @@
'''
Copyright 2014 Hewlett-Packard
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This product includes cryptographic software written by Eric Young
(eay@cryptsoft.com). This product includes software written by Tim
Hudson (tjh@cryptsoft.com).
========================================================================
Freezer Tar related functions
'''
from freezer.utils import (
validate_all_args, add_host_name_ts_level, create_dir)
from freezer.swift import object_to_file
import os
import logging
import subprocess
import time
def tar_restore(backup_opt_dict, read_pipe):
'''
Restore the provided file into backup_opt_dict.restore_abs_path
Descrypt the file if backup_opt_dict.encrypt_pass_file key is provided
'''
# Validate mandatory arguments
required_list = [
os.path.exists(backup_opt_dict.restore_abs_path)]
if not validate_all_args(required_list):
logging.critical("[*] Error: please provide ALL of the following \
arguments: {0}".format(' '.join(required_list)))
raise ValueError
# Set the default values for tar restore
tar_cmd = ' {0} -z --incremental --extract \
--unlink-first --ignore-zeros --warning=none --overwrite \
--directory {1} '.format(
backup_opt_dict.tar_path, backup_opt_dict.restore_abs_path)
# Check if encryption file is provided and set the openssl decrypt
# command accordingly
if backup_opt_dict.encrypt_pass_file:
openssl_cmd = " {0} enc -d -aes-256-cfb -pass file:{1}".format(
backup_opt_dict.openssl_path,
backup_opt_dict.encrypt_pass_file)
tar_cmd = ' {0} | {1} '.format(openssl_cmd, tar_cmd)
tar_cmd_proc = subprocess.Popen(
tar_cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True,
executable=backup_opt_dict.bash_path)
# Start loop reading the pipe and pass the data to the tar std input
while True:
data_stream = read_pipe.recv()
tar_cmd_proc.stdin.write(data_stream)
if len(data_stream) < int(backup_opt_dict.max_seg_size):
break
tar_err = tar_cmd_proc.communicate()[1]
if 'error' in tar_err.lower():
logging.critical('[*] Restore error: {0}'.format(tar_err))
raise Exception
def tar_incremental(
tar_cmd, backup_opt_dict, curr_tar_meta, remote_manifest_meta=None):
'''
Check if the backup already exist in swift. If the backup already
exists, the related meta data and the tar incremental meta file will be
downloaded. According to the meta data content, new options will be
provided for the next meta data upload to swift and the existing tar meta
file will be used in the current incremental backup. Also the level
options will be checked and updated respectively
'''
if not tar_cmd or not backup_opt_dict:
logging.error('[*] Error: tar_incremental, please provide tar_cmd \
and backup options')
raise ValueError
if not remote_manifest_meta:
remote_manifest_meta = dict()
# If returned object from check_backup is not a dict, the backup
# is considered at first run, so a backup level 0 will be executed
curr_backup_level = remote_manifest_meta.get(
'x-object-meta-backup-current-level', '0')
tar_meta = remote_manifest_meta.get(
'x-object-meta-tar-meta-obj-name')
tar_cmd_level = '--level={0} '.format(curr_backup_level)
# Write the tar meta data file in ~/.freezer. It will be
# removed later on. If ~/.freezer does not exists it will be created'.
create_dir(backup_opt_dict.workdir)
curr_tar_meta = '{0}/{1}'.format(
backup_opt_dict.workdir, curr_tar_meta)
tar_cmd_incr = ' --listed-incremental={0} '.format(curr_tar_meta)
if tar_meta:
# If tar meta data file is available, download it and use it
# as for tar incremental backup. Afte this, the
# remote_manifest_meta['x-object-meta-tar-meta-obj-name'] will be
# update with the current tar meta data name and uploaded again
tar_cmd_incr = ' --listed-incremental={0}/{1} '.format(
backup_opt_dict.workdir, tar_meta)
tar_meta_abs = "{0}/{1}".format(backup_opt_dict.workdir, tar_meta)
try:
object_to_file(
backup_opt_dict, tar_meta_abs)
except Exception:
logging.warning(
'[*] Tar metadata {0} not found. Executing level 0 backup\
'.format(tar_meta))
tar_cmd = ' {0} {1} {2} '.format(tar_cmd, tar_cmd_level, tar_cmd_incr)
return tar_cmd, backup_opt_dict, remote_manifest_meta
def gen_tar_command(
opt_dict, meta_data_backup_file=False, time_stamp=int(time.time()),
remote_manifest_meta=False):
'''
Generate tar command options.
'''
required_list = [
opt_dict.backup_name,
opt_dict.src_file,
os.path.exists(opt_dict.src_file)]
if not validate_all_args(required_list):
logging.critical(
'Error: Please ALL the following options: {0}'.format(
','.join(required_list)))
raise Exception
# Change che current working directory to op_dict.src_file
os.chdir(os.path.normpath(opt_dict.src_file.strip()))
logging.info('[*] Changing current working directory to: {0} \
'.format(opt_dict.src_file))
logging.info('[*] Backup started for: {0} \
'.format(opt_dict.src_file))
# Tar option for default behavoir. Please refer to man tar to have
# a better options explanation
tar_command = ' {0} --create -z --warning=none \
--dereference --hard-dereference --no-check-device --one-file-system \
--preserve-permissions --same-owner --seek \
--ignore-failed-read '.format(opt_dict.tar_path)
file_name = add_host_name_ts_level(opt_dict, time_stamp)
meta_data_backup_file = u'tar_metadata_{0}'.format(file_name)
# Incremental backup section
if not opt_dict.no_incremental:
(tar_command, opt_dict, remote_manifest_meta) = tar_incremental(
tar_command, opt_dict, meta_data_backup_file,
remote_manifest_meta)
# End incremental backup section
if opt_dict.exclude:
tar_command = ' {0} --exclude="{1}" '.format(
tar_command,
opt_dict.exclude)
tar_command = ' {0} . '.format(tar_command)
# Encrypt data if passfile is provided
if opt_dict.encrypt_pass_file:
openssl_cmd = "{0} enc -aes-256-cfb -pass file:{1}".format(
opt_dict.openssl_path, opt_dict.encrypt_pass_file)
tar_command = '{0} | {1} '.format(tar_command, openssl_cmd)
return opt_dict, tar_command, remote_manifest_meta
def tar_backup(opt_dict, tar_command, backup_queue):
'''
Execute an incremental backup using tar options, specified as
function arguments
'''
# Set counters, index, limits and bufsize for subprocess
buf_size = 65535
file_read_limit = 0
file_chunk_index = 00000000
file_block = b''
tar_chunk = b''
logging.info(
'[*] Archiving and compressing files from {0}'.format(
opt_dict.src_file))
tar_process = subprocess.Popen(
tar_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
bufsize=buf_size, shell=True, executable=opt_dict.bash_path)
# Iterate over tar process stdout
for file_block in tar_process.stdout:
tar_chunk += file_block
file_read_limit += len(file_block)
if file_read_limit >= opt_dict.max_seg_size:
backup_queue.put(
({file_chunk_index : tar_chunk}))
file_chunk_index += 1
tar_chunk = b''
file_read_limit = 0
# Upload segments smaller then opt_dict.max_seg_size
if len(tar_chunk) < opt_dict.max_seg_size:
backup_queue.put(
({file_chunk_index : tar_chunk}))
file_chunk_index += 1

0
freezer/test/__init__.py Normal file
View File

11
freezer/test/test_arguments.py Normal file
View File

@ -0,0 +1,11 @@
#!/usr/bin/env python
from freezer.arguments import backup_arguments
import pytest
import argparse
def test_backup_arguments():
backup_args, arg_parser = backup_arguments()
assert backup_args.tar_path is not False
assert backup_args.mode is ('fs' or 'mysql' or 'mongo')

16
freezer/test/test_backup.py Normal file
View File

@ -0,0 +1,16 @@
#!/usr/bin/env python
from freezer.backup import (
backup_mode_mysql, backup_mode_fs, backup_mode_mongo)
from freezer.arguments import backup_arguments
import time
import os
def test_backup_mode_mysql():
# THE WHOLE TEST NEED TO BE CHANGED USING MOCK!!
# Return backup options and arguments
backup_args = backup_arguments()

0
freezer/test/test_freezer.py Normal file
View File

524
freezer/utils.py Normal file
View File

@ -0,0 +1,524 @@
'''
Copyright 2014 Hewlett-Packard
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This product includes cryptographic software written by Eric Young
(eay@cryptsoft.com). This product includes software written by Tim
Hudson (tjh@cryptsoft.com).
========================================================================
Freezer general utils functions
'''
import logging
import os
import time
import datetime
import re
import subprocess
def gen_manifest_meta(
backup_opt_dict, manifest_meta_dict, meta_data_backup_file):
''' This function is used to load backup metadata information on Swift.
this is used to keep information between consecutive backup
executions.
If the manifest_meta_dict is available, most probably this is not
the first backup run for the provided backup name and host.
In this case we remove all the conflictive keys -> values from
the dictionary.
'''
if manifest_meta_dict.get('x-object-meta-tar-prev-meta-obj-name'):
tar_meta_prev = \
manifest_meta_dict['x-object-meta-tar-prev-meta-obj-name']
tar_meta_to_upload = \
manifest_meta_dict['x-object-meta-tar-meta-obj-name'] = \
manifest_meta_dict['x-object-meta-tar-prev-meta-obj-name'] = \
meta_data_backup_file
else:
manifest_meta_dict['x-object-meta-tar-prev-meta-obj-name'] = \
meta_data_backup_file
manifest_meta_dict['x-object-meta-backup-name'] = \
backup_opt_dict.backup_name
manifest_meta_dict['x-object-meta-src-file-to-backup'] = \
backup_opt_dict.src_file
manifest_meta_dict['x-object-meta-abs-file-path'] = ''
# Set manifest meta if encrypt_pass_file is provided
# The file will contain a plain password that will be used
# to encrypt and decrypt tasks
manifest_meta_dict['x-object-meta-encrypt-data'] = 'Yes'
if backup_opt_dict.encrypt_pass_file is False:
manifest_meta_dict['x-object-meta-encrypt-data'] = ''
manifest_meta_dict['x-object-meta-always-backup-level'] = ''
if backup_opt_dict.always_backup_level:
manifest_meta_dict['x-object-meta-always-backup-level'] = \
backup_opt_dict.always_backup_level
# Set manifest meta if max_backup_level argument is provided
# Once the incremental backup arrive to max_backup_level, it will
# restart from level 0
manifest_meta_dict['x-object-meta-maximum-backup-level'] = ''
if backup_opt_dict.max_backup_level is not False:
manifest_meta_dict['x-object-meta-maximum-backup-level'] = \
str(backup_opt_dict.max_backup_level)
# At the end of the execution, checks the objects ages for the
# specified swift container. If there are object older then the
# specified days they'll be removed.
# Unit is int and every int and 5 == five days.
manifest_meta_dict['x-object-meta-remove-backup-older-than-days'] = ''
if backup_opt_dict.remove_older_than is not False:
manifest_meta_dict['x-object-meta-remove-backup-older-than-days']\
= backup_opt_dict.remove_older_than
manifest_meta_dict['x-object-meta-hostname'] = backup_opt_dict.hostname
manifest_meta_dict['x-object-meta-segments-size-bytes'] = \
str(backup_opt_dict.max_seg_size)
manifest_meta_dict['x-object-meta-backup-created-timestamp'] = \
str(backup_opt_dict.time_stamp)
manifest_meta_dict['x-object-meta-providers-list'] = 'HP'
manifest_meta_dict['x-object-meta-tar-meta-obj-name'] = \
meta_data_backup_file
tar_meta_to_upload = tar_meta_prev = \
manifest_meta_dict['x-object-meta-tar-meta-obj-name'] = \
manifest_meta_dict['x-object-meta-tar-prev-meta-obj-name']
# Need to be processed from the last existing backup file found
# in Swift, matching with hostname and backup name
# the last existing file can be extracted from the timestamp
manifest_meta_dict['x-object-meta-container-segments'] = \
backup_opt_dict.container_segments
# Set the restart_always_backup value to n days. According
# to the following option, when the always_backup_level is set
# the backup will be reset to level 0 if the current backup
# times tamp is older then the days in x-object-meta-container-segments
manifest_meta_dict['x-object-meta-restart-always-backup'] = ''
if backup_opt_dict.restart_always_backup is not False:
manifest_meta_dict['x-object-meta-restart-always-backup'] = \
backup_opt_dict.restart_always_backup
return (
backup_opt_dict, manifest_meta_dict,
tar_meta_to_upload, tar_meta_prev)
def validate_all_args(required_list):
'''
Ensure ALL the elements of required_list are True. raise ValueError
Exception otherwise
'''
try:
for element in required_list:
if element is False or not element:
return False
except Exception as error:
logging.critical("[*] Error: {0} please provide ALL of the following \
arguments: {1}".format(error, ' '.join(required_list)))
raise Exception
return True
def validate_any_args(required_list):
'''
Ensure ANY of the elements of required_list are True. raise ValueError
Exception otherwise
'''
try:
for element in required_list:
if element:
return True
except Exception:
logging.critical("[*] Error: please provide ANY of the following \
arguments: {0}".format(' '.join(required_list)))
raise Exception
return False
def sort_backup_list(backup_opt_dict):
'''
Sort the backups by timestamp. The provided list contains strings in the
format hostname_backupname_timestamp_level
'''
# Remove duplicates objects
sorted_backups_list = list(set(backup_opt_dict.remote_match_backup))
sorted_backups_list.sort(key=lambda x: x.split('_')[2], reverse=True)
return sorted_backups_list
def create_dir(directory):
'''
Creates a directory if it doesn't exists and write the execution
in the logs
'''
try:
if not os.path.isdir(os.path.expanduser(directory)):
logging.warning('[*] Directory {0} does not exists,\
creating...'.format(os.path.expanduser(directory)))
os.makedirs(os.path.expanduser(directory))
else:
logging.warning('[*] Directory {0} found!'.format(
os.path.expanduser(directory)))
except Exception as error:
logging.warning('[*] Error while creating directory {0}: {1}\
'.format(os.path.expanduser(directory, error)))
raise Exception
def get_match_backup(backup_opt_dict):
'''
Return a dictionary containing a list of remote matching backups from
backup_opt_dict.remote_obj_list.
Backup have to exactly match against backup name and hostname of the
node where freezer is executed. The matching objects are stored and
available in backup_opt_dict.remote_match_backup
'''
if not backup_opt_dict.backup_name or not backup_opt_dict.container \
or not backup_opt_dict.remote_obj_list:
logging.critical("[*] Error: please provide a valid Swift container,\
backup name and the container contents")
raise Exception
backup_name = backup_opt_dict.backup_name.lower()
if backup_opt_dict.remote_obj_list:
hostname = backup_opt_dict.hostname
for container_object in backup_opt_dict.remote_obj_list:
object_name = container_object.get('name', None)
if object_name:
obj_name_match = re.search(r'{0}_({1})_\d+?_\d+?$'.format(
hostname, backup_name), object_name.lower(), re.I)
if obj_name_match:
backup_opt_dict.remote_match_backup.append(
object_name)
backup_opt_dict.remote_objects.append(container_object)
return backup_opt_dict
def get_newest_backup(backup_opt_dict):
'''
Return from backup_opt_dict.remote_match_backup, the newest backup
matching the provided backup name and hostname of the node where
freezer is executed. It correspond to the previous backup executed.
'''
if not backup_opt_dict.remote_match_backup:
return backup_opt_dict
backup_timestamp = 0
hostname = backup_opt_dict.hostname
# Sort remote backup list using timestamp in reverse order,
# that is from the newest to the oldest executed backup
sorted_backups_list = sort_backup_list(backup_opt_dict)
for remote_obj in sorted_backups_list:
obj_name_match = re.search(r'^{0}_({1})_(\d+)_\d+?$'.format(
hostname, backup_opt_dict.backup_name), remote_obj, re.I)
if not obj_name_match:
continue
remote_obj_timestamp = int(obj_name_match.group(2))
if remote_obj_timestamp > backup_timestamp:
backup_timestamp = remote_obj_timestamp
backup_opt_dict.remote_newest_backup = remote_obj
break
return backup_opt_dict
def get_rel_oldest_backup(backup_opt_dict):
'''
Return from swift, the relative oldest backup matching the provided
backup name and hostname of the node where freezer is executed.
The relative oldest backup correspond the oldest backup from the
last level 0 backup.
'''
if not backup_opt_dict.backup_name:
logging.critical("[*] Error: please provide a valid backup name in \
backup_opt_dict.backup_name")
raise Exception
backup_opt_dict.remote_rel_oldest = u''
backup_name = backup_opt_dict.backup_name
hostname = backup_opt_dict.hostname
first_backup_name = False
first_backup_ts = 0
for container_object in backup_opt_dict.remote_obj_list:
object_name = container_object.get('name', None)
if not object_name:
continue
obj_name_match = re.search(r'{0}_({1})_(\d+)_(\d+?)$'.format(
hostname, backup_name), object_name, re.I)
if not obj_name_match:
continue
remote_obj_timestamp = int(obj_name_match.group(2))
remote_obj_level = int(obj_name_match.group(3))
if remote_obj_level == 0 and (remote_obj_timestamp > first_backup_ts):
first_backup_name = object_name
first_backup_ts = remote_obj_timestamp
backup_opt_dict.remote_rel_oldest = first_backup_name
return backup_opt_dict
def get_abs_oldest_backup(backup_opt_dict):
'''
Return from swift, the absolute oldest backup matching the provided
backup name and hostname of the node where freezer is executed.
The absolute oldest backup correspond the oldest available level 0 backup.
'''
if not backup_opt_dict.backup_name:
logging.critical("[*] Error: please provide a valid backup name in \
backup_opt_dict.backup_name")
raise Exception
backup_opt_dict.remote_abs_oldest = u''
if len(backup_opt_dict.remote_match_backup) == 0:
return backup_opt_dict
backup_timestamp = 0
hostname = backup_opt_dict.hostname
for remote_obj in backup_opt_dict.remote_match_backup:
object_name = remote_obj.get('name', None)
obj_name_match = re.search(r'{0}_({1})_(\d+)_(\d+?)$'.format(
hostname, backup_opt_dict.backup_name), remote_obj, re.I)
if not obj_name_match:
continue
remote_obj_timestamp = int(obj_name_match.group(2))
if backup_timestamp == 0:
backup_timestamp = remote_obj_timestamp
if remote_obj_timestamp <= backup_timestamp:
backup_timestamp = remote_obj_timestamp
backup_opt_dict.remote_abs_oldest = object_name
return backup_opt_dict
def eval_restart_backup(backup_opt_dict):
'''
Restart backup level if the first backup execute with always_backup_level
is older then restart_always_backup
'''
if not backup_opt_dict.restart_always_backup:
logging.info('[*] No need to set Backup {0} to level 0.'.format(
backup_opt_dict.backup_name))
return False
logging.info('[*] Checking always backup level timestamp...')
# Compute the amount of seconds to be compared with
# the remote backup timestamp
max_time = int(float(backup_opt_dict.restart_always_backup) * 86400)
current_timestamp = backup_opt_dict.time_stamp
backup_name = backup_opt_dict.backup_name
hostname = backup_opt_dict.hostname
first_backup_ts = 0
# Get relative oldest backup by calling get_rel_oldes_backup()
backup_opt_dict = get_rel_oldest_backup(backup_opt_dict)
if not backup_opt_dict.remote_rel_oldest:
logging.info('[*] Relative oldest backup for backup name {0} on \
host {1} not available. The backup level is NOT restarted'.format(
backup_name, hostname))
return False
obj_name_match = re.search(r'{0}_({1})_(\d+)_(\d+?)$'.format(
hostname, backup_name), backup_opt_dict.remote_rel_oldest, re.I)
if not obj_name_match:
logging.info('[*] No backup match available for backup {0} \
and host {1}'.format(backup_name, hostname))
return Exception
first_backup_ts = int(obj_name_match.group(2))
if (current_timestamp - first_backup_ts) > max_time:
logging.info(
'[*] Backup {0} older then {1} days. Backup level set to 0'.format(
backup_name, backup_opt_dict.restart_always_backup))
return True
else:
logging.info('[*] No need to set level 0 for Backup {0}.'.format(
backup_name))
return False
def start_time():
'''
Compute start execution time, write it in the logs and return timestamp
'''
fmt = '%Y-%m-%d %H:%M:%S'
today_start = datetime.datetime.now()
time_stamp = int(time.mktime(today_start.timetuple()))
fmt_date_start = today_start.strftime(fmt)
logging.info('[*] Execution Started at: {0}'.format(fmt_date_start))
return time_stamp, today_start
def elapsed_time(today_start):
'''
Compute elapsed time from today_start and write basic stats
in the log file
'''
fmt = '%Y-%m-%d %H:%M:%S'
today_finish = datetime.datetime.now()
fmt_date_finish = today_finish.strftime(fmt)
time_elapsed = today_finish - today_start
# Logging end execution information
logging.info('[*] Execution Finished, at: {0}'.format(fmt_date_finish))
logging.info('[*] Time Elapsed: {0}'.format(time_elapsed))
def set_backup_level(backup_opt_dict, manifest_meta_dict):
'''
Set the backup level params in backup_opt_dict and the swift
manifest. This is a fundamental part of the incremental backup
'''
if manifest_meta_dict.get('x-object-meta-backup-name'):
backup_opt_dict.curr_backup_level = int(
manifest_meta_dict.get('x-object-meta-backup-current-level'))
max_backup_level = manifest_meta_dict.get(
'x-object-meta-maximum-backup-level')
always_backup_level = manifest_meta_dict.get(
'x-object-meta-always-backup-level')
restart_always_backup = manifest_meta_dict.get(
'x-object-meta-restart-always-backup')
if max_backup_level:
max_backup_level = int(max_backup_level)
if backup_opt_dict.curr_backup_level < max_backup_level:
backup_opt_dict.curr_backup_level += 1
manifest_meta_dict['x-object-meta-backup-current-level'] = \
str(backup_opt_dict.curr_backup_level)
else:
manifest_meta_dict['x-object-meta-backup-current-level'] = \
backup_opt_dict.curr_backup_level = '0'
elif always_backup_level:
always_backup_level = int(always_backup_level)
if backup_opt_dict.curr_backup_level < always_backup_level:
backup_opt_dict.curr_backup_level += 1
manifest_meta_dict['x-object-meta-backup-current-level'] = \
str(backup_opt_dict.curr_backup_level)
# If restart_always_backup is set, the backup_age will be computed
# and if the backup age in days is >= restart_always_backup, then
# backup-current-level will be set to 0
if restart_always_backup:
backup_opt_dict.restart_always_backup = restart_always_backup
if eval_restart_backup(backup_opt_dict):
backup_opt_dict.curr_backup_level = '0'
manifest_meta_dict['x-object-meta-backup-current-level'] \
= '0'
else:
backup_opt_dict.curr_backup_level = \
manifest_meta_dict['x-object-meta-backup-current-level'] = '0'
return backup_opt_dict, manifest_meta_dict
def get_vol_fs_type(backup_opt_dict):
'''
The argument need to be a full path lvm name i.e. /dev/vg0/var
or a disk partition like /dev/sda1. The returnet value is the
file system type
'''
vol_name = backup_opt_dict.lvm_srcvol
if os.path.exists(vol_name) is False:
logging.critical('[*] Provided volume name not found: {0} \
'.format(vol_name))
raise Exception
file_cmd = '{0} -0 -bLs --no-pad --no-buffer --preserve-date \
{1}'.format(backup_opt_dict.file_path, vol_name)
file_process = subprocess.Popen(
file_cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True,
executable=backup_opt_dict.bash_path)
(file_out, file_err) = file_process.communicate()
file_match = re.search(r'(\S+?) filesystem data', file_out, re.I)
if file_match is None:
logging.critical('[*] File system type not guessable: {0}\
'.format(file_err))
raise Exception
else:
filesys_type = file_match.group(1)
logging.info('[*] File system {0} found for volume {1}'.format(
filesys_type, vol_name))
return filesys_type.lower().strip()
raise Exception
def check_backup_existance(backup_opt_dict):
'''
Check if any backup is already available on Swift.
The verification is done by backup_name, which needs to be unique
in Swift. This function will return an empty dict if no backup are
found or the Manifest metadata if the backup_name is available
'''
if not backup_opt_dict.backup_name or not backup_opt_dict.container or \
not backup_opt_dict.remote_obj_list:
logging.warning("[*] A valid Swift container,\
or backup name or container content not available. \
Level 0 backup is being executed ")
return dict()
logging.info("[*] Retreiving backup name {0} on container \
{1}".format(
backup_opt_dict.backup_name.lower(), backup_opt_dict.container))
backup_opt_dict = get_match_backup(backup_opt_dict)
backup_opt_dict = get_newest_backup(backup_opt_dict)
if backup_opt_dict.remote_newest_backup:
sw_connector = backup_opt_dict.sw_connector
logging.info("[*] Backup {0} found!".format(
backup_opt_dict.backup_name))
backup_match = sw_connector.head_object(
backup_opt_dict.container, backup_opt_dict.remote_newest_backup)
return backup_match
else:
logging.warning("[*] No such backup {0} available... Executing \
level 0 backup".format(backup_opt_dict.backup_name))
return dict()
def add_host_name_ts_level(backup_opt_dict, time_stamp=int(time.time())):
'''
Create the object name as:
hostname_backupname_timestamp_backup_level
'''
if backup_opt_dict.backup_name is False:
logging.critical('[*] Error: Please specify the backup name with\
--backup-name option')
raise Exception
backup_name = u'{0}_{1}_{2}_{3}'.format(
backup_opt_dict.hostname,
backup_opt_dict.backup_name,
time_stamp, backup_opt_dict.curr_backup_level)
return backup_name

7
requirements.txt Normal file
View File

@ -0,0 +1,7 @@
python-swiftclient>=2.0.3
python-keystoneclient>=0.8.0
argparse>=1.2.1
docutils>=0.8.1
[testing]
pytest

79
setup.py Normal file
View File

@ -0,0 +1,79 @@
'''
Freezer Setup Python file.
'''
from setuptools import setup, find_packages
from setuptools.command.test import test as TestCommand
import freezer
import os
here = os.path.abspath(os.path.dirname(__file__))
class PyTest(TestCommand):
def finalize_options(self):
TestCommand.finalize_options(self)
self.test_args = []
self.test_suite = True
def run_tests(self):
import pytest
errcode = pytest.main(self.test_args)
sys.exit(errcode)
def read(*filenames, **kwargs):
encoding = kwargs.get('encoding', 'utf-8')
sep = kwargs.get('sep', '\n')
buf = []
for filename in filenames:
with io.open(filename, encoding=encoding) as f:
buf.append(f.read())
return sep.join(buf)
setup(
name='freezer',
version='1.0.8',
url='http://sourceforge.net/projects/openstack-freezer/',
license='Apache Software License',
author='Fausto Marzi, Ryszard Chojnacki, Emil Dimitrov',
author_email='fausto.marzi@hp.com, ryszard@hp.com, edimitrov@hp.com',
maintainer='Fausto Marzi, Ryszard Chojnacki, Emil Dimitrov',
maintainer_email='fausto.marzi@hp.com, ryszard@hp.com, edimitrov@hp.com',
tests_require=['pytest'],
description='''OpenStack incremental backup and restore automation tool for file system, MongoDB, MySQL. LVM snapshot and encryption support''',
long_description=open('README.rst').read(),
keywords="OpenStack Swift backup restore mongodb mysql lvm snapshot",
packages=find_packages(),
platforms='Linux, *BSD, OSX',
test_suite='freezer.test.test_freezer',
cmdclass={'test': PyTest},
scripts=['bin/freezerc'],
classifiers=[
'Programming Language :: Python',
'Development Status :: 5 - Production/Stable',
'Natural Language :: English',
'Environment :: OpenStack',
'Intended Audience :: Developers',
'Intended Audience :: Financial and Insurance Industry',
'Intended Audience :: Information Technology',
'Intended Audience :: System Administrators',
'Intended Audience :: Telecommunications Industry',
'License :: OSI Approved :: Apache Software License',
'Operating System :: MacOS',
'Operating System :: POSIX :: BSD :: FreeBSD',
'Operating System :: POSIX :: BSD :: NetBSD',
'Operating System :: POSIX :: BSD :: OpenBSD',
'Operating System :: POSIX :: Linux',
'Operating System :: Unix',
'Topic :: System :: Archiving :: Backup',
'Topic :: System :: Archiving :: Compression',
'Topic :: System :: Archiving',
],
install_requires=[
'python-swiftclient>=2.0.3',
'python-keystoneclient>=0.8.0',
'argparse>=1.2.1'],
extras_require={
'testing': ['pytest'],
}
)