Commit Graph

75 Commits (master)

Author SHA1 Message Date
caihui 02e435f0ed Remove six usage from freezer package
We don't support Python 2 anymore so we don't need this
compatibility library.

This patch remove six usage.

Change-Id: I661e9cdaf33e89bdb905d1f34a001f3853c4c366
3 years ago
Michal Arbet 89558a7202 Fix python3 compatibility
Change-Id: I01a142cab62253719f2534b0eda8e5954db76730
3 years ago
zhulingjie 63b5e271e4 Update json module to jsonutils
json is deprecated, should use oslo_serialization.jsonutils

Change-Id: Ib4b562c3f37a5c133f3a6ae8efa00a26c8b41846
4 years ago
zhufl 8d2cbeff00 Add missing ws seperator between words
This is to add missing ws seperator between words.

Change-Id: I55731b70dec4ecb19bf1f3ac2307091ba6a625ba
4 years ago
lu-meihong a41e228a1a file.close() isn't required because "
with open( as f:" is used before
Change-Id: I1168d039e17133f553dd36b424fdeb40fc1631ba
4 years ago
gengchc2 931e2a6926 Modify a few exceptions in
Change-Id: Iaafb29e1065b1e82e1c397c64a5aa2198408a93a
5 years ago
gengchc2 d59b138182 Modify a few bugs for supporting ftp/ftps
Change-Id: Ia0cd1a3561097f543cab932bf34cd8161754a0d4
5 years ago
gengchc2 d3854dde75 Add ftps subclass of BaseFtpStorage
The patch add ftps subclass of BaseFtpStorage.
Story: #2004332
Task: #27917

Change-Id: I36dc6286ec0b8dd5bd5663d974e48942536822ff
5 years ago
gecong1973 4d9cafb3fd Add ftp subclass of BaseFtpStorage
The patch add ftp subclass of BaseFtpStorage

Change-Id: Iec8bf1bb4e52e7320ae967c428fbb69bc607f4cd
5 years ago
gecong1973 75ca834282 Add BaseFtpStorage class fot freezer
The patch add BaseFtpStorage class fot freezer

Story: #2004332
Task: #27917
Change-Id: I5501d41dfcb19f9dac32e54770b5df5a9c14b3ab
5 years ago
gengchc2 93dbf58160 Add ftp support framework in freezer
The patch adds ftp support framework in freezer.


Change-Id: I68553e1b0d98997af4bff4ce57da81b1fb835491
Story: #2004332
Task: #27917
5 years ago
gengchc2 b876cbc11b Add sftp support with password for freezer
Add ftp support for freezer.


Change-Id: I21bc8e9d2398a7ed87bcc5d08954933b0dd123b0
Story: #2004332
Task: #27917
5 years ago
Chen 60c0fc6ffe Implement create_dirs and put_metadata methods for multiple storage
Change-Id: I59afa894fa435779fc3fed21e2aef45932af3618
Closes-Bug: #1593257
5 years ago
David Rabel 9f8d6c382e Fix license boilerplates
license boilerplates as docstring are included in automatically
generated documentation which leads to wrong formatting.

For example the indented hyperlink to the license is formatted as

Change-Id: I9d6088ab48d6787f29b593bfa5445b6a84e99de7
5 years ago
Zuul 12c9db8a3c Merge "Fix the path of RichQueue" 5 years ago
Zuul 4025729a86 Merge "Move the __init__ method to top to improve readability" 5 years ago
wu.chunyang 348fe7ea82 Fix the path of RichQueue
Change-Id: I448453e1d6d6459c81cb63f4e7a85994204f7a73
5 years ago
caoyuan 0eb17ac536 Move the __init__ method to top to improve readability
Change-Id: I0c6339161e2abeaa11479e70333fc17ac2d9aae1
5 years ago
caoyuan 0e45bc6daa Add a ssh key file checking
ssh is a storage used for linux destro, and the key file is
usually in '~/.ssh/' folder, default value is needed,
if the file is not generate by manually,
the freezer-agent will broken. Add checking is a
good idea for use. this patch is to:

1. add default ssh keyfile value
2. add ssh keyfile checking

Change-Id: Ic2d3cdd6f9d4c260cbf650f7aa532d5060778693
Closes-Bug: #1750566
5 years ago
Andreas Jaeger 037cf3fa81 Fix doc build warnings
Enable treading warnings as errors so that no new broken stuff gets
merged. Remove obsolete way of doing it, it's a nop.

Fix bugs found:
* Add missing sphinx modules to properly generate content.
* Misformatting of comments and files
* Do not index the windows modules, they import modules that do not
  exist on Linux.
* Place api docs in reference/api according to
  and reference it. Update .gitignore for this.
* Include missing files in tree.

Change-Id: I57d3124ac9571f189cd0e10e4cf7e805b8e09045
6 years ago
Pengju Jiao 86e04c00db Fix creating swift container error
Specify the swift storage as 'abc/123', it will
fail when 'abc' was not exist. This patch will fix it.

Change-Id: Id11d632741bbfc8da8ccb6180374fb6da59d32da
Closes-Bug: #1688442
6 years ago
shangxiaobj 248f73d4ea [Trivialfix]Fix typos in freezer
Fix the typos in project openstack-freezer

Change-Id: I84700c96ac19f14b786ffa10f79195b78ea2b366
6 years ago
Jenkins 303172d85a Merge "Increase AdminJob support for cinder mode backup" 6 years ago
Pengju Jiao 7378b6c177 Fix listing objects in s3 driver
When object count is bigger than 1000, IsTruncated is True, then
the restore will fail. (Although object number of the cinder mode
backup of a volume will never be bigger than 1000 with using
multipart upload in s3 driver, this should also be fixed)

Depends-On: I7586763cf25be7ca115c7325129a7c9aa7022603
Change-Id: I9e972023817f3c804ae02e925b93811e1e05cc52
6 years ago
Pengju Jiao df1119174d Increase AdminJob support for cinder mode backup
Currently, when use 'freezer-agent --action admin --remove-older-than
xx --mode cinder --cinder-vol-id xx --storage xx --container xx' to
remove the old backups, the result is successful, but the backup data
is not really deleted. This path will increase AdminJob support for
cinder mode backup and fix the issue.

Change-Id: I8c9bb86a6d6d82f0609f340df42e220204794940
Closes-Bug: #1701422
6 years ago
Pengju Jiao c4d1afac07 Increase s3 driver support
When use s3 driver, you should specify storage type with
'--storage s3', specify the auth info like '--access-key xxx'
and '--secret-key xxx', and specify the S3 compatible storage
endpoint with '--endpoint  http://ip:port'.

Change-Id: If35468b3bd3a156ac5ade60c771ec8ff1a320139
Implements: blueprint s3-driver-support
6 years ago
Saad Zaher f609f3b9ac Fix pylint checks
Change-Id: I784ee29952fc4cc1f605b60023a0c7ad30aef2e2
Partial-Bug: #1696455
6 years ago
Pengju Jiao 76cca40b29 Update 'admin action' of nova based backups
There are two issues in current 'admin action' of nova
based backups.

1. When remove old nova backups with command 'freezer-agent
--action admin --mode nova --engine nova --nova-inst-id xxxx
--no-incremental true --remove-older-than x" will fail cause
the incorrect swift path.
2. Can not do 'admin action' of nova tenant.

This patch contains two changes.
1. Fix remove old nova backups error.
2. Add option to do 'admin action' of nova tenant.

Change-Id: I60b3445905fab65725882fba9b0ca6b1e2abc408
Closes-Bug: #1684526
6 years ago
Mohammed Naser 4c83a3f7c3
Force Swift storage driver to always get a new client
At the moment, the Swift storage codebase does not have a consistent
way of using the Swift client, as it switches from grabbing an
existing client and creating new ones.

Due to the architecture of how restores happen, there are several
times where the same storage instance is held by the main process,
the process which reads from the storage and the engine (to retrieve
metadata).  All of these accesses result in unexpected behaviour which

The overall cost of creating a new client for every request is very
minimal and it eliminates all possibility of multithreading issues
that occur as per suggested by the author of the requests Python
library[0], which is used by the Swift client.

This patch eliminates all calls to create_swift() indirectly and makes
sure that all accesses to the Swift client create a new instance. In
my local testing, there was no noticable performance change but
no more SSL failures.

Closes-Bug: #1682606


Change-Id: I2d43f14943fc52fd6b6cb994251c97e00d3eb863
6 years ago
Gleb Stepanov b0c64f78cd Backup/Restore nova tenant
Implement nova tenant backup and restore to swift, local and ssh storage.
Add unit tests. Add new CLI option '--project-id' to perform tenant
backup/restore. Add new option - '--backup-media' as well.

Partially implements bp: tenant-backup

Change-Id: Ia8e4412dfaf0c24364db692de78c27cd0e3b3c4c
6 years ago
Pierre-Arthur MATHIEU 3e72f7d1aa Fix some issues with tempest test when not running with devstack
Change-Id: I51cc7867a60357d8e1c809d1ea985b0624546ebb
6 years ago
Jenkins 61975f593e Merge "Tune logging level for swift chunks" 6 years ago
Saad Zaher cca78b6ba0 Fix ssh connection while restore
ssh connection gets lost while restoring, so we check if the
connection is not  alive we reconnect again.

Change-Id: I2f6bce2555d89ac103dd77248013ddbdfa862bc4
6 years ago
Pierre-Arthur MATHIEU a1d56510b8 Tune logging level for swift chunks
Change-Id: Ifca7686a37b45ae9d34ca8b2417a5258e5e14cdc
6 years ago
Saad Zaher baa7942549 Added PrettyPrint for action: Info
Change-Id: Ia2fd5d2a9ec2ab93fe710384704f009e29339596
6 years ago
Ruslan Aliev 0df317c94b Block based incremental support - rsync
Added new engine type (rsync) to perform
space/bandwidth efficient backups.

Change-Id: I8390c9c85fc2478a4ad2fe7eb4e40f3e580da912
Signed-off-by: Ruslan Aliev <>
6 years ago
Dmitry Stepanenko 29f12645b4 Backup/restore cinder/nova to ssh
This patch implements ability to backup/restore
to remote ssh host

freezer-agent --storage ssh --ssh-key <pub_key> --ssh-username <user>
--ssh-host <host> --container <path-to-remote-backup> --mode cinder
--cinder-vol-id <cinder-id>

freezer-agent --action restore --storage ssh --ssh-key <pub_key>
--ssh-username <user> --ssh-host <host> --container
<path-to-remote-backup> --mode cinder --cinder-vol-id <cinder-id>

Change-Id: If79c3473646e32bcda14a83db53fe95479fffff4
7 years ago
yangyapeng eee345b4c6 Backup nova/cinder to local
Freezer-agent backup volume_by_glance upload data to swift by default.
This patch backup it and metadata to local,Now.

backup nova / cinder by glance to local:
freezer-agent --cinder-vol-id 0bd0d291-e75c-4fd9-b13b-058c335a8497
--storage local --container /home/nfs --mode cinder


freezer-agent --action restore
--cinder-vol-id 0bd0d291-e75c-4fd9-b13b-058c335a8497
--storage local --container /home/nfs
--restore-from-date 2016-05-10T00:01:00

Implements blueprint: backup-nova-cinder-to-non-swift

Change-Id: I9bc8496fea7727edbf35130b7dbf9bb435d7038b
7 years ago
Shangzhong Zhu 16e68e351a Select the most recent backup previous to the date
When restore data with --restore-from-date option from local storage,
The most recent backup subsequent to the specified date will be

According to the detailed desciptions from README.rst, the
--restore-from-date option will select the most recent backup previous
to the specified date.

The patch fixed the issue.

Change-Id: Iaac936c845ab5ccddd59c751a773e6f190dc064a
Closes-Bug: #1644478
Signed-off-by: Shangzhong Zhu <>
7 years ago
Jenkins a4dc47c413 Merge "Correct freezer reraising of exception" 7 years ago
Saad Zaher 2a871b0690 Fix upload Engine metafile to swift.
We need to pass the content length to swift in case of uploading
files, so we needed to calcuate the engine's metadata file size
and pass it to swift during upload.

Change-Id: I404401d0046c5aaff463043b82d40b48708b7149
7 years ago
Jenkins b80749a684 Merge "Fix manifest upload with ceph backend" 7 years ago
Jenkins 0b05f9859d Merge "failed to backup data to ssh storage" 7 years ago
Pierre-Arthur MATHIEU 7498a69821 Fix manifest upload with ceph backend
Change-Id: I937af1d3e30c0034680bfdc52cf2842824f88ac5
7 years ago
Jenkins f320b35137 Merge "Fix removing temp file" 7 years ago
Shangzhong Zhu 7cf0799408 failed to backup data to ssh storage
When backup data to ssh storage for the first time, because the sub
directory "metadata" isn't exist in the ssh storage directory, the
exception IOError will be raised.

The exception IOError should be caught in the funtion listdir() of the
SshStorage class, and the empty list should be returned when the errno
is ENOENT which indicates "No such file or directory". The other
exceptions, e.g. "Permission denied", should be raised.

Change-Id: Icddca7078a47fb82c12d4232065d51e2baa0ab60
Closes-Bug: #1638816
7 years ago
Cao Xuan Hoang 61ee6a331a Clean imports in code
This patch set modifies lines which are importing objects
instead of modules. As per openstack import guide lines, user should
import modules in a file not objects.
Also, imports was regrouped and placed in alphabetical order.

Change-Id: I192b8bf6f50a85e19640975f0902a014c4ba0878
Signed-off-by: Ruslan Aliev <>
7 years ago
Ruslan Aliev afca2bd2e4 Fix removing temp file
There is no need to call os.remove() with temp file,
it will be removed automatically after closed.
Actually, freezer.log has warning messages about it:
WARNING freezer.utils.utils [-] Error deleting file /tmp/tmpSNedJh
This patch will fix it.
Also, added newline to info message that using scheduler is deprecated
and more verbose output if there is a problem with deleting file.

Change-Id: I8dc31684bfece130eed6dd89806e965a4e1eabea
Signed-off-by: Ruslan Aliev <>
7 years ago
Saad Zaher 7658243238 Fixed nova backup and restore
Change-Id: Iea7f5fb0d0fb42d95e6c24d6d63693e272af3992
Closes-Bug: #1615461
7 years ago
gengchc2 f7d436ed9c Correct freezer reraising of exception
When an exception was caught and rethrown, it should call 'raise'
without any arguments because it shows the place where an exception
occured initially instead of place where the exception re-raised.

Change-Id: Ic071e958694166a7df87915f50fe0f4a70f01fab
7 years ago