Commit Graph

49 Commits (master)

Author SHA1 Message Date
ljhuang 3983e13908 Replace deprecated readfp method with read_file
The readfp method has been deprecated since version 3.2 [1].


Change-Id: I5246aef8e07b949273b6c3cc31435b4756ae9d1a
10 months ago
Caihui 7244bef734 Update to ReSizeSteam__next__()
1. The bug.
    Traceback (most recent call last):

      File "/opt/stack/freezer/freezer/tests/unit/openstack/", line 37, in test_backup_cinder_by_glance

      File "/opt/stack/freezer/freezer/openstack/", line 79, in backup_cinder_by_glance, package, headers=headers)

      File "/opt/stack/freezer/freezer/storage/", line 163, in add_stream
    for el in stream:

    TypeError: iter() returned non-iterator of type 'ReSizeStream'
2. Fix
In python3.x , a iterator object should be difined two funs " __iter__()" and "__next__()".

Update to ReSizeSteam.__next__().

Change-Id: I940841b94789d024d3b9447f96158963fe3e16ee
3 years ago
caihui 02e435f0ed Remove six usage from freezer package
We don't support Python 2 anymore so we don't need this
compatibility library.

This patch remove six usage.

Change-Id: I661e9cdaf33e89bdb905d1f34a001f3853c4c366
3 years ago
Michal Arbet bcaac1e3c1 Fix parsing config file for freezer job
Config parsers do not guess datatypes of values
in configuration files, always storing them internally
as strings. This means that we need to convert booleans, as
they are parsed as Strings from config file.

This patch is fixing typo from my previous commit where
None is needed to remove from second if clause as it is
already in first if clause and option_value should be False.

Change-Id: Id60e75b5f999b70626addaf47b4abd1ad15555d6
3 years ago
Michal Arbet 89558a7202 Fix python3 compatibility
Change-Id: I01a142cab62253719f2534b0eda8e5954db76730
3 years ago
zhulingjie 63b5e271e4 Update json module to jsonutils
json is deprecated, should use oslo_serialization.jsonutils

Change-Id: Ib4b562c3f37a5c133f3a6ae8efa00a26c8b41846
4 years ago
lu-meihong f8f1b59b7b The script name displayed is separated by spaces when the exec job is abnormal;
such as,when i execute 'freezer-agent --action exec --command /home/',
reported error:while executing script / h o m e / b a c k u p . s h return code was 1 instead of 0

Change-Id: Iea2f04af8672b3abd4f684b7cb5120a9e1b77cf9
4 years ago
Matthew Thode 7c16877195
switch to cryptography
Fernet could not be used as it is not a streaming cipher.
CFB mode does not require padding, so removed padding code.

Change-Id: I14f58d9d473c20ce7863baa0b2adadbfda268b7a
5 years ago
David Rabel 9f8d6c382e Fix license boilerplates
license boilerplates as docstring are included in automatically
generated documentation which leads to wrong formatting.

For example the indented hyperlink to the license is formatted as

Change-Id: I9d6088ab48d6787f29b593bfa5445b6a84e99de7
5 years ago
Pengju Jiao acd734bd51 [Trivialfix] Adjust the log level
Now freezer-agent log the transmitted info in freezer.log
This patch will adjust it to debug level

Change-Id: I7c9c4f5953ebbd7b457f82cfd4d66b4d8200682c
Closes-Bug: #1716548
6 years ago
Jenkins 419f1876bc Merge "Rsync V2 engine" 6 years ago
Ruslan Aliev b2fe5a8e70 Rsync V2 engine
There were fixed some bugs, improved performance and
reduced amount of code.
Also --rsync-block-size parameter was added.

Change-Id: I8201ddedbaa5950f9caeb7b10bd36ecce4d0eb61
Signed-off-by: Ruslan Aliev <>
6 years ago
Pengju Jiao c4d1afac07 Increase s3 driver support
When use s3 driver, you should specify storage type with
'--storage s3', specify the auth info like '--access-key xxx'
and '--secret-key xxx', and specify the S3 compatible storage
endpoint with '--endpoint  http://ip:port'.

Change-Id: If35468b3bd3a156ac5ade60c771ec8ff1a320139
Implements: blueprint s3-driver-support
6 years ago
DeepaJon 77be8fa29c Replace six.iteritems() with .items()
1.As mentioned in [1], we should avoid using six.iteritems to
achieve iterators. We can use dict.items instead, as it will
return iterators in PY3 as well. And dict.items/keys will more
2.In py2, the performance about list should be negligible,
see the link [2].

Change-Id: I522e8f06e8a31b0d591db825d0c1608f162cbbc6
6 years ago
Julia Varlamova 08b04e1c5d Fix ReSizeStream
Fix issue in 'next' method of ReSizeStream class. If StopIteration
error is rasen while doing "self.reminder +=", it
wasn't processed in 'except StopIteration' block.

We need to fix this as it leads to error when doing backup of small images,
which size is less than ReSizeStream chunk size.

Change-Id: I21dfba3967c6a582b273f0e342ca435b14db0a0a
6 years ago
Saad Zaher 6b71759ab1 Added Nova Engine to freezer
A complete engine to backup/restore nova instances through freezer.
This engine is able backup/restore signle server for now with
limited features. It can be used to backup to local/ssh/swift or any
storage driver available.

Change-Id: I317fc87d6f1f674f0eb7244e4b2a5d7a95743a5f
6 years ago
Jenkins 1a58dd23f5 Merge "Add flexible wait mechanism for events" 6 years ago
Saad Zaher 97b5c35f01 Fix the fetch of the stream generator
glanceclient is changed so we need to support different styles of
fetching the stream generator. either next(obj) or

Change-Id: I88b3485917649c06f724cdda288f91ab2e41ad66
6 years ago
Gleb Stepanov 43d631c616 Add flexible wait mechanism for events
As far as nova is not hard-realtime system it is
we should not depend on particular sleep time waiting
for some event.

Change-Id: I4271acbefbd735bb943263d664c5d68853ade0bc
6 years ago
Ruslan Aliev 0df317c94b Block based incremental support - rsync
Added new engine type (rsync) to perform
space/bandwidth efficient backups.

Change-Id: I8390c9c85fc2478a4ad2fe7eb4e40f3e580da912
Signed-off-by: Ruslan Aliev <>
6 years ago
Jenkins a4dc47c413 Merge "Correct freezer reraising of exception" 7 years ago
Ruslan Aliev 16b64376d7 Fix print warning when scheduler starts
Messages about creating new directory or finding already created
directory should not be treated as "WARNING", it's normal behavior.
Also, there is no need to print this messages every execution
of scheduler.

Change-Id: I31dc9acc06b0b1556b265da9fef4ef2a74c20e06
Closes-Bug: #1646851
Signed-off-by: Ruslan Aliev <>
7 years ago
Jenkins f320b35137 Merge "Fix removing temp file" 7 years ago
Jenkins 334caee452 Merge "Use ConfigParser instead of SafeConfigParser in Python 3" 7 years ago
Cao Xuan Hoang 61ee6a331a Clean imports in code
This patch set modifies lines which are importing objects
instead of modules. As per openstack import guide lines, user should
import modules in a file not objects.
Also, imports was regrouped and placed in alphabetical order.

Change-Id: I192b8bf6f50a85e19640975f0902a014c4ba0878
Signed-off-by: Ruslan Aliev <>
7 years ago
Ruslan Aliev afca2bd2e4 Fix removing temp file
There is no need to call os.remove() with temp file,
it will be removed automatically after closed.
Actually, freezer.log has warning messages about it:
WARNING freezer.utils.utils [-] Error deleting file /tmp/tmpSNedJh
This patch will fix it.
Also, added newline to info message that using scheduler is deprecated
and more verbose output if there is a problem with deleting file.

Change-Id: I8dc31684bfece130eed6dd89806e965a4e1eabea
Signed-off-by: Ruslan Aliev <>
7 years ago
Nguyen Hung Phuong 7cb4765e74 TrivialFix: Remove cfg import unused
This patch removes cfg import unused in


Change-Id: I8177734bf37107144a4f4edf5bd615dda06196bb
7 years ago
Pierre-Arthur MATHIEU c385a7953a Remove deprecated code
And deprecates freezer-scheduler cli, use python-freezerclient

Change-Id: I2960b1f21758a47e11a9df1723a962b745474bd8
7 years ago
Pierre-Arthur MATHIEU 95278ac06c Improve abort mechanism to support same functionality as stable/mitaka
Closes-bug: 1615662
Change-Id: Ifb5f312d4718329c1afad6c384fba1854e2b10ba
7 years ago
gecong1973 7a3a2cc478 Use ConfigParser instead of SafeConfigParser in Python 3
The SafeConfigParser class has been renamed to ConfigParser in
Python 3.2 [1]. The alias SafeConfigParser maybe removed in
future versions of Python 3. Use ConfigParser instead of
SafeConfigParser in Python 3

Change-Id: I1354b9fde203c9478bd77b71ec74a69304428385
7 years ago
Memo Garcia c4ea71a4d0 Enable parallel job execution for the scheduler to allow abort signals
Now the scheduler uses two pools of resources to handle execution of jobs.
The first one will handle the polling of jobs from the api.
The second one will handle the job exection. This way we avoid blocking the
scheduler while running jobs.

Abort a running job is now enabled thanks to the parallel execution.

Depends-on: I9e92ccdd7c592a8109fe8398db6d53c9c42dc10a
Closes-bug: 1615662
Change-Id: I48153dcb521b6a8f0138d5ef81239cf4cded4df0
7 years ago
gengchc2 f7d436ed9c Correct freezer reraising of exception
When an exception was caught and rethrown, it should call 'raise'
without any arguments because it shows the place where an exception
occured initially instead of place where the exception re-raised.

Change-Id: Ic071e958694166a7df87915f50fe0f4a70f01fab
7 years ago
Saad Zaher 4277d4a4f3 Added Validation layer for freezer jobs
validating arguments before starting freezer job. each job will
validate job-specific arguments and the _general_validate method
in base Job class will takecare of the common ones.

Implements: blueprint validation-layer-for-freezer-jobs
Change-Id: Ia964bcc72abe39408dd72338951e083926a130ca
7 years ago
Saad Zaher 9c94d30334 Freezer metadata part2
Change-Id: I9a6febb0ce427630c3e6602047613d6804b360dd
Implements-blueprint: freezer-metadata
7 years ago
Jenkins 574326c020 Merge "Correct reraising of exception" 7 years ago
Domhnall Walsh 98c93e68a5 Fix checksum generation so that it handles binary data
At the moment, if freezer attempts to perform a backup with
the consistency check option enabled, it will fail if the
files being backed up contain anything other than plain
ASCII (or at least some flavour of same, ISO-8859-x
etc.). This is because the hashlib module fails as it
expects a string to digest rather than a binary blob.

This should fix matters by escaping everything that might
cause an issue and works for Python 2 and 3. Because it
escapes strings in a different way to before, the
checksums it computes may be different to previously
in some cases, but at least now it should always produce

Change-Id: Idf62dc9e398c396c9de19cbff5622e79d4c7a198
Closes-Bug: #1595732
7 years ago
yuyafei 43f99362c0 Correct reraising of exception
When an exception was caught and rethrown, it should call 'raise'
without any arguments because it shows the place where an exception
occured initially instead of place where the exception re-raised.

Change-Id: I69e007566bf26fa2d4b6bcd1f996f4bba9f49049
7 years ago
Deklan Dieterly 00db2b0d95 Fix Freezer Agent Logging
Standardize on oslo.log.
Log exception stack traces where appropriate.
Add logging to all files.
Remove '[*]' from log messages.
Log command line arguments at Freezer Agent start
so that errors can be traced to specific invocations.

Leave scheduler module mostly unchanged because it
is doing python logging configuration.

Change-Id: I23de0558409e63978303963d592a4e5ee4dee2b5
7 years ago
Deklan Dieterly 196a93e1af Fix cStringIO usage
Change cStringIO.StringIO() to cStringIO().

Change-Id: Ic72fd2d650bdd656d8b0145f1c347f5b79e1d1ec
Closes-Bug: #1596074
7 years ago
Deklan Dieterly 694c98e063 Fix tox.ini file
Remove select line in tox.ini.
Currently checks are not being run.
Run pep8 checks.
Fix code to pass pep8 checks.

Change-Id: Iacef4386b1214f95b1bd106c17a5f714f948a1a1
Closes-Bug: #1593452
7 years ago
Deklan Dieterly 3cdfd8fba3 Fix exception propagation
Make exception propagation from subprocesses and
threads work correctly.
Make exceptions propagate to top level program.
Make exceptions in subprocesses and threads log correctly.
Fix Tempest README.rst

Change-Id: If9ebe36afdc02c2460a86da7b4d9fa15e12c5824
Closes-Bug: #1592589
7 years ago
Cynthia Lopes do Sacramento 1bf965888f Adds checksum before backup and after restore for consistency check.
In addition, implements global file walk function on utils module

Also adds unit tests for checksum module

Implements blueprint: backup-consistency

Change-Id: I3ab23c2dfaacbaf4f0b293afa0a204a76e928b04
Depends-On: I294d7d3ad023c38e0639baa5934731bb46e875a2
7 years ago
Saad Zaher 1b83ba7aef Refactor Freezer authentication with keystone
Supports v2 and v3 of keystone and using keystoneauth library for
authentication with sessions. Supports new OS environment variables

Change-Id: I5cf8a5460b81c071f5982c1a7b7750d1c861a86f
Implements: blueprint refactor-os-clients
7 years ago
Jenkins 06b7ae0ee5 Merge "Allow freezer to use trickle installed system wide" 7 years ago
Jenkins 550d088f24 Merge "Fix typos" 7 years ago
Saad Zaher de0344b4c8 Allow freezer to use trickle installed system wide
Freezer is using trickle only if it's installed system wide and not
looking for library anymore. Users have to
install trickle: apt-get install trickle or yum install trickle
for freezer to use it.

Change-Id: I49efff29876990dfd0782e3cf5ad68daf3e6055d
Closes-Bug: #1553159
7 years ago
eldar nugaev 9efa12fea6 Added --overwrite parameter for removing all files from restore directory.
Default value --overwrite=False

Change-Id: I5c1aea7000c888b966c2b6e57d22744e35fbdfae
Closes-bug: #1544713
7 years ago
daiki kato ecfa4f6f13 Fix typos
This patch fixes several typos.


Change-Id: Id5d5205f9234b1c8c8df6e865062261fcf960e8c
7 years ago
eldar nugaev c01d7c0a72 Refactoring of code:
1) new snapshot module with code vss,
lvm and part of -
create_snapshot and remove_snapshot

This module should contain all code related shadow,
vss and any new snapshots.

2) new openstack module with code from osclient,
restore and backup (BackupOS)

This module should contain code related to cinder/nova backups

3) new utils module with code from utils, win_utils,
bandwidth, config, exec_cmd, validator, streaming

This module should containd code related to anything
that cannot be extracted to its own module

4) new test modules - snapshot, utils, openstack and apiclient.

Test scheduler daemon is now located in scheduler module.

5) is deleted and code placed to
openstack.backup (BackupOS) and to job.BackupJob

Change-Id: I46194b40b3a65c15b6095cfb7388dcfd30d95997
7 years ago