1. The bug.
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/opt/stack/freezer/freezer/tests/unit/openstack/test_backup.py", line 37, in test_backup_cinder_by_glance
self.bakup_os.backup_cinder_by_glance(35)
File "/opt/stack/freezer/freezer/openstack/backup.py", line 79, in backup_cinder_by_glance
self.storage.add_stream(stream, package, headers=headers)
File "/opt/stack/freezer/freezer/storage/swift.py", line 163, in add_stream
for el in stream:
TypeError: iter() returned non-iterator of type 'ReSizeStream'
2. Fix
In python3.x , a iterator object should be difined two funs " __iter__()" and "__next__()".
Update ReSizeSteam.next() to ReSizeSteam.__next__().
Change-Id: I940841b94789d024d3b9447f96158963fe3e16ee
We don't support Python 2 anymore so we don't need this
compatibility library.
This patch remove six usage.
Change-Id: I661e9cdaf33e89bdb905d1f34a001f3853c4c366
Config parsers do not guess datatypes of values
in configuration files, always storing them internally
as strings. This means that we need to convert booleans, as
they are parsed as Strings from config file.
This patch is fixing typo from my previous commit where
None is needed to remove from second if clause as it is
already in first if clause and option_value should be False.
Change-Id: Id60e75b5f999b70626addaf47b4abd1ad15555d6
such as,when i execute 'freezer-agent --action exec --command /home/backup.sh',
reported error:while executing script / h o m e / b a c k u p . s h return code was 1 instead of 0
Change-Id: Iea2f04af8672b3abd4f684b7cb5120a9e1b77cf9
Fernet could not be used as it is not a streaming cipher.
CFB mode does not require padding, so removed padding code.
Depends-On: https://review.openstack.org/c/575199/
Change-Id: I14f58d9d473c20ce7863baa0b2adadbfda268b7a
license boilerplates as docstring are included in automatically
generated documentation which leads to wrong formatting.
For example the indented hyperlink to the license is formatted as
quotation.
Change-Id: I9d6088ab48d6787f29b593bfa5445b6a84e99de7
Now freezer-agent log the transmitted info in freezer.log
This patch will adjust it to debug level
Change-Id: I7c9c4f5953ebbd7b457f82cfd4d66b4d8200682c
Closes-Bug: #1716548
There were fixed some bugs, improved performance and
reduced amount of code.
Also --rsync-block-size parameter was added.
Change-Id: I8201ddedbaa5950f9caeb7b10bd36ecce4d0eb61
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
When use s3 driver, you should specify storage type with
'--storage s3', specify the auth info like '--access-key xxx'
and '--secret-key xxx', and specify the S3 compatible storage
endpoint with '--endpoint http://ip:port'.
Change-Id: If35468b3bd3a156ac5ade60c771ec8ff1a320139
Implements: blueprint s3-driver-support
1.As mentioned in [1], we should avoid using six.iteritems to
achieve iterators. We can use dict.items instead, as it will
return iterators in PY3 as well. And dict.items/keys will more
readable.
2.In py2, the performance about list should be negligible,
see the link [2].
[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Change-Id: I522e8f06e8a31b0d591db825d0c1608f162cbbc6
Fix issue in 'next' method of ReSizeStream class. If StopIteration
error is rasen while doing "self.reminder += self.stream.next()", it
wasn't processed in 'except StopIteration' block.
We need to fix this as it leads to error when doing backup of small images,
which size is less than ReSizeStream chunk size.
Change-Id: I21dfba3967c6a582b273f0e342ca435b14db0a0a
A complete engine to backup/restore nova instances through freezer.
This engine is able backup/restore signle server for now with
limited features. It can be used to backup to local/ssh/swift or any
storage driver available.
Change-Id: I317fc87d6f1f674f0eb7244e4b2a5d7a95743a5f
glanceclient is changed so we need to support different styles of
fetching the stream generator. either next(obj) or obj.next()
Change-Id: I88b3485917649c06f724cdda288f91ab2e41ad66
As far as nova is not hard-realtime system it is
we should not depend on particular sleep time waiting
for some event.
Change-Id: I4271acbefbd735bb943263d664c5d68853ade0bc
Added new engine type (rsync) to perform
space/bandwidth efficient backups.
Change-Id: I8390c9c85fc2478a4ad2fe7eb4e40f3e580da912
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Messages about creating new directory or finding already created
directory should not be treated as "WARNING", it's normal behavior.
Also, there is no need to print this messages every execution
of scheduler.
Change-Id: I31dc9acc06b0b1556b265da9fef4ef2a74c20e06
Closes-Bug: #1646851
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
This patch set modifies lines which are importing objects
instead of modules. As per openstack import guide lines, user should
import modules in a file not objects.
Also, imports was regrouped and placed in alphabetical order.
http://docs.openstack.org/developer/hacking/#imports
Change-Id: I192b8bf6f50a85e19640975f0902a014c4ba0878
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
There is no need to call os.remove() with temp file,
it will be removed automatically after closed.
Actually, freezer.log has warning messages about it:
WARNING freezer.utils.utils [-] Error deleting file /tmp/tmpSNedJh
This patch will fix it.
Also, added newline to info message that using scheduler is deprecated
and more verbose output if there is a problem with deleting file.
Change-Id: I8dc31684bfece130eed6dd89806e965a4e1eabea
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
The SafeConfigParser class has been renamed to ConfigParser in
Python 3.2 [1]. The alias SafeConfigParser maybe removed in
future versions of Python 3. Use ConfigParser instead of
SafeConfigParser in Python 3
[1] http://bugs.python.org/issue10627
Change-Id: I1354b9fde203c9478bd77b71ec74a69304428385
Now the scheduler uses two pools of resources to handle execution of jobs.
The first one will handle the polling of jobs from the api.
The second one will handle the job exection. This way we avoid blocking the
scheduler while running jobs.
Abort a running job is now enabled thanks to the parallel execution.
Depends-on: I9e92ccdd7c592a8109fe8398db6d53c9c42dc10a
Closes-bug: 1615662
Change-Id: I48153dcb521b6a8f0138d5ef81239cf4cded4df0
When an exception was caught and rethrown, it should call 'raise'
without any arguments because it shows the place where an exception
occured initially instead of place where the exception re-raised.
Change-Id: Ic071e958694166a7df87915f50fe0f4a70f01fab
validating arguments before starting freezer job. each job will
validate job-specific arguments and the _general_validate method
in base Job class will takecare of the common ones.
Implements: blueprint validation-layer-for-freezer-jobs
Change-Id: Ia964bcc72abe39408dd72338951e083926a130ca
At the moment, if freezer attempts to perform a backup with
the consistency check option enabled, it will fail if the
files being backed up contain anything other than plain
ASCII (or at least some flavour of same, ISO-8859-x
etc.). This is because the hashlib module fails as it
expects a string to digest rather than a binary blob.
This should fix matters by escaping everything that might
cause an issue and works for Python 2 and 3. Because it
escapes strings in a different way to before, the
checksums it computes may be different to previously
in some cases, but at least now it should always produce
one.
Change-Id: Idf62dc9e398c396c9de19cbff5622e79d4c7a198
Closes-Bug: #1595732
When an exception was caught and rethrown, it should call 'raise'
without any arguments because it shows the place where an exception
occured initially instead of place where the exception re-raised.
Change-Id: I69e007566bf26fa2d4b6bcd1f996f4bba9f49049
Standardize on oslo.log.
Log exception stack traces where appropriate.
Add logging to all files.
Remove '[*]' from log messages.
Log command line arguments at Freezer Agent start
so that errors can be traced to specific invocations.
Leave scheduler module mostly unchanged because it
is doing python logging configuration.
Change-Id: I23de0558409e63978303963d592a4e5ee4dee2b5
Remove select line in tox.ini.
Currently checks are not being run.
Run pep8 checks.
Fix code to pass pep8 checks.
Change-Id: Iacef4386b1214f95b1bd106c17a5f714f948a1a1
Closes-Bug: #1593452
Make exception propagation from subprocesses and
threads work correctly.
Make exceptions propagate to top level program.
Make exceptions in subprocesses and threads log correctly.
Fix Tempest README.rst
Change-Id: If9ebe36afdc02c2460a86da7b4d9fa15e12c5824
Closes-Bug: #1592589
In addition, implements global file walk function on utils module
Also adds unit tests for checksum module
Implements blueprint: backup-consistency
Change-Id: I3ab23c2dfaacbaf4f0b293afa0a204a76e928b04
Depends-On: I294d7d3ad023c38e0639baa5934731bb46e875a2
Supports v2 and v3 of keystone and using keystoneauth library for
authentication with sessions. Supports new OS environment variables
Change-Id: I5cf8a5460b81c071f5982c1a7b7750d1c861a86f
Implements: blueprint refactor-os-clients
Freezer is using trickle only if it's installed system wide and not
looking for trickle-overload.so library anymore. Users have to
install trickle: apt-get install trickle or yum install trickle
for freezer to use it.
Change-Id: I49efff29876990dfd0782e3cf5ad68daf3e6055d
Closes-Bug: #1553159
1) new snapshot module with code vss,
lvm and part of backup.py -
create_snapshot and remove_snapshot
This module should contain all code related shadow,
vss and any new snapshots.
2) new openstack module with code from osclient,
restore and backup (BackupOS)
This module should contain code related to cinder/nova backups
3) new utils module with code from utils, win_utils,
bandwidth, config, exec_cmd, validator, streaming
This module should containd code related to anything
that cannot be extracted to its own module
4) new test modules - snapshot, utils, openstack and apiclient.
Test scheduler daemon is now located in scheduler module.
5) Backup.py is deleted and code placed to
openstack.backup (BackupOS) and to job.BackupJob
Change-Id: I46194b40b3a65c15b6095cfb7388dcfd30d95997
ljhuang