openstack
/
freezer
Code Issues Proposed changes

Fix Backup removal is not working 

Argument --remove-before-date added
Argument --remove-from-date deprecated in favor of --remove-before-date
Argument --remove-before-date now accepts ISO datetime and timestamp

Closes-bug: 1599592
Change-Id: I5a2c535ca4bf837cd2dc63e9e664ffc979e516bd
This commit is contained in:
Memo Garcia 2016-07-11 12:19:02 +01:00 committed by Pierre-Arthur MATHIEU
parent d08a383679
commit 4ef18182e0
9 changed files with 126 additions and 255 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

223
README.rst
View File

@ -348,6 +348,7 @@ Freezer can use:
--storage ssh --ssh-username ubuntu --ssh-key ~/.ssh/id_rsa
--ssh-host 8.8.8.8
**Note** ssh keys with passphrase are not supported at the moment.
Restore
-------
@ -400,9 +401,9 @@ List remote objects in container::
$ sudo freezerc --action info --container freezer_testcontainer -l
Remove backups older then 1 day::
Remove backups older than a date::
$ freezerc --action admin --container freezer_dev-test --remove-older-then 1 --backup-name dev-test-01
$ freezer-agent --action admin --container freezer_dev-test --remove-before-date 2016-07-11T00:00:00 --backup-name dev-test-01
Cinder restore currently creates a volume with the contents of the saved one, but
@ -444,8 +445,9 @@ Freezer architectural components are the following:
Freezer uses GNU Tar under the hood to execute incremental backup and
restore. When a key is provided, it uses OpenSSL to encrypt data.
(AES-256-CFB)
=============
Freezer architecture is composed by the following components:
Freezer components.
-------------------
+-------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Component | Description |
@ -775,219 +777,6 @@ the installation.
Please check the FAQ to: FAQ.rst
Available options::
usage: freezerc [-h] [--config CONFIG]
[--action {backup,restore,info,admin,exec}]
[-F PATH_TO_BACKUP] [-N BACKUP_NAME] [-m MODE] [-C CONTAINER]
[-s] [--lvm-auto-snap LVM_AUTO_SNAP] [--lvm-srcvol LVM_SRCVOL]
[--lvm-snapname LVM_SNAPNAME] [--lvm-snap-perm {ro,rw}]
[--lvm-snapsize LVM_SNAPSIZE] [--lvm-dirmount LVM_DIRMOUNT]
[--lvm-volgroup LVM_VOLGROUP] [--max-level MAX_LEVEL]
[--always-level ALWAYS_LEVEL]
[--restart-always-level RESTART_ALWAYS_LEVEL]
[-R REMOVE_OLDER_THAN] [--remove-from-date REMOVE_FROM_DATE]
[--no-incremental] [--hostname HOSTNAME]
[--mysql-conf MYSQL_CONF] [--metadata-out METADATA_OUT]
[--log-file LOG_FILE]
[--log-level {all,debug,info,warn,error,critical}]
[--exclude EXCLUDE]
[--dereference-symlink {none,soft,hard,all}]
[--encrypt-pass-file ENCRYPT_PASS_FILE] [-M MAX_SEGMENT_SIZE]
[--restore-abs-path RESTORE_ABS_PATH]
[--restore-from-host HOSTNAME]
[--restore-from-date RESTORE_FROM_DATE] [--max-priority] [-V]
[-q] [--insecure] [--os-auth-ver {1,2,2.0,3}] [--proxy PROXY]
[--dry-run] [--upload-limit UPLOAD_LIMIT]
[--cinder-vol-id CINDER_VOL_ID] [--nova-inst-id NOVA_INST_ID]
[--cindernative-vol-id CINDERNATIVE_VOL_ID]
[--download-limit DOWNLOAD_LIMIT]
[--sql-server-conf SQL_SERVER_CONF]
[--command COMMAND] [--compression {gzip,bzip2,xz}]
[--storage {local,swift,ssh}] [--ssh-key SSH_KEY]
[--ssh-username SSH_USERNAME] [--ssh-host SSH_HOST]
[--ssh-port SSH_PORT]
optional arguments:
-h, --help show this help message and exit
--config CONFIG Config file abs path. Option arguments are provided
from config file. When config file is used any option
from command line provided take precedence.
--action {backup,restore,info,admin,exec}
Set the action to be taken. backup and restore are
self explanatory, info is used to retrieve info from
the storage media, exec is used to execute a script,
while admin is used to delete old backups and other
admin actions. Default backup.
-F PATH_TO_BACKUP, --path-to-backup PATH_TO_BACKUP, --file-to-backup PATH_TO_BACKUP
The file or directory you want to back up to Swift
-N BACKUP_NAME, --backup-name BACKUP_NAME
The backup name you want to use to identify your
backup on Swift
-m MODE, --mode MODE Set the technology to back from. Options are, fs
(filesystem), mongo (MongoDB), mysql (MySQL),
sqlserver (SQL Server) Default set to fs
-C CONTAINER, --container CONTAINER
The Swift container (or path to local storage) used to
upload files to
-s, --snapshot Create a snapshot of the fs containing the resource to
backup. When used, the lvm parameters will be guessed
and/or the default values will be used
--lvm-auto-snap LVM_AUTO_SNAP
Automatically guess the volume group and volume name
for given PATH.
--lvm-srcvol LVM_SRCVOL
Set the lvm volume you want to take a snaphost from.
Default no volume
--lvm-snapname LVM_SNAPNAME
Set the lvm snapshot name to use. If the snapshot name
already exists, the old one will be used a no new one
will be created. Default freezer_backup_snap.
--lvm-snap-perm {ro,rw}
Set the lvm snapshot permission to use. If the
permission is set to ro The snapshot will be immutable
- read only -. If the permission is set to rw it will
be mutable
--lvm-snapsize LVM_SNAPSIZE
Set the lvm snapshot size when creating a new
snapshot. Please add G for Gigabytes or M for
Megabytes, i.e. 500M or 8G. Default 1G.
--lvm-dirmount LVM_DIRMOUNT
Set the directory you want to mount the lvm snapshot
to. Default to /var/lib/freezer
--lvm-volgroup LVM_VOLGROUP
Specify the volume group of your logical volume. This
is important to mount your snapshot volume. Default
not set
--max-level MAX_LEVEL
Set the backup level used with tar to implement
incremental backup. If a level 1 is specified but no
level 0 is already available, a level 0 will be done
and subsequently backs to level 1. Default 0 (No
Incremental)
--always-level ALWAYS_LEVEL
Set backup maximum level used with tar to implement
incremental backup. If a level 3 is specified, the
backup will be executed from level 0 to level 3 and to
that point always a backup level 3 will be executed.
It will not restart from level 0. This option has
precedence over --max-backup-level. Default False
(Disabled)
--restart-always-level RESTART_ALWAYS_LEVEL
Restart the backup from level 0 after n days. Valid
only if --always-level option if set. If --always-
level is used together with --remove-older-then, there
might be the chance where the initial level 0 will be
removed Default False (Disabled)
-R REMOVE_OLDER_THAN, --remove-older-then REMOVE_OLDER_THAN, --remove-older-than REMOVE_OLDER_THAN
Checks in the specified container for object older
than the specified days.If i.e. 30 is specified, it
will remove the remote object older than 30 days.
Default False (Disabled) The option --remove-older-
then is deprecated and will be removed soon
--remove-from-date REMOVE_FROM_DATE
Checks the specified container and removes objects
older than the provided datetime in the form "YYYY-MM-
DDThh:mm:ss i.e. "1974-03-25T23:23:23". Make sure the
"T" is between date and time
--no-incremental Disable incremental feature. By default freezer build
the meta data even for level 0 backup. By setting this
option incremental meta data is not created at all.
Default disabled
--hostname HOSTNAME Set hostname to execute actions. If you are executing
freezer from one host but you want to delete objects
belonging to another host then you can set this option
that hostname and execute appropriate actions. Default
current node hostname.
--mysql-conf MYSQL_CONF
Set the MySQL configuration file where freezer
retrieve important information as db_name, user,
password, host, port. Following is an example of
config file: # backup_mysql_conf host = <db-host> user
= <mysqluser> password = <mysqlpass> port = <db-port>
--metadata-out METADATA_OUT
Set the filename to which write the metadata regarding
the backup metrics. Use "-" to output to standard
output.
--log-file LOG_FILE Set log file. By default logs to
/var/log/freezer.logIf that file is not writable,
freezer tries to logto ~/.freezer/freezer.log
--log-level {all,debug,info,warn,error,critical}
Set logging level. Can be all, debug, info,
warn,error, critical. Default value - info
--exclude EXCLUDE Exclude files, given as a PATTERN.Ex: --exclude
'*.log' will exclude any file with name ending with
.log. Default no exclude
--dereference-symlink {none,soft,hard,all}
Follow hard and soft links and archive and dump the
files they refer to. Default False.
--encrypt-pass-file ENCRYPT_PASS_FILE
Passing a private key to this option, allow you to
encrypt the files before to be uploaded in Swift.
Default do not encrypt.
-M MAX_SEGMENT_SIZE, --max-segment-size MAX_SEGMENT_SIZE
Set the maximum file chunk size in bytes to upload to
swift Default 33554432 bytes (32MB)
--restore-abs-path RESTORE_ABS_PATH
Set the absolute path where you want your data
restored. Default False.
--restore-from-host HOSTNAME
Set the hostname used to identify the data you want to
restore from. If you want to restore data in the same
host where the backup was executed just type from your
shell: "$ hostname" and the output is the value that
needs to be passed to this option. Mandatory with
Restore Default False. (Deprecated use "hostname"
instead)
--restore-from-date RESTORE_FROM_DATE
Set the absolute path where you want your data
restored. Please provide datetime in format "YYYY-MM-
DDThh:mm:ss" i.e. "1979-10-03T23:23:23". Make sure the
"T" is between date and time Default None.
--max-priority Set the cpu process to the highest priority (i.e. -20
on Linux) and real-time for I/O. The process priority
will be set only if nice and ionice are installed
Default disabled. Use with caution.
-V, --version Print the release version and exit
-q, --quiet Suppress error messages
--insecure Allow to access swift servers without checking SSL
certs.
--os-auth-ver {1,2,2.0,3}, --os-identity-api-version {1,2,2.0,3}
Openstack identity api version, can be 1, 2, 2.0 or 3
--proxy PROXY Enforce proxy that alters system HTTP_PROXY and
HTTPS_PROXY, use '' to eliminate all system proxies
--dry-run Do everything except writing or removing objects
--upload-limit UPLOAD_LIMIT
Upload bandwidth limit in Bytes per sec. Can be
invoked with dimensions (10K, 120M, 10G).
--cinder-vol-id CINDER_VOL_ID
Id of cinder volume for backup
--nova-inst-id NOVA_INST_ID
Id of nova instance for backup
--cindernative-vol-id CINDERNATIVE_VOL_ID
Id of cinder volume for native backup
--download-limit DOWNLOAD_LIMIT
Download bandwidth limit in Bytes per sec. Can be
invoked with dimensions (10K, 120M, 10G).
--sql-server-conf SQL_SERVER_CONF
Set the SQL Server configuration file where freezer
retrieve the sql server instance. Following is an
example of config file: instance = <db-instance>
--command COMMAND Command executed by exec action
--compression {gzip,bzip2,xz}
compression algorithm to use. gzip is default
algorithm
--storage {local,swift,ssh}
Storage for backups. Can be Swift or Local now. Swift
is defaultstorage now. Local stores backups on the
same defined path andswift will store files in
container.
--ssh-key SSH_KEY Path to ssh-key for ssh storage only
--ssh-username SSH_USERNAME
Remote username for ssh storage only
--ssh-host SSH_HOST Remote host for ssh storage only
--ssh-port SSH_PORT Remote port for ssh storage only (default 22)
Scheduler Options
-----------------

60
freezer/common/config.py
View File

@ -60,15 +60,15 @@ DEFAULT_PARAMS = {
'max_segment_size': 33554432, 'lvm_srcvol': False,
'download_limit': -1, 'hostname': False, 'remove_from_date': False,
'restart_always_level': False, 'lvm_dirmount': None,
'dereference_symlink': '',
'config': False, 'mysql_conf': False,
'dereference_symlink': '', 'remove_before_date': False,
'config': False, 'mysql_conf': False, 'remove_older_than': False,
'insecure': False, 'lvm_snapname': None,
'lvm_snapperm': 'ro', 'snapshot': False,
'max_priority': False, 'max_level': False, 'path_to_backup': False,
'encrypt_pass_file': False, 'volume': False, 'proxy': False,
'cinder_vol_id': '', 'cindernative_vol_id': '',
'nova_inst_id': '', '__version__': FREEZER_VERSION,
'remove_older_than': None, 'restore_from_date': False,
'restore_from_date': False,
'upload_limit': -1, 'always_level': False, 'version': False,
'dry_run': False, 'lvm_snapsize': DEFAULT_LVM_SNAPSIZE,
'restore_abs_path': False, 'log_file': None, 'log_level': "info",
@ -169,26 +169,28 @@ _COMMON = [
" --max-backup-level. Default False (Disabled)"),
cfg.FloatOpt('restart-always-level',
dest='restart_always_level',
help="Restart the backup from level 0 after n days. Valid only"
" if --always-level option if set. If --always-level is "
"used together with --remove-older-then, there might be "
help="Restart the backup from level 0 after n days. Valid "
"only if --always-level option is set. If "
"--always-level is used together with "
"--remove-older-than, there might be "
"the chance where the initial level 0 will be removed. "
"Default False (Disabled)"),
cfg.FloatOpt('remove-older-than',
short='R',
dest='remove_older_than',
help="Checks in the specified container for object older than "
"the specified days. If i.e. 30 is specified, it will "
"remove the remote object older than 30 days. Default "
"False (Disabled) The option --remove-older-then is "
"deprecated and will be removed soon",
deprecated_for_removal=True),
cfg.StrOpt('remove-from-date',
dest='remove_from_date',
help="Checks the specified container and removes objects older "
"than the provided datetime in the form "
"'YYYY-MM-DDThh:mm:ss' i.e. '1974-03-25T23:23:23'. "
"Make sure the 'T' is between date and time "),
"Make sure the 'T' is between date and time, Deprecated "
"favor of --remove-before-date",
deprecated_for_removal=True),
cfg.StrOpt('remove-before-date',
dest='remove_before_date',
help="Checks the specified container and removes objects older "
"than the provided datetime in the form "
"'YYYY-MM-DDThh:mm:ss' i.e. '1974-03-25T23:23:23'. "
"Make sure the 'T' is between date and time , it also "
"supports timestamp values",
deprecated_name='remove_from_date'),
cfg.StrOpt('no-incremental',
dest='no_incremental',
help="Disable incremental feature. By default freezer build the"
@ -260,8 +262,32 @@ _COMMON = [
"be found. "
"Please provide datetime in format 'YYYY-MM-DDThh:mm:ss' "
"i.e. '1979-10-03T23:23:23'. Make sure the 'T' is between "
"date and time Default None."
"date and time Default None.",
),
cfg.FloatOpt('remove-older-than',
short='R',
dest='remove_older_than',
help="Checks in the specified container for objects older "
"than the specified number of days. "
"If i.e. 30 is specified, it "
"will remove the remote object older than 30 days. "
"Default False (Disabled)"),
cfg.StrOpt('remove-before-date',
dest='remove_before_date',
help="Checks the specified container and removes objects older "
"than the provided datetime in the form "
"'YYYY-MM-DDThh:mm:ss' i.e. '1974-03-25T23:23:23'. "
"Make sure the 'T' is between date and time , it also "
"supports timestamp values",
deprecated_name='remove_from_date'),
cfg.StrOpt('remove-from-date',
dest='remove_from_date',
help="Checks the specified container and removes objects older "
"than the provided datetime in the form "
"'YYYY-MM-DDThh:mm:ss' i.e. '1974-03-25T23:23:23'. "
"Make sure the 'T' is between date and time, Deprecated "
"favor of --remove-before-date",
deprecated_for_removal=True),
cfg.StrOpt('max-priority',
dest='max_priority',
help="Set the cpu process to the highest priority (i.e. -20 on "

27
freezer/job.py
View File

@ -15,11 +15,9 @@ limitations under the License.
"""
import datetime
import os
from oslo_utils import importutils
import sys
import time
from freezer.openstack import backup
from freezer.openstack import restore
@ -201,16 +199,23 @@ class RestoreJob(Job):
class AdminJob(Job):
@Job.executemethod
def execute(self):
if self.conf.remove_from_date:
timestamp = utils.date_to_timestamp(self.conf.remove_from_date)
else:
timestamp = datetime.datetime.now() - \
datetime.timedelta(days=self.conf.remove_older_than)
timestamp = int(time.mktime(timestamp.timetuple()))
if self.conf.remove_before_date:
self.storage.remove_older_than(timestamp,
self.conf.hostname_backup_name)
return {}
if utils.is_iso_date(self.conf.remove_before_date):
timestamp = utils.date_to_timestamp(
self.conf.remove_before_date)
elif utils.is_timestamp(self.conf.remove_before_date):
timestamp = self.conf.remove_before_date
else:
raise Exception('Expecting ISO date or valid timestamp.')
self.storage.remove_before_date(timestamp,
self.conf.hostname_backup_name)
return {}
elif self.conf.remove_older_than:
self.storage.remove_older_than(self.conf.remove_before_date,
self.conf.hostname_backup_name)
return {}
class ExecJob(Job):

9
freezer/main.py
View File

@ -109,7 +109,6 @@ def freezer_main(backup_args):
return run_job(backup_args, storage)
else:
run_job(backup_args, storage)
if not backup_args.quiet:
@ -123,8 +122,16 @@ def run_job(conf, storage):
'info': job.InfoJob,
'admin': job.AdminJob,
'exec': job.ExecJob}[conf.action](conf, storage)
response = freezer_job.execute()
# Inject remove-from-date functionality for a backup and restore jobs
if ((conf.remove_before_date or conf.remove_from_date or
conf.remove_older_than) and conf.action != 'admin'):
# TODO(m3m0): return something here
job.AdminJob(conf, storage).execute()
# TODO(m3m0): delete all backups from the api.
if conf.metadata_out and response:
if conf.metadata_out == '-':
sys.stdout.write(json.dumps(response))

21
freezer/storage/base.py
View File

@ -13,8 +13,10 @@
# limitations under the License.
import abc
import os
import re
import time
from oslo_log import log
@ -137,7 +139,7 @@ class Storage(object):
"""
raise NotImplementedError("Should have implemented this")
def remove_older_than(self, remove_older_timestamp, hostname_backup_name):
def remove_before_date(self, remove_older_timestamp, hostname_backup_name):
"""
Removes backups which are older than the specified timestamp
:type remove_older_timestamp: int
@ -145,10 +147,23 @@ class Storage(object):
"""
backups = self.find_all(hostname_backup_name)
backups = [b for b in backups
if b.latest_update.timestamp < remove_older_timestamp]
if b.latest_update.timestamp <= remove_older_timestamp]
for b in backups:
b.storage.remove_backup(b)
def remove_older_than(self, days, hostname_backup_name):
"""Removes backups older than n amount of days.
:param days: int
:param hostname_backup_name: str
:return:
"""
now = time.time()
seconds_old = utils.days_to_seconds(days)
to_delete_timestamp = now - seconds_old
return self.remove_before_date(to_delete_timestamp,
hostname_backup_name)
@abc.abstractmethod
def info(self):
raise NotImplementedError("Should have implemented this")
@ -299,6 +314,7 @@ class Backup:
@staticmethod
def parse_backups(names, storage):
"""
:param names:
:type names: list[str] - file names of backups.
:type storage: freezer.storage.base.Storage
@ -306,6 +322,7 @@ class Backup:
:rtype: list[freezer.storage.base.Backup]
:return: list of zero level backups
"""
# TODO(m3m0): rename this function
prefix = 'tar_metadata_'
tar_names = set([x[len(prefix):]
for x in names if x.startswith(prefix)])

4
freezer/storage/swift.py
View File

@ -172,11 +172,11 @@ class SwiftStorage(base.Storage):
for i in range(backup.latest_update.level, -1, -1):
if i in backup.increments:
# remove segment
self.remove(self.segments, backup.increments[i])
self.remove(self.segments, backup.increments[i].__repr__())
# remove tar
self.remove(self.container, backup.increments[i].tar())
# remove manifest
self.remove(self.container, backup.increments[i])
self.remove(self.container, backup.increments[i].__repr__())
def add_stream(self, stream, package_name, headers=None):
i = 0

1
freezer/tests/commons.py
View File

@ -291,6 +291,7 @@ class BackupOpt1:
self.max_level = '0'
self.hostname_backup_name = "hostname_backup_name"
self.remove_older_than = '0'
self.remove_before_date = '2016-07-12T11:13:40'
self.max_segment_size = '0'
self.time_stamp = 123456789
self.container = 'test-container'

32
freezer/utils/utils.py
View File

@ -19,6 +19,7 @@ Freezer general utils functions
import datetime
import errno
import os
import re
import subprocess
import sys
import time
@ -193,9 +194,12 @@ def create_subprocess(cmd):
def date_to_timestamp(date):
fmt = '%Y-%m-%dT%H:%M:%S'
opt_backup_date = datetime.datetime.strptime(date, fmt)
return int(time.mktime(opt_backup_date.timetuple()))
try:
fmt = '%Y-%m-%dT%H:%M:%S'
opt_backup_date = datetime.datetime.strptime(date, fmt)
return int(time.mktime(opt_backup_date.timetuple()))
except Exception:
raise Exception('Invalid ISO date format')
class Bunch:
@ -464,3 +468,25 @@ def abort_subprocess(signum, frame):
LOG.error(traceback.print_exc())
finally:
sys.exit(33)
def days_to_seconds(n):
"""
86400 seconds in a day
:param n: number of days
:return: int
"""
return n * 86400
def is_iso_date(date):
iso_f = re.compile('^(\d{4})-0?(\d+)-0?(\d+)[T ]0?(\d+):0?(\d+):0?(\d+)$')
return re.match(iso_f, date)
def is_timestamp(ts):
try:
ts = int(ts)
return True
except ValueError:
raise Exception('Invalid timestamp')

4
tests/unit/storages/test_base.py
View File

@ -180,10 +180,10 @@ class TestBackup(unittest.TestCase):
base.Backup(t, "host_backup", 5000),
]
t.remove_backup = mock.Mock()
t.remove_older_than(3000, "host_backup")
t.remove_before_date(3000, "host_backup")
t.remove_backup.assert_any_call(r1)
t.remove_backup.assert_any_call(r2)
assert t.remove_backup.call_count == 2
assert t.remove_backup.call_count == 3
def test_create_backup(self):
t = base.Storage(None, skip_prepare=True)