Merge intree freezer tempest plugin

* It merges the intree freezer tempest plugin to a the newly
  seperated repo freezer-tempest-plugin

* Moves freezer/tests/integration/common.py to under freezer-tempest-plugin
  as it is consumed in tests/api/base.py .

* Excluding freezer/tests/freezer_tempest_plugin/tests/api/test_version.py
  as it is dependent on freezer project and it should be moved under
  integration tests.

Change-Id: I6967f915758728827e8ddcd1a45a7023904b694e
This commit is contained in:
Chandan Kumar 2017-12-08 16:42:44 +05:30
parent e6d6395a38
commit 65bdf3bcf9
16 changed files with 1693 additions and 26 deletions

View File

@ -11,8 +11,3 @@ More information can be found in the freezer developer documentation.
* Documentation: http://docs.openstack.org/developer/freezer-tempest-plugin * Documentation: http://docs.openstack.org/developer/freezer-tempest-plugin
* Source: http://git.openstack.org/cgit/openstack/freezer-tempest-plugin * Source: http://git.openstack.org/cgit/openstack/freezer-tempest-plugin
* Bugs: http://bugs.launchpad.net/freezer * Bugs: http://bugs.launchpad.net/freezer
Features
--------
* TODO

View File

@ -1,6 +1,259 @@
============================== Freezer Tempest Tests
Tempest Integration of Freezer =====================
==============================
This directory contains Tempest tests to cover the Freezer project. Integration tests in Freezer are implemented using tempest. This document describes different approaches to run these tests.
Where to start?
* If you just want to run the tests as quickly as possible, start with `Run tests inside a devstack VM`_.
* If you want to run tests on your local machine (with services running in devstack), start with `Run tests outside a devstack VM`_.
Alternatively there is a slightly different version that uses nose as a testrunner: `Run tests outside a devstack VM (alternative instructions using nose)`_.
* If you want to run tests on your local machine in PyCharm, start with `Run tests in PyCharm`_.
* If you want to run the tests on Mac OS X, start with `Mac OS X Instructions`_ and continue with one of the options above.
Setting up a devstack VM
------------------------
Install devstack with swift and the freezer [1]_ as well as the freezer-api [2]_ plugins by adding the following lines to you `local.conf`:
::
enable_plugin freezer https://git.openstack.org/openstack/freezer master
enable_plugin freezer-api https://git.openstack.org/openstack/freezer-api master
enable_service s-proxy s-object s-container s-account
.. [1] https://github.com/openstack/freezer/blob/master/devstack/README.rst
.. [2] https://github.com/openstack/freezer-api/blob/master/devstack/README.rst
Run tests inside a devstack VM
-------------------------------
#. Create a devstack VM as described in `Setting up a devstack VM`_
#. Inside your devstack VM, navigate to `/opt/stack/tempest`.
#. Run `ostestr -r freezer`
Debugging tests inside a devstack VM
------------------------------------
Often a devstack VM is used via SSH without graphical interface. Python has multiple command line debuggers. The out-of-the-box pdb works fine but I recommend pudb [3]_ which looks a bit like the old Turbo-Pascal/C IDE. The following steps are necessary to get it running:
#. Follow the steps in `Run tests inside a devstack VM`_.
#. Log into the devstack VM
#. Install pudb:
::
pip install pudb
#. Open the test file were you want to set the first breakpoint (more breakpoints can be set interactively later) and add the following line
::
import pudb;pu.db
#. Navigate to `/opt/stack/tempest`.
#. `ostestr` runs tests in parallel which causes issues with debuggers. To work around that you need to run the relevant test directly. E.g.:
::
python -m unittest freezer_tempest_plugin.tests.scenario.test_backups.TestFreezerScenario
#. It should drop you into the debugger!
.. [3] https://pypi.python.org/pypi/pudb
Run tests outside a devstack VM
-------------------------------
This section describes how to run the tests outside of a devstack VM (e.g. in PyCharm) while using services (keystone, swift, ...) inside a VM.
#. Create a devstack VM as described in `Setting up a devstack VM`_.
#. Create and activate a virtual environment for Tempest:
::
virtualenv --no-site-packages tempest-venv
. tempest-venv/bin/activate
#. Clone and install the Tempest project into the virtual environment:
::
git clone https://github.com/openstack/tempest
pip install tempest/
#. Clone and install the Freezer project into the virtual environment:
::
git clone https://github.com/openstack/freezer
pip install -e freezer/
#. Clone and install the Freezer API project into the virtual environment:
::
git clone https://github.com/openstack/freezer-api
pip install -e freezer-api/
#. Initialise a Tempest working directory:
::
mkdir tempest-working
cd tempest-working
tempest init .
#. Configure `tempest-working/etc/tempest.conf`. The easiest way to do this is to just copy the config from `/opt/stack/tempest/etc/tempest.conf` inside the devstack VM.
#. Run the freezer test inside the tempest working directory:
::
cd tempest-working
ostestr -r freezer
Run tests outside a devstack VM (alternative instructions using nose)
---------------------------------------------------------------------
#. Need to make sure that there is a Devstack or other environment for running Keystone and Swift.
#. Clone the Tempest Repo::
run 'git clone https://github.com/openstack/tempest.git'
#. Create a virtual environment for Tempest. In these instructions, the Tempest virtual environment is ``~/virtualenvs/tempest-freezer``.
#. Activate the Tempest virtual environment::
run 'source ~/virtualenvs/tempest-freezer/bin/activate'
#. Make sure you have latest pip installed::
run 'pip install --upgrade pip'
#. Install Tempest requirements.txt and test-requirements.txt in the Tempest virtual environment::
run 'pip install -r requirements.txt -r test-requirements.txt'
#. Install Tempest project into the virtual environment in develop mode::
run python setup.py develop
#. Create logging.conf in Tempest Repo home dir/etc
Make a copy of logging.conf.sample as logging.conf
In logging configuration
You will see this error on Mac OS X
socket.error: [Errno 2] No such file or directory
To fix this, edit logging.conf
Change /dev/log/ to '/var/run/syslog in logging.conf
see: https://github.com/baremetal/python-backoff/issues/1 for details
#. Create tempest.conf in Tempest Repo home dir/etc::
run 'oslo-config-generator --config-file etc/config-generator.tempest.conf --output-file etc/tempest.conf'
Add the following sections to tempest.conf and modify uri and uri_v3 to point to the host where Keystone is running::
[identity]
username = freezer
password = secretservice
tenant_name = service
domain_name = default
admin_username = admin
admin_password = secretadmin
admin_domain_name = default
admin_tenant_name = admin
alt_username = admin
alt_password = secretadmin
alt_tenant_name = admin
use_ssl = False
auth_version = v3
uri = http://10.10.10.6:5000/v2.0/
uri_v3 = http://10.10.10.6:35357/v3/
[auth]
allow_tenant_isolation = true
tempest_roles = admin
#. Clone freezer Repo::
run 'git clone https://github.com/openstack/freezer.git'
#. Set the virtual environment to the Tempest virtual environment::
run 'source ~/virtualenvs/tempest-freezer/bin/activate'
#. pip install freezer requirements.txt and test-requirements.txt in Tempest virtual environment::
run 'pip install -r requirements.txt -r test-requirements.txt'
#. Install nose in the Temptest virtual environment::
run 'pip install nose'
#. Install freezer project into the Tempest virtual environment in develop mode::
run python setup.py develop
#. Set project interpreter (pycharm) to Tempest virtual environment.
#. Create test config (pycharm) using the Tempest virtual environment as python interpreter::
Set the environment variable OS_AUTH_URL to the URI where Keystone is running. For example, OS_AUTH_URL=http://10.10.10.6:5000/v2.0.
Set the Working Directory to the Tempest home dir. This will allow Tempest to find the etc/tempest.conf file.
#. Run the tests in the api directory in the freezer_tempest_plugin directory.
Mac OS X Instructions
---------------------
For Mac OS X users you will need to install gnu-tar in ``/usr/local/bin`` and make sure that ``/usr/local/bin`` is in the PATH environment variable before any other directories where a different version of tar can be found. Gnu-tar can be installed as ``gtar`` or ``tar``, either name works.
Also, currently for Mac OS X users, the latest version of gnu-tar (1.29) will not allow ``--unlink-first`` and ``--overwrite`` options to be used together. Also, gnu-tar will complain about the ``--unlink-first`` argument. To get around these limitations, you will need to modify ``tar_builders.py`` and remove the ``--unlink-first`` option from the ``UNIX_TEMPLATE`` variable.
Run tests in PyCharm
--------------------
#. Set up the test environment as described in `Run tests outside a devstack VM`_.
#. Start PyCharm and open a new project pointing to the cloned freezer directory.
#. Click `File > Settings > Project: freezer > Project Interpreter`.
#. Click the gear-wheel icon next to `Project Interpreter` and choose `Add Local`.
#. Navigate to your virtual environment and select the Python interpreter under `bin/python` and confirm with `OK`
#. In the left pane, navigate to one of the test scripts in `freezer_tempest_plugin/tests/[api or scenario]/*.py`.
#. Right-click the file and choose `Run 'Unittests in [..]'`
#. This test run will most likely fail because it is started from the wrong directory. To fix this, open the dropdown box next to the run button in the top-right corner. Choose `Edit Configurations ..`
#. Point `Working directory:` to your tempest working directory.
#. Run the test again, this time it should work!
Troubleshooting
---------------
If tests fail these are good places to check:
* freezer-api log: `/var/log/apache2/freezer-api.log`
* freezer-agent log: `$HOME/.freezer/freezer.log`

View File

@ -0,0 +1,325 @@
# Copyright 2015 Hewlett-Packard
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import distutils.spawn
import hashlib
import itertools
import json
import os
import random
import shutil
import subprocess
import tempfile
import unittest
import paramiko
from six.moves import range
FREEZERC = distutils.spawn.find_executable('freezer-agent')
class CommandFailed(Exception):
def __init__(self, returncode, cmd, output, stderr):
super(CommandFailed, self).__init__()
self.returncode = returncode
self.cmd = cmd
self.stdout = output
self.stderr = stderr
def __str__(self):
return ("Command '%s' returned unexpected exit status %d.\n"
"stdout:\n%s\n"
"stderr:\n%s" % (self.cmd, self.returncode,
self.stdout, self.stderr))
def dict_to_args(d):
l = [['--' + k.replace('_', '-'), v] for k, v in d.items()]
return list(itertools.chain.from_iterable(l))
def execute_freezerc(dict, must_fail=False, merge_stderr=False):
"""
:param dict:
:type dict: dict[str, str]
:param must_fail:
:param merge_stderr:
:return:
"""
return execute([FREEZERC] + dict_to_args(dict), must_fail=must_fail,
merge_stderr=merge_stderr)
def execute(args, must_fail=False, merge_stderr=False):
"""
Executes specified command for the given action.
:param args:
:type args: list[str]
:param must_fail:
:param merge_stderr:
:return:
"""
stdout = subprocess.PIPE
stderr = subprocess.STDOUT if merge_stderr else subprocess.PIPE
proc = subprocess.Popen(args, stdout=stdout, stderr=stderr)
result, result_err = proc.communicate()
if not must_fail and proc.returncode != 0:
raise CommandFailed(proc.returncode, ' '.join(args), result,
result_err)
if must_fail and proc.returncode == 0:
raise CommandFailed(proc.returncode, ' '.join(args), result,
result_err)
return result
class Temp_Tree(object):
def __init__(self, suffix='', dir=None, create=True):
self.create = create
if create:
self.path = tempfile.mkdtemp(dir=dir, prefix='__freezer_',
suffix=suffix)
else:
self.path = dir
self.files = []
def __enter__(self):
return self
def cleanup(self):
if self.create and self.path:
shutil.rmtree(self.path)
def __exit__(self, exc_type, exc_val, exc_tb):
self.cleanup()
def add_random_data(self, ndir=5, nfile=5, size=1024):
"""
add some files containing random data
:param ndir: number of dirs to create
:param nfile: number of files to create in each dir
:param size: size of files
:return: None
"""
def create_file(path):
abs_pathname = self.create_file_with_random_data(
dir_path=path, size=size)
rel_path_name = abs_pathname[len(self.path) + 1:]
self.files.append(rel_path_name)
for _ in range(nfile):
create_file(self.path)
for _ in range(ndir):
subdir_path = tempfile.mkdtemp(dir=self.path)
for _ in range(nfile):
create_file(subdir_path)
def create_file_with_random_data(self, dir_path, size=1024):
handle, abs_pathname = tempfile.mkstemp(dir=dir_path)
with open(abs_pathname, 'wb') as fd:
fd.write(os.urandom(size))
return abs_pathname
def modify_random_files(self, count=1):
indexes = []
for _ in range(count):
indexes.append(random.randint(0, len(self.files) - 1))
for file_index in indexes:
file_name = self.files[file_index]
with open(os.path.join(self.path, file_name), 'ab') as fd:
size_to_add = int(fd.tell() * 0.5)
fd.write(os.urandom(size_to_add))
def delete_random_files(self, count=1):
indexes = []
for _ in range(count):
indexes.append(random.randint(0, len(self.files) - 1))
for file_index in indexes:
file_name = self.files[file_index]
os.unlink(os.path.join(self.path, file_name))
def get_file_hash(self, rel_filepath):
filepath = os.path.join(self.path, rel_filepath)
if os.path.isfile(filepath):
return self._filehash(filepath)
else:
return ''
def _filehash(self, filepath):
"""
Get GIT style sha1 hash for a file
:param filepath: path of file to hash
:return: hash of the file
"""
filesize_bytes = os.path.getsize(filepath)
hash_obj = hashlib.sha1()
hash_obj.update(("blob %u\0" % filesize_bytes).encode('utf-8'))
with open(filepath, 'rb') as handle:
hash_obj.update(handle.read())
return hash_obj.hexdigest()
def get_file_list(self):
"""
walks the dir tree and creates a list of relative pathnames
:return: list of relative file paths
"""
self.files = []
for root, dirs, files in os.walk(self.path):
rel_base = root[len(self.path) + 1:]
self.files.extend([os.path.join(rel_base, x) for x in files])
return self.files
def is_equal(self, other_tree):
"""
Checks whether two dir tree contain the same files
It checks the number of files and the hash of each file.
NOTE: tox puts .coverage files in the temp folder (?)
:param other_tree: dir tree to compare with
:return: true if the dir trees contain the same files
"""
lh_files = [x for x in sorted(self.get_file_list())
if not x.startswith('.coverage')]
rh_files = [x for x in sorted(other_tree.get_file_list())
if not x.startswith('.coverage')]
if lh_files != rh_files:
return False
for fname in lh_files:
if os.path.isfile(fname):
if self.get_file_hash(fname) != \
other_tree.get_file_hash(fname):
return False
return True
class TestFS(unittest.TestCase):
"""
Utility class for setting up the tests.
Type of tests depends (also) on the environment variables defined.
To enable the ssh storage testing, the following environment
variables need to be defined:
- FREEZER_TEST_SSH_KEY
- FREEZER_TEST_SSH_USERNAME
- FREEZER_TEST_SSH_HOST
- FREEZER_TEST_CONTAINER
To enable the swift storage testing, the following environment
variables need to be defined:
- FREEZER_TEST_OS_TENANT_NAME
- FREEZER_TEST_OS_USERNAME
- FREEZER_TEST_OS_REGION_NAME
- FREEZER_TEST_OS_PASSWORD
- FREEZER_TEST_OS_AUTH_URL
Tests involving LVM snapshots are evoided if:
- user is not root
- FREEZER_TEST_NO_LVM is set
"""
ssh_key = os.environ.get('FREEZER_TEST_SSH_KEY')
ssh_username = os.environ.get('FREEZER_TEST_SSH_USERNAME')
ssh_host = os.environ.get('FREEZER_TEST_SSH_HOST')
container = os.environ.get('FREEZER_TEST_CONTAINER')
use_ssh = ssh_key and ssh_username and ssh_host and container
os_tenant_name = os.environ.get('FREEZER_TEST_OS_TENANT_NAME')
os_user_name = os.environ.get('FREEZER_TEST_OS_USERNAME')
os_region = os.environ.get('FREEZER_TEST_OS_REGION_NAME')
os_password = os.environ.get('FREEZER_TEST_OS_PASSWORD')
os_auth_url = os.environ.get('FREEZER_TEST_OS_AUTH_URL')
use_os = (os_tenant_name and os_user_name and os_region and
os_password and os_auth_url)
if use_os:
os.environ['OS_USERNAME'] = os_user_name
os.environ['OS_TENANT_NAME'] = os_tenant_name
os.environ['OS_AUTH_URL'] = os_auth_url
os.environ['OS_PASSWORD'] = os_password
os.environ['OS_REGION_NAME'] = os_region
os.environ['OS_TENANT_ID'] = ''
openstack_executable = distutils.spawn.find_executable('openstack')
swift_executable = distutils.spawn.find_executable('swift')
use_lvm = (os.getuid() == 0 and 'FREEZER_TEST_NO_LVM' not in os.environ)
ssh_executable = distutils.spawn.find_executable('ssh')
def setUp(self):
self.source_tree = Temp_Tree()
self.dest_tree = Temp_Tree()
if TestFS.use_ssh:
self.ssh_client = paramiko.SSHClient()
self.ssh_client.set_missing_host_key_policy(
paramiko.AutoAddPolicy())
self.ssh_client.connect(TestFS.ssh_host,
username=TestFS.ssh_username,
key_filename=TestFS.ssh_key)
def tearDown(self):
self.source_tree.cleanup()
self.dest_tree.cleanup()
def assertTreesMatch(self):
self.assertTrue(self.source_tree.is_equal(self.dest_tree))
def assertTreesMatchNot(self):
self.assertFalse(self.source_tree.is_equal(self.dest_tree))
def get_file_list_ssh(self, sub_path=''):
ftp = self.ssh_client.open_sftp()
path = '{0}/{1}'.format(self.container, sub_path)
return ftp.listdir(path)
def remove_ssh_directory(self, sub_path=''):
cmd = 'rm -rf {0}/{1}'.format(self.container, sub_path)
self.ssh_client.exec_command(cmd)
def get_file_list_openstack(self, container):
if self.openstack_executable:
json_result = execute([self.openstack_executable, 'object', 'list',
container, '-f', json])
result = json.loads(json_result)
return [x['Name'] for x in result]
if self.swift_executable:
result = execute([self.swift_executable, 'list', container])
return result.split()
raise Exception(
"Unable to get container list using openstackclient/swiftclient")
def remove_swift_container(self, container):
if self.openstack_executable:
execute([self.openstack_executable, 'container',
'delete', container])
execute([self.openstack_executable, 'container',
'delete', container + '_segments'])
elif self.swift_executable:
execute([self.swift_executable, 'delete', container])
execute([self.swift_executable, 'delete', container + '_segments'])
return True
def do_backup_and_restore_with_check(self, backup_args, restore_args):
self.source_tree.add_random_data()
self.assertTreesMatchNot()
result = execute_freezerc(backup_args)
self.assertIsNotNone(result)
result = execute_freezerc(restore_args)
self.assertIsNotNone(result)
self.assertTreesMatch()

View File

@ -13,3 +13,9 @@
# License for the specific language governing permissions and limitations # License for the specific language governing permissions and limitations
# under the License. # under the License.
from oslo_config import cfg
service_option = cfg.BoolOpt('freezer',
default=True,
help="Whether or not freezer is expected to be "
"available")

View File

@ -13,13 +13,11 @@
# License for the specific language governing permissions and limitations # License for the specific language governing permissions and limitations
# under the License. # under the License.
import os import os
from tempest import config
from tempest.test_discover import plugins from tempest.test_discover import plugins
from freezer_tempest_plugin import config as project_config from freezer_tempest_plugin import config as freezer_config
class FreezerTempestPlugin(plugins.TempestPlugin): class FreezerTempestPlugin(plugins.TempestPlugin):
@ -31,7 +29,8 @@ class FreezerTempestPlugin(plugins.TempestPlugin):
return full_test_dir, base_path return full_test_dir, base_path
def register_opts(self, conf): def register_opts(self, conf):
pass conf.register_opt(freezer_config.service_option,
group='service_available')
def get_opt_lists(self): def get_opt_lists(self):
pass return [('service_available', [freezer_config.service_option])]

View File

@ -0,0 +1,269 @@
# (C) Copyright 2016 Hewlett Packard Enterprise Development Company LP
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from datetime import datetime
from datetime import timedelta
import json
import os
import subprocess
from time import mktime
from tempest import test
from freezer_tempest_plugin.common import Temp_Tree
def resolve_paths(metadata):
"""Find all paths associated with a particular backup
freezer-agent stores all backups in the timestamped sub-directory of the
first backup created with a particular (name, hostname) pair, so it isn't
possible to guess the true location of backup data. This function searches
the backup container to find both the true parent directory and a list of
all files associated with the given metadata.
:param metadata: the metadata associated with the backup to resolve
:return: a tuple containing the parent directory and a list of associated
files, or (None, None)
"""
base_name = '{}_{}'.format(metadata['hostname'], metadata['backup_name'])
expected_name = '{}_{}'.format(base_name, metadata['time_stamp'])
backup_base_path = os.path.join(metadata['container'], base_name)
for timestamp in os.listdir(backup_base_path):
timestamp_abs = os.path.join(backup_base_path, timestamp)
matching = filter(lambda p: expected_name in p,
os.listdir(timestamp_abs))
if matching:
return timestamp_abs, matching
return None, None
def mutate_timestamp(metadata, days_old):
"""Alter all timestamps of an existing backup
Since there's no proper way to assign a timestamp to a backup, this method
takes an existing backup and modifies all associated timestamps to make it
otherwise indistinguishable from a backup actually created in the past.
:param metadata: the metadata associated with the backup to mutate
:param days_old: the age (i.e. days before now) that should be set
"""
date = datetime.now() - timedelta(days=days_old)
old_time_stamp = metadata['time_stamp']
new_time_stamp = int(mktime(date.timetuple()))
parent_dir, files = resolve_paths(metadata)
if os.path.basename(parent_dir) == str(old_time_stamp):
# rename the parent dir, but only if it was created for this
# backup (the dir may contain other backups with different
# timestamps that we shouldn't touch)
new_path = os.path.join(os.path.dirname(parent_dir),
str(new_time_stamp))
os.rename(parent_dir, new_path)
parent_dir = new_path
# rename each file associated with the backup, since each filename
# contains the timestamp as well
for old_file in files:
new_file = old_file.replace(str(old_time_stamp), str(new_time_stamp))
os.rename(os.path.join(parent_dir, old_file),
os.path.join(parent_dir, new_file))
# update the metadata before saving to keep things consistent
metadata['time_stamp'] = new_time_stamp
def load_metadata(path):
"""Given a metadata path, return a dict containing parsed values.
:param path: the path to load
:return: a metadata dict
"""
with open(path, 'r') as f:
return json.load(f)
def save_metadata(metadata, path):
"""Write the given metadata object to the provided path.
:param metadata: the metadata dict to write
:param path: the path at which to write the metadata
"""
with open(path, 'w') as f:
json.dump(metadata, f)
class BaseFreezerTest(test.BaseTestCase):
credentials = ['primary']
def __init__(self, *args, **kwargs):
super(BaseFreezerTest, self).__init__(*args, **kwargs)
# noinspection PyAttributeOutsideInit
def setUp(self):
super(BaseFreezerTest, self).setUp()
self.storage = Temp_Tree()
self.source_trees = []
self.backup_count = 0
self.backup_name = 'backup_test'
self.get_environ()
def tearDown(self):
super(BaseFreezerTest, self).tearDown()
for tree in self.source_trees:
tree.cleanup()
self.storage.cleanup()
@classmethod
def get_auth_url(cls):
return cls.os_primary.auth_provider.auth_client.auth_url[:-len(
'/auth/tokens')]
@classmethod
def setup_clients(cls):
super(BaseFreezerTest, cls).setup_clients()
cls.get_environ()
@classmethod
def get_environ(cls):
os.environ['OS_PASSWORD'] = cls.os_primary.credentials.password
os.environ['OS_USERNAME'] = cls.os_primary.credentials.username
os.environ['OS_PROJECT_NAME'] = cls.os_primary.credentials.tenant_name
os.environ['OS_TENANT_NAME'] = cls.os_primary.credentials.tenant_name
os.environ['OS_PROJECT_DOMAIN_NAME'] = \
cls.os_primary.credentials.project_domain_name
os.environ['OS_USER_DOMAIN_NAME'] = \
cls.os_primary.credentials.user_domain_name
# Allow developers to set OS_AUTH_URL when developing so that
# Keystone may be on a host other than localhost.
if 'OS_AUTH_URL' not in os.environ:
os.environ['OS_AUTH_URL'] = cls.get_auth_url()
# Mac OS X uses gtar located in /usr/local/bin
os.environ['PATH'] = '/usr/local/bin:' + os.environ['PATH']
return os.environ
def run_subprocess(self, sub_process_args, fail_message):
proc = subprocess.Popen(sub_process_args,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
env=self.environ, shell=False)
out, err = proc.communicate()
self.assertEqual(0, proc.returncode,
fail_message + " Output: {0}. "
"Error: {1}".format(out, err))
self.assertEqual('', err,
fail_message + " Output: {0}. "
"Error: {1}".format(out, err))
def create_local_backup(self, hostname=None, compression=None,
consistency_check=None, incremental=True,
always_level=None, restart_always_level=None,
max_level=None):
"""Creates a new backup with the given parameters.
The backup will immediately be created using a randomly-generated
source tree on the local filesystem, and will be stored in a random
temporary directory using the 'local' storage mode. All generated data
files will be automatically removed during `tearDown()`, though
implementations are responsible for cleaning up any additional copies
or restores created via other methods.
:param hostname: if set, set `--hostname` to the given value
:param compression: if set, set `--compression` to the given value
:param consistency_check: if True, set `--consistency_check`
:param incremental: if False, set `--no-incremental`
:param always_level: sets `--always-level` to the given value
:param restart_always_level: sets `--restart-always-level`
:param max_level: sets `--max-level` to the given value
:return: the path to the stored backup metadata
"""
metadata_path = os.path.join(
self.storage.path,
'metadata-{}.json'.format(self.backup_count))
self.backup_count += 1
tree = Temp_Tree()
tree.add_random_data()
self.source_trees.append(tree)
backup_args = [
'freezer-agent',
'--path-to-backup', tree.path,
'--container', self.storage.path,
'--backup-name', self.backup_name,
'--storage', 'local',
'--metadata-out', metadata_path,
]
if hostname:
backup_args += ['--hostname', hostname]
if compression:
backup_args += ['--compression', compression]
if consistency_check:
backup_args += ['--consistency-check']
if incremental:
if always_level is not None:
backup_args += ['--always-level', str(always_level)]
if max_level is not None:
backup_args += ['--max-level', str(max_level)]
if restart_always_level:
backup_args += ['--restart-always-level',
str(restart_always_level)]
else:
backup_args += ['--no-incremental', 'NO_INCREMENTAL']
self.run_subprocess(backup_args, 'Test backup to local storage.')
return metadata_path
def create_mutated_backup(self, days_old=30, **kwargs):
"""Create a local backup with a mutated timestamp
This creates a new backup using `create_local_backup()`, modifies it
using `mutate_timestamp()`, and then returns the resulting (loaded)
metadata dict.
:param days_old: the age of the backup to create
:param kwargs: arguments to pass to `create_local_backup()`
:return: the loaded metadata
"""
metadata_path = self.create_local_backup(**kwargs)
metadata = load_metadata(metadata_path)
mutate_timestamp(metadata, days_old)
save_metadata(metadata, metadata_path)
return metadata

View File

@ -0,0 +1,188 @@
# (C) Copyright 2016 Hewlett Packard Enterprise Development Company LP
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import json
import os
import subprocess
from tempest.lib import decorators
from freezer_tempest_plugin import common
from freezer_tempest_plugin.tests.api import base
class TestFreezerCompressGzip(base.BaseFreezerTest):
def __init__(self, *args, **kwargs):
super(TestFreezerCompressGzip, self).__init__(*args, **kwargs)
# noinspection PyAttributeOutsideInit
def setUp(self):
super(TestFreezerCompressGzip, self).setUp()
# create a source tree to backup with a few empty files
# (files must be empty to avoid encoding errors with pure random data)
self.source_tree = common.Temp_Tree()
self.source_tree.add_random_data(size=0)
self.storage_tree = common.Temp_Tree()
self.dest_tree = common.Temp_Tree()
self.environ = super(TestFreezerCompressGzip, self).get_environ()
def tearDown(self):
super(TestFreezerCompressGzip, self).tearDown()
self.source_tree.cleanup()
self.dest_tree.cleanup()
self.storage_tree.cleanup()
def _backup(self, name, method):
# perform a normal backup, with gzip specified
backup_args = ['freezer-agent',
'--path-to-backup',
self.source_tree.path,
'--container',
self.storage_tree.path,
'--backup-name',
name,
'--storage',
'local',
'--compress',
method,
'--metadata-out',
os.path.join(self.storage_tree.path, 'metadata.json')]
self.run_subprocess(backup_args, 'Test gzip backup to local storage.')
def _restore(self, name, method):
restore_args = ['freezer-agent',
'--action',
'restore',
'--restore-abs-path',
self.dest_tree.path,
'--container',
self.storage_tree.path,
'--backup-name',
name,
'--storage',
'local',
'--compress',
method]
self.run_subprocess(restore_args, 'Test restore from local storage.')
def _metadata(self):
path = os.path.join(self.storage_tree.path, 'metadata.json')
with open(path, 'r') as f:
return json.load(f)
def _file_get_mimetype(self, metadata):
"""Given some file metadata, find its mimetype using the file command
:param metadata: the parsed json file metadata
:return: the mimetype
"""
"""
Data is stored like data/tar/localhost_False/1469786264/0_1469786264 so
we need build the same directory structure.
data: the directory that holds the backup data
tar: the engine used to create backup
localhost: the hostname of the machine where the backup was taken
False: it should be backup name or False is backup is not provided
1469786264: timestamp
0_1469786264: level zero timestamp
"""
data_file_path = 'data{0}{1}{0}{2}_{3}{0}{4}{0}{5}_{4}{0}data'.format(
os.path.sep,
"tar", # currently we support only tar
metadata['hostname'],
metadata['backup_name'],
metadata['time_stamp'],
metadata['curr_backup_level']
)
data_file_path = os.path.join(self.storage_tree.path,
data_file_path)
self.assertEqual(True, os.path.exists(data_file_path))
# run 'file' in brief mode to only output the values we want
proc = subprocess.Popen(['file', '-b', '--mime-type', data_file_path],
stdout=subprocess.PIPE)
out, err = proc.communicate()
self.assertEqual(0, proc.returncode)
return out.strip()
@decorators.attr(type="gate")
def test_freezer_backup_compress_gzip(self):
backup_name = 'freezer-test-backup-gzip-0'
self._backup(backup_name, 'gzip')
self._restore(backup_name, 'gzip')
# metadata should show the correct algorithm
metadata = self._metadata()
self.assertIn('compression', metadata)
self.assertEqual('gzip', metadata['compression'])
# file utility should detect the correct mimetype
gizp_mimetypes = ['application/gzip', 'application/x-gzip']
mimetype = self._file_get_mimetype(metadata)
self.assertIn(mimetype, gizp_mimetypes)
# actual contents should be the same
diff_args = ['diff', '-r', '-q',
self.source_tree.path,
self.dest_tree.path]
self.run_subprocess(diff_args, 'Verify restored copy is identical to '
'original.')
@decorators.attr(type="gate")
def test_freezer_backup_compress_bzip2(self):
backup_name = 'freezer-test-backup-bzip2-0'
self._backup(backup_name, 'bzip2')
self._restore(backup_name, 'bzip2')
metadata = self._metadata()
self.assertIn('compression', metadata)
self.assertEqual('bzip2', metadata['compression'])
mimetype = self._file_get_mimetype(metadata)
self.assertEqual('application/x-bzip2', mimetype)
diff_args = ['diff', '-r', '-q',
self.source_tree.path,
self.dest_tree.path]
self.run_subprocess(diff_args, 'Verify restored copy is identical to '
'original.')
@decorators.attr(type="gate")
def test_freezer_backup_compress_xz(self):
backup_name = 'freezer-test-backup-xz-0'
self._backup(backup_name, 'xz')
self._restore(backup_name, 'xz')
metadata = self._metadata()
self.assertIn('compression', metadata)
self.assertEqual('xz', metadata['compression'])
mimetype = self._file_get_mimetype(metadata)
self.assertEqual('application/x-xz', mimetype)
diff_args = ['diff', '-r', '-q',
self.source_tree.path,
self.dest_tree.path]
self.run_subprocess(diff_args, 'Verify restored copy is identical to '
'original.')

View File

@ -0,0 +1,110 @@
# (C) Copyright 2016 Hewlett Packard Enterprise Development Company LP
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import os
import shutil
from oslo_utils import uuidutils
from tempest.lib import decorators
from freezer_tempest_plugin.tests.api import base
class TestFreezerFSBackup(base.BaseFreezerTest):
def __init__(self, *args, **kwargs):
super(TestFreezerFSBackup, self).__init__(*args, **kwargs)
def setUp(self):
super(TestFreezerFSBackup, self).setUp()
test_id = uuidutils.generate_uuid(dashed=False)
self.backup_source_dir = (
"/tmp/freezer-test-backup-source/" + test_id
)
self.backup_source_sub_dir = self.backup_source_dir + "/subdir"
self.restore_target_dir = (
"/tmp/freezer-test-backup-restore/" + test_id
)
self.backup_local_storage_dir = (
"/tmp/freezer-test-backup-local-storage/" + test_id
)
self.freezer_backup_name = 'freezer-test-backup-fs-0'
shutil.rmtree(self.backup_source_dir, True)
os.makedirs(self.backup_source_dir)
open(self.backup_source_dir + "/a", 'w').close()
open(self.backup_source_dir + "/b", 'w').close()
open(self.backup_source_dir + "/c", 'w').close()
os.makedirs(self.backup_source_sub_dir)
open(self.backup_source_sub_dir + "/x", 'w').close()
open(self.backup_source_sub_dir + "/y", 'w').close()
open(self.backup_source_sub_dir + "/z", 'w').close()
shutil.rmtree(self.restore_target_dir, True)
os.makedirs(self.restore_target_dir)
shutil.rmtree(self.backup_local_storage_dir, True)
os.makedirs(self.backup_local_storage_dir)
self.environ = super(TestFreezerFSBackup, self).get_environ()
def tearDown(self):
super(TestFreezerFSBackup, self).tearDown()
shutil.rmtree(self.backup_source_dir, True)
shutil.rmtree(self.restore_target_dir, True)
shutil.rmtree(self.backup_local_storage_dir)
@decorators.attr(type="gate")
def test_freezer_fs_backup(self):
backup_args = ['freezer-agent',
'--path-to-backup',
self.backup_source_dir,
'--container',
self.backup_local_storage_dir,
'--backup-name',
self.freezer_backup_name,
'--storage',
'local']
self.run_subprocess(backup_args, "Test backup to local storage.")
restore_args = ['freezer-agent',
'--action',
'restore',
'--restore-abs-path',
self.restore_target_dir,
'--container',
self.backup_local_storage_dir,
'--backup-name',
self.freezer_backup_name,
'--storage',
'local']
self.run_subprocess(restore_args, "Test restore from local storage.")
diff_args = ['diff',
'-r',
'-q',
self.backup_source_dir,
self.restore_target_dir]
self.run_subprocess(diff_args,
"Test backup restore from local storage "
"diff.")

View File

@ -0,0 +1,94 @@
# (C) Copyright 2016 Hewlett Packard Enterprise Development Company LP
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import subprocess
from tempest.lib import decorators
from freezer_tempest_plugin import common
from freezer_tempest_plugin.tests.api import base
class TestFreezerMetadataChecksum(base.BaseFreezerTest):
def __init__(self, *args, **kwargs):
super(TestFreezerMetadataChecksum, self).__init__(*args, **kwargs)
# noinspection PyAttributeOutsideInit
def setUp(self):
super(TestFreezerMetadataChecksum, self).setUp()
self.environ = super(TestFreezerMetadataChecksum, self).get_environ()
self.dest_tree = common.Temp_Tree()
self.backup_name = 'backup_checksum_test'
def tearDown(self):
super(TestFreezerMetadataChecksum, self).tearDown()
self.dest_tree.cleanup()
@decorators.attr(type="gate")
def test_freezer_fs_backup_valid_checksum(self):
# perform a normal backup, but enable consistency checks and save the
# metadata to disk
metadata_path = self.create_local_backup(consistency_check=True)
metadata = base.load_metadata(metadata_path)
# load the stored metadata to retrieve the computed checksum
self.assertIn('consistency_checksum', metadata,
'Checksum must exist in stored metadata.')
checksum = metadata['consistency_checksum']
restore_args = ['freezer-agent',
'--action', 'restore',
'--restore-abs-path', self.dest_tree.path,
'--container', metadata['container'],
'--backup-name', self.backup_name,
'--storage', 'local',
'--consistency-checksum', checksum]
self.run_subprocess(restore_args,
'Test restore from local storage with '
'computed checksum.')
@decorators.attr(type="gate")
def test_freezer_fs_backup_bad_checksum(self):
# as above, but we'll ignore the computed checksum
metadata_path = self.create_local_backup(consistency_check=True)
metadata = base.load_metadata(metadata_path)
# make a failing sha256 checksum (assuming no added path string)
bad_checksum = '0' * 64
# attempt to restore using the bad checksum
restore_args = ['freezer-agent',
'--action', 'restore',
'--restore-abs-path', self.dest_tree.path,
'--container', metadata['container'],
'--backup-name', self.backup_name,
'--storage', 'local',
'--consistency-checksum', bad_checksum]
process = subprocess.Popen(restore_args,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
env=self.environ, shell=False)
out, err = process.communicate()
# make sure the subprocess exist with an error due to checksum mismatch
message = '{0} Output: {1} Error: {2}'.format(
'Restore process should fail with checksum error.',
out, err)
self.assertEqual(1, process.returncode, message)
self.assertEqual('', out, message)
self.assertNotEqual('', err, message)

View File

@ -0,0 +1,100 @@
# (C) Copyright 2016 Hewlett Packard Enterprise Development Company LP
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import os
import shutil
from oslo_utils import uuidutils
from tempest.lib import decorators
from freezer_tempest_plugin.tests.api import base
class TestFreezerSwiftBackup(base.BaseFreezerTest):
def __init__(self, *args, **kwargs):
super(TestFreezerSwiftBackup, self).__init__(*args, **kwargs)
def setUp(self):
super(TestFreezerSwiftBackup, self).setUp()
test_id = uuidutils.generate_uuid(dashed=False)
self.backup_source_dir = (
"/tmp/freezer-test-backup-source/" + test_id
)
self.backup_source_sub_dir = self.backup_source_dir + "/subdir"
self.restore_target_dir = (
"/tmp/freezer-test-backup-restore/" + test_id
)
self.freezer_container_name = 'freezer-test-container-0'
self.freezer_backup_name = 'freezer-test-backup-swift-0'
shutil.rmtree(self.backup_source_dir, True)
os.makedirs(self.backup_source_dir)
open(self.backup_source_dir + "/a", 'w').close()
open(self.backup_source_dir + "/b", 'w').close()
open(self.backup_source_dir + "/c", 'w').close()
os.makedirs(self.backup_source_sub_dir)
open(self.backup_source_sub_dir + "/x", 'w').close()
open(self.backup_source_sub_dir + "/y", 'w').close()
open(self.backup_source_sub_dir + "/z", 'w').close()
shutil.rmtree(self.restore_target_dir, True)
os.makedirs(self.restore_target_dir)
self.environ = super(TestFreezerSwiftBackup, self).get_environ()
def tearDown(self):
super(TestFreezerSwiftBackup, self).tearDown()
shutil.rmtree(self.backup_source_dir, True)
shutil.rmtree(self.restore_target_dir, True)
@decorators.attr(type="gate")
def test_freezer_swift_backup(self):
backup_args = ['freezer-agent',
'--path-to-backup',
self.backup_source_dir,
'--container',
self.freezer_container_name,
'--backup-name',
self.freezer_backup_name]
self.run_subprocess(backup_args, "Test backup to swift.")
restore_args = ['freezer-agent',
'--action',
'restore',
'--restore-abs-path',
self.restore_target_dir,
'--container',
self.freezer_container_name,
'--backup-name',
self.freezer_backup_name,
'--storage',
'swift']
self.run_subprocess(restore_args, "Test restore from swift.")
diff_args = ['diff',
'-r',
'-q',
self.backup_source_dir,
self.restore_target_dir]
self.run_subprocess(diff_args,
"Test backup to swift and restore diff.")

View File

@ -0,0 +1,25 @@
# (C) Copyright 2016 Hewlett Packard Enterprise Development Company LP
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from tempest.lib import decorators
from freezer_tempest_plugin.tests.api import base
class TestFreezerTestsRunning(base.BaseFreezerTest):
@decorators.attr(type="gate")
def test_tests_running(self):
# See if tempest plugin tests run.
self.assertEqual(1, 1, 'Tests are running')

View File

@ -0,0 +1,305 @@
# (C) Copyright 2016 Hewlett Packard Enterprise Development Company LP
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import hashlib
import json
import os
import shutil
import tempfile
import time
from tempest.lib.cli import base as cli_base
from tempest.lib.cli import output_parser
from freezer_tempest_plugin.tests.api import base
JOB_TABLE_RESULT_COLUMN = 3
class BaseFreezerCliTest(base.BaseFreezerTest):
"""Base test case class for all Freezer API tests."""
credentials = ['primary']
@classmethod
def setup_clients(cls):
super(BaseFreezerCliTest, cls).setup_clients()
cls.cli = CLIClientWithFreezer(
username=cls.os_primary.credentials.username,
# fails if the password contains an unescaped $ sign
password=cls.os_primary.credentials.password.replace('$', '$$'),
tenant_name=cls.os_primary.credentials.tenant_name,
uri=cls.get_auth_url(),
cli_dir='/usr/local/bin' # devstack default
)
cls.cli.cli_dir = ''
def delete_job(self, job_id):
self.cli.freezer_client(action='job-delete', params=job_id)
def create_job(self, job_json):
with tempfile.NamedTemporaryFile(delete=False) as job_file:
job_file.write(json.dumps(job_json))
job_file.flush()
output = self.cli.freezer_client(
action='job-create',
params='--file {} --client {}'.format(job_file.name,
job_json['client_id']))
job_id = output.split()[1]
expected = 'Job {} created'.format(job_id)
self.assertEqual(expected, output.strip())
self.addCleanup(self.delete_job, job_id)
return job_id
def find_job_in_job_list(self, job_id):
job_list = output_parser.table(
self.cli.freezer_client(action='job-list', params='-C test_node'))
for row in job_list['values']:
if row[0].strip() == job_id.strip():
return row
self.fail('Could not find job: {}'.format(job_id))
def wait_for_job_status(self, job_id, timeout=720):
start = time.time()
while True:
row = self.find_job_in_job_list(job_id)
if row[JOB_TABLE_RESULT_COLUMN]:
return
elif time.time() - start > timeout:
self.fail("Status of job '{}' is '{}'."
.format(job_id, row[JOB_TABLE_RESULT_COLUMN]))
else:
time.sleep(1)
def assertJobColumnEqual(self, job_id, column, expected):
row = self.find_job_in_job_list(job_id)
self.assertEqual(expected, row[column])
class CLIClientWithFreezer(cli_base.CLIClient):
def freezer_scheduler(self, action, flags='', params='', fail_ok=False,
endpoint_type='publicURL', merge_stderr=False):
"""Executes freezer-scheduler command for the given action.
:param action: the cli command to run using freezer-scheduler
:type action: string
:param flags: any optional cli flags to use
:type flags: string
:param params: any optional positional args to use :type params: string
:param fail_ok: if True an exception is not raised when the
cli return code is non-zero
:type fail_ok: boolean
:param endpoint_type: the type of endpoint for the service
:type endpoint_type: string
:param merge_stderr: if True the stderr buffer is merged into stdout
:type merge_stderr: boolean
"""
flags += ' --os-endpoint-type %s' % endpoint_type
flags += ' --os-cacert /etc/ssl/certs/ca-certificates.crt'
flags += ' --os-project-domain-name Default'
flags += ' --os-user-domain-name Default'
return self.cmd_with_auth(
'freezer-scheduler', action, flags, params, fail_ok, merge_stderr)
def freezer_client(self, action, flags='', params='', fail_ok=False,
endpoint_type='publicURL', merge_stderr=True):
flags += ' --os-endpoint-type %s' % endpoint_type
flags += ' --os-cacert /etc/ssl/certs/ca-certificates.crt'
flags += ' --os-project-domain-name Default'
flags += ' --os-user-domain-name Default'
return self.cmd_with_auth(
'freezer', action, flags, params, fail_ok, merge_stderr)
# This class is just copied from the freezer repo. Depending on where the
# scenario tests end up we may need to refactore this.
class Temp_Tree(object):
def __init__(self, suffix='', dir=None, create=True):
self.create = create
if create:
self.path = tempfile.mkdtemp(dir=dir, prefix='__freezer_',
suffix=suffix)
else:
self.path = dir
self.files = []
def __enter__(self):
return self
def cleanup(self):
if self.create and self.path:
shutil.rmtree(self.path)
def __exit__(self, exc_type, exc_val, exc_tb):
self.cleanup()
def add_random_data(self, ndir=5, nfile=5, size=1024):
"""
add some files containing randoma data
:param ndir: number of dirs to create
:param nfile: number of files to create in each dir
:param size: size of files
:return: None
"""
for x in range(ndir):
subdir_path = tempfile.mkdtemp(dir=self.path)
for y in range(nfile):
abs_pathname = self.create_file_with_random_data(
dir_path=subdir_path, size=size)
rel_path_name = abs_pathname[len(self.path) + 1:]
self.files.append(rel_path_name)
def create_file_with_random_data(self, dir_path, size=1024):
handle, abs_pathname = tempfile.mkstemp(dir=dir_path)
with open(abs_pathname, 'wb') as fd:
fd.write(os.urandom(size))
return abs_pathname
def get_file_hash(self, rel_filepath):
filepath = os.path.join(self.path, rel_filepath)
if os.path.isfile(filepath):
return self._filehash(filepath)
else:
return ''
def _filehash(self, filepath):
"""
Get GIT style sha1 hash for a file
:param filepath: path of file to hash
:return: hash of the file
"""
filesize_bytes = os.path.getsize(filepath)
hash_obj = hashlib.sha1()
hash_obj.update(("blob %u\0" % filesize_bytes).encode('utf-8'))
with open(filepath, 'rb') as handle:
hash_obj.update(handle.read())
return hash_obj.hexdigest()
def get_file_list(self):
"""
walks the dir tree and creates a list of relative pathnames
:return: list of relative file paths
"""
self.files = []
for root, dirs, files in os.walk(self.path):
rel_base = root[len(self.path) + 1:]
self.files.extend([os.path.join(rel_base, x) for x in files])
return self.files
def is_equal(self, other_tree):
"""
Checks whether two dir tree contain the same files
It checks the number of files and the hash of each file.
NOTE: tox puts .coverage files in the temp folder (?)
:param other_tree: dir tree to compare with
:return: true if the dir trees contain the same files
"""
lh_files = [x for x in sorted(self.get_file_list())
if not x.startswith('.coverage')]
rh_files = [x for x in sorted(other_tree.get_file_list())
if not x.startswith('.coverage')]
if lh_files != rh_files:
return False
for fname in lh_files:
if os.path.isfile(fname):
if self.get_file_hash(fname) != \
other_tree.get_file_hash(fname):
return False
return True
class TestFreezerScenario(BaseFreezerCliTest):
def setUp(self):
super(TestFreezerScenario, self).setUp()
self.source_tree = Temp_Tree()
self.source_tree.add_random_data()
self.dest_tree = Temp_Tree()
self.cli.freezer_scheduler(action='start',
flags='-c test_node '
'-f /tmp/freezer_tempest_job_dir/')
def tearDown(self):
super(TestFreezerScenario, self).tearDown()
self.source_tree.cleanup()
self.dest_tree.cleanup()
self.cli.freezer_scheduler(action='stop',
flags='-c test_node '
'-f /tmp/freezer_tempest_job_dir/')
def test_simple_backup(self):
backup_job = {
"client_id": "test_node",
"job_actions": [
{
"freezer_action": {
"action": "backup",
"mode": "fs",
"storage": "local",
"backup_name": "backup1",
"path_to_backup": self.source_tree.path,
"container": "/tmp/freezer_test/",
},
"max_retries": 3,
"max_retries_interval": 60
}
],
"description": "a test backup"
}
restore_job = {
"client_id": "test_node",
"job_actions": [
{
"freezer_action": {
"action": "restore",
"storage": "local",
"restore_abs_path": self.dest_tree.path,
"backup_name": "backup1",
"container": "/tmp/freezer_test/",
},
"max_retries": 3,
"max_retries_interval": 60
}
],
"description": "a test restore"
}
backup_job_id = self.create_job(backup_job)
self.cli.freezer_client(action='job-start', params=backup_job_id)
self.wait_for_job_status(backup_job_id)
self.assertJobColumnEqual(backup_job_id, JOB_TABLE_RESULT_COLUMN,
'success')
restore_job_id = self.create_job(restore_job)
self.wait_for_job_status(restore_job_id)
self.assertJobColumnEqual(restore_job_id, JOB_TABLE_RESULT_COLUMN,
'success')
self.assertTrue(self.source_tree.is_equal(self.dest_tree))

View File

@ -3,3 +3,8 @@
# process, which may cause wedges in the gate later. # process, which may cause wedges in the gate later.
pbr>=2.0 # Apache-2.0 pbr>=2.0 # Apache-2.0
oslo.utils>=3.31.0 # Apache-2.0
oslo.config>=5.1.0 # Apache-2.0
paramiko>=2.0.0 # LGPLv2.1+
six>=1.10.0 # MIT
tempest>=17.1.0 # Apache-2.0

View File

@ -1,5 +1,5 @@
[metadata] [metadata]
name = freezer-tempest-plugin name = freezer_tempest_plugin
summary = Tempest plugin for the freezer project. summary = Tempest plugin for the freezer project.
description-file = description-file =
README.rst README.rst
@ -48,4 +48,4 @@ output_file = freezer_tempest_plugin/locale/freezer_tempest_plugin.pot
[entry_points] [entry_points]
tempest.test_plugins = tempest.test_plugins =
freezer_tests = freezer_tempest_plugin.plugin:FreezerTempestPlugin freezer_tests = freezer_tempest_plugin.plugin:FreezerTempestPlugin

View File

@ -4,13 +4,8 @@
hacking>=0.12.0,<0.13 # Apache-2.0 hacking>=0.12.0,<0.13 # Apache-2.0
coverage>=4.0,!=4.4 # Apache-2.0
python-subunit>=0.0.18 # Apache-2.0/BSD
sphinx!=1.6.1,>=1.5.1 # BSD sphinx!=1.6.1,>=1.5.1 # BSD
oslosphinx>=4.7.0 # Apache-2.0 oslosphinx>=4.7.0 # Apache-2.0
oslotest>=1.10.0 # Apache-2.0
testrepository>=0.0.18 # Apache-2.0/BSD
testtools>=1.4.0 # MIT
# releasenotes # releasenotes
reno>=1.8.0 # Apache-2.0 reno>=1.8.0 # Apache-2.0

View File

@ -32,9 +32,7 @@ commands =
commands = oslo_debug_helper {posargs} commands = oslo_debug_helper {posargs}
[flake8] [flake8]
# E123, E125 skipped as they are invalid PEP-8. ignore = H405,H404,H403,H401
show-source = True show-source = True
ignore = E123,E125 enable-extensions = H203,H106
builtins = _ exclude = .venv,.tox,dist,doc,test,*egg,releasenotes
exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,build