Reorganize test module structure

Moved test modules:
 tests/             ->  tests/unit
 test_ci/           ->  tests/ci
 tests_functional   ->  tests/functional
 rally/hacking      ->  tests/hacking

Add testing read me file

Co-Authored-By: Boris Pavlovic <boris@pavlovic.me>
Co-Authored-By: Andrey Kurilin <akurilin@mirantis.com>

Change-Id: I57c09d892da4adf863c358a4d63e3543b50d10b7
This commit is contained in:
Sergey Skripnick 2014-10-06 21:18:24 +03:00 committed by Andrey Kurilin
parent 6121b49e74
commit c81e8a5f8b
18 changed files with 1990 additions and 2 deletions

View File

@ -42,6 +42,7 @@ Deeper in Rally:
verify verify
installation installation
usage usage
testing
feature_requests feature_requests
user_stories user_stories

75
tests/README.rst Normal file
View File

@ -0,0 +1,75 @@
Testing
=======
Please, don't hesitate to write tests ;)
Unit tests
----------
*Files: /tests/unit/**
The goal of unit tests is to ensure that internal parts of the code work properly.
All internal methods should be fully covered by unit tests with a reasonable mocks usage.
About Rally unit tests:
- All `unit tests <http://en.wikipedia.org/wiki/Unit_testing>`_ are located inside /tests/unit/*
- Tests are written on top of: *testtools*, *fixtures* and *mock* libs
- `Tox <https://tox.readthedocs.org/en/latest/>`_ is used to run unit tests
To run unit tests locally::
$ pip install tox
$ tox
To run py26, py27 or pep8 only::
$ tox -e <name>
#NOTE: <name> is one of py26, py27 or pep8
To get test coverage::
$ tox -e cover
#NOTE: Results will be in /cover/index.html
To generate docs::
$ tox -e docs
#NOTE: Documentation will be in doc/source/_build/html/index.html
Functional tests
----------------
*Files: /tests/functional/**
The goal of `functional tests <https://en.wikipedia.org/wiki/Functional_testing>`_ is to check that everything works well together.
Fuctional tests use Rally API only and check responses without touching internal parts.
To run functional tests locally::
$ source openrc
$ rally deployment create --from-env --name testing
$ tox -e cli
#NOTE: openrc file with OpenStack admin credentials
Rally CI scripts
----------------
*Files: /tests/ci/**
This directory contains scripts and files related to the Rally CI system.
Rally Style Commandments
------------------------
*File: /tests/hacking/checks.py*
This module contains Rally specific hacking rules for checking commandments.

15
tests/ci/README.rst Normal file
View File

@ -0,0 +1,15 @@
===============
Rally Gate Jobs
===============
For each patch submitted for review on Gerrit, there is a set of tests called **gate jobs** to be run against it. These tests check whether the Rally code works correctly after applying the patch and provide additional guarantees that it won't break the software when it gets merged. Rally gate jobs contain tests checking the codestyle (via *pep8*), unit tests suites, functional tests and a set of Rally benchmark tasks that are executed against a real *devstack* deployment.
rally-gate.sh
-------------
This script runs a set of real Rally benchmark tasks and fetches their results in textual / visualized form (available via a special html page by clicking the corresponding job title in Gerrit). It checks that scenarios don't fail while being executed against a devstack deployment and also tests SLA criteria to ensure that benchmark tasks have completed successfully.
rally-integrated.sh
-------------------
This script runs a functional tests suite for Rally CLI. The tests call a range of Rally CLI commands and check that their output contains the expected data.

0
tests/ci/__init__.py Normal file
View File

59
tests/ci/rally-gate.sh Executable file
View File

@ -0,0 +1,59 @@
#!/bin/bash -ex
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
# This script is executed by post_test_hook function in desvstack gate.
PROJECT=`echo $ZUUL_PROJECT | cut -d \/ -f 2`
SCENARIO=$BASE/new/$PROJECT/rally-scenarios/${RALLY_SCENARIO}.yaml
PLUGINS_DIR=$BASE/new/$PROJECT/rally-scenarios/plugins
EXTRA_DIR=$BASE/new/$PROJECT/rally-scenarios/extra
RALLY_PLUGINS_DIR=~/.rally/plugins/scenarios/
mkdir -p $RALLY_PLUGINS_DIR
if [ -d $PLUGINS_DIR ]; then
cp -r $PLUGINS_DIR/*.py $RALLY_PLUGINS_DIR
fi
if [ -d $EXTRA_DIR ]; then
mkdir -p ~/.rally/extra
cp -r $EXTRA_DIR/* ~/.rally/extra/
touch ~/.rally/extra/fake-image.img
fi
env
set -o pipefail
rally use deployment --name devstack
rally deployment check
rally show flavors
rally show images
rally show networks
rally show secgroups
rally show keypairs
rally -v task start --task $SCENARIO
mkdir -p rally-plot/extra
cp $BASE/new/rally/tests/ci/rally-gate/index.html rally-plot/extra/index.html
cp $SCENARIO rally-plot/task.txt
tar -czf rally-plot/plugins.tar.gz -C $RALLY_PLUGINS_DIR .
rally task plot2html --out rally-plot/results.html
gzip -9 rally-plot/results.html
rally task results | python -m json.tool > rally-plot/results.json
gzip -9 rally-plot/results.json
rally task detailed > rally-plot/detailed.txt
gzip -9 rally-plot/detailed.txt
rally task detailed --iterations-data > rally-plot/detailed_with_iterations.txt
gzip -9 rally-plot/detailed_with_iterations.txt
rally task sla_check | tee rally-plot/sla.txt

20
tests/ci/test_install.sh Executable file
View File

@ -0,0 +1,20 @@
#!/bin/sh -e
#
# Copyright 2013: Mirantis Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
sudo ./install_rally.sh
rally deployment list

View File

@ -0,0 +1,73 @@
# Copyright 2013: Mirantis Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import json
import unittest
import mock
import test_cli_utils as utils
class DeploymentTestCase(unittest.TestCase):
def setUp(self):
super(DeploymentTestCase, self).setUp()
self.rally = utils.Rally()
def test_create_fromenv_list_endpoint(self):
with mock.patch.dict("os.environ", utils.TEST_ENV):
self.rally("deployment create --name t_create_env --fromenv")
self.assertIn("t_create_env", self.rally("deployment list"))
self.assertIn(utils.TEST_ENV["OS_AUTH_URL"],
self.rally("deployment endpoint"))
def test_create_fromfile(self):
with mock.patch.dict("os.environ", utils.TEST_ENV):
self.rally("deployment create --name t_create_env --fromenv")
with open("/tmp/.tmp.deployment", "w") as f:
f.write(self.rally("deployment config"))
self.rally("deployment create --name t_create_file "
"--filename /tmp/.tmp.deployment")
self.assertIn("t_create_file", self.rally("deployment list"))
def test_config(self):
with mock.patch.dict("os.environ", utils.TEST_ENV):
self.rally("deployment create --name t_create_env --fromenv")
config = json.loads(self.rally("deployment config"))
self.assertEqual(utils.TEST_ENV["OS_USERNAME"],
config["admin"]["username"])
self.assertEqual(utils.TEST_ENV["OS_PASSWORD"],
config["admin"]["password"])
self.assertEqual(utils.TEST_ENV["OS_TENANT_NAME"],
config["admin"]["tenant_name"])
self.assertEqual(utils.TEST_ENV["OS_AUTH_URL"],
config["auth_url"])
def test_destroy(self):
with mock.patch.dict("os.environ", utils.TEST_ENV):
self.rally("deployment create --name t_create_env --fromenv")
self.assertIn("t_create_env", self.rally("deployment list"))
self.rally("deployment destroy")
self.assertNotIn("t_create_env", self.rally("deployment list"))
def test_check_success(self):
self.assertTrue(self.rally("deployment check"))
def test_check_fail(self):
with mock.patch.dict("os.environ", utils.TEST_ENV):
self.rally("deployment create --name t_create_env --fromenv")
self.assertRaises(utils.RallyCmdError, self.rally,
("deployment check"))

View File

@ -0,0 +1,134 @@
# Copyright 2013: Mirantis Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import os
import unittest
import mock
import test_cli_utils as utils
class TaskTestCase(unittest.TestCase):
def _get_sample_task_config(self):
return {
"Dummy.dummy_random_fail_in_atomic": [
{
"runner": {
"type": "constant",
"times": 100,
"concurrency": 5
}
}
]
}
def test_status(self):
rally = utils.Rally()
cfg = self._get_sample_task_config()
config = utils.TaskConfig(cfg)
rally("task start --task %s" % config.filename)
self.assertIn("finished", rally("task status"))
def test_detailed(self):
rally = utils.Rally()
cfg = self._get_sample_task_config()
config = utils.TaskConfig(cfg)
rally("task start --task %s" % config.filename)
detailed = rally("task detailed")
self.assertIn("Dummy.dummy_random_fail_in_atomic", detailed)
self.assertIn("dummy_fail_test (2)", detailed)
detailed_iterations_data = rally("task detailed --iterations-data")
self.assertIn("2. dummy_fail_test (2)", detailed_iterations_data)
def test_results(self):
rally = utils.Rally()
cfg = self._get_sample_task_config()
config = utils.TaskConfig(cfg)
rally("task start --task %s" % config.filename)
self.assertIn("result", rally("task results"))
def test_plot2html(self):
rally = utils.Rally()
cfg = self._get_sample_task_config()
config = utils.TaskConfig(cfg)
rally("task start --task %s" % config.filename)
if os.path.exists("/tmp/test_plot.html"):
os.remove("/tmp/test_plot.html")
rally("task plot2html /tmp/test_plot")
self.assertTrue(os.path.exists("/tmp/test_plot.html"))
def test_delete(self):
rally = utils.Rally()
cfg = self._get_sample_task_config()
config = utils.TaskConfig(cfg)
rally("task start --task %s" % config.filename)
self.assertIn("finished", rally("task status"))
rally("task delete")
self.assertNotIn("finishe", rally("task list"))
# NOTE(oanufriev): Not implemented
def test_abort(self):
pass
class SLATestCase(unittest.TestCase):
def _get_sample_task_config(self, max_seconds_per_iteration=4,
max_failure_percent=0):
return {
"KeystoneBasic.create_and_list_users": [
{
"args": {
"name_length": 10
},
"runner": {
"type": "constant",
"times": 5,
"concurrency": 5
},
"sla": {
"max_seconds_per_iteration": max_seconds_per_iteration,
"max_failure_percent": max_failure_percent,
}
}
]
}
def test_sla_fail(self):
rally = utils.Rally()
cfg = self._get_sample_task_config(max_seconds_per_iteration=0.001)
config = utils.TaskConfig(cfg)
rally("task start --task %s" % config.filename)
self.assertRaises(utils.RallyCmdError, rally, "task sla_check")
def test_sla_success(self):
rally = utils.Rally()
config = utils.TaskConfig(self._get_sample_task_config())
rally("task start --task %s" % config.filename)
rally("task sla_check")
expected = [
{"benchmark": "KeystoneBasic.create_and_list_users",
"criterion": "max_seconds_per_iteration",
"detail": mock.ANY,
"pos": 0, "success": True},
{"benchmark": "KeystoneBasic.create_and_list_users",
"criterion": "max_failure_percent",
"detail": mock.ANY,
"pos": 0, "success": True},
]
data = rally("task sla_check --json", getjson=True)
self.assertEqual(expected, data)

13
tests/hacking/README.rst Normal file
View File

@ -0,0 +1,13 @@
Rally Style Commandments
========================
- Step 1: Read the OpenStack Style Commandments
http://docs.openstack.org/developer/hacking/
- Step 2: Read on
Rally Specific Commandments
---------------------------
- [N301] Ensure that ``assert_*`` methods from ``mock`` library is used correctly
- [N302] Sub-error of N301, related to nonexistent "assert_called"
- [N303] Sub-error of N301, related to nonexistent "assert_called_once"

View File

83
tests/hacking/checks.py Normal file
View File

@ -0,0 +1,83 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
Guidelines for writing new hacking checks
- Use only for Rally specific tests. OpenStack general tests
should be submitted to the common 'hacking' module.
- Pick numbers in the range N3xx. Find the current test with
the highest allocated number and then pick the next value.
- Keep the test method code in the source file ordered based
on the N3xx value.
- List the new rule in the top level HACKING.rst file
- Add test cases for each new rule to tests/test_hacking.py
"""
def _parse_assert_mock_str(line):
point = line.find('.assert_')
if point != -1:
end_pos = line[point:].find('(') + point
return point, line[point + 1: end_pos], line[: point]
else:
return None, None, None
def check_assert_methods_from_mock(logical_line, filename):
"""Ensure that ``assert_*`` methods from ``mock`` library is used correctly
N301 - base error number
N302 - related to nonexistent "assert_called"
N303 - related to nonexistent "assert_called_once"
"""
correct_names = ["assert_any_call", "assert_called_once_with",
"assert_called_with", "assert_has_calls"]
if 'tests/' in filename:
pos, method_name, obj_name = _parse_assert_mock_str(logical_line)
if pos:
if method_name not in correct_names:
error_number = "N301"
msg = ("%(error_number)s:'%(method)s' is not present in `mock`"
" library. %(custom_msg)s For more details, visit "
"http://www.voidspace.org.uk/python/mock/ .")
if method_name == "assert_called":
error_number = "N302"
custom_msg = ("Maybe, you should try to use "
"'assertTrue(%s.called)' instead." %
obj_name)
elif method_name == "assert_called_once":
# For more details, see a bug in Rally:
# https://bugs.launchpad.net/rally/+bug/1305991
error_number = "N303"
custom_msg = ("Maybe, you should try to use "
"'assertEqual(1, %(obj_name)s.call_count)' "
"or '%(obj_name)s.assert_called_once_with()'"
" instead." % {"obj_name": obj_name})
else:
custom_msg = ("Correct 'assert_*' methods: '%s'."
% "', '".join(correct_names))
yield (pos, msg % {
"error_number": error_number,
"method": method_name,
"custom_msg": custom_msg})
def factory(register):
register(check_assert_methods_from_mock)

0
tests/unit/__init__.py Normal file
View File

View File

View File

@ -0,0 +1,118 @@
# Copyright 2014: Mirantis Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import json
import mock
import os
import re
import traceback
import yaml
from rally.benchmark.scenarios import base
from rally.benchmark import engine
from tests.unit import test
class TaskSampleTestCase(test.TestCase):
samples_path = os.path.join(
os.path.dirname(__file__),
os.pardir, os.pardir, os.pardir,
"doc", "samples", "tasks")
@mock.patch("rally.benchmark.engine.BenchmarkEngine"
"._validate_config_semantic")
def test_schema_is_valid(self, mock_semantic):
scenarios = set()
for dirname, dirnames, filenames in os.walk(self.samples_path):
for filename in filenames:
full_path = os.path.join(dirname, filename)
# NOTE(hughsaunders): Skip non config files
# (bug https://bugs.launchpad.net/rally/+bug/1314369)
if not re.search('\.(ya?ml|json)$', filename, flags=re.I):
continue
with open(full_path) as task_file:
try:
task_config = yaml.safe_load(task_file.read())
eng = engine.BenchmarkEngine(task_config,
mock.MagicMock())
eng.validate()
except Exception:
print(traceback.format_exc())
self.assertTrue(False,
"Wrong task config %s" % full_path)
else:
scenarios.update(task_config.keys())
# TODO(boris-42): We should refactor scenarios framework add "_" to
# all non-benchmark methods.. Then this test will pass.
missing = set(base.Scenario.list_benchmark_scenarios()) - scenarios
self.assertEqual(missing, set([]),
"These scenarios don't have samples: %s" % missing)
def test_json_correct_syntax(self):
for dirname, dirnames, filenames in os.walk(self.samples_path):
for filename in filenames:
if not filename.endswith(".json"):
continue
full_path = os.path.join(dirname, filename)
with open(full_path) as task_file:
json.load(task_file)
def test_task_config_pair_existance(self):
inexistent_paths = []
for dirname, dirnames, filenames in os.walk(self.samples_path):
# iterate over unique config names
for sample_name in set(
f[:-5] for f in filenames
if f.endswith(".json") or f.endswith(".yaml")):
partial_path = os.path.join(dirname, sample_name)
yaml_path = partial_path + ".yaml"
json_path = partial_path + ".json"
if not os.path.exists(yaml_path):
inexistent_paths.append(yaml_path)
elif not os.path.exists(json_path):
inexistent_paths.append(json_path)
if inexistent_paths:
self.fail("Sample task configs are missing:\n%r" % inexistent_paths)
def test_task_config_pairs_equality(self):
for dirname, dirnames, filenames in os.walk(self.samples_path):
# iterate over unique config names
for sample_name in set(
f[:-5] for f in filenames
if f.endswith(".json") or f.endswith(".yaml")):
partial_path = os.path.join(dirname, sample_name)
yaml_path = partial_path + ".yaml"
json_path = partial_path + ".json"
if os.path.exists(yaml_path) and os.path.exists(json_path):
with open(json_path) as json_file:
with open(yaml_path) as yaml_file:
json_config = yaml.safe_load(json_file.read())
yaml_config = yaml.safe_load(yaml_file.read())
self.assertEqual(
json_config,
yaml_config,
"Sample task configs are not equal:\n%s\n%s" %
(yaml_path, json_path))

1289
tests/unit/fakes.py Normal file

File diff suppressed because it is too large Load Diff

46
tests/unit/test.py Normal file
View File

@ -0,0 +1,46 @@
# Copyright 2013: Mirantis Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import mock
from oslotest import base
from rally import db
from rally.openstack.common.fixture import config
class DatabaseFixture(config.Config):
"""Create clean DB before starting test."""
def setUp(self):
super(DatabaseFixture, self).setUp()
db.db_cleanup()
self.conf.set_default('connection', "sqlite://", group='database')
db.db_drop()
db.db_create()
class TestCase(base.BaseTestCase):
"""Test case base class for all unit tests."""
def setUp(self):
super(TestCase, self).setUp()
self.addCleanup(mock.patch.stopall)
class DBTestCase(TestCase):
"""Base class for tests which use DB."""
def setUp(self):
super(TestCase, self).setUp()
self.useFixture(DatabaseFixture())

View File

@ -0,0 +1,62 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from tests.hacking import checks
from tests.unit import test
class HackingTestCase(test.TestCase):
def test__parse_assert_mock_str(self):
pos, method, obj = checks._parse_assert_mock_str(
"mock_clients.fake().quotas.delete.assert_called_once()")
self.assertEqual("assert_called_once", method)
self.assertEqual("mock_clients.fake().quotas.delete", obj)
def test__parse_assert_mock_str_no_assert(self):
pos, method, obj = checks._parse_assert_mock_str(
"mock_clients.fake().quotas.delete.")
self.assertIsNone(pos)
self.assertIsNone(method)
self.assertIsNone(obj)
def test_correct_usage_of_assert_from_mock(self):
correct_method_names = ["assert_any_call", "assert_called_once_with",
"assert_called_with", "assert_has_calls"]
for name in correct_method_names:
self.assertEqual(0, len(
list(checks.check_assert_methods_from_mock(
'some_mock.%s(asd)' % name, 'tests/fake/test'))))
def test_wrong_usage_of_broad_assert_from_mock(self):
fake_method = 'rtfm.assert_something()'
actual_number, actual_msg = next(checks.check_assert_methods_from_mock(
fake_method, 'tests/fake/test'))
self.assertEqual(4, actual_number)
self.assertTrue(actual_msg.startswith('N301'))
def test_wrong_usage_of_assert_called_from_mock(self):
fake_method = 'rtfm.assert_called()'
actual_number, actual_msg = next(checks.check_assert_methods_from_mock(
fake_method, 'tests/fake/test'))
self.assertEqual(4, actual_number)
self.assertTrue(actual_msg.startswith('N302'))
def test_wrong_usage_of_assert_called_once_from_mock(self):
fake_method = 'rtfm.assert_called_once()'
actual_number, actual_msg = next(checks.check_assert_methods_from_mock(
fake_method, 'tests/fake/test'))
self.assertEqual(4, actual_number)
self.assertTrue(actual_msg.startswith('N303'))

View File

@ -25,7 +25,7 @@ commands = {posargs}
[testenv:cli] [testenv:cli]
sitepackages = True sitepackages = True
commands = {toxinidir}/tests_ci/rally-integrated.sh commands = {toxinidir}/tests/ci/rally-integrated.sh
[testenv:cover] [testenv:cover]
commands = python setup.py testr --coverage --testr-args='{posargs}' commands = python setup.py testr --coverage --testr-args='{posargs}'
@ -45,4 +45,4 @@ exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,tools,*rally/verification/ver
[hacking] [hacking]
import_exceptions = rally.openstack.common.gettextutils._ import_exceptions = rally.openstack.common.gettextutils._
local-check-factory = rally.hacking.checks.factory local-check-factory = tests.hacking.checks.factory