Reorganize test module structure

Moved test modules:
 tests/             ->  tests/unit
 test_ci/           ->  tests/ci
 tests_functional   ->  tests/functional
 rally/hacking      ->  tests/hacking

Add testing read me file

Co-Authored-By: Boris Pavlovic <boris@pavlovic.me>
Co-Authored-By: Andrey Kurilin <akurilin@mirantis.com>

Change-Id: I57c09d892da4adf863c358a4d63e3543b50d10b7

doc/source/index.rst

@@ -42,6 +42,7 @@ Deeper in Rally:
    verify
    installation
    usage
+   testing
    feature_requests
    user_stories
 

tests/README.rst

@@ -0,0 +1,75 @@
+Testing
+=======
+
+Please, don't hesitate to write tests ;)
+
+
+Unit tests
+----------
+
+*Files: /tests/unit/**
+
+The goal of unit tests is to ensure that internal parts of the code work properly.
+All internal methods should be fully covered by unit tests with a reasonable mocks usage.
+
+
+About Rally unit tests:
+
+- All `unit tests <http://en.wikipedia.org/wiki/Unit_testing>`_ are located inside /tests/unit/*
+- Tests are written on top of: *testtools*, *fixtures* and *mock* libs
+- `Tox <https://tox.readthedocs.org/en/latest/>`_ is used to run unit tests
+
+
+To run unit tests locally::
+
+  $ pip install tox
+  $ tox
+
+To run py26, py27 or pep8 only::
+
+  $ tox -e <name>
+
+  #NOTE: <name> is one of py26, py27 or pep8
+
+To get test coverage::
+
+  $ tox -e cover
+
+  #NOTE: Results will be in /cover/index.html
+
+To generate docs::
+
+  $ tox -e docs
+
+  #NOTE: Documentation will be in doc/source/_build/html/index.html
+
+Functional tests
+----------------
+
+*Files: /tests/functional/**
+
+The goal of `functional tests <https://en.wikipedia.org/wiki/Functional_testing>`_ is to check that everything works well together.
+Fuctional tests use Rally API only and check responses without touching internal parts.
+
+To run functional tests locally::
+
+  $ source openrc
+  $ rally deployment create --from-env --name testing
+  $ tox -e cli
+
+  #NOTE: openrc file with OpenStack admin credentials
+
+Rally CI scripts
+----------------
+
+*Files: /tests/ci/**
+
+This directory contains scripts and files related to the Rally CI system.
+
+Rally Style Commandments
+------------------------
+
+*File: /tests/hacking/checks.py*
+
+This module contains Rally specific hacking rules for checking commandments.
+

tests/ci/README.rst

@@ -0,0 +1,15 @@
+===============
+Rally Gate Jobs
+===============
+
+For each patch submitted for review on Gerrit, there is a set of tests called **gate jobs** to be run against it. These tests check whether the Rally code works correctly after applying the patch and provide additional guarantees that it won't break the software when it gets merged. Rally gate jobs contain tests checking the codestyle (via *pep8*), unit tests suites, functional tests and a set of Rally benchmark tasks that are executed against a real *devstack* deployment.
+
+
+rally-gate.sh
+-------------
+This script runs a set of real Rally benchmark tasks and fetches their results in textual / visualized form (available via a special html page by clicking the corresponding job title in Gerrit). It checks that scenarios don't fail while being executed against a devstack deployment and also tests SLA criteria to ensure that benchmark tasks have completed successfully.
+
+
+rally-integrated.sh
+-------------------
+This script runs a functional tests suite for Rally CLI. The tests call a range of Rally CLI commands and check that their output contains the expected data.

tests/ci/rally-gate.sh

@@ -0,0 +1,59 @@
+#!/bin/bash -ex
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+# This script is executed by post_test_hook function in desvstack gate.
+
+PROJECT=`echo $ZUUL_PROJECT | cut -d \/ -f 2`
+SCENARIO=$BASE/new/$PROJECT/rally-scenarios/${RALLY_SCENARIO}.yaml
+PLUGINS_DIR=$BASE/new/$PROJECT/rally-scenarios/plugins
+EXTRA_DIR=$BASE/new/$PROJECT/rally-scenarios/extra
+
+RALLY_PLUGINS_DIR=~/.rally/plugins/scenarios/
+
+mkdir -p $RALLY_PLUGINS_DIR
+if [ -d $PLUGINS_DIR ]; then
+    cp -r $PLUGINS_DIR/*.py $RALLY_PLUGINS_DIR
+fi
+
+if [ -d $EXTRA_DIR ]; then
+ mkdir -p ~/.rally/extra
+ cp -r $EXTRA_DIR/* ~/.rally/extra/
+ touch ~/.rally/extra/fake-image.img
+fi
+
+env
+set -o pipefail
+
+rally use deployment --name devstack
+rally deployment check
+rally show flavors
+rally show images
+rally show networks
+rally show secgroups
+rally show keypairs
+rally -v task start --task $SCENARIO
+
+mkdir -p rally-plot/extra
+cp $BASE/new/rally/tests/ci/rally-gate/index.html rally-plot/extra/index.html
+cp $SCENARIO rally-plot/task.txt
+tar -czf rally-plot/plugins.tar.gz -C $RALLY_PLUGINS_DIR .
+rally task plot2html --out rally-plot/results.html
+gzip -9 rally-plot/results.html
+rally task results | python -m json.tool > rally-plot/results.json
+gzip -9 rally-plot/results.json
+rally task detailed > rally-plot/detailed.txt
+gzip -9 rally-plot/detailed.txt
+rally task detailed --iterations-data > rally-plot/detailed_with_iterations.txt
+gzip -9 rally-plot/detailed_with_iterations.txt
+rally task sla_check | tee rally-plot/sla.txt

tests/ci/test_install.sh

@@ -0,0 +1,20 @@
+#!/bin/sh -e
+#
+# Copyright 2013: Mirantis Inc.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+
+sudo ./install_rally.sh
+rally deployment list

tests/functional/test_cli_deployment.py

@@ -0,0 +1,73 @@
+# Copyright 2013: Mirantis Inc.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+
+import json
+import unittest
+
+import mock
+import test_cli_utils as utils
+
+
+class DeploymentTestCase(unittest.TestCase):
+
+    def setUp(self):
+        super(DeploymentTestCase, self).setUp()
+        self.rally = utils.Rally()
+
+    def test_create_fromenv_list_endpoint(self):
+        with mock.patch.dict("os.environ", utils.TEST_ENV):
+            self.rally("deployment create --name t_create_env --fromenv")
+        self.assertIn("t_create_env", self.rally("deployment list"))
+        self.assertIn(utils.TEST_ENV["OS_AUTH_URL"],
+                      self.rally("deployment endpoint"))
+
+    def test_create_fromfile(self):
+        with mock.patch.dict("os.environ", utils.TEST_ENV):
+            self.rally("deployment create --name t_create_env --fromenv")
+        with open("/tmp/.tmp.deployment", "w") as f:
+            f.write(self.rally("deployment config"))
+        self.rally("deployment create --name t_create_file "
+                   "--filename /tmp/.tmp.deployment")
+        self.assertIn("t_create_file", self.rally("deployment list"))
+
+    def test_config(self):
+        with mock.patch.dict("os.environ", utils.TEST_ENV):
+            self.rally("deployment create --name t_create_env --fromenv")
+        config = json.loads(self.rally("deployment config"))
+        self.assertEqual(utils.TEST_ENV["OS_USERNAME"],
+                         config["admin"]["username"])
+        self.assertEqual(utils.TEST_ENV["OS_PASSWORD"],
+                         config["admin"]["password"])
+        self.assertEqual(utils.TEST_ENV["OS_TENANT_NAME"],
+                         config["admin"]["tenant_name"])
+        self.assertEqual(utils.TEST_ENV["OS_AUTH_URL"],
+                         config["auth_url"])
+
+    def test_destroy(self):
+        with mock.patch.dict("os.environ", utils.TEST_ENV):
+            self.rally("deployment create --name t_create_env --fromenv")
+        self.assertIn("t_create_env", self.rally("deployment list"))
+        self.rally("deployment destroy")
+        self.assertNotIn("t_create_env", self.rally("deployment list"))
+
+    def test_check_success(self):
+        self.assertTrue(self.rally("deployment check"))
+
+    def test_check_fail(self):
+        with mock.patch.dict("os.environ", utils.TEST_ENV):
+            self.rally("deployment create --name t_create_env --fromenv")
+        self.assertRaises(utils.RallyCmdError, self.rally,
+                          ("deployment check"))

tests/functional/test_cli_task.py

@@ -0,0 +1,134 @@
+# Copyright 2013: Mirantis Inc.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+
+import os
+import unittest
+
+import mock
+import test_cli_utils as utils
+
+
+class TaskTestCase(unittest.TestCase):
+
+    def _get_sample_task_config(self):
+        return {
+            "Dummy.dummy_random_fail_in_atomic": [
+                {
+                    "runner": {
+                        "type": "constant",
+                        "times": 100,
+                        "concurrency": 5
+                    }
+                }
+            ]
+        }
+
+    def test_status(self):
+        rally = utils.Rally()
+        cfg = self._get_sample_task_config()
+        config = utils.TaskConfig(cfg)
+        rally("task start --task %s" % config.filename)
+        self.assertIn("finished", rally("task status"))
+
+    def test_detailed(self):
+        rally = utils.Rally()
+        cfg = self._get_sample_task_config()
+        config = utils.TaskConfig(cfg)
+        rally("task start --task %s" % config.filename)
+        detailed = rally("task detailed")
+        self.assertIn("Dummy.dummy_random_fail_in_atomic", detailed)
+        self.assertIn("dummy_fail_test (2)", detailed)
+        detailed_iterations_data = rally("task detailed --iterations-data")
+        self.assertIn("2. dummy_fail_test (2)", detailed_iterations_data)
+
+    def test_results(self):
+        rally = utils.Rally()
+        cfg = self._get_sample_task_config()
+        config = utils.TaskConfig(cfg)
+        rally("task start --task %s" % config.filename)
+        self.assertIn("result", rally("task results"))
+
+    def test_plot2html(self):
+        rally = utils.Rally()
+        cfg = self._get_sample_task_config()
+        config = utils.TaskConfig(cfg)
+        rally("task start --task %s" % config.filename)
+        if os.path.exists("/tmp/test_plot.html"):
+            os.remove("/tmp/test_plot.html")
+        rally("task plot2html /tmp/test_plot")
+        self.assertTrue(os.path.exists("/tmp/test_plot.html"))
+
+    def test_delete(self):
+        rally = utils.Rally()
+        cfg = self._get_sample_task_config()
+        config = utils.TaskConfig(cfg)
+        rally("task start --task %s" % config.filename)
+        self.assertIn("finished", rally("task status"))
+        rally("task delete")
+        self.assertNotIn("finishe", rally("task list"))
+
+    # NOTE(oanufriev): Not implemented
+    def test_abort(self):
+        pass
+
+
+class SLATestCase(unittest.TestCase):
+
+    def _get_sample_task_config(self, max_seconds_per_iteration=4,
+                                max_failure_percent=0):
+        return {
+            "KeystoneBasic.create_and_list_users": [
+                {
+                    "args": {
+                        "name_length": 10
+                    },
+                    "runner": {
+                        "type": "constant",
+                        "times": 5,
+                        "concurrency": 5
+                    },
+                    "sla": {
+                        "max_seconds_per_iteration": max_seconds_per_iteration,
+                        "max_failure_percent": max_failure_percent,
+                    }
+                }
+            ]
+        }
+
+    def test_sla_fail(self):
+        rally = utils.Rally()
+        cfg = self._get_sample_task_config(max_seconds_per_iteration=0.001)
+        config = utils.TaskConfig(cfg)
+        rally("task start --task %s" % config.filename)
+        self.assertRaises(utils.RallyCmdError, rally, "task sla_check")
+
+    def test_sla_success(self):
+        rally = utils.Rally()
+        config = utils.TaskConfig(self._get_sample_task_config())
+        rally("task start --task %s" % config.filename)
+        rally("task sla_check")
+        expected = [
+                {"benchmark": "KeystoneBasic.create_and_list_users",
+                 "criterion": "max_seconds_per_iteration",
+                 "detail": mock.ANY,
+                 "pos": 0, "success": True},
+                {"benchmark": "KeystoneBasic.create_and_list_users",
+                 "criterion": "max_failure_percent",
+                 "detail": mock.ANY,
+                 "pos": 0, "success": True},
+        ]
+        data = rally("task sla_check --json", getjson=True)
+        self.assertEqual(expected, data)

tests/hacking/README.rst

@@ -0,0 +1,13 @@
+Rally Style Commandments
+========================
+
+- Step 1: Read the OpenStack Style Commandments
+  http://docs.openstack.org/developer/hacking/
+- Step 2: Read on
+
+Rally Specific Commandments
+---------------------------
+
+- [N301] Ensure that ``assert_*`` methods from ``mock`` library is used correctly
+- [N302] Sub-error of N301, related to nonexistent "assert_called"
+- [N303] Sub-error of N301, related to nonexistent "assert_called_once"

tests/hacking/checks.py

@@ -0,0 +1,83 @@
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+"""
+Guidelines for writing new hacking checks
+
+ - Use only for Rally specific tests. OpenStack general tests
+   should be submitted to the common 'hacking' module.
+ - Pick numbers in the range N3xx. Find the current test with
+   the highest allocated number and then pick the next value.
+ - Keep the test method code in the source file ordered based
+   on the N3xx value.
+ - List the new rule in the top level HACKING.rst file
+ - Add test cases for each new rule to tests/test_hacking.py
+
+"""
+
+
+def _parse_assert_mock_str(line):
+    point = line.find('.assert_')
+
+    if point != -1:
+        end_pos = line[point:].find('(') + point
+        return point, line[point + 1: end_pos], line[: point]
+    else:
+        return None, None, None
+
+
+def check_assert_methods_from_mock(logical_line, filename):
+    """Ensure that ``assert_*`` methods from ``mock`` library is used correctly
+
+    N301 - base error number
+    N302 - related to nonexistent "assert_called"
+    N303 - related to nonexistent "assert_called_once"
+    """
+
+    correct_names = ["assert_any_call", "assert_called_once_with",
+                     "assert_called_with", "assert_has_calls"]
+
+    if 'tests/' in filename:
+        pos, method_name, obj_name = _parse_assert_mock_str(logical_line)
+
+        if pos:
+            if method_name not in correct_names:
+                error_number = "N301"
+                msg = ("%(error_number)s:'%(method)s' is not present in `mock`"
+                       " library. %(custom_msg)s For more details, visit "
+                       "http://www.voidspace.org.uk/python/mock/ .")
+
+                if method_name == "assert_called":
+                    error_number = "N302"
+                    custom_msg = ("Maybe, you should try to use "
+                                  "'assertTrue(%s.called)' instead." %
+                                  obj_name)
+                elif method_name == "assert_called_once":
+                    # For more details, see a bug in Rally:
+                    #    https://bugs.launchpad.net/rally/+bug/1305991
+                    error_number = "N303"
+                    custom_msg = ("Maybe, you should try to use "
+                                  "'assertEqual(1, %(obj_name)s.call_count)' "
+                                  "or '%(obj_name)s.assert_called_once_with()'"
+                                  " instead." % {"obj_name": obj_name})
+                else:
+                    custom_msg = ("Correct 'assert_*' methods: '%s'."
+                                  % "', '".join(correct_names))
+
+                yield (pos, msg % {
+                    "error_number": error_number,
+                    "method": method_name,
+                    "custom_msg": custom_msg})
+
+
+def factory(register):
+    register(check_assert_methods_from_mock)

tests/unit/doc/test_task_samples.py

@@ -0,0 +1,118 @@
+# Copyright 2014: Mirantis Inc.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import json
+import mock
+import os
+import re
+import traceback
+
+import yaml
+
+from rally.benchmark.scenarios import base
+from rally.benchmark import engine
+from tests.unit import test
+
+
+class TaskSampleTestCase(test.TestCase):
+    samples_path = os.path.join(
+        os.path.dirname(__file__),
+        os.pardir, os.pardir, os.pardir,
+        "doc", "samples", "tasks")
+
+    @mock.patch("rally.benchmark.engine.BenchmarkEngine"
+                "._validate_config_semantic")
+    def test_schema_is_valid(self, mock_semantic):
+        scenarios = set()
+
+        for dirname, dirnames, filenames in os.walk(self.samples_path):
+            for filename in filenames:
+                full_path = os.path.join(dirname, filename)
+
+                # NOTE(hughsaunders): Skip non config files
+                # (bug https://bugs.launchpad.net/rally/+bug/1314369)
+                if not re.search('\.(ya?ml|json)$', filename, flags=re.I):
+                    continue
+
+                with open(full_path) as task_file:
+                    try:
+                        task_config = yaml.safe_load(task_file.read())
+                        eng = engine.BenchmarkEngine(task_config,
+                                                     mock.MagicMock())
+                        eng.validate()
+                    except Exception:
+                        print(traceback.format_exc())
+                        self.assertTrue(False,
+                                        "Wrong task config %s" % full_path)
+                    else:
+                        scenarios.update(task_config.keys())
+
+        # TODO(boris-42): We should refactor scenarios framework add "_" to
+        #                 all non-benchmark methods.. Then this test will pass.
+        missing = set(base.Scenario.list_benchmark_scenarios()) - scenarios
+        self.assertEqual(missing, set([]),
+                         "These scenarios don't have samples: %s" % missing)
+
+    def test_json_correct_syntax(self):
+        for dirname, dirnames, filenames in os.walk(self.samples_path):
+            for filename in filenames:
+                if not filename.endswith(".json"):
+                    continue
+                full_path = os.path.join(dirname, filename)
+                with open(full_path) as task_file:
+                    json.load(task_file)
+
+    def test_task_config_pair_existance(self):
+        inexistent_paths = []
+
+        for dirname, dirnames, filenames in os.walk(self.samples_path):
+            # iterate over unique config names
+            for sample_name in set(
+                    f[:-5] for f in filenames
+                    if f.endswith(".json") or f.endswith(".yaml")):
+
+                partial_path = os.path.join(dirname, sample_name)
+                yaml_path = partial_path + ".yaml"
+                json_path = partial_path + ".json"
+
+                if not os.path.exists(yaml_path):
+                    inexistent_paths.append(yaml_path)
+                elif not os.path.exists(json_path):
+                    inexistent_paths.append(json_path)
+
+        if inexistent_paths:
+            self.fail("Sample task configs are missing:\n%r" % inexistent_paths)
+
+    def test_task_config_pairs_equality(self):
+        for dirname, dirnames, filenames in os.walk(self.samples_path):
+            # iterate over unique config names
+            for sample_name in set(
+                    f[:-5] for f in filenames
+                    if f.endswith(".json") or f.endswith(".yaml")):
+
+                partial_path = os.path.join(dirname, sample_name)
+                yaml_path = partial_path + ".yaml"
+                json_path = partial_path + ".json"
+
+                if os.path.exists(yaml_path) and os.path.exists(json_path):
+                    with open(json_path) as json_file:
+                        with open(yaml_path) as yaml_file:
+                            json_config = yaml.safe_load(json_file.read())
+                            yaml_config = yaml.safe_load(yaml_file.read())
+                            self.assertEqual(
+                                json_config,
+                                yaml_config,
+                                "Sample task configs are not equal:\n%s\n%s" %
+                                (yaml_path, json_path))

tests/unit/test.py

@@ -0,0 +1,46 @@
+# Copyright 2013: Mirantis Inc.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import mock
+from oslotest import base
+
+from rally import db
+from rally.openstack.common.fixture import config
+
+
+class DatabaseFixture(config.Config):
+    """Create clean DB before starting test."""
+    def setUp(self):
+        super(DatabaseFixture, self).setUp()
+        db.db_cleanup()
+        self.conf.set_default('connection', "sqlite://", group='database')
+        db.db_drop()
+        db.db_create()
+
+
+class TestCase(base.BaseTestCase):
+    """Test case base class for all unit tests."""
+
+    def setUp(self):
+        super(TestCase, self).setUp()
+        self.addCleanup(mock.patch.stopall)
+
+
+class DBTestCase(TestCase):
+    """Base class for tests which use DB."""
+
+    def setUp(self):
+        super(TestCase, self).setUp()
+        self.useFixture(DatabaseFixture())

tests/unit/test_hacking.py

@@ -0,0 +1,62 @@
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from tests.hacking import checks
+from tests.unit import test
+
+
+class HackingTestCase(test.TestCase):
+
+    def test__parse_assert_mock_str(self):
+        pos, method, obj = checks._parse_assert_mock_str(
+            "mock_clients.fake().quotas.delete.assert_called_once()")
+        self.assertEqual("assert_called_once", method)
+        self.assertEqual("mock_clients.fake().quotas.delete", obj)
+
+    def test__parse_assert_mock_str_no_assert(self):
+        pos, method, obj = checks._parse_assert_mock_str(
+            "mock_clients.fake().quotas.delete.")
+        self.assertIsNone(pos)
+        self.assertIsNone(method)
+        self.assertIsNone(obj)
+
+    def test_correct_usage_of_assert_from_mock(self):
+        correct_method_names = ["assert_any_call", "assert_called_once_with",
+                                "assert_called_with", "assert_has_calls"]
+        for name in correct_method_names:
+            self.assertEqual(0, len(
+                list(checks.check_assert_methods_from_mock(
+                    'some_mock.%s(asd)' % name, 'tests/fake/test'))))
+
+    def test_wrong_usage_of_broad_assert_from_mock(self):
+        fake_method = 'rtfm.assert_something()'
+
+        actual_number, actual_msg = next(checks.check_assert_methods_from_mock(
+            fake_method, 'tests/fake/test'))
+        self.assertEqual(4, actual_number)
+        self.assertTrue(actual_msg.startswith('N301'))
+
+    def test_wrong_usage_of_assert_called_from_mock(self):
+        fake_method = 'rtfm.assert_called()'
+
+        actual_number, actual_msg = next(checks.check_assert_methods_from_mock(
+            fake_method, 'tests/fake/test'))
+        self.assertEqual(4, actual_number)
+        self.assertTrue(actual_msg.startswith('N302'))
+
+    def test_wrong_usage_of_assert_called_once_from_mock(self):
+        fake_method = 'rtfm.assert_called_once()'
+
+        actual_number, actual_msg = next(checks.check_assert_methods_from_mock(
+            fake_method, 'tests/fake/test'))
+        self.assertEqual(4, actual_number)
+        self.assertTrue(actual_msg.startswith('N303'))

tox.ini

@@ -25,7 +25,7 @@ commands = {posargs}
 
 [testenv:cli]
 sitepackages = True
-commands = {toxinidir}/tests_ci/rally-integrated.sh
+commands = {toxinidir}/tests/ci/rally-integrated.sh
 
 [testenv:cover]
 commands = python setup.py testr --coverage --testr-args='{posargs}'
@@ -45,4 +45,4 @@ exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,tools,*rally/verification/ver
 
 [hacking]
 import_exceptions = rally.openstack.common.gettextutils._
-local-check-factory = rally.hacking.checks.factory
+local-check-factory = tests.hacking.checks.factory