From 61238e2e544e5ff5ecb288c26b4bf57953ec1c56 Mon Sep 17 00:00:00 2001 From: Sean McGinnis Date: Thu, 9 Jun 2016 11:23:44 -0500 Subject: [PATCH] Add test type clarification to devref We now have unit, functional, and tempest tests in tree. To avoid confusion, we need to make sure it's clear what each of these test types are intended to be used for. This adds a description for each test type and cleans up some of the documentation around testing in general. Change-Id: I15bba187889e058adbf03deb4308a41d0a6248e3 --- cinder/testing/README.rst | 35 --- doc/source/devref/development.environment.rst | 13 +- doc/source/devref/index.rst | 5 +- doc/source/devref/testing.rst | 158 +++++++++++++ doc/source/devref/unit_tests.rst | 209 ------------------ 5 files changed, 165 insertions(+), 255 deletions(-) delete mode 100644 cinder/testing/README.rst create mode 100644 doc/source/devref/testing.rst delete mode 100644 doc/source/devref/unit_tests.rst diff --git a/cinder/testing/README.rst b/cinder/testing/README.rst deleted file mode 100644 index e83a7cea4..000000000 --- a/cinder/testing/README.rst +++ /dev/null @@ -1,35 +0,0 @@ -===================================== -OpenStack Cinder Testing Infrastructure -===================================== - -A note of clarification is in order, to help those who are new to testing in -OpenStack cinder: - -- actual unit tests are created in the "tests" directory; -- the "testing" directory is used to house the infrastructure needed to support - testing in OpenStack Cinder. - -This README file attempts to provide current and prospective contributors with -everything they need to know in order to start creating unit tests and -utilizing the convenience code provided in cinder.testing. - -For more detailed information on cinder unit tests visit: -http://docs.openstack.org/developer/cinder/devref/unit_tests.html - -Running Tests ------------------------------------------------ - -In the root of the cinder source code run the run_tests.sh script. This will -offer to create a virtual environment and populate it with dependencies. -If you don't have dependencies installed that are needed for compiling cinder's -direct dependencies, you'll have to use your operating system's method of -installing extra dependencies. To get help using this script execute it with -the -h parameter to get options `./run_tests.sh -h` - -Writing Unit Tests ------------------- - -- All new unit tests are to be written in python-mock. -- Old tests that are still written in mox should be updated to use python-mock. - Usage of mox has been deprecated for writing Cinder unit tests. -- use addCleanup in favor of tearDown diff --git a/doc/source/devref/development.environment.rst b/doc/source/devref/development.environment.rst index fedbbb7d6..71553434b 100644 --- a/doc/source/devref/development.environment.rst +++ b/doc/source/devref/development.environment.rst @@ -94,17 +94,12 @@ Grab the code:: Running unit tests ------------------ -The unit tests will run by default inside a virtualenv in the ``.venv`` -directory. Run the unit tests by doing:: +Run the unit tests by doing:: - ./run_tests.sh + tox -e py34 + tox -e py27 -The first time you run them, you will be asked if you want to create a virtual -environment (hit "y"):: - - No virtual environment found...create one? (Y/n) - -See :doc:`unit_tests` for more details. +See :doc:`testing` for more details. .. _virtualenv: diff --git a/doc/source/devref/index.rst b/doc/source/devref/index.rst index c29e6fc93..4737a8bd5 100644 --- a/doc/source/devref/index.rst +++ b/doc/source/devref/index.rst @@ -18,7 +18,8 @@ Developer Guide =============== -In this section you will find information on Cinder's lower level programming APIs. +In this section you will find information on Cinder's lower level programming +APIs. Programming HowTos and Tutorials @@ -30,7 +31,7 @@ Programming HowTos and Tutorials api_microversion_dev api_conditional_updates api_microversion_history - unit_tests + testing addmethod.openstackapi drivers gmr diff --git a/doc/source/devref/testing.rst b/doc/source/devref/testing.rst new file mode 100644 index 000000000..f8f6a6b35 --- /dev/null +++ b/doc/source/devref/testing.rst @@ -0,0 +1,158 @@ +Testing +======= + +Cinder contains a few different test suites in the cinder/tests/ directory. The +different test suites are Unit Tests, Functional Tests, and Tempest Tests. + +Test Types +---------- + + +Unit Tests +~~~~~~~~~~ + +Unit tests are tests for individual methods, with at most a small handful of +modules involved. Mock should be used to remove any external dependencies. + +All significant code changes should have unit test coverage validating the code +happy path and any failure paths. + +Any proposed code change will be automatically rejected by the OpenStack +Jenkins server [#f1]_ if the change causes unit test failures. + +Functional Tests +~~~~~~~~~~~~~~~~ + +Functional tests validate a code path within Cinder. These tests should +validate the interaction of various modules within the project to verify the +code is logically correct. + +Functional tests run with a database present and may start Cinder services to +accept requests. These tests should not need to access an other OpenStack +non-Cinder services. + +Tempest Tests +~~~~~~~~~~~~~ + +The tempest tests in the Cinder tree validate the operational correctness +between Cinder and external components such as Nova, Glance, etc. These are +integration tests driven via public APIs to verify actual end user usage +scenarios. + +Running the tests +----------------- + +There are a number of ways to run tests currently, and there's a combination of +frameworks used depending on what commands you use. The preferred method is to +use tox, which calls ostestr via the tox.ini file. + +Unit Tests +~~~~~~~~~~ + +To run all unit tests simply run:: + + tox + +This will create a virtual environment, load all the packages from +test-requirements.txt and run all unit tests as well as run flake8 and hacking +checks against the code. + +You may run individual test targets, for example only py27 tests, by running:: + + tox -e py27 + +Note that you can inspect the tox.ini file to get more details on the available +options and what the test run does by default. + +Functional Tests +~~~~~~~~~~~~~~~~ + +To run all functional tests, run:: + + tox -e functional + +Tempest Tests +~~~~~~~~~~~~~ + +Tempest tests in the Cinder tree are "plugged in" to the normal tempest test +execution. To ensure the Cinder tests are picked up when running tempest, run:: + + cd /opt/stack/tempest + tox -e all-plugin + +More information about tempest can be found in the `Tempest Documentation +`_. + +Running a subset of tests using tox +----------------------------------- +One common activity is to just run a single test, you can do this with tox +simply by specifying to just run py27 or py34 tests against a single test:: + + tox -epy27 -- -n cinder.tests.unit.test_volume.AvailabilityZoneTestCase.test_list_availability_zones_cached + +Or all tests in the test_volume.py file:: + + tox -epy27 -- -n cinder.tests.unit.test_volume + +You may also use regular expressions to run any matching tests:: + + tox -epy27 -- -r test_volume + +For more information on these options and how to run tests, please see the +`ostestr documentation `_. + +Gotchas +------- + +**Running Tests from Shared Folders** + +If you are running the unit tests from a shared folder, you may see tests start +to fail or stop completely as a result of Python lockfile issues. You +can get around this by manually setting or updating the following line in +``cinder/tests/conf_fixture.py``:: + + CONF['lock_path'].SetDefault('/tmp') + +Note that you may use any location (not just ``/tmp``!) as long as it is not +a shared folder. + +**Running py34 tests** + +You will need to install python3-dev in order to get py34 tests to run. If you +do not have this, you will get the following:: + + netifaces.c:1:20: fatal error: Python.h: No such file or directory + #include + ^ + compilation terminated. + error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 + + ---------------------------------------- + + ERROR: could not install deps [-r/opt/stack/cinder/test-requirements.txt, + oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/ + .tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt + oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1) + _______________________________________________________________ summary _______________________________________________________________ + ERROR: py34: could not install deps [-r/opt/stack/cinder/test-requirements.txt, + oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/ + .tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt + oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1) + +To Fix: + +- On Ubuntu/Debian:: + + sudo apt-get install python3-dev + +- On Fedora 21/RHEL7/CentOS7:: + + sudo yum install python3-devel + +- On Fedora 22 and higher:: + + sudo dnf install python3-devel + +.. rubric:: Footnotes + +.. [#f1] See :doc:`jenkins`. diff --git a/doc/source/devref/unit_tests.rst b/doc/source/devref/unit_tests.rst deleted file mode 100644 index 495ef5fa2..000000000 --- a/doc/source/devref/unit_tests.rst +++ /dev/null @@ -1,209 +0,0 @@ -Unit Tests -========== - -Cinder contains a suite of unit tests, in the cinder/tests/unit directory. - -Any proposed code change will be automatically rejected by the OpenStack -Jenkins server [#f1]_ if the change causes unit test failures. - -Running the tests ------------------ -There are a number of ways to run unit tests currently, and there's a combination -of frameworks used depending on what commands you use. The preferred method -is to use tox, which calls ostestr via the tox.ini file. To run all tests simply run:: - - tox - -This will create a virtual environment, load all the packages from test-requirements.txt -and run all unit tests as well as run flake8 and hacking checks against the code. - -Note that you can inspect the tox.ini file to get more details on the available options -and what the test run does by default. - -Running a subset of tests using tox ------------------------------------ -One common activity is to just run a single test, you can do this with tox simply by -specifying to just run py27 or py34 tests against a single test:: - - tox -epy27 -- -n cinder.tests.unit.test_volume.AvailabilityZoneTestCase.test_list_availability_zones_cached - -Or all tests in the test_volume.py file:: - - tox -epy27 -- -n cinder.tests.unit.test_volume - -For more information on these options and how to run tests, please see the `ostestr -documentation `_. - -Run tests wrapper script ------------------------- - -In addition you can also use the wrapper script run_tests.sh by simply executing:: - - ./run_tests.sh - -This script is a wrapper around the testr testrunner and the flake8 checker. Note that -there has been talk around deprecating this wrapper and this method of testing, it's currently -available still but it may be good to get used to using tox or even ostestr directly. - -Documentation is left in place for those that still use it. - -Flags ------ - -The ``run_tests.sh`` script supports several flags. You can view a list of -flags by doing:: - - run_tests.sh -h - -This will show the following help information:: - - Usage: ./run_tests.sh [OPTION]... - Run Cinder's test suite(s) - - -V, --virtual-env Always use virtualenv. Install automatically if not present - -N, --no-virtual-env Don't use virtualenv. Run tests in local environment - -s, --no-site-packages Isolate the virtualenv from the global Python environment - -r, --recreate-db Recreate the test database (deprecated, as this is now the default). - -n, --no-recreate-db Don't recreate the test database. - -x, --stop Stop running tests after the first error or failure. - -f, --force Force a clean re-build of the virtual environment. Useful when dependencies have been added. - -p, --pep8 Just run pep8 - -P, --no-pep8 Don't run pep8 - -c, --coverage Generate coverage report - -h, --help Print this usage message - --hide-elapsed Don't print the elapsed time for each test along with slow test list - -Because ``run_tests.sh`` is a wrapper around testr, it also accepts the same -flags as testr. See the `testr documentation`_ for details about -these additional flags. - -.. _testr documentation: https://testrepository.readthedocs.org/en/latest/ -.. _nose options documentation: http://readthedocs.org/docs/nose/en/latest/usage.html#options - -Running a subset of tests -------------------------- - -Instead of running all tests, you can specify an individual directory, file, -class, or method that contains test code. - -To run the tests in the ``cinder/tests/scheduler`` directory:: - - ./run_tests.sh scheduler - -To run the tests in the ``cinder/tests/test_libvirt.py`` file:: - - ./run_tests.sh test_libvirt - -To run the tests in the `HostStateTestCase` class in -``cinder/tests/test_libvirt.py``:: - - ./run_tests.sh test_libvirt.HostStateTestCase - -To run the `ToPrimitiveTestCase.test_dict` test method in -``cinder/tests/test_utils.py``:: - - ./run_tests.sh test_utils.ToPrimitiveTestCase.test_dict - - -Virtualenv ----------- - -By default, the tests use the Python packages installed inside a -virtualenv [#f2]_. (This is equivalent to using the ``-V, --virtualenv`` flag). -If the virtualenv does not exist, it will be created the first time the tests are run. - -If you wish to recreate the virtualenv, call ``run_tests.sh`` with the flag:: - - -f, --force - -Recreating the virtualenv is useful if the package dependencies have changed -since the virtualenv was last created. If the ``requirements.txt`` or -``tools/install_venv.py`` files have changed, it's a good idea to recreate the -virtualenv. - -By default, the unit tests will see both the packages in the virtualenv and -the packages that have been installed in the Python global environment. In -some cases, the packages in the Python global environment may cause a conflict -with the packages in the virtualenv. If this occurs, you can isolate the -virtualenv from the global environment by using the flag:: - - -s, --no-site packages - -If you do not wish to use a virtualenv at all, use the flag:: - - -N, --no-virtual-env - -Database --------- - -Some of the unit tests make queries against an sqlite database. By -default, the test database (``tests.sqlite``) is deleted and recreated each -time ``run_tests.sh`` is invoked (This is equivalent to using the -``-r, --recreate-db`` flag). To reduce testing time if a database already -exists it can be reused by using the flag:: - - -n, --no-recreate-db - -Reusing an existing database may cause tests to fail if the schema has -changed. If any files in the ``cinder/db/sqlalchemy`` have changed, it's a good -idea to recreate the test database. - -Gotchas -------- - -**Running Tests from Shared Folders** - -If you are running the unit tests from a shared folder, you may see tests start -to fail or stop completely as a result of Python lockfile issues. You -can get around this by manually setting or updating the following line in -``cinder/tests/conf_fixture.py``:: - - CONF['lock_path'].SetDefault('/tmp') - -Note that you may use any location (not just ``/tmp``!) as long as it is not -a shared folder. - -.. rubric:: Footnotes - -.. [#f1] See :doc:`jenkins`. - -.. [#f2] See :doc:`development.environment` for more details about the use of - virtualenv. - -**Running py34 tests** - -You will need to install: -python3-dev -in order to get py34 tests to run. If you do not have this, you will get the following:: - - netifaces.c:1:20: fatal error: Python.h: No such file or directory - #include - ^ - compilation terminated. - error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 - - ---------------------------------------- - - ERROR: could not install deps [-r/opt/stack/cinder/test-requirements.txt, - oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/ - .tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt - oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1) - _______________________________________________________________ summary _______________________________________________________________ - ERROR: py34: could not install deps [-r/opt/stack/cinder/test-requirements.txt, - oslo.versionedobjects[fixtures]]; v = InvocationError('/opt/stack/cinder/ - .tox/py34/bin/pip install -r/opt/stack/cinder/test-requirements.txt - oslo.versionedobjects[fixtures] (see /opt/stack/cinder/.tox/py34/log/py34-1.log)', 1) - -To Fix: - -- On Ubuntu/Debian:: - - sudo apt-get install python3-dev - -- On Fedora 21/RHEL7/CentOS7:: - - sudo yum install python3-devel - -- On Fedora 22 and higher:: - - sudo dnf install python3-devel