12 KiB
Senlin testing
Overview of Testing
The Senlin project currently has five different types of testing facilities in place for developers to perform different kinds of tests:
- Unit Tests: These are source code level testing that verifies the classes and methods behave as implemented. Once implemented, these tests are also used to guarantee that code behavior won't change accidentally by other patches.
- API Tests: These tests treat the senlin-api and the senlin-engine as black boxes. The test cases focus more on the API surface rather than how each API is implemented. Once implemented, these tests help ensure that the user-visible service interface don't change without a good reason.
- Functional Tests: These tests also treat the senlin-api and the senlin-engine as block boxes. They focus more on the user perceivable service behavior. Most tests are anticipated to test a particular "story" and verify that the senlin-engine always behave consistently.
- Integration Tests: These are the tests that integrate senlin with other OpenStack services and verify the senlin service can perform its operations correctly when interacting with other services.
- Stress Tests: These are tests for measuring the performance of the senlin-api and senlin-engine under different workloads.
Cloud Backends
The senlin server is shipped with two collections of "cloud
backends": one for interacting with a real OpenStack deployment, the
other for running complex tests including api tests, functional tests,
stress tests. The first cloud backend is referred to as 'openstack' and the second is referred to as
'openstack_test'. While the openstack cloud backend contains full featured
drivers for senlin to talk to the OpenStack services supported, the
openstack_test backend contains some
"dummy" drivers that return fake responses for service requests. The
openstack_test driver is located at senlin/tests/drivers
subdirectory. It is provided to facilitate tests on the senlin service
itself without involving any other OpenStack services. Several types of
tests can benefit from these "dummy" drivers because 1) they can save
developers a lot time on debugging complex issues when interacting with
other OpenStack services, and 2) they make running those types of tests
much easier and quicker.
Note that "Integration Tests" are designed for senlin to interact with real services so we should use the openstack backend rather than the openstack_test backend.
To configure the backend to use before running tests, you can check
the [DEFAULT] section in the
configuration file /etc/senlin/senlin.conf
.
[DEFAULT]
cloud_backend = openstack_test # use this for api, functional tests;
# or 'openstack' for production environment
# and integration tests.
Unit Tests
All unit tests are to be placed in the senlin/tests/unit
sub-directory. Test cases are
organized by the targeted subsystems/modules. Each subsystem directory
must contain a separate blank __init__.py for tests discovery to
function properly.
An example directory structure:
senlin
`- tests
`- unit
|-- db
| |-- __init__.py
| |-- test_cluster_api.py
| `-- test_node_api.py
|-- engine
| |-- __init__.py
| |-- test_clusters.py
| `-- test_nodes.py
|-- __init__.py
`-- test_utils.py
Writing a Unit Test
The os-testr software (see: http://pypi.python.org/pypi/os-testr) is used to find and run tests, parallelize their runs, and record timing/results.
If new dependencies are introduced upon the development of a test, the test-requirements.txt file needs to be updated so that the virtual environment will be able to successfully execute all tests.
The test-requirements.txt file needs to be synchronized with the openstack/global-requirements project. Developers should try avoid introducing additional package dependencies unless forced to.
Running Unit Tests
Senlin uses tox for running unit tests, as practiced by many other OpenStack projects:
$ tox
This by default will run unit tests suite with Python 2.7 and PEP8/HACKING style checks. To run only one type of tests you can explicitly provide tox with the test environment to use:
$ tox -e py27 # test suite on python 2.7
$ tox -e pep8 # run full source code checker
To run only a subset of tests, you can provide tox with a regex argument:
$ tox -e py27 -- -r ClusterTest
To use debugger like pdb during test run, you have to run tests directly with other, non-concurrent test runner instead of testr. That also presumes that you have a virtual env with all senlin dependencies installed and configured.
A more convenient way to run specific test is to name the unit test directly, as shown below:
$ python -m testtools.run senlin.tests.unit.db.test_cluster_api
This command, however, is not using dependent packages in a particular virtual environment as the tox command does. It is using the system-wide Python package repository when running the tests.
API Tests
Senlin API test cases are written based on the tempest framework (see: tempest_overview). Test cases are developed using the Tempest Plugin Interface (see: tempest_plugin ).
Writing an API Test Case
When new APIs are added or existing APIs are changed, an API test
case should be added to the senlin/tests/tempest/api
sub-directory, based on the
resources impacted by the change.
Each test case should derive from the class senlin.tests.tempest.api.base.BaseSenlinAPITest
.
Positive test cases should be separated from negative ones. We don't
encourage combining more than one test case into a single method, unless
there is an obvious reason.
To improve the readability of the test cases, Senlin has provided a
utility module which can be leveraged - senlin/tests/tempest/common/utils.py
.
Running API Tests
Senlin API tests use fake OpenStack drivers to improve the throughput of test execution. This is because in API tests, we don't care about the details in how senlin-engine is interacting with other services. We care more about the APIs succeeds in an expected way or fails in a predictable manner.
Although the senlin engine is talking to fake drivers, the test cases
still need to communicate to the senlin API service as it would in a
real deployment. That means you will have to export your OpenStack
credentials before running the tests. For example, you will source the
openrc
file when using
a devstack environment:
$ source $HOME/devstack/openrc
This will ensure you have environment variables such as
OS_AUTH_URL
, OS_USERNAME
properly set and
exported. The next step is to enter the tempest
directory and run the tests there:
$ cd /opt/stack/tempest
$ nosetests -v -- senlin
To run a single test case, you can specify the test case name. For example:
$ cd /opt/stack/tempest
$ nosetests -v -- \
senlin.tests.tempest.api.clusters.test_cluster_create
If you prefer running API tests in a virtual environment, you can simply use the following command:
$ cd /opt/stack/senlin
$ tox -e api
Functional Tests
Similar to the API tests, senlin functional tests are also developed based on the tempest framework. Test cases are written using the Tempest Plugin Interface (see: tempest_plugin).
Writing Functional Tests
There are current a limited collection of functional test cases which
can be found under senlin/tests/tempest/functional/
subdirectory. In
future, we may add more test cases when needed. The above subdirectory
will remain the home of newly added functional tests.
When writing functional tests, it is highly desirable that each test case is designed for a specific use case or story line.
Running Functional Tests
Similar to API tests, you will need to export your OpenStack credentials before running any functional tests.
The most straight forward way to run functional tests is to use the
virtual environment defined in the tox.ini
file, that is:
$ cd /opt/stack/senlin
$ tox -e functional
If you prefer running a particular functional test case, you can do the following as well:
$ cd /opt/stack/senlin
$ python -m testtools.run senlin.tests.tempest.functional.test_cluster_basic
Integration Tests
Integration tests are basically another flavor of functional tests. The only difference from functional tests is that integration tests use real device drivers so the senlin-engine is talking to real services.
Writing Integration Tests
Integration tests are designed to be run at Gerrit gate to ensure that changes to senlin code won't break its interactions with other (backend) services. Since OpenStack gate infrastructure is a shared resource pool for all OpenStack projects, we are supposed to be very careful when adding new test cases. The test cases added are supposed to focus more on the interaction between senlin and other services than other things.
All integration test cases are to be placed under the subdirectory
senlin/tests/tempest/integration
. Test cases are
expected to be organized into a small number of story lines that can
exercise as many interactions between senlin and backend services as
possible.
Each "story line" should be organized into a separate class module
that inherits from the BaseSenlinIntegrationTest
class
which can be found at senlin/tests/tempest/integration/base.py
file. Each
test case should be annotated with a test.attr
annotator
and an idempotent ID as shown below:
from tempest.lib import decorators
from tempest import test
from senlin.tests.tempest.integration import base
class MyIntegrationTest(base.BaseSenlinIntegrationTest):
@test.attr(type=['integration'])
@decorators.idempotent_id('<A UUID for the test case>')
def test_a_sad_story(self):
# Test logic goes here
# ...
Running Integration Tests
The integration tests are designed to be executed at Gerrit gate. However, you can still run them locally in your development environment, i.e. a devstack installation.
To run integration tests, you will need to configure tempest
accounts by editing the /etc/tempest/accounts.yaml
file. For each entry of
the tempest account, you will need to provide values for
username
, tenant_name
, password
at least. For example:
- username: 'demo'
tenant_name: 'demo'
password: 'secretee'
After this is configured, you can run a specific test case using the following command:
$ cd /opt/stack/senlin
$ python -m testtools.run \
senlin.tests.tempest.integration.test_nova_server_cluster
Stress Tests
Stress tests are designed to measure a service's performance under certain workload pressure. In senlin, the stress tests are written to be executed using the Rally framework.
Writing Stress Test Cases
<TBD>
Running Stress Tests
<TBD>