openstack/tempest
Code Issues Proposed changes

Add tempest test writing guide

Browse Source 
This commit is the start of a new doc guide for helping people figure
out how to write good tempest tests. This is just a starting point for
the document and doesn't cover every aspect of writing a test case in
tempest. Right now it just covers the setupClass phase for writing
TestCases. It will be expanded upon in future commits to give a complete
guide to writing tests in Tempest.

Change-Id: Ie0db2079163f536ae72668c8fc9129ae4c6389e2
This commit is contained in:
Matthew Treinish 2017-03-01 15:25:39 -05:00
parent 90d505ae75
commit e8ab5f9b08
No known key found for this signature in database
GPG Key ID: FD12A0F214C9E177
4 changed files with 251 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
doc/source/configuration.rst
View File

@ -9,6 +9,8 @@ they are used in conjunction. The source of truth on each option is the sample
config file which explains the purpose of each individual option. You can see
the sample config file here: :ref:`tempest-sampleconf`
.. _tempest_cred_provider_conf:
Test Credentials
----------------
@ -232,6 +234,9 @@ Tempest's config around network configuration.
Enabling Remote Access to Created Servers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. _tempest_conf_network_allocation:
Network Creation/Usage for Servers
""""""""""""""""""""""""""""""""""
When Tempest creates servers for testing, some tests require being able to

1
doc/source/index.rst
View File

@ -68,6 +68,7 @@ Development
REVIEWING
microversion_testing
test-removal
write_tests
-------
Plugins

2
doc/source/plugin.rst
View File

@ -1,3 +1,5 @@
.. _tempest_plugin:
=============================
Tempest Test Plugin Interface
=============================

243
doc/source/write_tests.rst Normal file
View File

@ -0,0 +1,243 @@
.. _tempest_test_writing:
Tempest Test Writing Guide
==========================
This guide serves as a starting point for developers working on writing new
Tempest tests. At a high level tests in Tempest are just tests that conform to
the standard python `unit test`_ framework. But there are several aspects of
that are unique to tempest and it's role as an integration test suite running
against a real cloud.
.. _unit test: https://docs.python.org/3.6/library/unittest.html
.. note:: This guide is for writing tests in the tempest repository. While many
parts of this guide are also applicable to tempest plugins, not all
the APIs mentioned are considered stable or recommended for use in
plugins. Please refer to :ref:`tempest_plugin` for details about
writing plugins
Adding a New TestCase
=====================
The base unit of testing in Tempest is the `TestCase`_ (also called the test
class). Each TestCase contains test methods which are the individual tests that
will be executed by the test runner. But, the TestCase is the smallest self
contained unit for tests from the tempest perspective. It's also the level at
which tempest is parallel safe. In other words, multiple TestCases can be
executed in parallel, but individual test methods in the same TestCase can not.
Also, all test methods within a TestCase are assumed to be executed serially. As
such you can use the test case to store variables that are shared between
methods.
.. _TestCase: https://docs.python.org/3.6/library/unittest.html#unittest.TestCase
In standard unittest the lifecycle of a TestCase can be described in the
following phases:
#. setUpClass
#. setUp
#. Test Execution
#. tearDown
#. doCleanups
#. tearDownClass
setUpClass
----------
The setUpClass phase is the first phase executed by the test runner and is used
to perform any setup required for all the test methods to be executed. In
Tempest this is a very important step and will automatically do the necessary
setup for interacting with the configured cloud.
To accomplish this you do **not** define a setUpClass function, instead there
are a number of predefined phases to setUpClass that are used. The phases are:
* skip_checks
* setup_credentials
* setup_clients
* resource_setup
which is executed in that order. An example of a TestCase which defines all
of these would be::
from tempest import config
from tempest import test
CONF = config.CONF
class TestExampleCase(test.BaseTestCase):
@classmethod
def skip_checks(cls):
"""This section is used to evaluate config early and skip all test
methods based on these checks
"""
super(TestExampleCase, cls).skip_checks()
if not CONF.section.foo
cls.skip('A helpful message')
@classmethod
def setup_credentials(cls):
"""This section is used to do any manual credential allocation and also
in the case of dynamic credentials to override the default network
resource creation/auto allocation
"""
# This call is used to tell the credential allocator to not create any
# network resources for this test case. It also enables selective
# creation of other neutron resources. NOTE: it must go before the
# super call
cls.set_network_resources()
super(TestExampleCase, cls).setup_credentials()
@classmethod
def setup_clients(cls):
"""This section is used to setup client aliases from the manager object
or to initialize any additional clients. Except in a few very
specific situations you should not need to use this.
"""
super(TestExampleCase, cls).setup_clients()
cls.servers_client = cls.os.servers_client
@classmethod
def resource_setup(cls):
"""This section is used to create any resources or objects which are
going to be used and shared by **all** test methods in the
TestCase. Note then anything created in this section must also be
destroyed in the corresponding resource_cleanup() method (which will
be run during tearDownClass())
"""
super(TestExampleCase, cls).resource_setup()
cls.shared_server = cls.servers_client.create_server(...)
Allocating Credentials
''''''''''''''''''''''
Since Tempest tests are all about testing a running cloud, every test will need
credentials to be able to make API requests against the cloud. Since this is
critical to operation and, when running in parallel, easy to make a mistake,
the base TestCase class will automatically allocate a regular user for each
TestCase during the setup_credentials() phase. During this process it will also
initialize a client manager object using those credentials, which will be your
entry point into interacting with the cloud. For more details on how credentials
are allocated the :ref:`tempest_cred_provider_conf` section of the Tempest
Configuration Guide provides more details on the operation of this.
There are some cases when you need more than a single set of credentials, or
credentials with a more specialized set of roles. To accomplish this you have
to set a class variable ``credentials`` on the TestCase directly. For example::
from tempest import test
class TestExampleAdmin(test.BaseTestCase):
credentials = ['primary', 'admin']
@classmethod
def skip_checks(cls):
...
In this example the ``TestExampleAdmin`` TestCase will allocate 2 sets of
credentials, one regular user and one admin user. The corresponding manager
objects will be set as class variables cls.os and cls.os_adm respectively. You
can also allocate a second user by putting **'alt'** in the list too. A set of
alt credentials are the same as primary but can be used for tests cases that
need a second user/project.
You can also specify credentials with specific roles assigned. This is useful
for cases where there are specific RBAC requirements hard coded into an API.
The canonical example of this are swift tests which often want to test swift's
concepts of operator and reseller_admin. An actual example from tempest on how
to do this is::
class PublicObjectTest(base.BaseObjectTest):
credentials = [['operator', CONF.object_storage.operator_role],
['operator_alt', CONF.object_storage.operator_role]]
@classmethod
def setup_credentials(cls):
super(PublicObjectTest, cls).setup_credentials()
...
In this case the manager objects will be set to ``cls.os_roles_operator`` and
``cls.os_roles_operator_alt`` respectively.
There is no limit to how many credentials you can allocate in this manner,
however in almost every case you should **not** need more than 3 sets of
credentials per test case.
To figure out the mapping of manager objects set on the TestCase and the
requested credentials you can reference:
+-------------------+---------------------+
| Credentials Entry | Manager Variable |
+===================+=====================+
| primary | cls.os |
+-------------------+---------------------+
| admin | cls.os_adm |
+-------------------+---------------------+
| alt | cls.os_alt |
+-------------------+---------------------+
| [$label, $role] | cls.os_roles_$label |
+-------------------+---------------------+
By default cls.os is available since it is allocated in the base tempest test
class. (located in tempest/test.py) If your TestCase inherits from a different
direct parent class (it'll still inherit from the BaseTestCase, just not
directly) be sure to check if that class overrides allocated credentials.
Dealing with Network Allocation
'''''''''''''''''''''''''''''''
When neutron is enabled and a testing requires networking this isn't normally
automatically setup when a tenant is created. Since tempest needs isolated
tenants to function properly it also needs to handle network allocation. By
default the base test class will allocate a network, subnet, and router
automatically. (this depends on the configured credential provider, for more
details see: :ref:`tempest_conf_network_allocation`) However, there are
situations where you do no need all of these resources allocated. (or your
TestCase inherits from a class that overrides the default in tempest/test.py)
There is a class level mechanism to override this allocation and specify which
resources you need. To do this you need to call `cls.set_network_resources()`
in the `setup_credentials()` method before the `super()`. For example::
from tempest import test
class TestExampleCase(test.BaseTestCase):
@classmethod
def setup_credentials(cls):
cls.set_network_resources(network=True, subnet=True, router=False)
super(TestExampleCase, cls).setup_credentials()
There are 2 quirks with the usage here. First for the set_network_resources
function to work properly it **must be called before super()**. This is so
that children classes' settings are always used instead of a parent classes'.
The other quirk here is that if you do not want to allocate any network
resources for your test class simply call `set_network_resources()` without
any arguments. For example::
from tempest import test
class TestExampleCase(test.BaseTestCase):
@classmethod
def setup_credentials(cls):
cls.set_network_resources()
super(TestExampleCase, cls).setup_credentials()
This will not allocate any networking resources. This is because by default all
the arguments default to False.
It's also worth pointing out that it is common for base test classes for
different services (and scenario tests) to override this setting. When
inheriting from classes other than the base TestCase in tempest/test.py it is
worth checking the immediate parent for what is set to determine if your
class needs to override that setting.