openstack
/
tempest
Code Issues Proposed changes

docs: Add HACKING information about test attributes 

Tempest uses test attributes [0] in many places for different
reasons, but there is no documentation to explain why.
For example, Tempest uses these attributes to customize
Zuul jobs to selectively run only slow scenario tests. Tempest
also has a tox entrypoint for just running smoke tests.

Further, there is no documentation explaining:

* what is a smoke test
* what is a negative test
* what is a slow test

And more importantly: When should each attribute be applied?

So this information is not just useful for beginners (helping
them understand the different types of testing and what they
mean) but also for users who would like to understand how to
correctly apply attributes to new tests in either Tempest
or Tempest plugins.

[0] https://testtools.readthedocs.io/en/latest/for-test-authors.html?highlight=attr#test-attributes

Change-Id: I7be0bebaab56a418616bea01e90a437b23499234
This commit is contained in:
Felipe Monteiro 2018-07-09 23:58:20 -04:00
parent e6c330892f
commit 46920b8af9
1 changed files with 54 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

55
HACKING.rst
View File

@ -106,7 +106,7 @@ Service tagging is used to specify which services are exercised by a particular
test method. You specify the services with the ``tempest.common.utils.services``
decorator. For example:
@utils.services('compute', 'image')
``@utils.services('compute', 'image')``
Valid service tag names are the same as the list of directories in tempest.api
that have tests.
@ -118,6 +118,59 @@ name. For example, any test that make an API call to a service other than Nova
in ``tempest.api.compute`` would require a service tag for those services,
however they do not need to be tagged as ``compute``.
Test Attributes
---------------
Tempest leverages `test attributes`_ which are a simple but effective way of
distinguishing between different "types" of API tests. A test can be "tagged"
with such attributes using the ``decorators.attr`` decorator, for example::
@decorators.attr(type=['negative'])
def test_aggregate_create_aggregate_name_length_less_than_1(self):
[...]
These test attributes can be used for test selection via regular expressions.
For example, ``(?!.*\[.*\bslow\b.*\])(^tempest\.scenario)`` runs all the tests
in the ``scenario`` test module, *except* for those tagged with the ``slow``
attribute (via a negative lookahead in the regular expression). These
attributes are used in Tempest's ``tox.ini`` as well as Tempest's Zuul job
definitions for specifying particular batches of Tempest test suites to run.
.. _test attributes: https://testtools.readthedocs.io/en/latest/for-test-authors.html?highlight=attr#test-attributes
Negative Attribute
^^^^^^^^^^^^^^^^^^
The ``type='negative'`` attribute is used to signify that a test is a negative
test, which is a test that handles invalid input gracefully. This attribute
should be applied to all negative test scenarios.
This attribute must be applied to each test that belongs to a negative test
class, i.e. a test class name ending with "Negative.*" substring.
.. todo::
Add a hacking check for ensuring that all classes that contain substring
"Negative" have the negative attribute decorator applied above each test.
Slow Attribute
^^^^^^^^^^^^^^
The ``type='slow'`` attribute is used to signify that a test takes a long time
to run, relatively speaking. This attribute is usually applied to
:ref:`scenario tests <scenario_field_guide>`, which involve a complicated
series of API operations, the total runtime of which can be relatively long.
This long runtime has performance implications on `Zuul`_ jobs, which is why
the ``slow`` attribute is leveraged to run slow tests on a selective basis,
to keep total `Zuul`_ job runtime down to a reasonable time frame.
.. _Zuul: https://docs.openstack.org/infra/zuul/
Smoke Attribute
^^^^^^^^^^^^^^^
The ``type='smoke'`` attribute is used to signify that a test is a so-called
smoke test, which is a type of test that tests the most vital OpenStack
functionality, like listing servers or flavors or creating volumes. The
attribute should be sparingly applied to only the tests that sanity-check the
most essential functionality of an OpenStack cloud.
Test fixtures and resources
---------------------------
Test level resources should be cleaned-up after the test execution. Clean-up