Tempest provides a LockFixture to avoid two potentially interfering
tests to run in parallel. However, this solution does not scale when
we want to separate a set of tests from many other test cases. For
example, host aggregate and availability zone testing needs compute
hosts without any nova servers to be able to test moving computes
between aggregates but a lot of other tests are creating nova
servers. To fully separate these aggregate tests from the rest of
the tempest test cases, this patch proposes a @serial class decorator
to mark a test class to be run totally independently of any other test
Under the hood, the @serial decorator is implemented with a tempest-wide
interprocess read-write lock. The serial test classes always take the
write lock, while the non-serial classes take the read lock. The lock
allows in many readers OR a single writer. So the serial tests are run
independently from the rest.
To minimize the time a serial test blocks other tempest tests run in
parallel, this patch also introduced a serial_tests test directory to
store the serial tests. The current test ordering in a fresh env
uses alphabetical order so the serial tests will run at the end of
the execution not randomly in the middle. The gate uses fresh VMs
for every run so we can rely on this optimization there. In local
envs where tests are re-run, the subsequent runs will be ordered at
runtime by stestr. Therfore, a longer runtime might be observed due to
locking, but the correctness of the test execution is still kept.
Now we have more documents for plugins on stable release,
support and their testing policy. Let's prepare a
index page for it will be easy for plugins maintainer.
Also renaming the stable support policy section to have a clear
Adding redirect links for moved documents.
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.
Fix problems found.
Update local hacking checks for new flake8, remove vi check, it's now in
hacking as H106.
The tempest plugin is used on older branches as well.
We really only need hacking on this repo anyways,
where we no longer support python 2, so here we
make the requirement specific to python 3.
This documentation is included in HACKING at the top to let
newcomers know that Tempest's testing scope is confined to the
The user is directed to  for additional Tempest plugin
repos where they can add their tests for other projects. Note
that  currently appears to not be rendering correctly,
but this is an unrelated issue.
I61ec91ba4a17c6797f1c626c9b09e4fdb381c357 moves scenario
docstring under the relevant test method, but the reference
to that docstring kept unchanged in HACKING.rst. Besides,
this is no longer a good example of "class level docstring",
so we just pick another one for the example.
This patchset adds 2 hacking checks for making sure negative
tests have correct conventions applied.
* T117: Check that each negative test has the
This patch set adds both hacking checks, adds unit tests
and updates HACKING.rst documentation with both new checks.
Tempest uses test attributes  in many places for different
reasons, but there is no documentation to explain why.
For example, Tempest uses these attributes to customize
Zuul jobs to selectively run only slow scenario tests. Tempest
also has a tox entrypoint for just running smoke tests.
Further, there is no documentation explaining:
* what is a smoke test
* what is a negative test
* what is a slow test
And more importantly: When should each attribute be applied?
So this information is not just useful for beginners (helping
them understand the different types of testing and what they
mean) but also for users who would like to understand how to
correctly apply attributes to new tests in either Tempest
or Tempest plugins.
Current doc explaining the stable branches support policy is under
coding guide. But support policy should be at higher level which
can be useful for user or maintainer of Tempest.
This commit adds a note to the documentation to clarify which branches
tempest will support under the new extended maintenance stable branch
policy. The previous policy of supporting all active branches will not
scale under the new policy as we do not have enough resources to gate
all propsed changes against more than 3 stable branches at a time. With
the addition of this to the documentation we'll clearly define what
branches are in scope for testing of tempest and how/when we should
handle dropping support for a branch.
This is to elaborate on feature flag functionality in Tempest's
HACKING.rst. Currently, there isn't much documentation on feature
flags in Tempest or how to add them. The only place feature
flag is mentioned is in this file. Since testtools.skip(Unless|If)
is a common way of using a feature flag, it is mentioned. Also,
Tempest's interaction with DevStack is an important thing to mention
when explaining how to introduce a new feature flag.
This commit fixes docs consistency about markups, mainly. Originally,
some of command options were written with strong emphasis. However,
double dashes are converted to a single dash with it. And this commit
also fixes some inconsistencies and weird indentations.
Most of them can still be visited through http, but the following
one is necessary to update, so I change them all by this chance.
$ git clone http://git.openstack.org/openstack/tempest
Add docstrings, unit tests and an helper for class resource
cleanup. Check super's resource_cleanup is invoked when
overriden to ensure the cleanup stack is processed.
services and extension decorators are based on CONF and needed by
Tempest only, so moving them to tempest.common.
Since a few plugnis use them today, use debtcollector to avoid
breaking all dependencies.
* Fix unsupported 'message' Exception attribute in PY3
The 'message' attribute has been deprecated and removed from Python3.
Use six.text_type(e) instead of e.message. For more details, please check :
* Add hacking to prevent this from happening in the future.
I've been reading our entire documentation to see where it could
be improved. It's guide good actually. While reading I've fixed
some typos, added some capitalization to project names, mostly
What's worth reviewing is the 2 paragraphs I added to the REVIEWING
Tempest tests should use the stable method defined in tempest.lib.
This patch is to use data_utils from tempest.lib.common.utils in
OAUTHConsumersV3Test and HACKING.rst.
Sometimes commiters tried to add tempest tests which require admin
credential under non-admin test path and that caused confusions to
tempest users. This patch adds some coding rule to make test path
clear for the maintenance.
NOTE: This patch adds #noqa to AbsoluteLimitsTests because the test
class needs force_tenant_isolation which requires admin
credential indirectly but the test itself is not admin test.
The history is Id71a705cf9b1dd0c0d41a2fb45ab77c95430a123
It's not used anymore. There was general consensus in Feb 2016
to deprecate it (see ) and remove it in Newton.
 [qa] deprecating Tempest stress framework
We have discussed negative tests at OpenStack Summit Austin before
The conclusion is
* Keep the existing negative tests
Tempest should not cover all negative test cases as integration tests.
Such tests should be covered with function tests in each projects.
Function tests are implemented in Nova, [Ceilometer -> Gabbi], Barbican
36 negative tests are used in Refstack as required tests and they are valuable
So we need some guideline about negative tests for our future, and this
patch adds it.
This commit adds a hacking rule to enforce that we never add a config
dependency on tempest/lib. Right now we're completely dependent on
reviewers catching this, it is a strong rule so we should ensure we
can't ever land a change that does this.
This commit fixes docs errors and warnings. However, 'WARNING: nonlocal
image URI found' is still remaining because it needs an ugly workaround
that is replacing it to raw html.
There is a common method for generating random uuid value.
So it is nice to avoid pointing it out on human review to reduce
In addition, this patch removes str() around the method calls
because str() is called in the method.
OpenStack has settled (a very long time ago) on project to represent
the container which has users. We should make tempest configs always
use project versions of these names. This deprecates the old tenant
versions of these names, and makes project version the prefered
The idempotent id decorator and the check-uuid script were migrated to
tempest-lib as part of tempest-lib 0.10.0. However, tempest was never
updated to use the library version of idempotent id or check-uuid.
This commit updates tempest to remove it's local copies of the
idempotent_id decorator and the check-uuid tool and use them from
Also, instead of updating the import of every single test module in
tempest an alias is just added to tempest.test to let all the old
imports work seamlessly. Updating the imports and removing the alias
may or may not be worth it in the future, but that will be a follow
on patch or discussion.