Tempest provides a LockFixture to avoid two potentially interfering
tests to run in parallel. However, this solution does not scale when
we want to separate a set of tests from many other test cases. For
example, host aggregate and availability zone testing needs compute
hosts without any nova servers to be able to test moving computes
between aggregates but a lot of other tests are creating nova
servers. To fully separate these aggregate tests from the rest of
the tempest test cases, this patch proposes a @serial class decorator
to mark a test class to be run totally independently of any other test
classes.
Under the hood, the @serial decorator is implemented with a tempest-wide
interprocess read-write lock. The serial test classes always take the
write lock, while the non-serial classes take the read lock. The lock
allows in many readers OR a single writer. So the serial tests are run
independently from the rest.
To minimize the time a serial test blocks other tempest tests run in
parallel, this patch also introduced a serial_tests test directory to
store the serial tests. The current test ordering in a fresh env
uses alphabetical order so the serial tests will run at the end of
the execution not randomly in the middle. The gate uses fresh VMs
for every run so we can rely on this optimization there. In local
envs where tests are re-run, the subsequent runs will be ordered at
runtime by stestr. Therfore, a longer runtime might be observed due to
locking, but the correctness of the test execution is still kept.
Related-Bug: #821732
Change-Id: I0181517edab75f586464a38c4811417f888783b1
dfb304355b