In out functional tests we run nova services as eventlets. Also those
services can spawn there own eventlets for RPC or other parallel
processing. The test case executor only sees and tracks the main
eventlet where the code of the test case is running. When that is
finishes the test executor considers the test case to be finished
regardless of the other spawned eventlets. This could lead to leaked
eventlets that are running in parallel with later test cases.
One way that it can cause trouble is via the global variables in
nova.rpc module. Those globals are re-initialized for each test case so
they are not directly leaking information between test cases. However if
a late eventlet calls nova.rpc.get_versioned_notifier() it will get a
totally usable FakeVersionedNotifier object regardless of which test
case this notifier is belongs to or which test case the eventlet belongs
to. This way the late eventlet can send a notification to the currently
running test case and therefore can make it fail.
The current case we saw is the following:
1) The test case
nova.tests.functional.test_servers.ServersTestV219.test_description_errors
creates a server but don't wait for it to reach terminal state (ACTIVE
/ ERROR). This test case finishes quickly but leaks running eventlets
in the background waiting for some RPC call to return.
2) As the test case finished the cleanup code deletes the test case
specific setup, including the DB.
3) The test executor moves forward and starts running another test case
4) 60 seconds later the leaked eventlet times out waiting for the RPC
call to return and tries doing things, but fails as the DB is already
gone. Then it tries to report this as an error notification. It calls
nova.rpc.get_versioned_notifier() and gets a fresh notifier that is
connected to the currently running test case. Then emits the error
notification there.
5) The currently running test case also waits for an error notification
to be triggered by the currently running test code. But it gets the
notification form the late eventlet first. As the content of the
notification does not match with the expectations the currently
running test case fails. The late eventlet prints a lot of
error about the DB being gone making the troubleshooting pretty hard.
This patch proposes a way to fix this by marking each eventlet at spawn
time with the id of the test case that was directly or indirectly
started it.
Then when the NotificationFixture gets a notification it compares the
test case id stored in the calling eventlet with the id of the test case
initialized the NotificationFixture. If the two ids do not match then
the fixture ignores the notification and raises an exception to the
caller eventlet to make it terminate.
Change-Id: I012dcf63306bae624dc4f66aae6c6d96a20d4327
Closes-Bug: #1946339
61fc81a676