f24e0c1da2
In out functional tests we run nova services as eventlets. Also those
services can spawn there own eventlets for RPC or other parallel
processing. The test case executor only sees and tracks the main
eventlet where the code of the test case is running. When that is
finishes the test executor considers the test case to be finished
regardless of the other spawned eventlets. This could lead to leaked
eventlets that are running in parallel with later test cases.
One way that it can cause trouble is via the global variables in
nova.rpc module. Those globals are re-initialized for each test case so
they are not directly leaking information between test cases. However if
a late eventlet calls nova.rpc.get_versioned_notifier() it will get a
totally usable FakeVersionedNotifier object regardless of which test
case this notifier is belongs to or which test case the eventlet belongs
to. This way the late eventlet can send a notification to the currently
running test case and therefore can make it fail.
The current case we saw is the following:
1) The test case
nova.tests.functional.test_servers.ServersTestV219.test_description_errors
creates a server but don't wait for it to reach terminal state (ACTIVE
/ ERROR). This test case finishes quickly but leaks running eventlets
in the background waiting for some RPC call to return.
2) As the test case finished the cleanup code deletes the test case
specific setup, including the DB.
3) The test executor moves forward and starts running another test case
4) 60 seconds later the leaked eventlet times out waiting for the RPC
call to return and tries doing things, but fails as the DB is already
gone. Then it tries to report this as an error notification. It calls
nova.rpc.get_versioned_notifier() and gets a fresh notifier that is
connected to the currently running test case. Then emits the error
notification there.
5) The currently running test case also waits for an error notification
to be triggered by the currently running test code. But it gets the
notification form the late eventlet first. As the content of the
notification does not match with the expectations the currently
running test case fails. The late eventlet prints a lot of
error about the DB being gone making the troubleshooting pretty hard.
This patch proposes a way to fix this by marking each eventlet at spawn
time with the id of the test case that was directly or indirectly
started it.
Then when the NotificationFixture gets a notification it compares the
test case id stored in the calling eventlet with the id of the test case
initialized the NotificationFixture. If the two ids do not match then
the fixture ignores the notification and raises an exception to the
caller eventlet to make it terminate.
Change-Id: I012dcf63306bae624dc4f66aae6c6d96a20d4327
Closes-Bug: #1946339
(cherry picked from commit
|
||
|---|---|---|
| api-guide/source | ||
| api-ref/source | ||
| devstack | ||
| doc | ||
| etc/nova | ||
| gate | ||
| nova | ||
| playbooks | ||
| releasenotes | ||
| roles | ||
| tools | ||
| .coveragerc | ||
| .gitignore | ||
| .gitreview | ||
| .mailmap | ||
| .pre-commit-config.yaml | ||
| .stestr.conf | ||
| .zuul.yaml | ||
| bindep.txt | ||
| CONTRIBUTING.rst | ||
| HACKING.rst | ||
| LICENSE | ||
| lower-constraints.txt | ||
| MAINTAINERS | ||
| mypy-files.txt | ||
| README.rst | ||
| requirements.txt | ||
| setup.cfg | ||
| setup.py | ||
| test-requirements.txt | ||
| tox.ini | ||
OpenStack Nova
OpenStack Nova provides a cloud computing fabric controller, supporting a wide variety of compute technologies, including: libvirt (KVM, Xen, LXC and more), Hyper-V, VMware, OpenStack Ironic and PowerVM.
Use the following resources to learn more.
API
To learn how to use Nova's API, consult the documentation available online at:
For more information on OpenStack APIs, SDKs and CLIs in general, refer to:
Operators
To learn how to deploy and configure OpenStack Nova, consult the documentation available online at:
In the unfortunate event that bugs are discovered, they should be reported to the appropriate bug tracker. If you obtained the software from a 3rd party operating system vendor, it is often wise to use their own bug tracker for reporting problems. In all other cases use the master OpenStack bug tracker, available at:
Developers
For information on how to contribute to Nova, please see the contents of the CONTRIBUTING.rst.
Any new code must follow the development guidelines detailed in the HACKING.rst file, and pass all unit tests.
Further developer focused documentation is available at:
Other Information
During each Summit and Project Team Gathering, we agree on what the whole community wants to focus on for the upcoming release. The plans for nova can be found at: