OpenStack Compute (Nova)
Go to file
Lee Yarwood 4819f694b2 libvirt: Increase incremental and max sleep time during device detach
Bug #1894804 outlines how DEVICE_DELETED events were often missing from
QEMU on Focal based OpenStack CI hosts as originally seen in bug
 #1882521. This has eventually been tracked down to some undefined QEMU
behaviour when a new device_del QMP command is received while another is
still being processed, causing the original attempt to be aborted.

We hit this race in slower OpenStack CI envs as n-cpu rather crudely
retries attempts to detach devices using the RetryDecorator from
oslo.service. The default incremental sleep time currently being tight
enough to ensure QEMU is still processing the first device_del request
on these slower CI hosts when n-cpu asks libvirt to retry the detach,
sending another device_del to QEMU hitting the above behaviour.

Additionally we have also seen the following check being hit when
testing with QEMU >= v5.0.0. This check now rejects overlapping
device_del requests in QEMU rather than aborting the original:

cce8944cc9

This change aims to avoid this situation entirely by raising the default
incremental sleep time between detach requests from 2 seconds to 10,
leaving enough time for the first attempt to complete. The overall
maximum sleep time is also increased from 30 to 60 seconds.

Future work will aim to entirely remove this retry logic with a libvirt
event driven approach, polling for the the
VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED and
VIR_DOMAIN_EVENT_ID_DEVICE_REMOVAL_FAILED events before retrying.

Finally, the cleanup of unused arguments in detach_device_with_retry is
left for a follow up change in order to keep this initial change small
enough to quickly backport.

Closes-Bug: #1882521
Related-Bug: #1894804
Change-Id: Ib9ed7069cef5b73033351f7a78a3fb566753970d
(cherry picked from commit dd1e6d4b0c)
2020-10-10 19:40:20 +01:00
api-guide/source doc: Fix rendering in the accelerator support doc 2020-09-18 09:03:15 +00:00
api-ref/source trivial: Final cleanup 2020-09-11 14:09:06 +01:00
devstack Merge "Find instance in another cell during floating IP re-association" 2019-09-13 15:19:55 +00:00
doc Merge "docs: fix aggregate weight multiplier property names" 2020-09-22 15:27:51 +00:00
etc/nova Allow versioned discovery unauthenticated 2020-04-03 21:24:28 +00:00
gate test_evacuate.sh: Stop using libvirt-bin 2020-09-22 10:29:37 +01:00
nova libvirt: Increase incremental and max sleep time during device detach 2020-10-10 19:40:20 +01:00
playbooks Make our ceph job test with glance in multistore mode 2020-07-14 09:11:42 -07:00
releasenotes libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION 2020-09-22 10:29:37 +01:00
roles Enable cross-cell resize in the nova-multi-cell job 2019-12-23 10:10:57 -05:00
tools tools: Remove xenserver tooling 2020-08-31 15:53:35 +01:00
.coveragerc Remove nova/openstack/* from .coveragerc 2016-10-12 16:20:49 -04:00
.gitignore tox: Integrate mypy 2020-05-15 15:59:53 +01:00
.gitreview [stable-only] Update .gitreview for stable/victoria 2020-09-25 14:16:20 +02:00
.mailmap Add mailmap entry 2014-05-07 12:14:26 -07:00
.pre-commit-config.yaml Switch to hacking 2.x 2020-01-17 11:30:40 +00:00
.stestr.conf Finish stestr migration 2017-11-24 16:51:12 -05:00
.zuul.yaml Run voting jobs in gate pipeline also 2020-09-05 22:35:14 +00:00
CONTRIBUTING.rst [Community goal] Update contributor documentation 2020-03-25 12:01:37 +00:00
HACKING.rst Remove hacking rules for python 2/3 compatibility 2020-06-17 08:19:13 +00:00
LICENSE initial commit 2010-05-27 23:05:26 -07:00
MAINTAINERS Fix broken URLs 2017-09-07 15:42:31 +02:00
README.rst docs: Remove references to XenAPI driver 2020-08-31 15:53:31 +01:00
bindep.txt Add lsscsi to bindep 2020-08-05 16:16:49 -05:00
lower-constraints.txt Migrate default policy file from JSON to YAML 2020-09-09 08:09:38 -05:00
mypy-files.txt Add type hints to 'nova.virt.libvirt.utils' 2020-09-04 14:42:18 +01:00
requirements.txt Migrate default policy file from JSON to YAML 2020-09-09 08:09:38 -05:00
setup.cfg tox: Integrate mypy 2020-05-15 15:59:53 +01:00
setup.py Updated from global requirements 2017-03-02 11:50:48 +00:00
test-requirements.txt [goal] Prepare for job migration to Ubuntu Focal (20.04) 2020-08-18 11:28:32 +00:00
tox.ini [stable-only] Update TOX_CONSTRAINTS_FILE for stable/victoria 2020-09-25 14:18:11 +02:00

README.rst

OpenStack Nova

image

OpenStack Nova provides a cloud computing fabric controller, supporting a wide variety of compute technologies, including: libvirt (KVM, Xen, LXC and more), Hyper-V, VMware, OpenStack Ironic and PowerVM.

Use the following resources to learn more.

API

To learn how to use Nova's API, consult the documentation available online at:

For more information on OpenStack APIs, SDKs and CLIs in general, refer to:

Operators

To learn how to deploy and configure OpenStack Nova, consult the documentation available online at:

In the unfortunate event that bugs are discovered, they should be reported to the appropriate bug tracker. If you obtained the software from a 3rd party operating system vendor, it is often wise to use their own bug tracker for reporting problems. In all other cases use the master OpenStack bug tracker, available at:

Developers

For information on how to contribute to Nova, please see the contents of the CONTRIBUTING.rst.

Any new code must follow the development guidelines detailed in the HACKING.rst file, and pass all unit tests.

Further developer focused documentation is available at:

Other Information

During each Summit and Project Team Gathering, we agree on what the whole community wants to focus on for the upcoming release. The plans for nova can be found at: