57985 Commits

Author SHA1 Message Date
Lee Yarwood
b62f9a04a0 Add os-volume_attachments reference docs
This change adds a simple sequence diagram showing the flow of a volume
attachment between the various services, using the libvirt driver as an
example virt driver.

Change-Id: I631ac9de3d48aa0ad849f6615d0ad2052cb63e80
2020-11-09 17:12:48 +00:00
Zuul
edd8fefe3f Merge "optimize the shelve code flow" 2020-11-03 09:22:00 +00:00
Zuul
527d6a5b7d Merge "Modify glance's copy_image permission for nova-ceph-multistore" 2020-11-02 22:15:22 +00:00
Zuul
926af1faf1 Merge "objects: Fix issue in exception type" 2020-10-31 11:20:21 +00:00
Zuul
0b7c00b566 Merge "Add regression test for bug #1899649" 2020-10-31 08:14:30 +00:00
Zuul
74bf2a714c Merge "Fix virsh domifstat to get vhostuser vif statistics" 2020-10-31 06:31:40 +00:00
Ghanshyam Mann
e7c7a2851e Modify glance's copy_image permission for nova-ceph-multistore
nova-ceph-multistore setup needs non-admin users to copy the image.
To allow that glance's policy was overriden to allow public
images to copy. This restriction again can cause issue if there
is any new copy image tempest test try to copy private image with
admin users.
- https://review.opendev.org/#/c/742546/

Let's allow everyone to copy every image to make it work
for all type of test credentials.

Change-Id: Ia65afdfb8989909441dba55faeed2d78cc7f1ee7
2020-10-29 12:20:17 -05:00
Zuul
7dcc4cfea7 Merge "Add placeholder migrations for Victoria backports" 2020-10-28 16:28:56 +00:00
Lee Yarwood
0d2ca53bb8 libvirt: Only ask tpool.Proxy to autowrap vir* classes
I668643c836d46a25df46d4c99a973af5e50a39db attempted to fix service wide
pauses by providing a more complete list of classes to tpool.Proxy.

While this excluded libvirtError it can include internal libvirt-python
classes pointed to by private globals that have been introduced with the
use of type checking within the module.

Any attempt to wrap these internal classes will result in the failure
seen in bug #1901383. As a result this change simply ignores any class
found during inspection that doesn't start with the `vir` string, used
by libvirt to denote public methods and classes.

Closes-Bug: #1901383
Co-Authored-By: Daniel Berrange <berrange@redhat.com>
Change-Id: I568b0c4fd6069b9118ff116532f14abb46cc42ab
2020-10-27 09:38:18 +00:00
Zuul
a83001a903 Merge "CinderFixture: Return a driver_type of fake within connection_info" 2020-10-24 01:46:23 +00:00
Zuul
e2a1bc7641 Merge "[doc]: Fix glance image_metadata link" 2020-10-24 01:23:53 +00:00
Stephen Finucane
b32a685595 docs: Make JSON valid
Pygments 2.7.x is stricter in how it validates JSON escapes, aligning it
closer with the spec [1]. Turns out we have some invalid JSON in our
docs, meaning builds are now failing with the following error:

  doc/source/user/metadata.rst:262: WARNING: Could not lex literal_block
  as "json". Highlighting skipped.

Resolve this.

[1] 9514e794e0

Change-Id: Ic50e29e9c7817744ad0b4f9de309aa3e96a09505
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
2020-10-23 17:15:43 +01:00
Zuul
6a5e158756 Merge "Remove self.server dependency form _evacuate_server" 2020-10-22 13:13:04 +00:00
Zuul
2e684ca9f6 Merge "fixtures: Handle binding of first port" 2020-10-22 13:12:57 +00:00
Zuul
5f7e622448 Merge "virt: Remove 'post_interrupted_snapshot_cleanup' API" 2020-10-22 10:41:20 +00:00
Zuul
f62b87cb07 Merge "virt: Remove 'get_console_pool_info' API" 2020-10-21 23:31:50 +00:00
Balazs Gibizer
f5a68826c7 [doc]: Fix glance image_metadata link
Change-Id: Ib1e3798992696cf5f271b20bc731b8b766d173ac
Closes-Bug: #1900731
2020-10-21 18:19:13 +02:00
Zuul
719dfbb98e Merge "Handle disabled CPU features to fix live migration failures" 2020-10-19 20:54:42 +00:00
Zuul
20ca7c4337 Merge "Prevent archiving of pci_devices records because of 'instance_uuid'" 2020-10-18 04:48:04 +00:00
Zuul
7f4afc1a0f Merge "compute: Lock by instance.uuid lock during swap_volume" 2020-10-17 09:54:23 +00:00
Zuul
3d8972612b Merge "Clean up the DynamicVendorData constructor" 2020-10-17 09:45:31 +00:00
Zuul
136cd65b7d Merge "Use cell targeted context to query BDMs for metadata" 2020-10-17 09:45:23 +00:00
Takashi Natsume
78ad4a4efd Add placeholder migrations for Victoria backports
Change-Id: I3f9c9deb11c45e6a252effa3c82951e02d9f4334
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
2020-10-17 07:53:08 +00:00
Zuul
d3b4c0bc93 Merge "bindep: Install python3 and python3-devel on CentOS 8 and Fedora" 2020-10-17 06:22:54 +00:00
Andrew Bonney
eeeca4ceff Handle disabled CPU features to fix live migration failures
When performing a live migration between hypervisors running
libvirt, where one or more CPU features are disabled, nova does
not take account of these. This results in migration failures
as none of the available hypervisor targets appear compatible.

This patch ensures that the libvirt 'disable' poicy is taken
account of, at least in a basic sense, by explicitly ignoring
items flagged in this way when enumerating CPU features.

Closes-Bug: #1898715
Change-Id: Iaf14ca97cfac99dd280d1114123f2d4bb6292b63
2020-10-16 11:25:14 +01:00
Lee Yarwood
e620e45790 CinderFixture: Return a driver_type of fake within connection_info
Previously the attachment info returned by the fixture would incorrectly
list the driver_type outside of the connection_info dict [1][2]. This
changes moves it within the connection_info dict and updates the type to
`fake` for use by the libvirt driver that provides a compatible volume
driver [3] for use in our functional tests.

[1] https://docs.openstack.org/api-ref/block-storage/v3/index.html?expanded=create-attachment-detail#id407
[2] https://docs.openstack.org/api-ref/block-storage/v3/index.html?expanded=update-an-attachment-detail#id411
[3] 2c3ff1004e/nova/virt/libvirt/volume/volume.py (L171-L184)

Change-Id: I73ff0a7508d9f8f6f3d498be13f6f7c505b05e5c
2020-10-14 21:34:33 +01:00
Zuul
2c3ff1004e Merge "libvirt: Log exception when unable to import rbd or rados" 2020-10-14 18:02:26 +00:00
Zuul
16019dac13 Merge "Add a hacking rule for assert_has_calls" 2020-10-14 14:31:15 +00:00
Balazs Gibizer
1f06418549 Remove self.server dependency form _evacuate_server
This has not been caused trouble yet as the only caller called it with
self.server anyhow.

Change-Id: Id8dd11dc45bfc090b966ead0c69e46a17154dadb
2020-10-14 16:29:47 +02:00
Lee Yarwood
770c139edb Add regression test for bug #1899649
Related-Bug: #1899649
Change-Id: If795b603d812917c9257ee5166a979c1fd6687f3
2020-10-14 08:58:55 +01:00
Zuul
1b26ec4cf9 Merge "Fix a hacking test" 2020-10-13 20:46:32 +00:00
zhanghao
4a4f12678a Fix virsh domifstat to get vhostuser vif statistics
When a DPDK VM attaches an interface, libvirt can not automatically
generate "<target dev='vhuXXX'/>" in XML file, which causes virsh
domifstat query to fail.

Change-Id: Id4e5b52af521b5f3a206e87bf024fd1e47fc4824
Closes-Bug: #1899431
2020-10-13 07:17:51 +00:00
melanie witt
1c256cf774 Prevent archiving of pci_devices records because of 'instance_uuid'
Currently in the archive_deleted_rows code, we will attempt to clean up
"residue" of deleted instance records by assuming any table with a
'instance_uuid' column represents data tied to an instance's lifecycle
and delete such records.

This behavior poses a problem in the case where an instance has a PCI
device allocated and someone deletes the instance. The 'instance_uuid'
column in the pci_devices table is used to track the allocation
association of a PCI with an instance. There is a small time window
during which the instance record has been deleted but the PCI device
has not yet been freed from a database record perspective as PCI
devices are freed during the _complete_deletion method in the compute
manager as part of the resource tracker update call.

Records in the pci_devices table are anyway not related to the
lifecycle of instances so they should not be considered residue to
clean up if an instance is deleted. This adds a condition to avoid
archiving pci_devices on the basis of an instance association.

Closes-Bug: #1899541

Change-Id: Ie62d3566230aa3e2786d129adbb2e3570b06e4c6
2020-10-12 22:52:00 +00:00
Stephen Finucane
12cc805d36 objects: Fix issue in exception type
We were attempting to pass a 'target_version' variable into an exception
message. 'target_version' is a tuple which means it's expanded out
resulting in the following error:

  TypeError: not all arguments converted during string formatting

Fix this.

Change-Id: I6063b2108ae38776d034fd7a4c3aa88dc66a084f
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Closes-Bug: #1898554
2020-10-12 10:09:10 +01:00
Balazs Gibizer
450213f1f0 Clean up the DynamicVendorData constructor
There are a bunch of unused parameters in DynamicVendorData constructor
and a comment that they cannot be removed due to JsonFileVendorData. But
JsonFileVendorData does not depends on those paramters and both the
base class and JsonFileVendorData uses *args **kwargs. So it is safe to
remove the unused params.

The context field of DynamicVendorData is also removed as it is unused.
This makes the request_context parameter of the InstanceMeta
constructor also unused so that is removed.

Change-Id: Ie27fd6a5513e53903b9acd5d63038b3b484acbde
2020-10-12 08:25:32 +00:00
Balazs Gibizer
1390eecf8d Use cell targeted context to query BDMs for metadata
The metadata service supports a multicell deployment in a configuration
where the nova-api service implements the metadata API. In this case the
metadata query needs to be cell targeted. This was partly implemented
already. The instance itself is queried from the cell DB properly.
However the BDM data used the non targeted context resulting in an empty
BDM returned by the metadata service.

Functional reproduction test is not added as I did not find a way to
have a cell setup in the functional test that reproduce the problem. I
reproduced the bug and tested the fix in a devstack.

Change-Id: I48f57082edaef3ec4722bd31ce29a90b94d32523
Closes-Bug: #1881944
2020-10-12 08:25:24 +00:00
Zuul
261de76104 Merge "Follow up for cherry-pick check for merge patch" 2020-10-12 07:26:50 +00:00
Zuul
3756f4ffa6 Merge "libvirt: 'video.vram' property must be an integer" 2020-10-10 14:40:49 +00:00
Zuul
745f835e9d Merge "libvirt: Increase incremental and max sleep time during device detach" 2020-10-10 14:40:42 +00:00
Lee Yarwood
6cf449bdd0 compute: Lock by instance.uuid lock during swap_volume
The libvirt driver is currently the only virt driver implementing swap
volume within Nova. While libvirt itself does support moving between
multiple volumes attached to the same instance at the same time the
current logic within the libvirt driver makes a call to
virDomainGetXMLDesc that fails if there are active block jobs against
any disk attached to the domain.

This change simply uses an instance.uuid based lock in the compute layer
to serialise requests to swap_volume to avoid this from being possible.

Closes-Bug: #1896621
Change-Id: Ic5ce2580e7638a47f1ffddb4edbb503bf490504c
2020-10-09 10:48:34 +00:00
dengzhaosen
383bcbca70 optimize the shelve code flow
Change-Id: I23d59ba4a33a183f99f651cc04fadefb3ea68722
2020-10-08 16:48:15 +08:00
melanie witt
1e10461c71 Follow up for cherry-pick check for merge patch
This is a follow up to change
I8e4e5afc773d53dee9c1c24951bb07a45ddc2f1a which fixed an issue with
validation when the topmost patch after a Zuul rebase is a merge
patch.

We need to also use the $commit_hash variable for the check for
stable-only patches, else it will incorrectly fail because it is
checking the merge patch's commit message.

Change-Id: Ia725346b65dd5e2f16aa049c74b45d99e22b3524
2020-10-08 05:45:24 +00:00
Zuul
2745e68537 Merge "Add Python3 wallaby unit tests" 2020-10-06 23:01:27 +00:00
Stephen Finucane
f2ca089bce libvirt: 'video.vram' property must be an integer
The 'vram' property of the 'video' device must be an integer else
libvirt will spit the dummy out, e.g.

  libvirt.libvirtError: XML error: cannot parse video vram '8192.0'

The division operator in Python 3 results in a float, not an integer
like in Python 2. Use the truncation division operator instead.

Change-Id: Iebf678c229da4f455459d068cafeee5f241aea1f
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Closes-Bug: #1896496
2020-10-06 09:05:23 +00:00
Lee Yarwood
dd1e6d4b0c libvirt: Increase incremental and max sleep time during device detach
Bug #1894804 outlines how DEVICE_DELETED events were often missing from
QEMU on Focal based OpenStack CI hosts as originally seen in bug
 #1882521. This has eventually been tracked down to some undefined QEMU
behaviour when a new device_del QMP command is received while another is
still being processed, causing the original attempt to be aborted.

We hit this race in slower OpenStack CI envs as n-cpu rather crudely
retries attempts to detach devices using the RetryDecorator from
oslo.service. The default incremental sleep time currently being tight
enough to ensure QEMU is still processing the first device_del request
on these slower CI hosts when n-cpu asks libvirt to retry the detach,
sending another device_del to QEMU hitting the above behaviour.

Additionally we have also seen the following check being hit when
testing with QEMU >= v5.0.0. This check now rejects overlapping
device_del requests in QEMU rather than aborting the original:

cce8944cc9

This change aims to avoid this situation entirely by raising the default
incremental sleep time between detach requests from 2 seconds to 10,
leaving enough time for the first attempt to complete. The overall
maximum sleep time is also increased from 30 to 60 seconds.

Future work will aim to entirely remove this retry logic with a libvirt
event driven approach, polling for the the
VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED and
VIR_DOMAIN_EVENT_ID_DEVICE_REMOVAL_FAILED events before retrying.

Finally, the cleanup of unused arguments in detach_device_with_retry is
left for a follow up change in order to keep this initial change small
enough to quickly backport.

Closes-Bug: #1882521
Related-Bug: #1894804
Change-Id: Ib9ed7069cef5b73033351f7a78a3fb566753970d
2020-10-06 09:42:26 +01:00
Zuul
655cd30c9f Merge "Update master for stable/victoria" 2020-10-06 02:28:54 +00:00
Zuul
07c68e45dd Merge "Log stack trace when fails launching a defined domain" 2020-10-06 00:30:48 +00:00
Lee Yarwood
aa16dd09eb libvirt: Log exception when unable to import rbd or rados
This should help provide some context when the RbdDriver later raises a
RuntimeError if rbd or rados hasn't been imported correctly.

Change-Id: Ie8bb5e5622bd37dfe8073cca12f77174e8e7d98c
2020-10-05 16:27:04 +01:00
Zuul
43801cf24a Merge "libvirt: Remove MIN_{LIBVIRT,QEMU}_PMEM_SUPPORT" 2020-10-05 14:08:13 +00:00
Zuul
8bb44fd545 Merge "nova-evacuate: Disable libvirtd service and sockets during negative tests" 2020-10-04 18:14:56 +00:00