Merge "Adds Pick guest CPU architecture based on host arch in libvirt driver support"

This commit is contained in:
Zuul 2022-01-13 14:40:57 +00:00 committed by Gerrit Code Review
commit 669a4f68a1
1 changed files with 349 additions and 0 deletions

View File

@ -0,0 +1,349 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
================================================================
Pick guest CPU architecture based on host arch in libvirt driver
================================================================
https://blueprints.launchpad.net/nova/+spec/pick-guest-arch-based-on-host-arch-in-libvirt-driver
Implement new image meta property that allows for the selection of the
correct QEMU binary, cpu architecture, and machine type for a guest
architecture that is different than the host architecture; An x86_64
guest running on an AArch64 host, and vice versa.
Problem description
===================
Currently, in many places, Nova's libvirt driver makes decisions on how
to configure guest XML based on *host* CPU architecture
``caps.host.cpu.arch``. That is not optimal in all cases where physical
hardware support is limited for non-traditional architectures.
So all of the said code needs to be reworked to make those decisions
based on ``guest`` CPU architecture (i.e. ``guest.arch``, which should be
set based on the image metadata property, ``hw_emulation_architecture``).
A related piece of work is to distinguish between hosts that can do AArch64,
PPC64, Etc. via KVM (which is hardware-accelerated) vs. those that can only
do it via plain emulation ``TCG`` — this is to ensure that guests are not
arbitrarily scheduled on hosts that are incapable of hardware acceleration,
thus losing out on performance-related benefits.
Use Cases
---------
* As an admin/operator I want to allow for cpu architecture emulation due to
constraints of or lack with alternate physical architecture types.
* As an admin/operator I want to deploy AArch64, PPC64, MIPs, RISC-V, and
s390x as an emulated architecture on x86_64.
* As an admin/operator I want to deploy x86_64, PPC64, MIPs, RISC-V, and
s390x as an emulated architecture on AArch64.
Proposed change
===============
To enable this new cpu architecture spec, an image property will
be introduced and an additional function which allows for checks and
comparisions between the host architecture and desired emulation architecture
.. note::
The following ``hw_architecture`` image property relates to the physical
architecture of the compute hosts. If physical nodes are not present for
the desired architecture then the instance will not be provisioned.
Retrieve OS architecture for LibvirtConfigGuest
-----------------------------------------------
This leverages nova virt libvirt config to grab the ``os_arch`` and update
the ``hw_architecture`` image meta property with the retrieved value. With
this change we can perform the required comparisons within nova virt libvirt
driver for the ``hw_architecture`` and ``hw_emulation_architecture`` values.
.. code-block:: python
if self.os_arch is not None:
type_node.set("arch", self.os_arch)
Allow emulation architecture to be defined by image property
------------------------------------------------------------
To enable defining the guest architecture the following string based image
meta property will be introduced:
* ``hw_emulation_architecture``
When this image property is not defined then instance provisioning will
occur as normal. The process is demonstrated below via the 3 examples.
**Example 1** When both image meta properties are set the emulation
architecture will take precedent, and it will build on a X86_64 host that
supports emulatating AARCH64 or whatever supported architecture is inputted
in place of AARCH64.
* ``hw_emulation_architecture = AARCH64``
* ``hw_architecture = X86_64``
**Example 2** When the emulation image meta property is set the emulation
architecture will take precedent, and it will build on any host that
supports emulating X86_64 or whatever supported architecture is inputted
in place of X86_64.
* ``hw_emulation_architecture = X86_64``
* ``hw_architecture = <unset>``
**Example 3** When the ``hw_emulation_architecture`` property is unset it
will build on any host that natively supports the specified architecture.
* ``hw_emulation_architecture = <unset>``
* ``hw_architecture = AARCH64`` OR ``hw_architecture = X86_64``
Update scheduler request_filter to handle both architecture fields
------------------------------------------------------------------
Within the ``transform_image_metadata`` function, we will add the two
architecture properties to the ``prefix_map``. this in itself also requires
additional os-traits to be added for both **hw** and **compute**.
.. code-block:: python
def transform_image_metadata(ctxt, request_spec):
"""Transform image metadata to required traits.
This will modify the request_spec to request hosts that support
virtualisation capabilities based on the image metadata properties.
"""
if not CONF.scheduler.image_metadata_prefilter:
return False
prefix_map = {
'hw_cdrom_bus': 'COMPUTE_STORAGE_BUS',
'hw_disk_bus': 'COMPUTE_STORAGE_BUS',
'hw_video_model': 'COMPUTE_GRAPHICS_MODEL',
'hw_vif_model': 'COMPUTE_NET_VIF_MODEL',
'hw_architecture': 'HW_ARCH',
'hw_emulation_architecture': 'COMPUTE_ARCH',
}
Update os-traits
----------------
Below are the os-traits proposed for the compute cpu architectures to be
supported for emulatation, where as the hardware architecture includes all
current nova supported architectures within nova objects fields.
.. code-block:: python
TRAITS = [
'AARCH64',
'PPC64LE',
'MIPSEL',
'S390X',
'RISCV64',
'X86_64',
]
To account for the emulation of these architectures, updates will be made
to the nova virt libvirt driver ensuring that compute capability traits
are reported for each architecture emulator that is available on the hosts.
Perform architecture test against emulation
-------------------------------------------
To facilitate a simple check throughout the nova virt libvirt driver the
following function does a check and will set the appropriate guest
architecture based on emulation, if defined.
.. code-block:: python
def _check_emulation_arch(self, image_meta):
emulation_arch = image_meta.properties.get("hw_emulation_architecture")
if emulation_arch:
arch = emulation_arch
else:
arch = libvirt_utils.get_arch(image_meta)
return arch
Utilization of the actual check performed through processing the image_meta
dictionary values.
.. code-block:: python
arch = self._check_emulation_arch(image_meta)
Proposed emulated architectures and current support level
---------------------------------------------------------
All testing performed with changes proposed in this spec demonstrated that
the emulated guests maintain current support for all basic lifecycle actions.
Listed below are the proposed architectures and there current functional
level with the spec, with the plan of all being ``Tested and validated for
functional support``.
* ``X86_64`` - Tested and validated for functional support
* ``AARCH64`` - Tested and validated for functional support
* ``PPC64LE`` - Tested and validated for functional support
* ``MIPSEL`` - Awaiting libvirt patch for PCI support
* ``S390X`` - Troubleshooting guest kernel crash for functional support
* ``RISCV64`` - To be Tested
Alternatives
------------
Other attempts have been made leverage existing image meta properties such
as ``hw_architecture`` only; however, this opens various other issues with
conflicting check and alterations of core code. This also runs into issues
during the scheduling of instances as there will be no matching physical
host architectures, which is what this spec aims to solves.
While the best option is providing actual physical support for the
cpu architectures you want to test, this opens the ability to a wider
audience to perform the same type of local emulation they can with QEMU
within an openstack environment.
Data model impact
-----------------
* Adds a new set of standard traits to os-traits.
* Adds new property to image_meta objects.
* The OS arch value will be pulled into the ``LibvirtConfigGuest``.
REST API impact
---------------
None
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
None
Performance Impact
------------------
This is expected to improve boot performance in a heterogeneous cloud
by reducing reschedules. By passing a more constrained request to
placement this feature should also reduce the resulting set of
allocation_candidates that are returned.
This will also ensure that native support is handled first over emulation
as it requires a specific property to be set in order to perform the
required checks.
Other deployer impact
---------------------
Ensure that all the desired QEMU binaries are installed on the physical
nodes for the cpu architectures that you would like to support.
Developer impact
----------------
None
Upgrade impact
--------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
chateaulav - Jonathan Race
Feature Liaison
---------------
Feature liaison:
Liaison Needed
Work Items
----------
- Add new traits
- Update prefilter
- Modify nova libvirt virt driver to perform checks for emulation architecture
- Add new property to image_meta objects
- Modify nova libvirt virt config to pull OS arch into LibvirtConfigGuest
- Tests
Dependencies
============
**Blueprint**
* https://blueprints.launchpad.net/nova/+spec/pick-guest-arch-based-on-host-arch-in-libvirt-driver
**Project Changesets**
* https://review.opendev.org/c/openstack/nova/+/822053
* https://review.opendev.org/c/openstack/os-traits/+/824050
**Libvirt MIPs PCI Bug**
* https://bugzilla.redhat.com/show_bug.cgi?id=1432101
Testing
=======
Unit tests will be added for validation of the following proposed changes:
* **nova virt libvirt driver** to validate handling of the
``hw_emulation_architecture`` image property value and associated checks.
* **nova scheduler request_filter** to ensure proper handling of the
prefilter, with added the two new values.
Proposed updates to tempest will account for the non-native architectures
being supported through emulation.
* AARCH64 architecture will be tested with every patch
* Remaining architectures will be tested with the ``periodic-weekly`` and
``experimental`` pipelines.
Documentation Impact
====================
A release note will be added. As there is enduser impact, user facing
documentation will be required for the supported emulation architecture
types and the required image meta properties to need to be set.
References
==========
* http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026544.html
* https://blueprints.launchpad.net/nova/+spec/pick-guest-arch-based-on-host-arch-in-libvirt-driver
History
=======
.. list-table:: Revisions
:header-rows: 1
* - Release Name
- Description
* - Yoga
- Introduced