Merge "Adds Pick guest CPU architecture based on host arch in libvirt driver support"
This commit is contained in:
commit
669a4f68a1
|
@ -0,0 +1,349 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
================================================================
|
||||
Pick guest CPU architecture based on host arch in libvirt driver
|
||||
================================================================
|
||||
|
||||
https://blueprints.launchpad.net/nova/+spec/pick-guest-arch-based-on-host-arch-in-libvirt-driver
|
||||
|
||||
Implement new image meta property that allows for the selection of the
|
||||
correct QEMU binary, cpu architecture, and machine type for a guest
|
||||
architecture that is different than the host architecture; An x86_64
|
||||
guest running on an AArch64 host, and vice versa.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Currently, in many places, Nova's libvirt driver makes decisions on how
|
||||
to configure guest XML based on *host* CPU architecture
|
||||
``caps.host.cpu.arch``. That is not optimal in all cases where physical
|
||||
hardware support is limited for non-traditional architectures.
|
||||
|
||||
So all of the said code needs to be reworked to make those decisions
|
||||
based on ``guest`` CPU architecture (i.e. ``guest.arch``, which should be
|
||||
set based on the image metadata property, ``hw_emulation_architecture``).
|
||||
|
||||
A related piece of work is to distinguish between hosts that can do AArch64,
|
||||
PPC64, Etc. via KVM (which is hardware-accelerated) vs. those that can only
|
||||
do it via plain emulation ``TCG`` — this is to ensure that guests are not
|
||||
arbitrarily scheduled on hosts that are incapable of hardware acceleration,
|
||||
thus losing out on performance-related benefits.
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
* As an admin/operator I want to allow for cpu architecture emulation due to
|
||||
constraints of or lack with alternate physical architecture types.
|
||||
|
||||
* As an admin/operator I want to deploy AArch64, PPC64, MIPs, RISC-V, and
|
||||
s390x as an emulated architecture on x86_64.
|
||||
|
||||
* As an admin/operator I want to deploy x86_64, PPC64, MIPs, RISC-V, and
|
||||
s390x as an emulated architecture on AArch64.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
To enable this new cpu architecture spec, an image property will
|
||||
be introduced and an additional function which allows for checks and
|
||||
comparisions between the host architecture and desired emulation architecture
|
||||
|
||||
.. note::
|
||||
|
||||
The following ``hw_architecture`` image property relates to the physical
|
||||
architecture of the compute hosts. If physical nodes are not present for
|
||||
the desired architecture then the instance will not be provisioned.
|
||||
|
||||
Retrieve OS architecture for LibvirtConfigGuest
|
||||
-----------------------------------------------
|
||||
|
||||
This leverages nova virt libvirt config to grab the ``os_arch`` and update
|
||||
the ``hw_architecture`` image meta property with the retrieved value. With
|
||||
this change we can perform the required comparisons within nova virt libvirt
|
||||
driver for the ``hw_architecture`` and ``hw_emulation_architecture`` values.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
if self.os_arch is not None:
|
||||
type_node.set("arch", self.os_arch)
|
||||
|
||||
Allow emulation architecture to be defined by image property
|
||||
------------------------------------------------------------
|
||||
|
||||
To enable defining the guest architecture the following string based image
|
||||
meta property will be introduced:
|
||||
|
||||
* ``hw_emulation_architecture``
|
||||
|
||||
When this image property is not defined then instance provisioning will
|
||||
occur as normal. The process is demonstrated below via the 3 examples.
|
||||
|
||||
**Example 1** When both image meta properties are set the emulation
|
||||
architecture will take precedent, and it will build on a X86_64 host that
|
||||
supports emulatating AARCH64 or whatever supported architecture is inputted
|
||||
in place of AARCH64.
|
||||
|
||||
* ``hw_emulation_architecture = AARCH64``
|
||||
* ``hw_architecture = X86_64``
|
||||
|
||||
**Example 2** When the emulation image meta property is set the emulation
|
||||
architecture will take precedent, and it will build on any host that
|
||||
supports emulating X86_64 or whatever supported architecture is inputted
|
||||
in place of X86_64.
|
||||
|
||||
* ``hw_emulation_architecture = X86_64``
|
||||
* ``hw_architecture = <unset>``
|
||||
|
||||
**Example 3** When the ``hw_emulation_architecture`` property is unset it
|
||||
will build on any host that natively supports the specified architecture.
|
||||
|
||||
* ``hw_emulation_architecture = <unset>``
|
||||
* ``hw_architecture = AARCH64`` OR ``hw_architecture = X86_64``
|
||||
|
||||
Update scheduler request_filter to handle both architecture fields
|
||||
------------------------------------------------------------------
|
||||
|
||||
Within the ``transform_image_metadata`` function, we will add the two
|
||||
architecture properties to the ``prefix_map``. this in itself also requires
|
||||
additional os-traits to be added for both **hw** and **compute**.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def transform_image_metadata(ctxt, request_spec):
|
||||
"""Transform image metadata to required traits.
|
||||
|
||||
This will modify the request_spec to request hosts that support
|
||||
virtualisation capabilities based on the image metadata properties.
|
||||
"""
|
||||
if not CONF.scheduler.image_metadata_prefilter:
|
||||
return False
|
||||
|
||||
prefix_map = {
|
||||
'hw_cdrom_bus': 'COMPUTE_STORAGE_BUS',
|
||||
'hw_disk_bus': 'COMPUTE_STORAGE_BUS',
|
||||
'hw_video_model': 'COMPUTE_GRAPHICS_MODEL',
|
||||
'hw_vif_model': 'COMPUTE_NET_VIF_MODEL',
|
||||
'hw_architecture': 'HW_ARCH',
|
||||
'hw_emulation_architecture': 'COMPUTE_ARCH',
|
||||
}
|
||||
|
||||
|
||||
Update os-traits
|
||||
----------------
|
||||
|
||||
Below are the os-traits proposed for the compute cpu architectures to be
|
||||
supported for emulatation, where as the hardware architecture includes all
|
||||
current nova supported architectures within nova objects fields.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
TRAITS = [
|
||||
'AARCH64',
|
||||
'PPC64LE',
|
||||
'MIPSEL',
|
||||
'S390X',
|
||||
'RISCV64',
|
||||
'X86_64',
|
||||
]
|
||||
|
||||
To account for the emulation of these architectures, updates will be made
|
||||
to the nova virt libvirt driver ensuring that compute capability traits
|
||||
are reported for each architecture emulator that is available on the hosts.
|
||||
|
||||
Perform architecture test against emulation
|
||||
-------------------------------------------
|
||||
|
||||
To facilitate a simple check throughout the nova virt libvirt driver the
|
||||
following function does a check and will set the appropriate guest
|
||||
architecture based on emulation, if defined.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def _check_emulation_arch(self, image_meta):
|
||||
emulation_arch = image_meta.properties.get("hw_emulation_architecture")
|
||||
if emulation_arch:
|
||||
arch = emulation_arch
|
||||
else:
|
||||
arch = libvirt_utils.get_arch(image_meta)
|
||||
return arch
|
||||
|
||||
|
||||
Utilization of the actual check performed through processing the image_meta
|
||||
dictionary values.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
arch = self._check_emulation_arch(image_meta)
|
||||
|
||||
Proposed emulated architectures and current support level
|
||||
---------------------------------------------------------
|
||||
|
||||
All testing performed with changes proposed in this spec demonstrated that
|
||||
the emulated guests maintain current support for all basic lifecycle actions.
|
||||
Listed below are the proposed architectures and there current functional
|
||||
level with the spec, with the plan of all being ``Tested and validated for
|
||||
functional support``.
|
||||
|
||||
* ``X86_64`` - Tested and validated for functional support
|
||||
* ``AARCH64`` - Tested and validated for functional support
|
||||
* ``PPC64LE`` - Tested and validated for functional support
|
||||
* ``MIPSEL`` - Awaiting libvirt patch for PCI support
|
||||
* ``S390X`` - Troubleshooting guest kernel crash for functional support
|
||||
* ``RISCV64`` - To be Tested
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
Other attempts have been made leverage existing image meta properties such
|
||||
as ``hw_architecture`` only; however, this opens various other issues with
|
||||
conflicting check and alterations of core code. This also runs into issues
|
||||
during the scheduling of instances as there will be no matching physical
|
||||
host architectures, which is what this spec aims to solves.
|
||||
|
||||
While the best option is providing actual physical support for the
|
||||
cpu architectures you want to test, this opens the ability to a wider
|
||||
audience to perform the same type of local emulation they can with QEMU
|
||||
within an openstack environment.
|
||||
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
* Adds a new set of standard traits to os-traits.
|
||||
* Adds new property to image_meta objects.
|
||||
* The OS arch value will be pulled into the ``LibvirtConfigGuest``.
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
This is expected to improve boot performance in a heterogeneous cloud
|
||||
by reducing reschedules. By passing a more constrained request to
|
||||
placement this feature should also reduce the resulting set of
|
||||
allocation_candidates that are returned.
|
||||
|
||||
This will also ensure that native support is handled first over emulation
|
||||
as it requires a specific property to be set in order to perform the
|
||||
required checks.
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
Ensure that all the desired QEMU binaries are installed on the physical
|
||||
nodes for the cpu architectures that you would like to support.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
None
|
||||
|
||||
Upgrade impact
|
||||
--------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
chateaulav - Jonathan Race
|
||||
|
||||
Feature Liaison
|
||||
---------------
|
||||
|
||||
Feature liaison:
|
||||
Liaison Needed
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
- Add new traits
|
||||
- Update prefilter
|
||||
- Modify nova libvirt virt driver to perform checks for emulation architecture
|
||||
- Add new property to image_meta objects
|
||||
- Modify nova libvirt virt config to pull OS arch into LibvirtConfigGuest
|
||||
- Tests
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
**Blueprint**
|
||||
|
||||
* https://blueprints.launchpad.net/nova/+spec/pick-guest-arch-based-on-host-arch-in-libvirt-driver
|
||||
|
||||
**Project Changesets**
|
||||
|
||||
* https://review.opendev.org/c/openstack/nova/+/822053
|
||||
* https://review.opendev.org/c/openstack/os-traits/+/824050
|
||||
|
||||
**Libvirt MIPs PCI Bug**
|
||||
|
||||
* https://bugzilla.redhat.com/show_bug.cgi?id=1432101
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Unit tests will be added for validation of the following proposed changes:
|
||||
|
||||
* **nova virt libvirt driver** to validate handling of the
|
||||
``hw_emulation_architecture`` image property value and associated checks.
|
||||
* **nova scheduler request_filter** to ensure proper handling of the
|
||||
prefilter, with added the two new values.
|
||||
|
||||
Proposed updates to tempest will account for the non-native architectures
|
||||
being supported through emulation.
|
||||
|
||||
* AARCH64 architecture will be tested with every patch
|
||||
* Remaining architectures will be tested with the ``periodic-weekly`` and
|
||||
``experimental`` pipelines.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
A release note will be added. As there is enduser impact, user facing
|
||||
documentation will be required for the supported emulation architecture
|
||||
types and the required image meta properties to need to be set.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
* http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026544.html
|
||||
* https://blueprints.launchpad.net/nova/+spec/pick-guest-arch-based-on-host-arch-in-libvirt-driver
|
||||
|
||||
History
|
||||
=======
|
||||
|
||||
.. list-table:: Revisions
|
||||
:header-rows: 1
|
||||
|
||||
* - Release Name
|
||||
- Description
|
||||
* - Yoga
|
||||
- Introduced
|
Loading…
Reference in New Issue