Libvirt driver emulator threads placement policy
The Nova schedular determines CPU resource utilization and instance CPU placement based on the number of vCPUs in the flavour. A number of hypervisors have work that is performed in the host OS on behalf of a guest instance, which does not take place in association with a vCPU. This is currently unaccounted for in Nova scheduling and cannot have any placement policy controls applied. Blueprint: libvirt-emulator-threads-policy Change-Id: I069cd14ea89045136ae8a29c2d2f6c8c17157533
This commit is contained in:
Daniel P. Berrange
committed by
Sahid Orentino Ferdjaoui
Sahid Orentino Ferdjaoui
parent
5e5b627aee
commit
ad39cf6b87
256
specs/ocata/approved/libvirt-emulator-threads-policy.rst
Normal file
256
specs/ocata/approved/libvirt-emulator-threads-policy.rst
Normal file
@@ -0,0 +1,256 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
================================================
|
||||
Libvirt driver emulator threads placement policy
|
||||
================================================
|
||||
|
||||
https://blueprints.launchpad.net/nova/+spec/libvirt-emulator-threads-policy
|
||||
|
||||
The Nova scheduler determines CPU resource utilization and instance
|
||||
CPU placement based on the number of vCPUs in the flavor. A number of
|
||||
hypervisors have operations that are being performed on behalf of the
|
||||
guest instance in the host OS. These operations should be accounted
|
||||
and scheduled separately, as well as have their own placement policy
|
||||
controls applied.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
The Nova scheduler determines CPU resource utilization by counting the
|
||||
number of vCPUs allocated for each guest. When doing overcommit, as
|
||||
opposed to dedicated resources, this vCPU count is multiplied by an
|
||||
overcommit ratio. This utilization is then used to determine optimal
|
||||
guest placement across compute nodes, or within NUMA nodes.
|
||||
|
||||
A number of hypervisors, however, perform work on behalf of a guest
|
||||
instance in an execution context that is not associated with the
|
||||
virtual instance vCPUs. With KVM / QEMU, there are one or more threads
|
||||
associated with the QEMU process which are used for the QEMU main
|
||||
event loop, asynchronous I/O operation completion, migration data
|
||||
transfer, SPICE display I/O and more. With Xen, if the stub-domain
|
||||
feature is in use, there is an entire domain used to provide I/O
|
||||
backends for the main domain.
|
||||
|
||||
Nova does not have any current mechanism to either track this extra
|
||||
guest instance compute requirement in order to measure utilization,
|
||||
nor to place any control over its execution policy.
|
||||
|
||||
The libvirt driver has implemented a generic placement policy for KVM
|
||||
whereby the QEMU emulator threads are allowed to float across the same
|
||||
pCPUs that the instance vCPUs are running on. In other words, the
|
||||
emulator threads will steal some time from the vCPUs whenever they
|
||||
have work to do. This is just about acceptable in the case where CPU
|
||||
overcommit is being used. However, when guests want dedicated vCPU
|
||||
allocation though, there is a desire to be able to express other
|
||||
placement policies, for example, to allocate one or more pCPUs to be
|
||||
dedicated to a guest's emulator threads. This becomes critical as Nova
|
||||
continues to implement support for real-time workloads, as it will not
|
||||
be acceptable to allow emulator threads to steal time from real-time
|
||||
vCPUs.
|
||||
|
||||
While it would be possible for the libvirt driver to add different
|
||||
placement policies, unless the concept of emulator threads is exposed
|
||||
to the scheduler in some manner, CPU usage cannot be expressed in a
|
||||
satisfactory manner. Thus there needs to be a way to describe to the
|
||||
scheduler what other CPU usage may be associated with a guest, and
|
||||
account for that during placement.
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
With current Nova real time support in libvirt, there is a requirement
|
||||
to reserve one vCPU for running non-realtime workloads. The QEMU
|
||||
emulator threads are pinned to run on the same host pCPU as this
|
||||
vCPU. While this requirement is just about acceptable for Linux
|
||||
guests, it prevents use of Nova to run other real time operating
|
||||
systems which require realtime response for all vCPUs. To broaden the
|
||||
realtime support it is necessary to pin emulator threads separately
|
||||
from vCPUs, which requires that the scheduler be able to account for
|
||||
extra pCPU usage per guest.
|
||||
|
||||
Project Priority
|
||||
----------------
|
||||
|
||||
None
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
A pre-requisite for enabling the emulator threads placement policy
|
||||
feature on a flavor is that it must also have ‘hw:cpu_policy’ set to
|
||||
‘dedicated’.
|
||||
|
||||
Each hypervisor has a different architecture, for example QEMU has
|
||||
emulator threads, while Xen has stub-domains. To avoid favoring any
|
||||
specific implementation, the idea is to extend
|
||||
`estimate_instance_overhead` to return 1 additional host CPU to take
|
||||
into account during claim. A user which expresses the desire to
|
||||
isolate emulator threads must use a flavor configured to accept that
|
||||
specification as:
|
||||
|
||||
* hw:cpu_emulator_threads=isolate
|
||||
|
||||
Would say that this instance is to be considered to consume 1
|
||||
additional host CPU. That pCPU used to make running emulator threads
|
||||
is going to always be configured on the related guest NUMA node ID 0,
|
||||
to make it predictable for users. Currently there is no desire to make
|
||||
customizable the number of host CPUs running emulator threads since
|
||||
only one should work for almost every use case. If in the future there
|
||||
is a desire to isolate more than one host CPU to run emulator threads,
|
||||
we would implement instead I/O threads to add granularity on
|
||||
dedicating used resources to run guests on host CPUs.
|
||||
|
||||
As we said an additional pCPU is going to be consumed but this first
|
||||
implementation is not going to update the user quotas, that in a
|
||||
spirit of simplicity since quotas already leak on different scenarios.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
We could use a host level tunable to just reserve a set of host pCPUs
|
||||
for running emulator threads globally, instead of trying to account
|
||||
for it per instance. This would work in the simple case, but when NUMA
|
||||
is used, it is highly desirable to have more fine grained config to
|
||||
control emulator thread placement. When real-time or dedicated CPUs
|
||||
are used, it will be critical to separate emulator threads for
|
||||
different KVM instances.
|
||||
|
||||
Another option is to hardcode an assumption that the vCPUs number set
|
||||
against the flavour implicitly includes 1 vCPUs for emulator. eg a
|
||||
vCPU value of 5 would imply 4 actual vCPUs and 1 system pseudo-vCPU.
|
||||
This would likely be extremely confusing to tenant users, and
|
||||
developers alike.
|
||||
|
||||
Do nothing is always an option. If we did nothing, then it would limit
|
||||
the types of workload that can be run on Nova. This would have a
|
||||
negative impact inparticular on users making use of the dedicated vCPU
|
||||
feature, as there would be no way to guarantee their vCPUs are not
|
||||
pre-empted by emulator threads. It can be worked around to some degree
|
||||
with realtime by setting a fixed policy that the emulator threads only
|
||||
run on the vCPUs that have non-realtime policy. This requires that all
|
||||
guest OS using realtime are SMP, but some guest OS want realtime, but
|
||||
are only UP.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
The InstanceNUMATopology object will be extended to have a new field
|
||||
|
||||
* cpu_emulator_threads=CPUEmulatorThreadsField()
|
||||
|
||||
This field will be implemented as an enum with two options:
|
||||
|
||||
* shared - The emulator threads float across the pCPUs associated to
|
||||
the guest.
|
||||
* isolate - The emulator threads are isolated on a single pCPU.
|
||||
|
||||
By default 'shared' will be used. It's important to note that: Since
|
||||
[1] on kernel the load-balancing on CPU isolated from the kernel
|
||||
command line using 'isolcpus=' has been removed. It means that the
|
||||
emulator threads are not going to float on the union of pCPUs
|
||||
dedicated to the guest but instead be constrained to the pCPU running
|
||||
vCPU 0.
|
||||
|
||||
[1] https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux-stable/+/47b8ea7186aae7f474ec4c98f43eaa8da719cd83%5E%21/#F0
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
For end users, using the option 'cpu_emulator_threads' is going to
|
||||
consume an additional host CPU on the resources quota regarding the
|
||||
guest vCPUs allocated.
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
The NUMA and compute scheduler filters will have some changes to them,
|
||||
but it is not anticipated that they will become more computationally
|
||||
expensive to any measurable degree.
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
Deployers who want to use that new feature will have to configure
|
||||
their flavors to use a dedicated cpu policy (hw:cpu_policy=dedicated),
|
||||
in a same time set 'hw:cpu_emulator_threads' to 'isolate'.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
* Developers of other virtualization drivers may wish to make use of
|
||||
the new flavor extra spec property and scheduler accounting. This
|
||||
will be of particular interest to the Xen hypervisor, if using the
|
||||
stub domain feature.
|
||||
|
||||
* Developers of metric or GUI systems have to take into account that
|
||||
host CPU overhead which are going to be consumed by instances with a
|
||||
`cpu_emulator_threads` set as `isolate`.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
sahid-ferdjaoui
|
||||
|
||||
Other contributors:
|
||||
berrange
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
* Enhance flavor extra spec to take into account hw:cpu_emulator_threads
|
||||
* Enhance InstanceNUMATopology to take into account cpu_emulator_threads
|
||||
* Make resource tracker to handle 'estimate_instance_overhead' with vcpus
|
||||
* Extend estimate_instance_overhead for libvirt
|
||||
* Make libvirt to corretly pin emulator threads if requested.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
The realtime spec is not a pre-requisite, but is complementary to
|
||||
this work
|
||||
|
||||
* https://blueprints.launchpad.net/nova/+spec/libvirt-real-time
|
||||
* https://review.openstack.org/#/c/139688/
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
This can be tested in any CI system that is capable of testing the
|
||||
current NUMA and dedicated CPUs policy. i.e. It requires ability to
|
||||
use KVM and not merely QEMU. Functionnal tests for the scheduling and
|
||||
driver bits (libvirt) are going to be added.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
The documentation detailing NUMA and dedicated CPU policy usage will need
|
||||
to be extended to also describe the new options this work introduces.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
History
|
||||
=======
|
||||
Reference in New Issue
Block a user