Use PCPU and VCPU in one instance
We'd like to propose a way to create instance to take using of PCPU and VCPU resource in one instance. Re-proposed in U release. Change-Id: I0c4ffe349dc9381c83c836efb4af99d072169c88
This commit is contained in:
Wang Huaqiang
397
specs/ussuri/approved/use-pcpu-vcpu-in-one-instance.rst
Normal file
397
specs/ussuri/approved/use-pcpu-vcpu-in-one-instance.rst
Normal file
@@ -0,0 +1,397 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
=========================================
|
||||||
|
Use ``PCPU`` and ``VCPU`` in One Instance
|
||||||
|
=========================================
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/nova/+spec/use-pcpu-and-vcpu-in-one-instance
|
||||||
|
|
||||||
|
The spec `CPU resource tracking`_ splits host CPUs into ``PCPU`` and ``VCPU``
|
||||||
|
resources, making it possible to run instances of ``dedicated`` CPU allocation
|
||||||
|
policy and instances of ``shared`` CPU allocation policy in the same host.
|
||||||
|
This spec aims to create such kind of instance that some of the CPUs are
|
||||||
|
dedicated (``PCPU``) CPUs and the remaining CPUs are shared (``VCPU``) CPUs
|
||||||
|
and expose this information via the metadata API.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
The current CPU allocation policy, ``dedicated`` or ``shared``, is applied to
|
||||||
|
all CPUs of an instance. However, with the introduction of
|
||||||
|
`CPU resource tracking`_, it is possible to propose a more fine-grained CPU
|
||||||
|
allocation policy, which is based on the control over individual instance CPU,
|
||||||
|
and specifying the ``dedicated`` or ``shared`` CPU allocation policy to each
|
||||||
|
instance CPU.
|
||||||
|
|
||||||
|
Use Cases
|
||||||
|
---------
|
||||||
|
|
||||||
|
As an operator, I would like to have an instance with some realtime CPUs for
|
||||||
|
high performance, and at the same time, in order to increase instance density,
|
||||||
|
I wish to make the remaining CPUs, which do not demand high performance,
|
||||||
|
shared with other instances because I only care about the performance of
|
||||||
|
realtime CPUs. One example is deploying the NFV task that is enhanced with
|
||||||
|
DPDK framework in the instance, in which the data plane threads could be
|
||||||
|
processed with the realtime CPUs and the control-plane tasks are scheduled
|
||||||
|
on CPUs that may be shared with other instances.
|
||||||
|
|
||||||
|
As a Kubernetes administrator, I wish to run a multi-tier or auto-scaling
|
||||||
|
application in Kubernetes, which is running in single OpenStack VM, with
|
||||||
|
the expectation that using dedicated high-performance CPUs for application
|
||||||
|
itself and deploying the containers on shared cores.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
Introduce a new CPU allocation policy ``mixed``
|
||||||
|
-----------------------------------------------
|
||||||
|
|
||||||
|
``dedicated`` and ``shared`` are the existing instance CPU allocation policies
|
||||||
|
that determine how instance CPU is scheduled on host CPU. This specification
|
||||||
|
proposes a new CPU allocation policy, with the name ``mixed``, to
|
||||||
|
create a CPU *mixed* instance in such way that some instance CPUs are
|
||||||
|
allocated from computing node's ``PCPU`` resource, and the rest of instance
|
||||||
|
CPUs are allocated from the ``VCPU`` resources. The CPU allocated from
|
||||||
|
``PCPU`` resource will be pinned on particular host CPUs which are defined in
|
||||||
|
``CONF.compute.dedicated_cpu_set``, and the CPU from ``VCPU`` resource will be
|
||||||
|
floating on the host CPUs which are defined in ``CONF.compute.shared_cpu_set``.
|
||||||
|
In this proposal, we call these two kinds of CPUs as *dedicated* vCPU and
|
||||||
|
*shared* vCPU respectively.
|
||||||
|
|
||||||
|
Instance CPU policy matrix
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Nova operator may set the instance CPU allocation policy through the
|
||||||
|
``hw:cpu_policy`` and ``hw_cpu_policy`` interfaces, which may raise conflict.
|
||||||
|
The CPU policy conflict is proposed to be solved with the following policy
|
||||||
|
matrix:
|
||||||
|
|
||||||
|
+---------------------------+-----------+-----------+-----------+-----------+
|
||||||
|
| | hw:cpu_policy |
|
||||||
|
+ INSTANCE CPU POLICY +-----------+-----------+-----------+-----------+
|
||||||
|
| | DEDICATED | MIXED | SHARED | undefined |
|
||||||
|
+---------------+-----------+-----------+-----------+-----------+-----------+
|
||||||
|
| hw_cpu_policy | DEDICATED | dedicated | conflict | conflict | dedicated |
|
||||||
|
+ +-----------+-----------+-----------+-----------+-----------+
|
||||||
|
| | MIXED | dedicated | mixed | conflict | mixed |
|
||||||
|
+ +-----------+-----------+-----------+-----------+-----------+
|
||||||
|
| | SHARED | dedicated | conflict | shared | shared |
|
||||||
|
+ +-----------+-----------+-----------+-----------+-----------+
|
||||||
|
| | undefined | dedicated | mixed | shared | undefined |
|
||||||
|
+---------------+-----------+-----------+-----------+-----------+-----------+
|
||||||
|
|
||||||
|
For example, if a ``dedicated`` CPU policy is specified in instance flavor
|
||||||
|
``hw:cpu_policy``, then the instance CPU policy is ``dedicated``, regardless
|
||||||
|
of the setting specified in image property ``hw_cpu_policy``. If ``shared``
|
||||||
|
is explicitly set in ``hw:cpu_policy``, then a ``mixed`` policy specified
|
||||||
|
in ``hw_cpu_policy`` is conflict, which will throw an exception, the instance
|
||||||
|
booting request will be rejected.
|
||||||
|
|
||||||
|
If there is no explicit instance CPU policy specified in flavor or image
|
||||||
|
property, the flavor matrix result would be 'undefined', and the final
|
||||||
|
instance policy is further determined and resolved by ``resources:PCPU``
|
||||||
|
and ``resources:VCPU`` specified in flavor extra specs. Refer to
|
||||||
|
:ref:`section<mixed-instance-via-PCPU-VCPU>` and the spec
|
||||||
|
`CPU resource tracking`_.
|
||||||
|
|
||||||
|
Affect over real-time vCPUs
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
It's also possible to set some *mixed* instance vCPUs as real-time vCPU,
|
||||||
|
*realtime* CPUs must be chosen from the instance *dedicated* CPU set.
|
||||||
|
|
||||||
|
Affect over emulator thread policy
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
If emulator thread policy is ``ISOLATE``, the *mixed* instance will look for
|
||||||
|
a *dedicated* host CPU for instance emulator thread, which is very similar
|
||||||
|
to the case introduced by ``dedicated`` policy instance.
|
||||||
|
|
||||||
|
If the emulator thread policy is ``SHARE``, then the instance emulator thread
|
||||||
|
will float over the host CPUs defined in configuration
|
||||||
|
``CONF.compute.cpu_shared_set``.
|
||||||
|
|
||||||
|
Set dedicated CPU bit-mask in ``hw:pinned_cpus`` for ``mixed`` instance
|
||||||
|
-----------------------------------------------------------------------
|
||||||
|
|
||||||
|
As an interface to create the ``mixed`` policy instance through legacy flavor
|
||||||
|
extra specs or image properties, the flavor extra spec ``hw:pinned_cpus`` is
|
||||||
|
introduced. If the extra spec ``hw:pinned_cpus`` is found in the instance
|
||||||
|
flavor, then the information of the *dedicated* CPU could be found through
|
||||||
|
parsing ``hw:pinned_cpus``.
|
||||||
|
Here is the example to create an instance with ``mixed`` policy:
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
$ openstack flavor set <flavor_id> \
|
||||||
|
--property hw:cpu_policy=mixed \
|
||||||
|
--property hw:pinned_cpus=0-3,7
|
||||||
|
|
||||||
|
And, following is the proposing command to create a *mixed* instance which
|
||||||
|
consists of multiple NUMA nodes by setting the *dedicated* CPUs in
|
||||||
|
``hw:pinned_cpus``:
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
$ openstack flavor set <flavor_id> \
|
||||||
|
--property hw:cpu_policy=mixed \
|
||||||
|
--property hw:pinned_cpus=2,7 \
|
||||||
|
--property hw:numa_nodes=2 \
|
||||||
|
--property hw:numa_cpus.0=0-2 \
|
||||||
|
--property hw:numa_cpus.1=3-7 \
|
||||||
|
--property hw:numa_mem.0=1024 \
|
||||||
|
--property hw:numa_mem.1=2048
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
Please be aware that there is no equivalent setting in image properties
|
||||||
|
for flavor extra spec ``hw:pinned_cpus``. It will not be supported to
|
||||||
|
create *mixed* instance through image properties.
|
||||||
|
|
||||||
|
.. _mixed-instance-via-PCPU-VCPU:
|
||||||
|
|
||||||
|
Create *mixed* instance via ``resources:PCPU`` and ``resources:VCPU``
|
||||||
|
---------------------------------------------------------------------
|
||||||
|
|
||||||
|
`CPU resource tracking`_ introduced a way to create an instance with
|
||||||
|
``dedicated`` or ``shared`` CPU allocation policy through ``resources:PCPU``
|
||||||
|
and ``resources:VCPU`` interfaces, but did not allow requesting both ``PCPU``
|
||||||
|
resource and ``VCPU`` resource for one instance.
|
||||||
|
|
||||||
|
This specification proposes to let an instance request ``PCPU`` resource along
|
||||||
|
with ``VCPU``, and effectively applying for the ``mixed`` CPU allocation
|
||||||
|
policy if the ``cpu_policy`` is not explicitly specified in the flavor list.
|
||||||
|
So an instance with such flavors potentially creates a ``mixed`` policy
|
||||||
|
instance:
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
$ openstack flavor set \
|
||||||
|
--property "resources:PCPU"="<dedicated CPU number>" \
|
||||||
|
--property "resources:VCPU"="<shared CPU number>" \
|
||||||
|
<flavor_id>
|
||||||
|
|
||||||
|
For *mixed* instance created in such way, both <shared CPU number> and
|
||||||
|
<dedicated CPU number> must be greater than zero. Otherwise, it effectively
|
||||||
|
creates the ``dedicated`` or ``shared`` policy instance, that all CPUs in the
|
||||||
|
instance is in a same allocation policy.
|
||||||
|
|
||||||
|
The ``resources:PCPU`` and ``resources::VCPU`` interfaces only put the request
|
||||||
|
toward ``Placement`` service for how many ``PCPU`` and ``VCPU`` resources are
|
||||||
|
required to fulfill the instance vCPU thread and emulator thread requirement.
|
||||||
|
The ``PCPU`` and ``VCPU`` distribution on the instance, especially on the
|
||||||
|
instance with multiple NUMA nodes, will be spread across the NUMA nodes in the
|
||||||
|
round-robin way, and ``VCPU`` will be put ahead of ``PCPU``. Here is one
|
||||||
|
example and the instance is created with flavor below::
|
||||||
|
|
||||||
|
flavor:
|
||||||
|
vcpus:8
|
||||||
|
memory_mb=512
|
||||||
|
extra_specs:
|
||||||
|
hw:numa_nodes:2
|
||||||
|
resources:VCPU=3
|
||||||
|
resources:PCPU=5
|
||||||
|
|
||||||
|
Instance emulator thread policy is not specified in the flavor, so it does not
|
||||||
|
occupy any dedicated ``PCPU`` resource for it, all ``PCPU`` and ``VCPU``
|
||||||
|
resources will be used by vCPU threads, and the expected distribution on NUMA
|
||||||
|
nodes is::
|
||||||
|
|
||||||
|
NUMA node 0: VCPU VCPU PCPU PCPU
|
||||||
|
NUMA node 1: VCPU PCPU PCPU PCPU
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
The demanding instance CPU number is the number of vCPU, specified by
|
||||||
|
``flavor.vcpus``, plus the number of CPU that is special for emulator
|
||||||
|
thread, and if the emulator thread policy is ``ISOLATE``, the instance
|
||||||
|
requests ``flavor.vcpus`` + 1 vCPUs, if the policy is not ``ISOLATE``,
|
||||||
|
the instance just requests ``flavor.vcpus`` vCPU.
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
Creating CPU mixed instance by extending the ``dedicated`` policy
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Instead of adding a special instance CPU allocation policy, the CPU mixed
|
||||||
|
instance is supported by extending the existing ``dedicated`` policy and
|
||||||
|
specifying the CPUs that are pinned to the host CPUs chosen from ``PCPU``
|
||||||
|
resource.
|
||||||
|
|
||||||
|
Following extra spec and the image property are defined to keep the
|
||||||
|
*dedicated* CPUs of a ``mixed`` policy instance::
|
||||||
|
|
||||||
|
hw:cpu_dedicated_mask=<cpu set string>
|
||||||
|
hw_cpu_dedicated_mask=<cpu set string>
|
||||||
|
|
||||||
|
The ``<cpu set string>`` shares the same definition defined above.
|
||||||
|
|
||||||
|
This was rejected at it overloads the ``dedicated`` policy to mean two things,
|
||||||
|
depending on the value of another configuration option.
|
||||||
|
|
||||||
|
Creating ``mixed`` instance with ``hw:cpu_policy`` and ``resources:(P|V)CPU``
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Following commands was proposed as an example to create a *mixed* instance by
|
||||||
|
an explicit request of ``PCPU`` resources, and infer the ``VCPU`` count by
|
||||||
|
``flavor::vcpus`` and ``PCPU`` count:
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
$ openstack flavor create mixed_vmf --vcpus 4 --ram 512 --disk 1
|
||||||
|
$ openstack flavor set mixed_vmf \
|
||||||
|
--property hw:cpu_policy=mixed \
|
||||||
|
--property resources:PCPU=2
|
||||||
|
|
||||||
|
This was rejected due to the mixing use of ``hw:cpu_policy`` and
|
||||||
|
``resources:PCPU``. It is not recommended to mix placement style syntax with
|
||||||
|
traditional extra specs.
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
No change to data objects.
|
||||||
|
|
||||||
|
..note::
|
||||||
|
The ``InstanceNUMACell.cpu_pinning`` field keeps the CPU pinning
|
||||||
|
information with the instance CPU ID and the host CPU ID that the
|
||||||
|
instance CPU is pinning on. This field is used and filled after the CPU
|
||||||
|
pair information is determined for the instance taking the ``dedicated``
|
||||||
|
policy.
|
||||||
|
This field will also be used for ``mixed`` policy instance with the same
|
||||||
|
purpose, but will be initialized with a dictionary keyed by the instance
|
||||||
|
CPU IDs that wants to be pinned on host CPUs at the instance object
|
||||||
|
creating stage. The value of this field will be initialized with *-1*,
|
||||||
|
which means host CPU ID is not valid because the host accommodating the
|
||||||
|
instance is not determined in instance creating stage. The value is
|
||||||
|
replaced with appropriate host CPU ID by nova-scheduler.
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
The metadata API will be extended with the *dedicated* vCPU info and a new
|
||||||
|
OpenStack metadata version will be added to indicate this is a new metadata
|
||||||
|
API.
|
||||||
|
|
||||||
|
The new field will be added to the ``meta_data.json``::
|
||||||
|
|
||||||
|
dedicated_cpus=<cpu set string>
|
||||||
|
|
||||||
|
The ``<cpu set string>`` indicated the vCPU IDs which are running on dedicated
|
||||||
|
host CPUs.
|
||||||
|
|
||||||
|
The new cpu policy ``mixed`` is added to extra spec ``hw:cpu_policy``.
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Notifications impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
If the end user wants to create an instance with a ``mixed`` CPU allocation
|
||||||
|
policy, the user is required to set corresponding flavor extra specs or image
|
||||||
|
properties.
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
This proposal affects the selection of instance CPU allocation policy, but the
|
||||||
|
performance impact is trivial.
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Upgrade impact
|
||||||
|
--------------
|
||||||
|
|
||||||
|
The ``mixed`` cpu policy is only available when the whole cluster upgrade
|
||||||
|
finished. A service version will be bumped for detecting the upgrade.
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
Wang, Huaqiang <huaqiang.wang@intel.com>
|
||||||
|
|
||||||
|
Feature Liaison
|
||||||
|
---------------
|
||||||
|
|
||||||
|
Feature liaison:
|
||||||
|
Stephen Finucane <stephenfin@redhat.com>
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
* Add new instance CPU allocation policy ``mixed`` property and resolve
|
||||||
|
conflicts
|
||||||
|
* Bump nova service version.
|
||||||
|
* Add flavor extra spec ``hw:pinned_cpus`` and create *mixed* instance
|
||||||
|
* Translate *dedicated* and *shared* CPU request to placement ``PCPU`` and
|
||||||
|
``VCPU`` resources request.
|
||||||
|
* Change libvirt driver to create ``PCPU`` mapping and ``VCPU`` mapping
|
||||||
|
* Add nova metadata service by offering final pCPU layout in
|
||||||
|
``dedicated_cpus`` field
|
||||||
|
* Validate real-time CPU mask for ``mixed`` instance.
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
None
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
Functional and unit tests are required to cover:
|
||||||
|
|
||||||
|
* Ensure to solve the conflicts between the CPU policy matrix
|
||||||
|
* Ensure only *dedicated* vCPUs are possible to be real-time vCPUs
|
||||||
|
* Ensure creating ``mixed`` policy instance properly either by flavor
|
||||||
|
settings or by ``resources::PCPU=xx`` and ``resources::VCPU=xx`` settings.
|
||||||
|
* Ensure *shared* vCPUs is placed before the ``dedicated`` vCPUs
|
||||||
|
* Ensure the emulator CPU is properly scheduled according to its policy.
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
The documents should be changed to introduce the usage of new ``mixed`` CPU
|
||||||
|
allocation policy and the new flavor extra specs.
|
||||||
|
|
||||||
|
Metadata service will be updated accordingly.
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
* `CPU resource tracking`_
|
||||||
|
|
||||||
|
.. _CPU resource tracking: http://specs.openstack.org/openstack/nova-specs/specs/train/approved/cpu-resources.html
|
||||||
|
|
||||||
|
History
|
||||||
|
=======
|
||||||
|
|
||||||
|
.. list-table:: Revisions
|
||||||
|
:header-rows: 1
|
||||||
|
|
||||||
|
* - Release Name
|
||||||
|
- Description
|
||||||
|
* - Train
|
||||||
|
- Introduced, abandoned
|
||||||
|
* - Ussuri
|
||||||
|
- Re-proposed
|
||||||
Reference in New Issue
Block a user