From fba813daf40726659575718669c5bd5d33c0466a Mon Sep 17 00:00:00 2001 From: Wang Huaqiang Date: Tue, 15 Oct 2019 02:48:01 +0800 Subject: [PATCH] Use PCPU and VCPU in one instance We'd like to propose a way to create instance to take using of PCPU and VCPU resource in one instance. Re-proposed in U release. Change-Id: I0c4ffe349dc9381c83c836efb4af99d072169c88 --- .../use-pcpu-vcpu-in-one-instance.rst | 397 ++++++++++++++++++ 1 file changed, 397 insertions(+) create mode 100644 specs/ussuri/approved/use-pcpu-vcpu-in-one-instance.rst diff --git a/specs/ussuri/approved/use-pcpu-vcpu-in-one-instance.rst b/specs/ussuri/approved/use-pcpu-vcpu-in-one-instance.rst new file mode 100644 index 000000000..c54065bd9 --- /dev/null +++ b/specs/ussuri/approved/use-pcpu-vcpu-in-one-instance.rst @@ -0,0 +1,397 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + +========================================= +Use ``PCPU`` and ``VCPU`` in One Instance +========================================= + +https://blueprints.launchpad.net/nova/+spec/use-pcpu-and-vcpu-in-one-instance + +The spec `CPU resource tracking`_ splits host CPUs into ``PCPU`` and ``VCPU`` +resources, making it possible to run instances of ``dedicated`` CPU allocation +policy and instances of ``shared`` CPU allocation policy in the same host. +This spec aims to create such kind of instance that some of the CPUs are +dedicated (``PCPU``) CPUs and the remaining CPUs are shared (``VCPU``) CPUs +and expose this information via the metadata API. + +Problem description +=================== + +The current CPU allocation policy, ``dedicated`` or ``shared``, is applied to +all CPUs of an instance. However, with the introduction of +`CPU resource tracking`_, it is possible to propose a more fine-grained CPU +allocation policy, which is based on the control over individual instance CPU, +and specifying the ``dedicated`` or ``shared`` CPU allocation policy to each +instance CPU. + +Use Cases +--------- + +As an operator, I would like to have an instance with some realtime CPUs for +high performance, and at the same time, in order to increase instance density, +I wish to make the remaining CPUs, which do not demand high performance, +shared with other instances because I only care about the performance of +realtime CPUs. One example is deploying the NFV task that is enhanced with +DPDK framework in the instance, in which the data plane threads could be +processed with the realtime CPUs and the control-plane tasks are scheduled +on CPUs that may be shared with other instances. + +As a Kubernetes administrator, I wish to run a multi-tier or auto-scaling +application in Kubernetes, which is running in single OpenStack VM, with +the expectation that using dedicated high-performance CPUs for application +itself and deploying the containers on shared cores. + +Proposed change +=============== + +Introduce a new CPU allocation policy ``mixed`` +----------------------------------------------- + +``dedicated`` and ``shared`` are the existing instance CPU allocation policies +that determine how instance CPU is scheduled on host CPU. This specification +proposes a new CPU allocation policy, with the name ``mixed``, to +create a CPU *mixed* instance in such way that some instance CPUs are +allocated from computing node's ``PCPU`` resource, and the rest of instance +CPUs are allocated from the ``VCPU`` resources. The CPU allocated from +``PCPU`` resource will be pinned on particular host CPUs which are defined in +``CONF.compute.dedicated_cpu_set``, and the CPU from ``VCPU`` resource will be +floating on the host CPUs which are defined in ``CONF.compute.shared_cpu_set``. +In this proposal, we call these two kinds of CPUs as *dedicated* vCPU and +*shared* vCPU respectively. + +Instance CPU policy matrix +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Nova operator may set the instance CPU allocation policy through the +``hw:cpu_policy`` and ``hw_cpu_policy`` interfaces, which may raise conflict. +The CPU policy conflict is proposed to be solved with the following policy +matrix: + ++---------------------------+-----------+-----------+-----------+-----------+ +| | hw:cpu_policy | ++ INSTANCE CPU POLICY +-----------+-----------+-----------+-----------+ +| | DEDICATED | MIXED | SHARED | undefined | ++---------------+-----------+-----------+-----------+-----------+-----------+ +| hw_cpu_policy | DEDICATED | dedicated | conflict | conflict | dedicated | ++ +-----------+-----------+-----------+-----------+-----------+ +| | MIXED | dedicated | mixed | conflict | mixed | ++ +-----------+-----------+-----------+-----------+-----------+ +| | SHARED | dedicated | conflict | shared | shared | ++ +-----------+-----------+-----------+-----------+-----------+ +| | undefined | dedicated | mixed | shared | undefined | ++---------------+-----------+-----------+-----------+-----------+-----------+ + +For example, if a ``dedicated`` CPU policy is specified in instance flavor +``hw:cpu_policy``, then the instance CPU policy is ``dedicated``, regardless +of the setting specified in image property ``hw_cpu_policy``. If ``shared`` +is explicitly set in ``hw:cpu_policy``, then a ``mixed`` policy specified +in ``hw_cpu_policy`` is conflict, which will throw an exception, the instance +booting request will be rejected. + +If there is no explicit instance CPU policy specified in flavor or image +property, the flavor matrix result would be 'undefined', and the final +instance policy is further determined and resolved by ``resources:PCPU`` +and ``resources:VCPU`` specified in flavor extra specs. Refer to +:ref:`section` and the spec +`CPU resource tracking`_. + +Affect over real-time vCPUs +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It's also possible to set some *mixed* instance vCPUs as real-time vCPU, +*realtime* CPUs must be chosen from the instance *dedicated* CPU set. + +Affect over emulator thread policy +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If emulator thread policy is ``ISOLATE``, the *mixed* instance will look for +a *dedicated* host CPU for instance emulator thread, which is very similar +to the case introduced by ``dedicated`` policy instance. + +If the emulator thread policy is ``SHARE``, then the instance emulator thread +will float over the host CPUs defined in configuration +``CONF.compute.cpu_shared_set``. + +Set dedicated CPU bit-mask in ``hw:pinned_cpus`` for ``mixed`` instance +----------------------------------------------------------------------- + +As an interface to create the ``mixed`` policy instance through legacy flavor +extra specs or image properties, the flavor extra spec ``hw:pinned_cpus`` is +introduced. If the extra spec ``hw:pinned_cpus`` is found in the instance +flavor, then the information of the *dedicated* CPU could be found through +parsing ``hw:pinned_cpus``. +Here is the example to create an instance with ``mixed`` policy: + +.. code:: + + $ openstack flavor set \ + --property hw:cpu_policy=mixed \ + --property hw:pinned_cpus=0-3,7 + +And, following is the proposing command to create a *mixed* instance which +consists of multiple NUMA nodes by setting the *dedicated* CPUs in +``hw:pinned_cpus``: + +.. code:: + + $ openstack flavor set \ + --property hw:cpu_policy=mixed \ + --property hw:pinned_cpus=2,7 \ + --property hw:numa_nodes=2 \ + --property hw:numa_cpus.0=0-2 \ + --property hw:numa_cpus.1=3-7 \ + --property hw:numa_mem.0=1024 \ + --property hw:numa_mem.1=2048 + +.. note:: + Please be aware that there is no equivalent setting in image properties + for flavor extra spec ``hw:pinned_cpus``. It will not be supported to + create *mixed* instance through image properties. + +.. _mixed-instance-via-PCPU-VCPU: + +Create *mixed* instance via ``resources:PCPU`` and ``resources:VCPU`` +--------------------------------------------------------------------- + +`CPU resource tracking`_ introduced a way to create an instance with +``dedicated`` or ``shared`` CPU allocation policy through ``resources:PCPU`` +and ``resources:VCPU`` interfaces, but did not allow requesting both ``PCPU`` +resource and ``VCPU`` resource for one instance. + +This specification proposes to let an instance request ``PCPU`` resource along +with ``VCPU``, and effectively applying for the ``mixed`` CPU allocation +policy if the ``cpu_policy`` is not explicitly specified in the flavor list. +So an instance with such flavors potentially creates a ``mixed`` policy +instance: + +.. code:: + + $ openstack flavor set \ + --property "resources:PCPU"="" \ + --property "resources:VCPU"="" \ + + +For *mixed* instance created in such way, both and + must be greater than zero. Otherwise, it effectively +creates the ``dedicated`` or ``shared`` policy instance, that all CPUs in the +instance is in a same allocation policy. + +The ``resources:PCPU`` and ``resources::VCPU`` interfaces only put the request +toward ``Placement`` service for how many ``PCPU`` and ``VCPU`` resources are +required to fulfill the instance vCPU thread and emulator thread requirement. +The ``PCPU`` and ``VCPU`` distribution on the instance, especially on the +instance with multiple NUMA nodes, will be spread across the NUMA nodes in the +round-robin way, and ``VCPU`` will be put ahead of ``PCPU``. Here is one +example and the instance is created with flavor below:: + + flavor: + vcpus:8 + memory_mb=512 + extra_specs: + hw:numa_nodes:2 + resources:VCPU=3 + resources:PCPU=5 + +Instance emulator thread policy is not specified in the flavor, so it does not +occupy any dedicated ``PCPU`` resource for it, all ``PCPU`` and ``VCPU`` +resources will be used by vCPU threads, and the expected distribution on NUMA +nodes is:: + + NUMA node 0: VCPU VCPU PCPU PCPU + NUMA node 1: VCPU PCPU PCPU PCPU + +.. note:: + The demanding instance CPU number is the number of vCPU, specified by + ``flavor.vcpus``, plus the number of CPU that is special for emulator + thread, and if the emulator thread policy is ``ISOLATE``, the instance + requests ``flavor.vcpus`` + 1 vCPUs, if the policy is not ``ISOLATE``, + the instance just requests ``flavor.vcpus`` vCPU. + +Alternatives +------------ + +Creating CPU mixed instance by extending the ``dedicated`` policy +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Instead of adding a special instance CPU allocation policy, the CPU mixed +instance is supported by extending the existing ``dedicated`` policy and +specifying the CPUs that are pinned to the host CPUs chosen from ``PCPU`` +resource. + +Following extra spec and the image property are defined to keep the +*dedicated* CPUs of a ``mixed`` policy instance:: + + hw:cpu_dedicated_mask= + hw_cpu_dedicated_mask= + +The ```` shares the same definition defined above. + +This was rejected at it overloads the ``dedicated`` policy to mean two things, +depending on the value of another configuration option. + +Creating ``mixed`` instance with ``hw:cpu_policy`` and ``resources:(P|V)CPU`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Following commands was proposed as an example to create a *mixed* instance by +an explicit request of ``PCPU`` resources, and infer the ``VCPU`` count by +``flavor::vcpus`` and ``PCPU`` count: + +.. code:: + + $ openstack flavor create mixed_vmf --vcpus 4 --ram 512 --disk 1 + $ openstack flavor set mixed_vmf \ + --property hw:cpu_policy=mixed \ + --property resources:PCPU=2 + +This was rejected due to the mixing use of ``hw:cpu_policy`` and +``resources:PCPU``. It is not recommended to mix placement style syntax with +traditional extra specs. + +Data model impact +----------------- + +No change to data objects. + +..note:: + The ``InstanceNUMACell.cpu_pinning`` field keeps the CPU pinning + information with the instance CPU ID and the host CPU ID that the + instance CPU is pinning on. This field is used and filled after the CPU + pair information is determined for the instance taking the ``dedicated`` + policy. + This field will also be used for ``mixed`` policy instance with the same + purpose, but will be initialized with a dictionary keyed by the instance + CPU IDs that wants to be pinned on host CPUs at the instance object + creating stage. The value of this field will be initialized with *-1*, + which means host CPU ID is not valid because the host accommodating the + instance is not determined in instance creating stage. The value is + replaced with appropriate host CPU ID by nova-scheduler. + +REST API impact +--------------- + +The metadata API will be extended with the *dedicated* vCPU info and a new +OpenStack metadata version will be added to indicate this is a new metadata +API. + +The new field will be added to the ``meta_data.json``:: + + dedicated_cpus= + +The ```` indicated the vCPU IDs which are running on dedicated +host CPUs. + +The new cpu policy ``mixed`` is added to extra spec ``hw:cpu_policy``. + +Security impact +--------------- + +None + +Notifications impact +-------------------- + +None + +Other end user impact +--------------------- + +If the end user wants to create an instance with a ``mixed`` CPU allocation +policy, the user is required to set corresponding flavor extra specs or image +properties. + +Performance Impact +------------------ + +This proposal affects the selection of instance CPU allocation policy, but the +performance impact is trivial. + +Other deployer impact +--------------------- + +None + +Developer impact +---------------- + +None + +Upgrade impact +-------------- + +The ``mixed`` cpu policy is only available when the whole cluster upgrade +finished. A service version will be bumped for detecting the upgrade. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + Wang, Huaqiang + +Feature Liaison +--------------- + +Feature liaison: + Stephen Finucane + +Work Items +---------- + +* Add new instance CPU allocation policy ``mixed`` property and resolve + conflicts +* Bump nova service version. +* Add flavor extra spec ``hw:pinned_cpus`` and create *mixed* instance +* Translate *dedicated* and *shared* CPU request to placement ``PCPU`` and + ``VCPU`` resources request. +* Change libvirt driver to create ``PCPU`` mapping and ``VCPU`` mapping +* Add nova metadata service by offering final pCPU layout in + ``dedicated_cpus`` field +* Validate real-time CPU mask for ``mixed`` instance. + +Dependencies +============ + +None + +Testing +======= + +Functional and unit tests are required to cover: + +* Ensure to solve the conflicts between the CPU policy matrix +* Ensure only *dedicated* vCPUs are possible to be real-time vCPUs +* Ensure creating ``mixed`` policy instance properly either by flavor + settings or by ``resources::PCPU=xx`` and ``resources::VCPU=xx`` settings. +* Ensure *shared* vCPUs is placed before the ``dedicated`` vCPUs +* Ensure the emulator CPU is properly scheduled according to its policy. + +Documentation Impact +==================== + +The documents should be changed to introduce the usage of new ``mixed`` CPU +allocation policy and the new flavor extra specs. + +Metadata service will be updated accordingly. + +References +========== + +* `CPU resource tracking`_ + +.. _CPU resource tracking: http://specs.openstack.org/openstack/nova-specs/specs/train/approved/cpu-resources.html + +History +======= + +.. list-table:: Revisions + :header-rows: 1 + + * - Release Name + - Description + * - Train + - Introduced, abandoned + * - Ussuri + - Re-proposed