nova/doc/source/admin/cpu-topologies.rst

==============
CPU topologies
==============

The NUMA topology and CPU pinning features in OpenStack provide high-level
control over how instances run on hypervisor CPUs and the topology of virtual
CPUs available to instances. These features help minimize latency and maximize
performance.

SMP, NUMA, and SMT
~~~~~~~~~~~~~~~~~~

Symmetric multiprocessing (SMP)
  SMP is a design found in many modern multi-core systems. In an SMP system,
  there are two or more CPUs and these CPUs are connected by some interconnect.
  This provides CPUs with equal access to system resources like memory and
  input/output ports.

Non-uniform memory access (NUMA)
  NUMA is a derivative of the SMP design that is found in many multi-socket
  systems. In a NUMA system, system memory is divided into cells or nodes that
  are associated with particular CPUs. Requests for memory on other nodes are
  possible through an interconnect bus. However, bandwidth across this shared
  bus is limited. As a result, competition for this resource can incur
  performance penalties.

Simultaneous Multi-Threading (SMT)
  SMT is a design complementary to SMP. Whereas CPUs in SMP systems share a bus
  and some memory, CPUs in SMT systems share many more components. CPUs that
  share components are known as thread siblings.  All CPUs appear as usable
  CPUs on the system and can execute workloads in parallel. However, as with
  NUMA, threads compete for shared resources.

In OpenStack, SMP CPUs are known as *cores*, NUMA cells or nodes are known as
*sockets*, and SMT CPUs are known as *threads*. For example, a quad-socket,
eight core system with Hyper-Threading would have four sockets, eight cores per
socket and two threads per core, for a total of 64 CPUs.

Customizing instance NUMA placement policies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. important::

   The functionality described below is currently only supported by the
   libvirt/KVM and Hyper-V driver. The Hyper-V driver may require :ref:`some
   host configuration <configure-hyperv-numa>` for this to work.

When running workloads on NUMA hosts, it is important that the vCPUs executing
processes are on the same NUMA node as the memory used by these processes.
This ensures all memory accesses are local to the node and thus do not consume
the limited cross-node memory bandwidth, adding latency to memory accesses.
Similarly, large pages are assigned from memory and benefit from the same
performance improvements as memory allocated using standard pages. Thus, they
also should be local. Finally, PCI devices are directly associated with
specific NUMA nodes for the purposes of DMA. Instances that use PCI or SR-IOV
devices should be placed on the NUMA node associated with these devices.

NUMA topology can exist on both the physical hardware of the host and the
virtual hardware of the instance. In OpenStack, when booting a process, the
hypervisor driver looks at the NUMA topology field of both the instance and
the host it is being booted on, and uses that information to generate an
appropriate configuration.

By default, an instance floats across all NUMA nodes on a host. NUMA awareness
can be enabled implicitly through the use of huge pages or pinned CPUs or
explicitly through the use of flavor extra specs or image metadata. If the
instance has requested a specific NUMA topology, compute will try to pin the
vCPUs of different NUMA cells on the instance to the corresponding NUMA cells
on the host. It will also expose the NUMA topology of the instance to the
guest OS.

If you want compute to pin a particular vCPU as part of this process,
set the ``vcpu_pin_set`` parameter in the ``nova.conf`` configuration
file. For more information about the ``vcpu_pin_set`` parameter, see the
:doc:`/configuration/config`.

In all cases where NUMA awareness is used, the ``NUMATopologyFilter``
filter must be enabled. Details on this filter are provided in
:doc:`/admin/configuration/schedulers`.

.. caution::

   The NUMA node(s) used are normally chosen at random. However, if a PCI
   passthrough or SR-IOV device is attached to the instance, then the NUMA
   node that the device is associated with will be used. This can provide
   important performance improvements. However, booting a large number of
   similar instances can result in unbalanced NUMA node usage. Care should
   be taken to mitigate this issue. See this `discussion`_ for more details.

.. caution::

   Inadequate per-node resources will result in scheduling failures. Resources
   that are specific to a node include not only CPUs and memory, but also PCI
   and SR-IOV resources. It is not possible to use multiple resources from
   different nodes without requesting a multi-node layout. As such, it may be
   necessary to ensure PCI or SR-IOV resources are associated with the same
   NUMA node or force a multi-node layout.

When used, NUMA awareness allows the operating system of the instance to
intelligently schedule the workloads that it runs and minimize cross-node
memory bandwidth. To restrict an instance's vCPUs to a single host NUMA node,
run:

.. code-block:: console

   $ openstack flavor set m1.large --property hw:numa_nodes=1

Some workloads have very demanding requirements for memory access latency or
bandwidth that exceed the memory bandwidth available from a single NUMA node.
For such workloads, it is beneficial to spread the instance across multiple
host NUMA nodes, even if the instance's RAM/vCPUs could theoretically fit on a
single NUMA node. To force an instance's vCPUs to spread across two host NUMA
nodes, run:

.. code-block:: console

   $ openstack flavor set m1.large --property hw:numa_nodes=2

The allocation of instances vCPUs and memory from different host NUMA nodes can
be configured. This allows for asymmetric allocation of vCPUs and memory, which
can be important for some workloads. To spread the 6 vCPUs and 6 GB of memory
of an instance across two NUMA nodes and create an asymmetric 1:2 vCPU and
memory mapping between the two nodes, run:

.. code-block:: console

   $ openstack flavor set m1.large --property hw:numa_nodes=2
   $ openstack flavor set m1.large \  # configure guest node 0
     --property hw:numa_cpus.0=0,1 \
     --property hw:numa_mem.0=2048
   $ openstack flavor set m1.large \  # configure guest node 1
     --property hw:numa_cpus.1=2,3,4,5 \
     --property hw:numa_mem.1=4096

.. note::

    Hyper-V does not support asymmetric NUMA topologies, and the Hyper-V
    driver will not spawn instances with such topologies.

For more information about the syntax for ``hw:numa_nodes``, ``hw:numa_cpus.N``
and ``hw:num_mem.N``, refer to the :ref:`NUMA
topology <extra-specs-numa-topology>` guide.

Customizing instance CPU pinning policies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. important::

   The functionality described below is currently only supported by the
   libvirt/KVM driver. Hyper-V does not support CPU pinning.

By default, instance vCPU processes are not assigned to any particular host
CPU, instead, they float across host CPUs like any other process. This allows
for features like overcommitting of CPUs. In heavily contended systems, this
provides optimal system performance at the expense of performance and latency
for individual instances.

Some workloads require real-time or near real-time behavior, which is not
possible with the latency introduced by the default CPU policy. For such
workloads, it is beneficial to control which host CPUs are bound to an
instance's vCPUs. This process is known as pinning. No instance with pinned
CPUs can use the CPUs of another pinned instance, thus preventing resource
contention between instances. To configure a flavor to use pinned vCPUs, a
use a dedicated CPU policy. To force this, run:

.. code-block:: console

   $ openstack flavor set m1.large --property hw:cpu_policy=dedicated

.. caution::

   Host aggregates should be used to separate pinned instances from unpinned
   instances as the latter will not respect the resourcing requirements of
   the former.

When running workloads on SMT hosts, it is important to be aware of the impact
that thread siblings can have. Thread siblings share a number of components
and contention on these components can impact performance. To configure how
to use threads, a CPU thread policy should be specified. For workloads where
sharing benefits performance, use thread siblings. To force this, run:

.. code-block:: console

   $ openstack flavor set m1.large \
     --property hw:cpu_policy=dedicated \
     --property hw:cpu_thread_policy=require

For other workloads where performance is impacted by contention for resources,
use non-thread siblings or non-SMT hosts. To force this, run:

.. code-block:: console

   $ openstack flavor set m1.large \
     --property hw:cpu_policy=dedicated \
     --property hw:cpu_thread_policy=isolate

Finally, for workloads where performance is minimally impacted, use thread
siblings if available. This is the default, but it can be set explicitly:

.. code-block:: console

   $ openstack flavor set m1.large \
     --property hw:cpu_policy=dedicated \
     --property hw:cpu_thread_policy=prefer

For more information about the syntax for ``hw:cpu_policy`` and
``hw:cpu_thread_policy``, refer to the :doc:`/admin/flavors` guide.

Applications are frequently packaged as images. For applications that require
real-time or near real-time behavior, configure image metadata to ensure
created instances are always pinned regardless of flavor. To configure an
image to use pinned vCPUs and avoid thread siblings, run:

.. code-block:: console

   $ openstack image set [IMAGE_ID] \
     --property hw_cpu_policy=dedicated \
     --property hw_cpu_thread_policy=isolate

If the flavor specifies a CPU policy of ``dedicated`` then that policy will be
used. If the flavor explicitly specifies a CPU policy of ``shared`` and the
image specifies no policy or a policy of ``shared`` then the ``shared`` policy
will be used, but if the image specifies a policy of ``dedicated`` an exception
will be raised. By setting a ``shared`` policy through flavor extra-specs,
administrators can prevent users configuring CPU policies in images and
impacting resource utilization. To configure this policy, run:

.. code-block:: console

   $ openstack flavor set m1.large --property hw:cpu_policy=shared

If the flavor does not specify a CPU thread policy then the CPU thread policy
specified by the image (if any) will be used. If both the flavor and image
specify a CPU thread policy then they must specify the same policy, otherwise
an exception will be raised.

.. note::

   There is no correlation required between the NUMA topology exposed in the
   instance and how the instance is actually pinned on the host. This is by
   design. See this `invalid bug
   <https://bugs.launchpad.net/nova/+bug/1466780>`_ for more information.

For more information about image metadata, refer to the `Image metadata`_
guide.

Customizing instance CPU topologies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. important::

   The functionality described below is currently only supported by the
   libvirt/KVM driver.

.. note::
   Currently it also works with libvirt/QEMU driver but we don't recommend
   it in production use cases. This is because vCPUs are actually running
   in one thread on host in qemu TCG (Tiny Code Generator), which is the
   backend for libvirt/QEMU driver. Work to enable full multi-threading
   support for TCG (a.k.a. MTTCG) is on going in QEMU community. Please see
   this `MTTCG project`_ page for detail.

In addition to configuring how an instance is scheduled on host CPUs, it is
possible to configure how CPUs are represented in the instance itself. By
default, when instance NUMA placement is not specified, a topology of N
sockets, each with one core and one thread, is used for an instance, where N
corresponds to the number of instance vCPUs requested. When instance NUMA
placement is specified, the number of sockets is fixed to the number of host
NUMA nodes to use and the total number of instance CPUs is split over these
sockets.

Some workloads benefit from a custom topology. For example, in some operating
systems, a different license may be needed depending on the number of CPU
sockets. To configure a flavor to use a maximum of two sockets, run:

.. code-block:: console

   $ openstack flavor set m1.large --property hw:cpu_sockets=2

Similarly, to configure a flavor to use one core and one thread, run:

.. code-block:: console

   $ openstack flavor set m1.large \
     --property hw:cpu_cores=1 \
     --property hw:cpu_threads=1

.. caution::

   If specifying all values, the product of sockets multiplied by cores
   multiplied by threads must equal the number of instance vCPUs. If specifying
   any one of these values or the multiple of two values, the values must be a
   factor of the number of instance vCPUs to prevent an exception. For example,
   specifying ``hw:cpu_sockets=2`` on a host with an odd number of cores fails.
   Similarly, specifying ``hw:cpu_cores=2`` and ``hw:cpu_threads=4`` on a host
   with ten cores fails.

For more information about the syntax for ``hw:cpu_sockets``, ``hw:cpu_cores``
and ``hw:cpu_threads``, refer to the :doc:`/admin/flavors` guide.

It is also possible to set upper limits on the number of sockets, cores, and
threads used. Unlike the hard values above, it is not necessary for this exact
number to used because it only provides a limit. This can be used to provide
some flexibility in scheduling, while ensuring certain limits are not
exceeded. For example, to ensure no more than two sockets are defined in the
instance topology, run:

.. code-block:: console

   $ openstack flavor set m1.large --property hw:cpu_max_sockets=2

For more information about the syntax for ``hw:cpu_max_sockets``,
``hw:cpu_max_cores``, and ``hw:cpu_max_threads``, refer to the
:doc:`/admin/flavors` guide.

Applications are frequently packaged as images. For applications that prefer
certain CPU topologies, configure image metadata to hint that created instances
should have a given topology regardless of flavor. To configure an image to
request a two-socket, four-core per socket topology, run:

.. code-block:: console

   $ openstack image set [IMAGE_ID] \
     --property hw_cpu_sockets=2 \
     --property hw_cpu_cores=4

To constrain instances to a given limit of sockets, cores or threads, use the
``max_`` variants. To configure an image to have a maximum of two sockets and a
maximum of one thread, run:

.. code-block:: console

   $ openstack image set [IMAGE_ID] \
     --property hw_cpu_max_sockets=2 \
     --property hw_cpu_max_threads=1

The value specified in the flavor is treated as the absolute limit.  The image
limits are not permitted to exceed the flavor limits, they can only be equal
to or lower than what the flavor defines. By setting a ``max`` value for
sockets, cores, or threads, administrators can prevent users configuring
topologies that might, for example, incur an additional licensing fees.

For more information about image metadata, refer to the `Image metadata`_
guide.

.. _configure-hyperv-numa:

Configuring Hyper-V compute nodes for instance NUMA policies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Hyper-V is configured by default to allow instances to span multiple NUMA
nodes, regardless if the instances have been configured to only span N NUMA
nodes. This behaviour allows Hyper-V instances to have up to 64 vCPUs and 1 TB
of memory.

Checking NUMA spanning can easily be done by running this following PowerShell
command:

.. code-block:: console

   (Get-VMHost).NumaSpanningEnabled

In order to disable this behaviour, the host will have to be configured to
disable NUMA spanning. This can be done by executing these following
PowerShell commands:

.. code-block:: console

   Set-VMHost -NumaSpanningEnabled $false
   Restart-Service vmms

In order to restore this behaviour, execute these PowerShell commands:

.. code-block:: console

   Set-VMHost -NumaSpanningEnabled $true
   Restart-Service vmms

The *Virtual Machine Management Service* (*vmms*) is responsible for managing
the Hyper-V VMs. The VMs will still run while the service is down or
restarting, but they will not be manageable by the ``nova-compute`` service. In
order for the effects of the host NUMA spanning configuration to take effect,
the VMs will have to be restarted.

Hyper-V does not allow instances with a NUMA topology to have dynamic
memory allocation turned on. The Hyper-V driver will ignore the configured
``dynamic_memory_ratio`` from the given ``nova.conf`` file when spawning
instances with a NUMA topology.

.. Links
.. _`Image metadata`: https://docs.openstack.org/image-guide/image-metadata.html
.. _`discussion`: http://lists.openstack.org/pipermail/openstack-dev/2016-March/090367.html
.. _`MTTCG project`: http://wiki.qemu.org/Features/tcg-multithread