Merge "[admin-guide] Rework CPU topology guide"
This commit is contained in:
Jenkins
Gerrit Code Review
commit
d465f42169
|
|
@ -24,4 +24,4 @@ instance for these kind of workloads.
|
|||
:maxdepth: 2
|
||||
|
||||
compute-pci-passthrough.rst
|
||||
compute-numa-cpu-pinning.rst
|
||||
compute-cpu-topologies.rst
|
||||
|
|
|
|||
|
|
@ -1,35 +1,37 @@
|
|||
.. _section-compute-numa-cpu-pinning:
|
||||
.. _compute-cpu-topologies:
|
||||
|
||||
==========================================
|
||||
Enabling advanced CPU topologies in guests
|
||||
==========================================
|
||||
==============
|
||||
CPU topologies
|
||||
==============
|
||||
|
||||
The NUMA topology and CPU pinning features in OpenStack provide high level
|
||||
control over how instances run on host CPUs, and the topology of CPUs inside
|
||||
the instance. These features can be used to minimize latency and maximize
|
||||
per-instance performance.
|
||||
The NUMA topology and CPU pinning features in OpenStack provide high-level
|
||||
control over how instances run on hypervisor CPUs and the topology of virtual
|
||||
CPUs available to instances. These features help minimize latency and maximize
|
||||
performance.
|
||||
|
||||
SMP, NUMA, and SMT overviews
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
SMP, NUMA, and SMT
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Symmetric multiprocessing (SMP) is a design found in many modern multi-core
|
||||
systems. In an SMP system, there are two or more CPUs and these CPUs are
|
||||
connected by some interconnect. This provides CPUs with equal access to
|
||||
system resources like memory and IO ports.
|
||||
Symmetric multiprocessing (SMP)
|
||||
SMP is a design found in many modern multi-core systems. In an SMP system,
|
||||
there are two or more CPUs and these CPUs are connected by some interconnect.
|
||||
This provides CPUs with equal access to system resources like memory and
|
||||
input/output ports.
|
||||
|
||||
Non-uniform memory access (NUMA) is a derivative of the SMP design that is
|
||||
found in many multi-socket systems. In a NUMA system, system memory is divided
|
||||
into cells or nodes that are associated with particular CPUs. Requests for
|
||||
memory on other nodes are possible through an interconnect bus, however,
|
||||
bandwidth across this shared bus is limited. As a result, competition for this
|
||||
this resource can incur performance penalties.
|
||||
Non-uniform memory access (NUMA)
|
||||
NUMA is a derivative of the SMP design that is found in many multi-socket
|
||||
systems. In a NUMA system, system memory is divided into cells or nodes that
|
||||
are associated with particular CPUs. Requests for memory on other nodes are
|
||||
possible through an interconnect bus. However, bandwidth across this shared
|
||||
bus is limited. As a result, competition for this resource can incur
|
||||
performance penalties.
|
||||
|
||||
Simultaneous Multi-Threading (SMT), known as as Hyper-Threading on Intel
|
||||
platforms, is a design that is complementary to SMP. Whereas CPUs in SMP
|
||||
systems share a bus and some memory, CPUs in SMT systems share many more
|
||||
components. CPUs that share components are known as thread siblings. All CPUs
|
||||
appear as usable CPUs on the system and can execute workloads in parallel,
|
||||
however, as with NUMA, threads compete for shared resources.
|
||||
Simultaneous Multi-Threading (SMT)
|
||||
SMT is a design complementary to SMP. Whereas CPUs in SMP systems share a bus
|
||||
and some memory, CPUs in SMT systems share many more components. CPUs that
|
||||
share components are known as thread siblings. All CPUs appear as usable
|
||||
CPUs on the system and can execute workloads in parallel. However, as with
|
||||
NUMA, threads compete for shared resources.
|
||||
|
||||
In OpenStack, SMP CPUs are known as *cores*, NUMA cells or nodes are known as
|
||||
*sockets*, and SMT CPUs are known as *threads*. For example, a quad-socket,
|
||||
|
|
@ -49,14 +51,14 @@ processes are on the same NUMA node as the memory used by these processes.
|
|||
This ensures all memory accesses are local to the node and thus do not consume
|
||||
the limited cross-node memory bandwidth, adding latency to memory accesses.
|
||||
Similarly, large pages are assigned from memory and benefit from the same
|
||||
performance improvements as memory allocated using standard pages, thus, they
|
||||
performance improvements as memory allocated using standard pages. Thus, they
|
||||
also should be local. Finally, PCI devices are directly associated with
|
||||
specific NUMA nodes for the purposes of DMA. Instances that use PCI or SR-IOV
|
||||
devices should be placed on the NUMA node associated with the said devices.
|
||||
devices should be placed on the NUMA node associated with these devices.
|
||||
|
||||
By default, an instance will float across all NUMA nodes on a host. NUMA
|
||||
awareness can be enabled implicitly, through the use of hugepages or pinned
|
||||
CPUs, or explicitly, through the use of flavor extra specs or image metadata.
|
||||
By default, an instance floats across all NUMA nodes on a host. NUMA
|
||||
awareness can be enabled implicitly through the use of hugepages or pinned
|
||||
CPUs or explicitly through the use of flavor extra specs or image metadata.
|
||||
In all cases, the ``NUMATopologyFilter`` filter must be enabled. Details on
|
||||
this filter are provided in `Scheduling`_ configuration guide.
|
||||
|
||||
|
|
@ -65,13 +67,13 @@ this filter are provided in `Scheduling`_ configuration guide.
|
|||
The NUMA node(s) used are normally chosen at random. However, if a PCI
|
||||
passthrough or SR-IOV device is attached to the instance, then the NUMA
|
||||
node that the device is associated with will be used. This can provide
|
||||
important performance improvements, however, booting a large number of
|
||||
important performance improvements. However, booting a large number of
|
||||
similar instances can result in unbalanced NUMA node usage. Care should
|
||||
be taken to mitigate this issue. See this `discussion`_ for more details.
|
||||
|
||||
.. caution::
|
||||
|
||||
Inadequate per-node resources will result in scheduling failures. Resources
|
||||
Inadequate per-node resources will result in scheduling failures. Resources
|
||||
that are specific to a node include not only CPUs and memory, but also PCI
|
||||
and SR-IOV resources. It is not possible to use multiple resources from
|
||||
different nodes without requesting a multi-node layout. As such, it may be
|
||||
|
|
@ -85,10 +87,10 @@ run:
|
|||
|
||||
.. code-block:: console
|
||||
|
||||
# openstack flavor set m1.large --property hw:numa_nodes=1
|
||||
$ openstack flavor set m1.large --property hw:numa_nodes=1
|
||||
|
||||
Some workloads have very demanding requirements for memory access latency or
|
||||
bandwidth which exceed the memory bandwidth available from a single NUMA node.
|
||||
bandwidth that exceed the memory bandwidth available from a single NUMA node.
|
||||
For such workloads, it is beneficial to spread the instance across multiple
|
||||
host NUMA nodes, even if the instance's RAM/vCPUs could theoretically fit on a
|
||||
single NUMA node. To force an instance's vCPUs to spread across two host NUMA
|
||||
|
|
@ -96,23 +98,23 @@ nodes, run:
|
|||
|
||||
.. code-block:: console
|
||||
|
||||
# openstack flavor set m1.large --property hw:numa_nodes=2
|
||||
$ openstack flavor set m1.large --property hw:numa_nodes=2
|
||||
|
||||
The allocation of instances vCPUs and memory from different host NUMA nodes can
|
||||
be configured. This allows for asymmetric allocation of vCPUs and memory, which
|
||||
can be important for some workloads. To spread the six vCPUs and 6 GB of memory
|
||||
can be important for some workloads. To spread the 6 vCPUs and 6 GB of memory
|
||||
of an instance across two NUMA nodes and create an asymmetric 1:2 vCPU and
|
||||
memory mapping between the two nodes, run:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# openstack flavor set m1.large --property hw:numa_nodes=2
|
||||
# openstack flavor set m1.large \ # configure guest node 0
|
||||
--property hw:numa_cpus.0=0,1 \
|
||||
--property hw:numa_mem.0=2048
|
||||
# openstack flavor set m1.large \ # configure guest node 1
|
||||
--property hw:numa_cpus.1=2,3,4,5 \
|
||||
--property hw:numa_mem.1=4096
|
||||
$ openstack flavor set m1.large --property hw:numa_nodes=2
|
||||
$ openstack flavor set m1.large \ # configure guest node 0
|
||||
--property hw:numa_cpus.0=0,1 \
|
||||
--property hw:numa_mem.0=2048
|
||||
$ openstack flavor set m1.large \ # configure guest node 1
|
||||
--property hw:numa_cpus.1=2,3,4,5 \
|
||||
--property hw:numa_mem.1=4096
|
||||
|
||||
For more information about the syntax for ``hw:numa_nodes``, ``hw:numa_cpus.N``
|
||||
and ``hw:num_mem.N``, refer to the `Flavors`_ guide.
|
||||
|
|
@ -141,7 +143,7 @@ use a dedicated CPU policy. To force this, run:
|
|||
|
||||
.. code-block:: console
|
||||
|
||||
# openstack flavor set m1.large --property hw:cpu_policy=dedicated
|
||||
$ openstack flavor set m1.large --property hw:cpu_policy=dedicated
|
||||
|
||||
.. caution::
|
||||
|
||||
|
|
@ -157,27 +159,27 @@ sharing benefits performance, use thread siblings. To force this, run:
|
|||
|
||||
.. code-block:: console
|
||||
|
||||
# openstack flavor set m1.large \
|
||||
--property hw:cpu_policy=dedicated \
|
||||
--property hw:cpu_thread_policy=require
|
||||
$ openstack flavor set m1.large \
|
||||
--property hw:cpu_policy=dedicated \
|
||||
--property hw:cpu_thread_policy=require
|
||||
|
||||
For other workloads where performance is impacted by contention for resources,
|
||||
use non-thread siblings or non-SMT hosts. To force this, run:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# openstack flavor set m1.large \
|
||||
--property hw:cpu_policy=dedicated \
|
||||
--property hw:cpu_thread_policy=isolate
|
||||
$ openstack flavor set m1.large \
|
||||
--property hw:cpu_policy=dedicated \
|
||||
--property hw:cpu_thread_policy=isolate
|
||||
|
||||
Finally, for workloads where performance is minimally impacted, use thread
|
||||
siblings if available. This is the default, but it can be set explicitly:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# openstack flavor set m1.large \
|
||||
--property hw:cpu_policy=dedicated \
|
||||
--property hw:cpu_thread_policy=prefer
|
||||
$ openstack flavor set m1.large \
|
||||
--property hw:cpu_policy=dedicated \
|
||||
--property hw:cpu_thread_policy=prefer
|
||||
|
||||
For more information about the syntax for ``hw:cpu_policy`` and
|
||||
``hw:cpu_thread_policy``, refer to the `Flavors`_ guide.
|
||||
|
|
@ -189,25 +191,26 @@ image to use pinned vCPUs and avoid thread siblings, run:
|
|||
|
||||
.. code-block:: console
|
||||
|
||||
# openstack image set [IMAGE_ID] \
|
||||
--property hw_cpu_policy=dedicated \
|
||||
--property hw_cpu_thread_policy=isolate
|
||||
$ openstack image set [IMAGE_ID] \
|
||||
--property hw_cpu_policy=dedicated \
|
||||
--property hw_cpu_thread_policy=isolate
|
||||
|
||||
Image metadata takes precedence over flavor extra specs: configuring competing
|
||||
policies will result in an exception. By setting a ``shared`` policy through
|
||||
image metadata, administrators can prevent users configuring CPU policies in
|
||||
flavors and impacting resource utilization. To configure this policy, run:
|
||||
Image metadata takes precedence over flavor extra specs. Thus, configuring
|
||||
competing policies causes an exception. By setting a ``shared`` policy
|
||||
through image metadata, administrators can prevent users configuring CPU
|
||||
policies in flavors and impacting resource utilization. To configure this
|
||||
policy, run:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# openstack image set [IMAGE_ID] --property hw_cpu_policy=shared
|
||||
$ openstack image set [IMAGE_ID] --property hw_cpu_policy=shared
|
||||
|
||||
.. note::
|
||||
|
||||
There is no correlation required between the NUMA topology exposed in the
|
||||
instance and how the instance is actually pinned on the host. This is by
|
||||
design. See this `bug <https://bugs.launchpad.net/nova/+bug/1466780>`_ for
|
||||
more information.
|
||||
design. See this `invalid bug
|
||||
<https://bugs.launchpad.net/nova/+bug/1466780>`_ for more information.
|
||||
|
||||
For more information about image metadata, refer to the `Image metadata`_
|
||||
guide.
|
||||
|
|
@ -235,15 +238,15 @@ sockets. To configure a flavor to use a maximum of two sockets, run:
|
|||
|
||||
.. code-block:: console
|
||||
|
||||
# openstack flavor set m1.large --property hw:cpu_sockets=2
|
||||
$ openstack flavor set m1.large --property hw:cpu_sockets=2
|
||||
|
||||
Similarly, to configure a flavor to use one core and one thread, run:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# openstack flavor set m1.large \
|
||||
--property hw:cpu_cores=1 \
|
||||
--property hw:cpu_threads=1
|
||||
$ openstack flavor set m1.large \
|
||||
--property hw:cpu_cores=1 \
|
||||
--property hw:cpu_threads=1
|
||||
|
||||
.. caution::
|
||||
|
||||
|
|
@ -260,14 +263,14 @@ and ``hw:cpu_threads``, refer to the `Flavors`_ guide.
|
|||
|
||||
It is also possible to set upper limits on the number of sockets, cores, and
|
||||
threads used. Unlike the hard values above, it is not necessary for this exact
|
||||
number to used because it only provides a a limit. This can be used to provide
|
||||
number to used because it only provides a limit. This can be used to provide
|
||||
some flexibility in scheduling, while ensuring certains limits are not
|
||||
exceeded. For example, to ensure no more than two sockets are defined in the
|
||||
instance topology, run:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# openstack flavor set m1.large --property=hw:cpu_max_sockets=2
|
||||
$ openstack flavor set m1.large --property=hw:cpu_max_sockets=2
|
||||
|
||||
For more information about the syntax for ``hw:cpu_max_sockets``,
|
||||
``hw:cpu_max_cores``, and ``hw:cpu_max_threads``, refer to the `Flavors`_
|
||||
|
|
@ -280,9 +283,9 @@ request a two-socket, four-core per socket topology, run:
|
|||
|
||||
.. code-block:: console
|
||||
|
||||
# openstack image set [IMAGE_ID] \
|
||||
--property hw_cpu_sockets=2 \
|
||||
--property hw_cpu_cores=4
|
||||
$ openstack image set [IMAGE_ID] \
|
||||
--property hw_cpu_sockets=2 \
|
||||
--property hw_cpu_cores=4
|
||||
|
||||
To constrain instances to a given limit of sockets, cores or threads, use the
|
||||
``max_`` variants. To configure an image to have a maximum of two sockets and a
|
||||
|
|
@ -290,9 +293,9 @@ maximum of one thread, run:
|
|||
|
||||
.. code-block:: console
|
||||
|
||||
# openstack image set [IMAGE_ID] \
|
||||
--property hw_cpu_max_sockets=2 \
|
||||
--property hw_cpu_max_threads=1
|
||||
$ openstack image set [IMAGE_ID] \
|
||||
--property hw_cpu_max_sockets=2 \
|
||||
--property hw_cpu_max_threads=1
|
||||
|
||||
Image metadata takes precedence over flavor extra specs. Configuring competing
|
||||
constraints causes an exception. By setting a ``max`` value for sockets, cores,
|
||||
|
|
@ -207,6 +207,7 @@ redirect 301 /admin-guide/cli_nova_specify_host.html /admin-guide/cli-nova-speci
|
|||
redirect 301 /admin-guide/cli_set_compute_quotas.html /admin-guide/cli-set-compute-quotas.html
|
||||
redirect 301 /admin-guide/cli_set_quotas.html /admin-guide/cli-set-quotas.html
|
||||
redirect 301 /admin-guide/compute_arch.html /admin-guide/compute-arch.html
|
||||
redirect 301 /admin-guide/compute-numa-cpu-pinning.html /admin-guide/compute-cpu-topologies.html
|
||||
redirect 301 /admin-guide/cross_project.html /admin-guide/cross-project.html
|
||||
redirect 301 /admin-guide/cross_project_cors.html /admin-guide/cross-project-cors.html
|
||||
redirect 301 /admin-guide/dashboard_admin_manage_roles.html /admin-guide/dashboard-admin-manage-roles.html
|
||||
|
|
|
|||
Loading…
Reference in New Issue