Merge "[admin-guide] Rework CPU topology guide"
This commit is contained in:
commit
d465f42169
@ -24,4 +24,4 @@ instance for these kind of workloads.
|
|||||||
:maxdepth: 2
|
:maxdepth: 2
|
||||||
|
|
||||||
compute-pci-passthrough.rst
|
compute-pci-passthrough.rst
|
||||||
compute-numa-cpu-pinning.rst
|
compute-cpu-topologies.rst
|
||||||
|
@ -1,35 +1,37 @@
|
|||||||
.. _section-compute-numa-cpu-pinning:
|
.. _compute-cpu-topologies:
|
||||||
|
|
||||||
==========================================
|
==============
|
||||||
Enabling advanced CPU topologies in guests
|
CPU topologies
|
||||||
==========================================
|
==============
|
||||||
|
|
||||||
The NUMA topology and CPU pinning features in OpenStack provide high level
|
The NUMA topology and CPU pinning features in OpenStack provide high-level
|
||||||
control over how instances run on host CPUs, and the topology of CPUs inside
|
control over how instances run on hypervisor CPUs and the topology of virtual
|
||||||
the instance. These features can be used to minimize latency and maximize
|
CPUs available to instances. These features help minimize latency and maximize
|
||||||
per-instance performance.
|
performance.
|
||||||
|
|
||||||
SMP, NUMA, and SMT overviews
|
SMP, NUMA, and SMT
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
Symmetric multiprocessing (SMP) is a design found in many modern multi-core
|
Symmetric multiprocessing (SMP)
|
||||||
systems. In an SMP system, there are two or more CPUs and these CPUs are
|
SMP is a design found in many modern multi-core systems. In an SMP system,
|
||||||
connected by some interconnect. This provides CPUs with equal access to
|
there are two or more CPUs and these CPUs are connected by some interconnect.
|
||||||
system resources like memory and IO ports.
|
This provides CPUs with equal access to system resources like memory and
|
||||||
|
input/output ports.
|
||||||
|
|
||||||
Non-uniform memory access (NUMA) is a derivative of the SMP design that is
|
Non-uniform memory access (NUMA)
|
||||||
found in many multi-socket systems. In a NUMA system, system memory is divided
|
NUMA is a derivative of the SMP design that is found in many multi-socket
|
||||||
into cells or nodes that are associated with particular CPUs. Requests for
|
systems. In a NUMA system, system memory is divided into cells or nodes that
|
||||||
memory on other nodes are possible through an interconnect bus, however,
|
are associated with particular CPUs. Requests for memory on other nodes are
|
||||||
bandwidth across this shared bus is limited. As a result, competition for this
|
possible through an interconnect bus. However, bandwidth across this shared
|
||||||
this resource can incur performance penalties.
|
bus is limited. As a result, competition for this resource can incur
|
||||||
|
performance penalties.
|
||||||
|
|
||||||
Simultaneous Multi-Threading (SMT), known as as Hyper-Threading on Intel
|
Simultaneous Multi-Threading (SMT)
|
||||||
platforms, is a design that is complementary to SMP. Whereas CPUs in SMP
|
SMT is a design complementary to SMP. Whereas CPUs in SMP systems share a bus
|
||||||
systems share a bus and some memory, CPUs in SMT systems share many more
|
and some memory, CPUs in SMT systems share many more components. CPUs that
|
||||||
components. CPUs that share components are known as thread siblings. All CPUs
|
share components are known as thread siblings. All CPUs appear as usable
|
||||||
appear as usable CPUs on the system and can execute workloads in parallel,
|
CPUs on the system and can execute workloads in parallel. However, as with
|
||||||
however, as with NUMA, threads compete for shared resources.
|
NUMA, threads compete for shared resources.
|
||||||
|
|
||||||
In OpenStack, SMP CPUs are known as *cores*, NUMA cells or nodes are known as
|
In OpenStack, SMP CPUs are known as *cores*, NUMA cells or nodes are known as
|
||||||
*sockets*, and SMT CPUs are known as *threads*. For example, a quad-socket,
|
*sockets*, and SMT CPUs are known as *threads*. For example, a quad-socket,
|
||||||
@ -49,14 +51,14 @@ processes are on the same NUMA node as the memory used by these processes.
|
|||||||
This ensures all memory accesses are local to the node and thus do not consume
|
This ensures all memory accesses are local to the node and thus do not consume
|
||||||
the limited cross-node memory bandwidth, adding latency to memory accesses.
|
the limited cross-node memory bandwidth, adding latency to memory accesses.
|
||||||
Similarly, large pages are assigned from memory and benefit from the same
|
Similarly, large pages are assigned from memory and benefit from the same
|
||||||
performance improvements as memory allocated using standard pages, thus, they
|
performance improvements as memory allocated using standard pages. Thus, they
|
||||||
also should be local. Finally, PCI devices are directly associated with
|
also should be local. Finally, PCI devices are directly associated with
|
||||||
specific NUMA nodes for the purposes of DMA. Instances that use PCI or SR-IOV
|
specific NUMA nodes for the purposes of DMA. Instances that use PCI or SR-IOV
|
||||||
devices should be placed on the NUMA node associated with the said devices.
|
devices should be placed on the NUMA node associated with these devices.
|
||||||
|
|
||||||
By default, an instance will float across all NUMA nodes on a host. NUMA
|
By default, an instance floats across all NUMA nodes on a host. NUMA
|
||||||
awareness can be enabled implicitly, through the use of hugepages or pinned
|
awareness can be enabled implicitly through the use of hugepages or pinned
|
||||||
CPUs, or explicitly, through the use of flavor extra specs or image metadata.
|
CPUs or explicitly through the use of flavor extra specs or image metadata.
|
||||||
In all cases, the ``NUMATopologyFilter`` filter must be enabled. Details on
|
In all cases, the ``NUMATopologyFilter`` filter must be enabled. Details on
|
||||||
this filter are provided in `Scheduling`_ configuration guide.
|
this filter are provided in `Scheduling`_ configuration guide.
|
||||||
|
|
||||||
@ -65,13 +67,13 @@ this filter are provided in `Scheduling`_ configuration guide.
|
|||||||
The NUMA node(s) used are normally chosen at random. However, if a PCI
|
The NUMA node(s) used are normally chosen at random. However, if a PCI
|
||||||
passthrough or SR-IOV device is attached to the instance, then the NUMA
|
passthrough or SR-IOV device is attached to the instance, then the NUMA
|
||||||
node that the device is associated with will be used. This can provide
|
node that the device is associated with will be used. This can provide
|
||||||
important performance improvements, however, booting a large number of
|
important performance improvements. However, booting a large number of
|
||||||
similar instances can result in unbalanced NUMA node usage. Care should
|
similar instances can result in unbalanced NUMA node usage. Care should
|
||||||
be taken to mitigate this issue. See this `discussion`_ for more details.
|
be taken to mitigate this issue. See this `discussion`_ for more details.
|
||||||
|
|
||||||
.. caution::
|
.. caution::
|
||||||
|
|
||||||
Inadequate per-node resources will result in scheduling failures. Resources
|
Inadequate per-node resources will result in scheduling failures. Resources
|
||||||
that are specific to a node include not only CPUs and memory, but also PCI
|
that are specific to a node include not only CPUs and memory, but also PCI
|
||||||
and SR-IOV resources. It is not possible to use multiple resources from
|
and SR-IOV resources. It is not possible to use multiple resources from
|
||||||
different nodes without requesting a multi-node layout. As such, it may be
|
different nodes without requesting a multi-node layout. As such, it may be
|
||||||
@ -85,10 +87,10 @@ run:
|
|||||||
|
|
||||||
.. code-block:: console
|
.. code-block:: console
|
||||||
|
|
||||||
# openstack flavor set m1.large --property hw:numa_nodes=1
|
$ openstack flavor set m1.large --property hw:numa_nodes=1
|
||||||
|
|
||||||
Some workloads have very demanding requirements for memory access latency or
|
Some workloads have very demanding requirements for memory access latency or
|
||||||
bandwidth which exceed the memory bandwidth available from a single NUMA node.
|
bandwidth that exceed the memory bandwidth available from a single NUMA node.
|
||||||
For such workloads, it is beneficial to spread the instance across multiple
|
For such workloads, it is beneficial to spread the instance across multiple
|
||||||
host NUMA nodes, even if the instance's RAM/vCPUs could theoretically fit on a
|
host NUMA nodes, even if the instance's RAM/vCPUs could theoretically fit on a
|
||||||
single NUMA node. To force an instance's vCPUs to spread across two host NUMA
|
single NUMA node. To force an instance's vCPUs to spread across two host NUMA
|
||||||
@ -96,23 +98,23 @@ nodes, run:
|
|||||||
|
|
||||||
.. code-block:: console
|
.. code-block:: console
|
||||||
|
|
||||||
# openstack flavor set m1.large --property hw:numa_nodes=2
|
$ openstack flavor set m1.large --property hw:numa_nodes=2
|
||||||
|
|
||||||
The allocation of instances vCPUs and memory from different host NUMA nodes can
|
The allocation of instances vCPUs and memory from different host NUMA nodes can
|
||||||
be configured. This allows for asymmetric allocation of vCPUs and memory, which
|
be configured. This allows for asymmetric allocation of vCPUs and memory, which
|
||||||
can be important for some workloads. To spread the six vCPUs and 6 GB of memory
|
can be important for some workloads. To spread the 6 vCPUs and 6 GB of memory
|
||||||
of an instance across two NUMA nodes and create an asymmetric 1:2 vCPU and
|
of an instance across two NUMA nodes and create an asymmetric 1:2 vCPU and
|
||||||
memory mapping between the two nodes, run:
|
memory mapping between the two nodes, run:
|
||||||
|
|
||||||
.. code-block:: console
|
.. code-block:: console
|
||||||
|
|
||||||
# openstack flavor set m1.large --property hw:numa_nodes=2
|
$ openstack flavor set m1.large --property hw:numa_nodes=2
|
||||||
# openstack flavor set m1.large \ # configure guest node 0
|
$ openstack flavor set m1.large \ # configure guest node 0
|
||||||
--property hw:numa_cpus.0=0,1 \
|
--property hw:numa_cpus.0=0,1 \
|
||||||
--property hw:numa_mem.0=2048
|
--property hw:numa_mem.0=2048
|
||||||
# openstack flavor set m1.large \ # configure guest node 1
|
$ openstack flavor set m1.large \ # configure guest node 1
|
||||||
--property hw:numa_cpus.1=2,3,4,5 \
|
--property hw:numa_cpus.1=2,3,4,5 \
|
||||||
--property hw:numa_mem.1=4096
|
--property hw:numa_mem.1=4096
|
||||||
|
|
||||||
For more information about the syntax for ``hw:numa_nodes``, ``hw:numa_cpus.N``
|
For more information about the syntax for ``hw:numa_nodes``, ``hw:numa_cpus.N``
|
||||||
and ``hw:num_mem.N``, refer to the `Flavors`_ guide.
|
and ``hw:num_mem.N``, refer to the `Flavors`_ guide.
|
||||||
@ -141,7 +143,7 @@ use a dedicated CPU policy. To force this, run:
|
|||||||
|
|
||||||
.. code-block:: console
|
.. code-block:: console
|
||||||
|
|
||||||
# openstack flavor set m1.large --property hw:cpu_policy=dedicated
|
$ openstack flavor set m1.large --property hw:cpu_policy=dedicated
|
||||||
|
|
||||||
.. caution::
|
.. caution::
|
||||||
|
|
||||||
@ -157,27 +159,27 @@ sharing benefits performance, use thread siblings. To force this, run:
|
|||||||
|
|
||||||
.. code-block:: console
|
.. code-block:: console
|
||||||
|
|
||||||
# openstack flavor set m1.large \
|
$ openstack flavor set m1.large \
|
||||||
--property hw:cpu_policy=dedicated \
|
--property hw:cpu_policy=dedicated \
|
||||||
--property hw:cpu_thread_policy=require
|
--property hw:cpu_thread_policy=require
|
||||||
|
|
||||||
For other workloads where performance is impacted by contention for resources,
|
For other workloads where performance is impacted by contention for resources,
|
||||||
use non-thread siblings or non-SMT hosts. To force this, run:
|
use non-thread siblings or non-SMT hosts. To force this, run:
|
||||||
|
|
||||||
.. code-block:: console
|
.. code-block:: console
|
||||||
|
|
||||||
# openstack flavor set m1.large \
|
$ openstack flavor set m1.large \
|
||||||
--property hw:cpu_policy=dedicated \
|
--property hw:cpu_policy=dedicated \
|
||||||
--property hw:cpu_thread_policy=isolate
|
--property hw:cpu_thread_policy=isolate
|
||||||
|
|
||||||
Finally, for workloads where performance is minimally impacted, use thread
|
Finally, for workloads where performance is minimally impacted, use thread
|
||||||
siblings if available. This is the default, but it can be set explicitly:
|
siblings if available. This is the default, but it can be set explicitly:
|
||||||
|
|
||||||
.. code-block:: console
|
.. code-block:: console
|
||||||
|
|
||||||
# openstack flavor set m1.large \
|
$ openstack flavor set m1.large \
|
||||||
--property hw:cpu_policy=dedicated \
|
--property hw:cpu_policy=dedicated \
|
||||||
--property hw:cpu_thread_policy=prefer
|
--property hw:cpu_thread_policy=prefer
|
||||||
|
|
||||||
For more information about the syntax for ``hw:cpu_policy`` and
|
For more information about the syntax for ``hw:cpu_policy`` and
|
||||||
``hw:cpu_thread_policy``, refer to the `Flavors`_ guide.
|
``hw:cpu_thread_policy``, refer to the `Flavors`_ guide.
|
||||||
@ -189,25 +191,26 @@ image to use pinned vCPUs and avoid thread siblings, run:
|
|||||||
|
|
||||||
.. code-block:: console
|
.. code-block:: console
|
||||||
|
|
||||||
# openstack image set [IMAGE_ID] \
|
$ openstack image set [IMAGE_ID] \
|
||||||
--property hw_cpu_policy=dedicated \
|
--property hw_cpu_policy=dedicated \
|
||||||
--property hw_cpu_thread_policy=isolate
|
--property hw_cpu_thread_policy=isolate
|
||||||
|
|
||||||
Image metadata takes precedence over flavor extra specs: configuring competing
|
Image metadata takes precedence over flavor extra specs. Thus, configuring
|
||||||
policies will result in an exception. By setting a ``shared`` policy through
|
competing policies causes an exception. By setting a ``shared`` policy
|
||||||
image metadata, administrators can prevent users configuring CPU policies in
|
through image metadata, administrators can prevent users configuring CPU
|
||||||
flavors and impacting resource utilization. To configure this policy, run:
|
policies in flavors and impacting resource utilization. To configure this
|
||||||
|
policy, run:
|
||||||
|
|
||||||
.. code-block:: console
|
.. code-block:: console
|
||||||
|
|
||||||
# openstack image set [IMAGE_ID] --property hw_cpu_policy=shared
|
$ openstack image set [IMAGE_ID] --property hw_cpu_policy=shared
|
||||||
|
|
||||||
.. note::
|
.. note::
|
||||||
|
|
||||||
There is no correlation required between the NUMA topology exposed in the
|
There is no correlation required between the NUMA topology exposed in the
|
||||||
instance and how the instance is actually pinned on the host. This is by
|
instance and how the instance is actually pinned on the host. This is by
|
||||||
design. See this `bug <https://bugs.launchpad.net/nova/+bug/1466780>`_ for
|
design. See this `invalid bug
|
||||||
more information.
|
<https://bugs.launchpad.net/nova/+bug/1466780>`_ for more information.
|
||||||
|
|
||||||
For more information about image metadata, refer to the `Image metadata`_
|
For more information about image metadata, refer to the `Image metadata`_
|
||||||
guide.
|
guide.
|
||||||
@ -235,15 +238,15 @@ sockets. To configure a flavor to use a maximum of two sockets, run:
|
|||||||
|
|
||||||
.. code-block:: console
|
.. code-block:: console
|
||||||
|
|
||||||
# openstack flavor set m1.large --property hw:cpu_sockets=2
|
$ openstack flavor set m1.large --property hw:cpu_sockets=2
|
||||||
|
|
||||||
Similarly, to configure a flavor to use one core and one thread, run:
|
Similarly, to configure a flavor to use one core and one thread, run:
|
||||||
|
|
||||||
.. code-block:: console
|
.. code-block:: console
|
||||||
|
|
||||||
# openstack flavor set m1.large \
|
$ openstack flavor set m1.large \
|
||||||
--property hw:cpu_cores=1 \
|
--property hw:cpu_cores=1 \
|
||||||
--property hw:cpu_threads=1
|
--property hw:cpu_threads=1
|
||||||
|
|
||||||
.. caution::
|
.. caution::
|
||||||
|
|
||||||
@ -260,14 +263,14 @@ and ``hw:cpu_threads``, refer to the `Flavors`_ guide.
|
|||||||
|
|
||||||
It is also possible to set upper limits on the number of sockets, cores, and
|
It is also possible to set upper limits on the number of sockets, cores, and
|
||||||
threads used. Unlike the hard values above, it is not necessary for this exact
|
threads used. Unlike the hard values above, it is not necessary for this exact
|
||||||
number to used because it only provides a a limit. This can be used to provide
|
number to used because it only provides a limit. This can be used to provide
|
||||||
some flexibility in scheduling, while ensuring certains limits are not
|
some flexibility in scheduling, while ensuring certains limits are not
|
||||||
exceeded. For example, to ensure no more than two sockets are defined in the
|
exceeded. For example, to ensure no more than two sockets are defined in the
|
||||||
instance topology, run:
|
instance topology, run:
|
||||||
|
|
||||||
.. code-block:: console
|
.. code-block:: console
|
||||||
|
|
||||||
# openstack flavor set m1.large --property=hw:cpu_max_sockets=2
|
$ openstack flavor set m1.large --property=hw:cpu_max_sockets=2
|
||||||
|
|
||||||
For more information about the syntax for ``hw:cpu_max_sockets``,
|
For more information about the syntax for ``hw:cpu_max_sockets``,
|
||||||
``hw:cpu_max_cores``, and ``hw:cpu_max_threads``, refer to the `Flavors`_
|
``hw:cpu_max_cores``, and ``hw:cpu_max_threads``, refer to the `Flavors`_
|
||||||
@ -280,9 +283,9 @@ request a two-socket, four-core per socket topology, run:
|
|||||||
|
|
||||||
.. code-block:: console
|
.. code-block:: console
|
||||||
|
|
||||||
# openstack image set [IMAGE_ID] \
|
$ openstack image set [IMAGE_ID] \
|
||||||
--property hw_cpu_sockets=2 \
|
--property hw_cpu_sockets=2 \
|
||||||
--property hw_cpu_cores=4
|
--property hw_cpu_cores=4
|
||||||
|
|
||||||
To constrain instances to a given limit of sockets, cores or threads, use the
|
To constrain instances to a given limit of sockets, cores or threads, use the
|
||||||
``max_`` variants. To configure an image to have a maximum of two sockets and a
|
``max_`` variants. To configure an image to have a maximum of two sockets and a
|
||||||
@ -290,9 +293,9 @@ maximum of one thread, run:
|
|||||||
|
|
||||||
.. code-block:: console
|
.. code-block:: console
|
||||||
|
|
||||||
# openstack image set [IMAGE_ID] \
|
$ openstack image set [IMAGE_ID] \
|
||||||
--property hw_cpu_max_sockets=2 \
|
--property hw_cpu_max_sockets=2 \
|
||||||
--property hw_cpu_max_threads=1
|
--property hw_cpu_max_threads=1
|
||||||
|
|
||||||
Image metadata takes precedence over flavor extra specs. Configuring competing
|
Image metadata takes precedence over flavor extra specs. Configuring competing
|
||||||
constraints causes an exception. By setting a ``max`` value for sockets, cores,
|
constraints causes an exception. By setting a ``max`` value for sockets, cores,
|
@ -207,6 +207,7 @@ redirect 301 /admin-guide/cli_nova_specify_host.html /admin-guide/cli-nova-speci
|
|||||||
redirect 301 /admin-guide/cli_set_compute_quotas.html /admin-guide/cli-set-compute-quotas.html
|
redirect 301 /admin-guide/cli_set_compute_quotas.html /admin-guide/cli-set-compute-quotas.html
|
||||||
redirect 301 /admin-guide/cli_set_quotas.html /admin-guide/cli-set-quotas.html
|
redirect 301 /admin-guide/cli_set_quotas.html /admin-guide/cli-set-quotas.html
|
||||||
redirect 301 /admin-guide/compute_arch.html /admin-guide/compute-arch.html
|
redirect 301 /admin-guide/compute_arch.html /admin-guide/compute-arch.html
|
||||||
|
redirect 301 /admin-guide/compute-numa-cpu-pinning.html /admin-guide/compute-cpu-topologies.html
|
||||||
redirect 301 /admin-guide/cross_project.html /admin-guide/cross-project.html
|
redirect 301 /admin-guide/cross_project.html /admin-guide/cross-project.html
|
||||||
redirect 301 /admin-guide/cross_project_cors.html /admin-guide/cross-project-cors.html
|
redirect 301 /admin-guide/cross_project_cors.html /admin-guide/cross-project-cors.html
|
||||||
redirect 301 /admin-guide/dashboard_admin_manage_roles.html /admin-guide/dashboard-admin-manage-roles.html
|
redirect 301 /admin-guide/dashboard_admin_manage_roles.html /admin-guide/dashboard-admin-manage-roles.html
|
||||||
|
Loading…
Reference in New Issue
Block a user