From 777c02485f0bd1d99d12594dd0ebce24c1bdbb55 Mon Sep 17 00:00:00 2001 From: Stephen Finucane Date: Mon, 22 Feb 2021 12:52:17 +0000 Subject: [PATCH] docs: Add a real-time guide This beefy patch closes a long-standing TODO and allows us to move yet more information out of the flavors guide and into specific documents. This, combined with existing documentation in place, means we can remove the sections for various extra specs from the 'user/flavors' guide: - hw:cpu_realtime -> doc/source/admin/real-time.rst - hw:cpu_realtime_mask -> doc/source/admin/real-time.rst - hw:emulator_threads_policy -> doc/source/admin/cpu-topologies.rst - hw:cpu_policy -> doc/source/admin/cpu-topologies.rst - hw:cpu_thread_policy -> doc/source/admin/cpu-topologies.rst - hw:cpu_sockets -> doc/source/admin/cpu-topologies.rst - hw:cpu_cores -> doc/source/admin/cpu-topologies.rst - hw:cpu_threads -> doc/source/admin/cpu-topologies.rst - hw:cpu_max_sockets -> doc/source/admin/cpu-topologies.rst - hw:cpu_max_cores -> doc/source/admin/cpu-topologies.rst - hw:cpu_max_threads -> doc/source/admin/cpu-topologies.rst - hw:numa_nodes -> doc/source/admin/cpu-topologies.rst - hw:numa_cpus.N -> doc/source/admin/cpu-topologies.rst - hw:numa_mem.N -> doc/source/admin/cpu-topologies.rst - hw:mem_page_size -> doc/source/admin/huge-pages.rst Multiple improvements to the libvirt extra spec docs are included here, for want of a better place to include them. Change-Id: I02b044f8246f4a42481bb5f00259842692b29b71 Signed-off-by: Stephen Finucane --- doc/source/admin/cpu-topologies.rst | 158 ++++++++---- doc/source/admin/huge-pages.rst | 5 +- doc/source/admin/index.rst | 1 + doc/source/admin/networking.rst | 5 +- doc/source/admin/real-time.rst | 153 ++++++++++++ doc/source/configuration/extra-specs.rst | 5 +- doc/source/user/flavors.rst | 291 ----------------------- 7 files changed, 270 insertions(+), 348 deletions(-) create mode 100644 doc/source/admin/real-time.rst diff --git a/doc/source/admin/cpu-topologies.rst b/doc/source/admin/cpu-topologies.rst index 5c2a1c974018..8f41ddfed8df 100644 --- a/doc/source/admin/cpu-topologies.rst +++ b/doc/source/admin/cpu-topologies.rst @@ -56,6 +56,8 @@ VCPU Resource class representing a unit of CPU resources for a single guest approximating the processing power of a single physical processor. +.. _numa-topologies: + Customizing instance NUMA placement policies -------------------------------------------- @@ -113,12 +115,14 @@ filter must be enabled. Details on this filter are provided in When used, NUMA awareness allows the operating system of the instance to intelligently schedule the workloads that it runs and minimize cross-node -memory bandwidth. To restrict an instance's vCPUs to a single host NUMA node, +memory bandwidth. To configure guest NUMA nodes, you can use the +:nova:extra-spec:`hw:numa_nodes` flavor extra spec. +For example, to restrict an instance's vCPUs to a single host NUMA node, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] --property hw:numa_nodes=1 + $ openstack flavor set $FLAVOR --property hw:numa_nodes=1 Some workloads have very demanding requirements for memory access latency or bandwidth that exceed the memory bandwidth available from a single NUMA node. @@ -129,34 +133,56 @@ nodes, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] --property hw:numa_nodes=2 + $ openstack flavor set $FLAVOR --property hw:numa_nodes=2 -The allocation of instances vCPUs and memory from different host NUMA nodes can +The allocation of instance vCPUs and memory from different host NUMA nodes can be configured. This allows for asymmetric allocation of vCPUs and memory, which -can be important for some workloads. To spread the 6 vCPUs and 6 GB of memory +can be important for some workloads. You can configure the allocation of +instance vCPUs and memory across each **guest** NUMA node using the +:nova:extra-spec:`hw:numa_cpus.{id}` and :nova:extra-spec:`hw:numa_mem.{id}` +extra specs respectively. +For example, to spread the 6 vCPUs and 6 GB of memory of an instance across two NUMA nodes and create an asymmetric 1:2 vCPU and memory mapping between the two nodes, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] --property hw:numa_nodes=2 + $ openstack flavor set $FLAVOR --property hw:numa_nodes=2 # configure guest node 0 - $ openstack flavor set [FLAVOR_ID] \ + $ openstack flavor set $FLAVOR \ --property hw:numa_cpus.0=0,1 \ --property hw:numa_mem.0=2048 # configure guest node 1 - $ openstack flavor set [FLAVOR_ID] \ + $ openstack flavor set $FLAVOR \ --property hw:numa_cpus.1=2,3,4,5 \ --property hw:numa_mem.1=4096 +.. note:: + + The ``{num}`` parameter is an index of *guest* NUMA nodes and may not + correspond to *host* NUMA nodes. For example, on a platform with two NUMA + nodes, the scheduler may opt to place guest NUMA node 0, as referenced in + ``hw:numa_mem.0`` on host NUMA node 1 and vice versa. Similarly, the + CPUs bitmask specified in the value for ``hw:numa_cpus.{num}`` refer to + *guest* vCPUs and may not correspond to *host* CPUs. As such, this feature + cannot be used to constrain instances to specific host CPUs or NUMA nodes. + +.. warning:: + + If the combined values of ``hw:numa_cpus.{num}`` or ``hw:numa_mem.{num}`` + are greater than the available number of CPUs or memory respectively, an + exception will be raised. + .. note:: Hyper-V does not support asymmetric NUMA topologies, and the Hyper-V driver will not spawn instances with such topologies. For more information about the syntax for ``hw:numa_nodes``, ``hw:numa_cpus.N`` -and ``hw:num_mem.N``, refer to the :ref:`NUMA -topology ` guide. +and ``hw:num_mem.N``, refer to :doc:`/configuration/extra-specs`. + + +.. _cpu-pinning-policies: Customizing instance CPU pinning policies ----------------------------------------- @@ -189,14 +215,16 @@ CPUs can use the CPUs of another pinned instance, thus preventing resource contention between instances. CPU pinning policies can be used to determine whether an instance should be -pinned or not. There are three policies: ``dedicated``, ``mixed`` and +pinned or not. They can be configured using the +:nova:extra-spec:`hw:cpu_policy` extra spec and equivalent image metadata +property. There are three policies: ``dedicated``, ``mixed`` and ``shared`` (the default). The ``dedicated`` CPU policy is used to specify that all CPUs of an instance should use pinned CPUs. To configure a flavor to use the ``dedicated`` CPU policy, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] --property hw:cpu_policy=dedicated + $ openstack flavor set $FLAVOR --property hw:cpu_policy=dedicated This works by ensuring ``PCPU`` allocations are used instead of ``VCPU`` allocations. As such, it is also possible to request this resource type @@ -204,9 +232,9 @@ explicitly. To configure this, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] --property resources:PCPU=N + $ openstack flavor set $FLAVOR --property resources:PCPU=N -where ``N`` is the number of vCPUs defined in the flavor. +(where ``N`` is the number of vCPUs defined in the flavor). .. note:: @@ -218,28 +246,28 @@ use pinned CPUs. To configure a flavor to use the ``shared`` CPU policy, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] --property hw:cpu_policy=shared + $ openstack flavor set $FLAVOR --property hw:cpu_policy=shared The ``mixed`` CPU policy is used to specify that an instance use pinned CPUs along with unpinned CPUs. The instance pinned CPU could be specified in the -``hw:cpu_dedicated_mask`` or, if real-time is enabled -(``hw:cpu_realtime``\ = yes), in the ``hw:cpu_realtime_mask`` extra spec. For +:nova:extra-spec:`hw:cpu_dedicated_mask` or, if :doc:`real-time ` is +enabled, in the :nova:extra-spec:`hw:cpu_realtime_mask` extra spec. For example, to configure a flavor to use the ``mixed`` CPU policy with 4 vCPUs in -total and the first 2 vCPUs as pinned CPUs, with the ``hw:cpu_realtime_mask`` -extra spec, run: +total and the first 2 vCPUs as pinned CPUs, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] \ + $ openstack flavor set $FLAVOR \ --vcpus=4 \ --property hw:cpu_policy=mixed \ --property hw:cpu_dedicated_mask=0-1 -To create the mixed instance with the real-time extra specs, run: +To configure a flavor to use the ``mixed`` CPU policy with 4 vCPUs in total and +the first 2 vCPUs as pinned **real-time** CPUs, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] \ + $ openstack flavor set $FLAVOR \ --vcpus=4 \ --property hw:cpu_policy=mixed \ --property hw:cpu_realtime=yes \ @@ -249,7 +277,12 @@ To create the mixed instance with the real-time extra specs, run: For more information about the syntax for ``hw:cpu_policy``, ``hw:cpu_dedicated_mask``, ``hw:realtime_cpu`` and ``hw:cpu_realtime_mask``, - refer to the :doc:`/user/flavors` guide. + refer to :doc:`/configuration/extra-specs` + +.. note:: + + For more information about real-time functionality, refer to the + :doc:`documentation `. It is also possible to configure the CPU policy via image metadata. This can be useful when packaging applications that require real-time or near real-time @@ -259,13 +292,13 @@ policy, run: .. code-block:: console - $ openstack image set [IMAGE_ID] --property hw_cpu_policy=dedicated + $ openstack image set $IMAGE --property hw_cpu_policy=dedicated Likewise, to configure an image to use the ``shared`` CPU policy, run: .. code-block:: console - $ openstack image set [IMAGE_ID] --property hw_cpu_policy=shared + $ openstack image set $IMAGE --property hw_cpu_policy=shared .. note:: @@ -312,7 +345,7 @@ performance, you can request hosts **with** SMT. To configure this, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] \ + $ openstack flavor set $FLAVOR \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=require @@ -322,7 +355,7 @@ request this trait explicitly. To configure this, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] \ + $ openstack flavor set $FLAVOR \ --property resources:PCPU=N \ --property trait:HW_CPU_HYPERTHREADING=required @@ -331,7 +364,7 @@ you can request hosts **without** SMT. To configure this, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] \ + $ openstack flavor set $FLAVOR \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=isolate @@ -341,7 +374,7 @@ possible to request this trait explicitly. To configure this, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] \ + $ openstack flavor set $FLAVOR \ --property resources:PCPU=N \ --property trait:HW_CPU_HYPERTHREADING=forbidden @@ -351,7 +384,7 @@ is the default, but it can be set explicitly: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] \ + $ openstack flavor set $FLAVOR \ --property hw:cpu_policy=dedicated \ --property hw:cpu_thread_policy=prefer @@ -360,7 +393,7 @@ This does not utilize traits and, as such, there is no trait-based equivalent. .. note:: For more information about the syntax for ``hw:cpu_thread_policy``, refer to - the :doc:`/user/flavors` guide. + :doc:`/configuration/extra-specs`. As with CPU policies, it also possible to configure the CPU thread policy via image metadata. This can be useful when packaging applications that require @@ -370,7 +403,7 @@ image are always pinned regardless of flavor. To configure an image to use the .. code-block:: console - $ openstack image set [IMAGE_ID] \ + $ openstack image set $IMAGE \ --property hw_cpu_policy=dedicated \ --property hw_cpu_thread_policy=require @@ -378,7 +411,7 @@ Likewise, to configure an image to use the ``isolate`` CPU thread policy, run: .. code-block:: console - $ openstack image set [IMAGE_ID] \ + $ openstack image set $IMAGE \ --property hw_cpu_policy=dedicated \ --property hw_cpu_thread_policy=isolate @@ -386,7 +419,7 @@ Finally, to configure an image to use the ``prefer`` CPU thread policy, run: .. code-block:: console - $ openstack image set [IMAGE_ID] \ + $ openstack image set $IMAGE \ --property hw_cpu_policy=dedicated \ --property hw_cpu_thread_policy=prefer @@ -400,6 +433,8 @@ an exception will be raised. For more information about image metadata, refer to the `Image metadata`_ guide. +.. _emulator-thread-pinning-policies: + Customizing instance emulator thread pinning policies ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -429,25 +464,45 @@ the ``isolate`` emulator thread policy, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] \ + $ openstack flavor set $FLAVOR \ --property hw:cpu_policy=dedicated \ --property hw:emulator_threads_policy=isolate The ``share`` policy is used to specify that emulator threads from a given instance should be run on the pool of host cores listed in -:oslo.config:option:`compute.cpu_shared_set`. To configure a flavor to use the -``share`` emulator thread policy, run: +:oslo.config:option:`compute.cpu_shared_set` if configured, else across all +pCPUs of the instance. +To configure a flavor to use the ``share`` emulator thread policy, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] \ + $ openstack flavor set $FLAVOR \ --property hw:cpu_policy=dedicated \ --property hw:emulator_threads_policy=share +The above behavior can be summarized in this helpful table: + +.. list-table:: + :header-rows: 1 + :stub-columns: 1 + + * - + - :oslo.config:option:`compute.cpu_shared_set` set + - :oslo.config:option:`compute.cpu_shared_set` unset + * - ``hw:emulator_treads_policy`` unset (default) + - Pinned to all of the instance's pCPUs + - Pinned to all of the instance's pCPUs + * - ``hw:emulator_threads_policy`` = ``share`` + - Pinned to :oslo.config:option:`compute.cpu_shared_set` + - Pinned to all of the instance's pCPUs + * - ``hw:emulator_threads_policy`` = ``isolate`` + - Pinned to a single pCPU distinct from the instance's pCPUs + - Pinned to a single pCPU distinct from the instance's pCPUs + .. note:: For more information about the syntax for ``hw:emulator_threads_policy``, - refer to the :doc:`/user/flavors` guide. + refer to :nova:extra-spec:`the documentation `. Customizing instance CPU topologies ----------------------------------- @@ -476,17 +531,17 @@ sockets. Some workloads benefit from a custom topology. For example, in some operating systems, a different license may be needed depending on the number of CPU -sockets. To configure a flavor to use a maximum of two sockets, run: +sockets. To configure a flavor to use two sockets, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] --property hw:cpu_sockets=2 + $ openstack flavor set $FLAVOR --property hw:cpu_sockets=2 Similarly, to configure a flavor to use one core and one thread, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] \ + $ openstack flavor set $FLAVOR \ --property hw:cpu_cores=1 \ --property hw:cpu_threads=1 @@ -501,22 +556,25 @@ Similarly, to configure a flavor to use one core and one thread, run: with ten cores fails. For more information about the syntax for ``hw:cpu_sockets``, ``hw:cpu_cores`` -and ``hw:cpu_threads``, refer to the :doc:`/user/flavors` guide. +and ``hw:cpu_threads``, refer to :doc:`/configuration/extra-specs`. It is also possible to set upper limits on the number of sockets, cores, and threads used. Unlike the hard values above, it is not necessary for this exact number to used because it only provides a limit. This can be used to provide some flexibility in scheduling, while ensuring certain limits are not -exceeded. For example, to ensure no more than two sockets are defined in the -instance topology, run: +exceeded. For example, to ensure no more than two sockets, eight cores and one +thread are defined in the instance topology, run: .. code-block:: console - $ openstack flavor set [FLAVOR_ID] --property hw:cpu_max_sockets=2 + $ openstack flavor set $FLAVOR \ + --property hw:cpu_max_sockets=2 \ + --property hw:cpu_max_cores=8 \ + --property hw:cpu_max_threads=1 For more information about the syntax for ``hw:cpu_max_sockets``, -``hw:cpu_max_cores``, and ``hw:cpu_max_threads``, refer to the -:doc:`/user/flavors` guide. +``hw:cpu_max_cores``, and ``hw:cpu_max_threads``, refer to +:doc:`/configuration/extra-specs`. Applications are frequently packaged as images. For applications that prefer certain CPU topologies, configure image metadata to hint that created instances @@ -525,7 +583,7 @@ request a two-socket, four-core per socket topology, run: .. code-block:: console - $ openstack image set [IMAGE_ID] \ + $ openstack image set $IMAGE \ --property hw_cpu_sockets=2 \ --property hw_cpu_cores=4 @@ -535,7 +593,7 @@ maximum of one thread, run: .. code-block:: console - $ openstack image set [IMAGE_ID] \ + $ openstack image set $IMAGE \ --property hw_cpu_max_sockets=2 \ --property hw_cpu_max_threads=1 diff --git a/doc/source/admin/huge-pages.rst b/doc/source/admin/huge-pages.rst index 3a77756564a5..73f6c5dd2dbf 100644 --- a/doc/source/admin/huge-pages.rst +++ b/doc/source/admin/huge-pages.rst @@ -56,6 +56,7 @@ Enabling huge pages on the host ------------------------------- .. important:: + Huge pages may not be used on a host configured for file-backed memory. See :doc:`file-backed-memory` for details @@ -163,7 +164,7 @@ By default, an instance does not use huge pages for its underlying memory. However, huge pages can bring important or required performance improvements for some workloads. Huge pages must be requested explicitly through the use of flavor extra specs or image metadata. To request an instance use huge pages, -run: +you can use the :nova:extra-spec:`hw:mem_page_size` flavor extra spec: .. code-block:: console @@ -205,7 +206,7 @@ run: $ openstack flavor set m1.large --property hw:mem_page_size=any For more information about the syntax for ``hw:mem_page_size``, refer to -:doc:`flavors`. +:nova:extra-spec:`the documentation `. Applications are frequently packaged as images. For applications that require the IO performance improvements that huge pages provides, configure image diff --git a/doc/source/admin/index.rst b/doc/source/admin/index.rst index 3509233761be..c5c3c296f1db 100644 --- a/doc/source/admin/index.rst +++ b/doc/source/admin/index.rst @@ -107,6 +107,7 @@ instance for these kind of workloads. pci-passthrough cpu-topologies + real-time huge-pages virtual-gpu file-backed-memory diff --git a/doc/source/admin/networking.rst b/doc/source/admin/networking.rst index 626bb8959226..8b5a296d18fd 100644 --- a/doc/source/admin/networking.rst +++ b/doc/source/admin/networking.rst @@ -187,8 +187,9 @@ with ``provider:physical_network=foo`` must be scheduled on host cores from NUMA nodes 0, while instances using one or more networks with ``provider:physical_network=bar`` must be scheduled on host cores from both NUMA nodes 2 and 3. For the latter case, it will be necessary to split the -guest across two or more host NUMA nodes using the ``hw:numa_nodes`` -:ref:`flavor extra spec `. +guest across two or more host NUMA nodes using the +:nova:extra-spec:`hw:numa_nodes` extra spec, as discussed :ref:`here +`. Now, take an example for a deployment using L3 networks. diff --git a/doc/source/admin/real-time.rst b/doc/source/admin/real-time.rst new file mode 100644 index 000000000000..fdebf6bcc9c8 --- /dev/null +++ b/doc/source/admin/real-time.rst @@ -0,0 +1,153 @@ +========= +Real Time +========= + +.. versionadded:: 13.0.0 (Mitaka) + +Nova supports configuring `real-time policies`__ for instances. This builds upon +the improved performance offered by :doc:`CPU pinning ` by +providing stronger guarantees for worst case scheduler latency for vCPUs. + +.. __: https://en.wikipedia.org/wiki/Real-time_computing + + +Enabling Real-Time +------------------ + +Currently the creation of real-time instances is only supported when using the +libvirt compute driver with a :oslo.config:option:`libvirt.virt_type` of +``kvm`` or ``qemu``. It requires extensive configuration of the host and this +document provides but a rough overview of the changes required. Configuration +will vary depending on your hardware, BIOS configuration, host and guest OS' +and application. + +BIOS configuration +~~~~~~~~~~~~~~~~~~ + +Configure your host BIOS as recommended in the `rt-wiki`__ page. +The most important steps are: + +- Disable power management, including CPU sleep states +- Disable SMT (hyper-threading) or any option related to logical processors + +These are standard steps used in benchmarking as both sets of features can +result in non-deterministic behavior. + +.. __: https://rt.wiki.kernel.org/index.php/HOWTO:_Build_an_RT-application + +OS configuration +~~~~~~~~~~~~~~~~ + +This is inherently specific to the distro used, however, there are some common +steps: + +- Install the real-time (preemptible) kernel (``PREEMPT_RT_FULL``) and + real-time KVM modules +- Configure hugepages +- Isolate host cores to be used for instances from the kernel +- Disable features like CPU frequency scaling (e.g. P-States on Intel + processors) + +RHEL and RHEL-derived distros like CentOS provide packages in their +repositories to accomplish. The ``kernel-rt`` and ``kernel-rt-kvm`` +packages will provide the real-time kernel and real-time KVM module, +respectively, while the ``tuned-profiles-realtime`` package will provide +`tuned`__ profiles to configure the host for real-time workloads. You should +refer to your distro documentation for more information. + +.. __: https://tuned-project.org/ + +Validation +~~~~~~~~~~ + +Once your BIOS and the host OS have been configured, you can validate +"real-time readiness" using the ``hwlatdetect`` and ``rteval`` utilities. On +RHEL and RHEL-derived hosts, you can install these using the ``rt-tests`` +package. More information about the ``rteval`` tool can be found `here`__. + +.. __: https://git.kernel.org/pub/scm/utils/rteval/rteval.git/tree/README + + +Configuring a flavor or image +----------------------------- + +.. versionchanged:: 22.0.0 (Victoria) + + Previously, it was necessary to specify + :nova:extra-spec:`hw:cpu_realtime_mask` when + :nova:extra-spec:`hw:cpu_realtime` was set to ``true``. + Starting in Victoria, it is possible + to omit this when an emulator thread policy is configured using the + :nova:extra-spec:`hw:emulator_threads_policy` extra spec, thus allowing all + guest cores to be be allocated as real-time cores. + +.. versionchanged:: 22.0.0 (Victoria) + + Previously, a leading caret was necessary when specifying the value for + :nova:extra-spec:`hw:cpu_realtime_mask` and omitting it would be equivalent + to not setting the mask, resulting in a failure to spawn the instance. + +Compared to configuring the host, configuring the guest is relatively trivial +and merely requires a combination of flavor extra specs and image metadata +properties, along with a suitable real-time guest OS. + +Enable real-time by setting the :nova:extra-spec:`hw:cpu_realtime` flavor extra +spec to ``true``. When this is configured, it is necessary to specify where +guest overhead processes should be scheduled to. This can be accomplished in +one of three ways. Firstly, the :nova:extra-spec:`hw:cpu_realtime_mask` extra +spec or equivalent image metadata property can be used to indicate which guest +cores should be scheduled as real-time cores, leaving the remainder to be +scheduled as non-real-time cores and to handle overhead processes. For example, +to allocate the first two cores of an 8 core instance as the non-real-time +cores: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --property hw:cpu_realtime=yes \ + --property hw:cpu_realtime_mask=2-7 # so 0,1 are non-real-time + +In this configuration, any non-real-time cores configured will have an implicit +``dedicated`` :ref:`CPU pinning policy ` applied. It is +possible to apply a ``shared`` policy for these non-real-time cores by +specifying the ``mixed`` :ref:`CPU pinning policy ` via +the :nova:extra-spec:`hw:cpu_policy` extra spec. This can be useful to increase +resource utilization of the host. For example: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --property hw:cpu_policy=mixed \ + --property hw:cpu_realtime=yes \ + --property hw:cpu_realtime_mask=2-7 # so 0,1 are non-real-time and unpinned + +Finally, you can explicitly :ref:`offload guest overhead processes to another +host core ` using the +:nova:extra-spec:`hw:emulator_threads_policy` extra spec. For example: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --property hw:cpu_realtime=yes \ + --property hw:emulator_thread_policy=share + +.. note:: + + Emulator thread pinning requires additional host configuration. + Refer to :ref:`the documentation ` for + more information. + +In addition to configuring the instance CPUs, it is also likely that you will +need to configure guest huge pages. For information on how to configure these, +refer to :doc:`the documentation ` + +References +---------- + +* `Libvirt real time instances (spec)`__ +* `The Real Time Linux collaborative project`__ +* `Deploying Real Time OpenStack`__ + +.. __: https://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/libvirt-real-time.html +.. __: https://wiki.linuxfoundation.org/realtime/start +.. __: https://that.guru/blog/deploying-real-time-openstack/ diff --git a/doc/source/configuration/extra-specs.rst b/doc/source/configuration/extra-specs.rst index 6cd8788e144f..8e7c8614df5f 100644 --- a/doc/source/configuration/extra-specs.rst +++ b/doc/source/configuration/extra-specs.rst @@ -114,9 +114,8 @@ driver-specific. ~~~~~~~~~ The following extra specs are used to configure quotas for various -paravirtualized devices. - -They are only supported by the libvirt virt driver. +paravirtualized devices. Different quotas are supported by different virt +drivers, as noted below. .. extra-specs:: quota diff --git a/doc/source/user/flavors.rst b/doc/source/user/flavors.rst index 84624f8ea2b9..7eb243478d78 100644 --- a/doc/source/user/flavors.rst +++ b/doc/source/user/flavors.rst @@ -184,103 +184,6 @@ Performance Monitoring Unit (vPMU) required, such workloads should set ``hw:pmu=False``. For most workloads the default of unset or enabling the vPMU ``hw:pmu=True`` will be correct. -.. _extra-specs-cpu-topology: - -CPU topology - For the libvirt driver, you can define the topology of the processors in the - virtual machine using properties. The properties with ``max`` limit the - number that can be selected by the user with image properties. - - .. code-block:: console - - $ openstack flavor set FLAVOR-NAME \ - --property hw:cpu_sockets=FLAVOR-SOCKETS \ - --property hw:cpu_cores=FLAVOR-CORES \ - --property hw:cpu_threads=FLAVOR-THREADS \ - --property hw:cpu_max_sockets=FLAVOR-SOCKETS \ - --property hw:cpu_max_cores=FLAVOR-CORES \ - --property hw:cpu_max_threads=FLAVOR-THREADS - - Where: - - - FLAVOR-SOCKETS: (integer) The number of sockets for the guest VM. By - default, this is set to the number of vCPUs requested. - - FLAVOR-CORES: (integer) The number of cores per socket for the guest VM. By - default, this is set to ``1``. - - FLAVOR-THREADS: (integer) The number of threads per core for the guest VM. - By default, this is set to ``1``. - -.. _extra-specs-cpu-policy: - -CPU pinning policy - For the libvirt driver, you can pin the virtual CPUs (vCPUs) of instances to - the host's physical CPU cores (pCPUs) using properties. You can further - refine this by stating how hardware CPU threads in a simultaneous - multithreading-based (SMT) architecture be used. These configurations will - result in improved per-instance determinism and performance. - - .. note:: - - SMT-based architectures include Intel processors with Hyper-Threading - technology. In these architectures, processor cores share a number of - components with one or more other cores. Cores in such architectures are - commonly referred to as hardware threads, while the cores that a given - core share components with are known as thread siblings. - - .. note:: - - Host aggregates should be used to separate these pinned instances from - unpinned instances as the latter will not respect the resourcing - requirements of the former. - - .. code:: console - - $ openstack flavor set FLAVOR-NAME \ - --property hw:cpu_policy=CPU-POLICY \ - --property hw:cpu_thread_policy=CPU-THREAD-POLICY - - Valid CPU-POLICY values are: - - - ``shared``: (default) The guest vCPUs will be allowed to freely float - across host pCPUs, albeit potentially constrained by NUMA policy. - - ``dedicated``: The guest vCPUs will be strictly pinned to a set of host - pCPUs. In the absence of an explicit vCPU topology request, the drivers - typically expose all vCPUs as sockets with one core and one thread. When - strict CPU pinning is in effect the guest CPU topology will be setup to - match the topology of the CPUs to which it is pinned. This option implies - an overcommit ratio of 1.0. For example, if a two vCPU guest is pinned to a - single host core with two threads, then the guest will get a topology of - one socket, one core, two threads. - - ``mixed``: This policy will create an instance combined with the ``shared`` - policy vCPUs and ``dedicated`` policy vCPUs, as a result, some guest vCPUs - will be freely float across host pCPUs and the rest of guest vCPUs will be - pinned to host pCPUs. The pinned guest vCPUs are configured using the - ``hw:cpu_dedicated_mask`` extra spec. - - .. note:: - - The ``hw:cpu_dedicated_mask`` option is only valid if ``hw:cpu_policy`` - is set to ``mixed`` and cannot be configured with ``hw:cpu_realtime_mask`` - at the same time. - - Valid CPU-THREAD-POLICY values are: - - - ``prefer``: (default) The host may or may not have an SMT architecture. - Where an SMT architecture is present, thread siblings are preferred. - - ``isolate``: The host must not have an SMT architecture or must emulate a - non-SMT architecture. Hosts that support SMT (by reporting the - ``HW_CPU_HYPERTHREADING`` trait) are excluded. - - ``require``: The host must have an SMT architecture and must report the - ``HW_CPU_HYPERTHREADING`` trait. Each vCPU is allocated on thread siblings. - If the host does not have an SMT architecture, then it is not used. If the - host has an SMT architecture, but not enough cores with free thread - siblings are available, then scheduling fails. - - .. note:: - - The ``hw:cpu_thread_policy`` option is valid if ``hw:cpu_policy`` is set - to ``dedicated`` or ``mixed``. - .. _pci_numa_affinity_policy: PCI NUMA Affinity Policy @@ -336,60 +239,6 @@ PCI NUMA Affinity Policy * There is no information about PCI-NUMA affinity available -.. _extra-specs-numa-topology: - -NUMA topology - For the libvirt driver, you can define the host NUMA placement for the - instance vCPU threads as well as the allocation of instance vCPUs and memory - from the host NUMA nodes. For flavors whose memory and vCPU allocations are - larger than the size of NUMA nodes in the compute hosts, the definition of a - NUMA topology allows hosts to better utilize NUMA and improve performance of - the instance OS. - - .. code-block:: console - - $ openstack flavor set FLAVOR-NAME \ - --property hw:numa_nodes=FLAVOR-NODES \ - --property hw:numa_cpus.N=FLAVOR-CORES \ - --property hw:numa_mem.N=FLAVOR-MEMORY - - Where: - - - FLAVOR-NODES: (integer) The number of host NUMA nodes to restrict execution - of instance vCPU threads to. If not specified, the vCPU threads can run on - any number of the host NUMA nodes available. - - N: (integer) The instance NUMA node to apply a given CPU or memory - configuration to, where N is in the range ``0`` to ``FLAVOR-NODES - 1``. - - FLAVOR-CORES: (comma-separated list of integers) A list of instance vCPUs - to map to instance NUMA node N. If not specified, vCPUs are evenly divided - among available NUMA nodes. - - FLAVOR-MEMORY: (integer) The number of MB of instance memory to map to - instance NUMA node N. If not specified, memory is evenly divided among - available NUMA nodes. - - .. note:: - - ``hw:numa_cpus.N`` and ``hw:numa_mem.N`` are only valid if - ``hw:numa_nodes`` is set. Additionally, they are only required if the - instance's NUMA nodes have an asymmetrical allocation of CPUs and RAM - (important for some NFV workloads). - - .. note:: - - The ``N`` parameter is an index of *guest* NUMA nodes and may not - correspond to *host* NUMA nodes. For example, on a platform with two NUMA - nodes, the scheduler may opt to place guest NUMA node 0, as referenced in - ``hw:numa_mem.0`` on host NUMA node 1 and vice versa. Similarly, the - integers used for ``FLAVOR-CORES`` are indexes of *guest* vCPUs and may - not correspond to *host* CPUs. As such, this feature cannot be used to - constrain instances to specific host CPUs or NUMA nodes. - - .. warning:: - - If the combined values of ``hw:numa_cpus.N`` or ``hw:numa_mem.N`` are - greater than the available number of CPUs or memory respectively, an - exception is raised. - .. _extra-specs-memory-encryption: Hardware encryption of guest memory @@ -402,146 +251,6 @@ Hardware encryption of guest memory $ openstack flavor set FLAVOR-NAME \ --property hw:mem_encryption=True -.. _extra-specs-realtime-policy: - -CPU real-time policy - For the libvirt driver, you can state that one or more of your instance - virtual CPUs (vCPUs), though not all of them, run with a real-time policy. - When used on a correctly configured host, this provides stronger guarantees - for worst case scheduler latency for vCPUs and is a requirement for certain - applications. - - .. todo:: - - Document the required steps to configure hosts and guests. There are a lot - of things necessary, from isolating hosts and configuring the - ``[compute] cpu_dedicated_set`` nova configuration option on the host, to - choosing a correctly configured guest image. - - .. important:: - - While most of your instance vCPUs can run with a real-time policy, you must - either mark at least one vCPU as non-real-time to be account for emulator - overhead (housekeeping) or explicitly configure an :ref:`emulator thread - policy `. - - .. important:: - - To use this extra spec, you must enable pinned CPUs. Refer to - :ref:`CPU policy ` for more information. - - .. code:: console - - $ openstack flavor set FLAVOR-NAME \ - --property hw:cpu_realtime=CPU-REALTIME-POLICY \ - --property hw:cpu_realtime_mask=CPU-REALTIME-MASK - - Where: - - CPU-REALTIME-POLICY (enum): - One of: - - - ``no``: (default) The guest vCPUs will not have a real-time policy - - ``yes``: The guest vCPUs will have a real-time policy - - CPU-REALTIME-MASK (coremask): - A coremask indicating which vCPUs **will** or, if starting with a ``^``, - **will not** have a real-time policy. For example, a value of ``0-5`` - indicates that vCPUs ``0`` to ``5`` will have a real-time policy. - Conversely, a value of ``^0-1`` indicates that all vCPUs *except* vCPUs - ``0`` and ``1`` will have a real-time policy. - - .. note:: - - The ``hw:cpu_realtime_mask`` option is only valid if ``hw:cpu_realtime`` - is set to ``yes``. - - .. versionchanged:: 22.0.0 (Victoria) - - Previously, it was necessary to specify ``hw:cpu_realtime_mask`` when - ``hw:cpu_realtime`` was set to yes. Starting in Victoria, it is possible - to omit this when an emulator thread policy is configured using the - ``hw:emulator_threads_policy`` extra spec. - - .. versionchanged:: 22.0.0 (Victoria) - - Previously, the leading caret was necessary and omitting it would be - equivalent to not setting the mask, resulting in a failure to spawn - the instance. - -.. _extra-specs-emulator-threads-policy: - -Emulator threads policy - For the libvirt driver, you can assign a separate pCPU to an instance that - will be used for emulator threads, which are emulator processes not directly - related to the guest OS. This pCPU will used in addition to the pCPUs used - for the guest. This is generally required for use with a :ref:`real-time - workload `. - - .. important:: - - To use this extra spec, you must enable pinned CPUs. Refer to :ref:`CPU - policy ` for more information. - - .. code:: console - - $ openstack flavor set FLAVOR-NAME \ - --property hw:emulator_threads_policy=THREAD-POLICY - - The expected behavior of emulator threads depends on the value of the - ``hw:emulator_threads_policy`` flavor extra spec and the value of - :oslo.config:option:`compute.cpu_shared_set`. It is presented in the - following table: - - .. list-table:: - :header-rows: 1 - :stub-columns: 1 - - * - - - :oslo.config:option:`compute.cpu_shared_set` set - - :oslo.config:option:`compute.cpu_shared_set` unset - * - ``hw:emulator_treads_policy`` unset (default) - - Pinned to all of the instance's pCPUs - - Pinned to all of the instance's pCPUs - * - ``hw:emulator_threads_policy`` = ``share`` - - Pinned to :oslo.config:option:`compute.cpu_shared_set` - - Pinned to all of the instance's pCPUs - * - ``hw:emulator_threads_policy`` = ``isolate`` - - Pinned to a single pCPU distinct from the instance's pCPUs - - Pinned to a single pCPU distinct from the instance's pCPUs - -.. _extra-specs-large-pages-allocation: - -Large pages allocation - You can configure the size of large pages used to back the VMs. - - .. code:: console - - $ openstack flavor set FLAVOR-NAME \ - --property hw:mem_page_size=PAGE_SIZE - - Valid ``PAGE_SIZE`` values are: - - - ``small``: (default) The smallest page size is used. Example: 4 KB on x86. - - ``large``: Only use larger page sizes for guest RAM. Example: either 2 MB - or 1 GB on x86. - - ``any``: It is left up to the compute driver to decide. In this case, the - libvirt driver might try to find large pages, but fall back to small pages. - Other drivers may choose alternate policies for ``any``. - - pagesize: (string) An explicit page size can be set if the workload has - specific requirements. This value can be an integer value for the page size - in KB, or can use any standard suffix. Example: ``4KB``, ``2MB``, - ``2048``, ``1GB``. - - .. note:: - - Large pages can be enabled for guest RAM without any regard to whether the - guest OS will use them or not. If the guest OS chooses not to use huge - pages, it will merely see small pages as before. Conversely, if a guest OS - does intend to use huge pages, it is very important that the guest RAM be - backed by huge pages. Otherwise, the guest OS will not be getting the - performance benefit it is expecting. - .. _extra-spec-pci-passthrough: PCI passthrough