diff --git a/doc/source/admin/index.rst b/doc/source/admin/index.rst index 861f9971cee3..3509233761be 100644 --- a/doc/source/admin/index.rst +++ b/doc/source/admin/index.rst @@ -116,6 +116,7 @@ instance for these kind of workloads. uefi secure-boot managing-resource-providers + resource-limits Additional guides diff --git a/doc/source/admin/resource-limits.rst b/doc/source/admin/resource-limits.rst new file mode 100644 index 000000000000..c74ad31c17bc --- /dev/null +++ b/doc/source/admin/resource-limits.rst @@ -0,0 +1,312 @@ +=============== +Resource Limits +=============== + +Nova supports configuring limits on individual resources including CPU, memory, +disk and network. These limits can be used to enforce basic Quality-of-Service +(QoS) policies on such resources. + +.. note:: + + Hypervisor-enforced resource limits are distinct from API-enforced user and + project quotas. For information on the latter, refer to :doc:`quotas`. + +.. warning:: + + This feature is poorly tested and poorly maintained. It may no longer work + as expected. Where possible, consider using the QoS policies provided by + other services, such as + :cinder-doc:`Cinder ` and + :neutron-doc:`Neutron `. + + +Configuring resource limits +--------------------------- + +Resource quota enforcement support is specific to the virt driver in use on +compute hosts. + +libvirt +~~~~~~~ + +The libvirt driver supports CPU, disk and VIF limits. Unfortunately all of +these work quite differently, as discussed below. + +CPU limits +^^^^^^^^^^ + +Libvirt enforces CPU limits in terms of *shares* and *quotas*, configured +via :nova:extra-spec:`quota:cpu_shares` and :nova:extra-spec:`quota:cpu_period` +/ :nova:extra-spec:`quota:cpu_quota`, respectively. Both are implemented using +the `cgroups v1 cpu controller`__. + +CPU shares are a proportional weighted share of total CPU resources relative to +other instances. It does not limit CPU usage if CPUs are not busy. There is no +unit and the value is purely relative to other instances, so an instance +configured with value of 2048 will get twice as much CPU time as a VM +configured with the value 1024. For example, to configure a CPU share of 1024 +for a flavor: + +.. code-block:: console + + $ openstack flavor set $FLAVOR --property quota:cpu_shares=1024 + +The CPU quotas require both a period and quota. The CPU period specifies the +enforcement interval in microseconds, while the CPU quota specifies the maximum +allowed bandwidth in microseconds that the each vCPU of the instance can +consume. The CPU period must be in the range 1000 (1mS) to 1,000,000 (1s) or 0 +(disabled). The CPU quota must be in the range 1000 (1mS) to 2^64 or 0 +(disabled). Where the CPU quota exceeds the CPU period, this means the guest +vCPU process is able to consume multiple pCPUs worth of bandwidth. For example, +to limit each guest vCPU to 1 pCPU worth of runtime per period: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --property quota:cpu_period=1000 \ + --property quota:cpu_quota=1000 + +To limit each guest vCPU to 2 pCPUs worth of runtime per period: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --property quota:cpu_period=1000 \ + --property quota:cpu_quota=2000 + +Finally, to limit each guest vCPU to 0.5 pCPUs worth of runtime per period: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --property quota:cpu_period=1000 \ + --property quota:cpu_quota=500 + +.. note:: + + Smaller periods will ensure a consistent latency response at the expense of + burst capacity. + +CPU shares and CPU quotas can work hand-in-hand. For example, if two instances +were configured with :nova:extra-spec:`quota:cpu_shares`\ =1024 and +:nova:extra-spec:`quota:cpu_period`\ =100000 (100mS) for both, then configuring +both with a :nova:extra-spec:`quota:cpu_quota`\ =75000 (75mS) will result in +them sharing a host CPU equally, with both getting exactly 50mS of CPU time. +If instead only one instance gets :nova:extra-spec:`quota:cpu_quota`\ =75000 +(75mS) while the other gets :nova:extra-spec:`quota:cpu_quota`\ =25000 (25mS), +then the first will get 3/4 of the time per period. + +.. __: https://man7.org/linux/man-pages/man7/cgroups.7.html + +Memory Limits +^^^^^^^^^^^^^ + +The libvirt driver does not support memory limits. + +Disk I/O Limits +^^^^^^^^^^^^^^^ + +Libvirt enforces disk limits through maximum disk read, write and total bytes +per second, using the :nova:extra-spec:`quota:disk_read_bytes_sec`, +:nova:extra-spec:`quota:disk_write_bytes_sec` and +:nova:extra-spec:`quota:disk_total_bytes_sec` extra specs, respectively. It can +also enforce disk limits through maximum disk read, write and total I/O +operations per second, using the :nova:extra-spec:`quota:disk_read_iops_sec`, +:nova:extra-spec:`quota:disk_write_iops_sec` and +:nova:extra-spec:`quota:disk_total_iops_sec` extra specs, respectively. For +example, to set a maximum disk write of 10 MB/sec for a flavor: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --property quota:disk_write_bytes_sec=10485760 + +Network bandwidth limits +^^^^^^^^^^^^^^^^^^^^^^^^ + +.. warning:: + + These limits are enforced via libvirt and will only work where the network + is connect to the instance using a tap interface. It will not work for + things like :doc:`SR-IOV VFs `. + :neutron-doc:`Neutron's QoS policies ` should be + preferred wherever possible. + +Libvirt enforces network bandwidth limits through inbound and outbound average, +using the :nova:extra-spec:`quota:vif_inbound_average` and +:nova:extra-spec:`quota:vif_outbound_average` extra specs, respectively. +In addition, optional *peak* values, which specifies the maximum rate at which +a bridge can send data (kB/s), and *burst* values, which specifies the amount +of bytes that can be burst at peak speed (kilobytes), can be specified for both +inbound and outbound traffic, using the +:nova:extra-spec:`quota:vif_inbound_peak` / +:nova:extra-spec:`quota:vif_outbound_peak` and +:nova:extra-spec:`quota:vif_inbound_burst` / +:nova:extra-spec:`quota:vif_outbound_burst` extra specs, respectively. + +For example, to configure **outbound** traffic to an average of 262 Mbit/s +(32768 kB/s), a peak of 524 Mbit/s, and burst of 65536 kilobytes: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --property quota:vif_outbound_average=32768 \ + --property quota:vif_outbound_peak=65536 \ + --property quota:vif_outbound_burst=65536 + +.. note:: + + The speed limit values in above example are specified in kilobytes/second, + whle the burst value is in kilobytes. + +VMWare +~~~~~~ + +In contrast to libvirt, the VMWare virt driver enforces resource limits using +consistent terminology, specifically through relative allocation levels, hard +upper limits and minimum reservations configured via, for example, the +:nova:extra-spec:`quota:cpu_shares_level` / +:nova:extra-spec:`quota:cpu_shares_share`, :nova:extra-spec:`quota:cpu_limit`, +and :nova:extra-spec:`quota:cpu_reservation` extra specs, respectively. + +Allocation levels can be specified using one of ``high``, ``normal``, ``low``, +or ``custom``. When ``custom`` is specified, the number of shares must be +specified using e.g. :nova:extra-spec:`quota:cpu_shares_share`. There is no +unit and the values are relative to other instances on the host. The upper +limits and reservations, by comparison, are measure in resource-specific units, +such as MHz for CPUs and will ensure that the instance never used more than or +gets less than the specified amount of the resource. + +CPU limits +^^^^^^^^^^ + +CPU limits are configured via the :nova:extra-spec:`quota:cpu_shares_level` / +:nova:extra-spec:`quota:cpu_shares_share`, :nova:extra-spec:`quota:cpu_limit`, +and :nova:extra-spec:`quota:cpu_reservation` extra specs. + +For example, to configure a CPU allocation level of ``custom`` with 1024 +shares: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --quota:cpu_shares_level=custom \ + --quota:cpu_shares_share=1024 + +To configure a minimum CPU allocation of 1024 MHz and a maximum of 2048 MHz: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --quota:cpu_reservation=1024 \ + --quota:cpu_limit=2048 + +Memory limits +^^^^^^^^^^^^^ + +Memory limits are configured via the +:nova:extra-spec:`quota:memory_shares_level` / +:nova:extra-spec:`quota:memory_shares_share`, +:nova:extra-spec:`quota:memory_limit`, and +:nova:extra-spec:`quota:memory_reservation` extra specs. + +For example, to configure a memory allocation level of ``custom`` with 1024 +shares: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --quota:memory_shares_level=custom \ + --quota:memory_shares_share=1024 + +To configure a minimum memory allocation of 1024 MB and a maximum of 2048 MB: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --quota:memory_reservation=1024 \ + --quota:memory_limit=2048 + +Disk I/O limits +^^^^^^^^^^^^^^^ + +Disk I/O limits are configured via the +:nova:extra-spec:`quota:disk_io_shares_level` / +:nova:extra-spec:`quota:disk_io_shares_share`, +:nova:extra-spec:`quota:disk_io_limit`, and +:nova:extra-spec:`quota:disk_io_reservation` extra specs. + +For example, to configure a disk I/O allocation level of ``custom`` with 1024 +shares: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --quota:disk_io_shares_level=custom \ + --quota:disk_io_shares_share=1024 + +To configure a minimum disk I/O allocation of 1024 MB and a maximum of 2048 MB: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --quota:disk_io_reservation=1024 \ + --quota:disk_io_limit=2048 + +Network bandwidth limits +^^^^^^^^^^^^^^^^^^^^^^^^ + +Network bandwidth limits are configured via the +:nova:extra-spec:`quota:vif_shares_level` / +:nova:extra-spec:`quota:vif_shares_share`, +:nova:extra-spec:`quota:vif_limit`, and +:nova:extra-spec:`quota:vif_reservation` extra specs. + +For example, to configure a network bandwidth allocation level of ``custom`` +with 1024 shares: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --quota:vif_shares_level=custom \ + --quota:vif_shares_share=1024 + +To configure a minimum bandwidth allocation of 1024 Mbits/sec and a maximum of +2048 Mbits/sec: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --quota:vif_reservation=1024 \ + --quota:vif_limit=2048 + +Hyper-V +~~~~~~~ + +CPU limits +^^^^^^^^^^ + +The Hyper-V driver does not support CPU limits. + +Memory limits +^^^^^^^^^^^^^ + +The Hyper-V driver does not support memory limits. + +Disk I/O limits +^^^^^^^^^^^^^^^ + +Hyper-V enforces disk limits through maximum total bytes and total I/O +operations per second, using the :nova:extra-spec:`quota:disk_total_bytes_sec` +and :nova:extra-spec:`quota:disk_total_iops_sec` extra specs, respectively. For +example, to set a maximum disk read/write of 10 MB/sec for a flavor: + +.. code-block:: console + + $ openstack flavor set $FLAVOR \ + --property quota:disk_total_bytes_sec=10485760 + +Network bandwidth limits +^^^^^^^^^^^^^^^^^^^^^^^^ + +The Hyper-V driver does not support network bandwidth limits. diff --git a/doc/source/user/flavors.rst b/doc/source/user/flavors.rst index cafdc6430a0e..84624f8ea2b9 100644 --- a/doc/source/user/flavors.rst +++ b/doc/source/user/flavors.rst @@ -94,220 +94,11 @@ Description Extra Specs ~~~~~~~~~~~ -.. todo:: - - A lot of these need investigation - for example, I can find no reference to - the ``cpu_shares_level`` option outside of documentation and (possibly) - useless tests. We should assess which drivers each option actually apply to. - .. todo:: This is now documented in :doc:`/configuration/extra-specs`, so this should be removed and the documentation moved to those specs. -.. _extra-specs-CPU-limits: - -CPU limits - You can configure the CPU limits with control parameters. For example, to - configure the I/O limit, use: - - .. code-block:: console - - $ openstack flavor set FLAVOR-NAME \ - --property quota:read_bytes_sec=10240000 \ - --property quota:write_bytes_sec=10240000 - - Use these optional parameters to control weight shares, enforcement intervals - for runtime quotas, and a quota for maximum allowed bandwidth: - - - ``cpu_shares``: Specifies the proportional weighted share for the domain. - If this element is omitted, the service defaults to the OS provided - defaults. There is no unit for the value; it is a relative measure based on - the setting of other VMs. For example, a VM configured with value 2048 gets - twice as much CPU time as a VM configured with value 1024. - - - ``cpu_shares_level``: On VMware, specifies the allocation level. Can be - ``custom``, ``high``, ``normal``, or ``low``. If you choose ``custom``, set - the number of shares using ``cpu_shares_share``. - - - ``cpu_period``: Specifies the enforcement interval (unit: microseconds) - for QEMU and LXC hypervisors. Within a period, each VCPU of the domain is - not allowed to consume more than the quota worth of runtime. The value - should be in range ``[1000, 1000000]``. A period with value 0 means no - value. - - - ``cpu_limit``: Specifies the upper limit for VMware machine CPU allocation - in MHz. This parameter ensures that a machine never uses more than the - defined amount of CPU time. It can be used to enforce a limit on the - machine's CPU performance. - - - ``cpu_reservation``: Specifies the guaranteed minimum CPU reservation in - MHz for VMware. This means that if needed, the machine will definitely get - allocated the reserved amount of CPU cycles. - - - ``cpu_quota``: Specifies the maximum allowed bandwidth (unit: - microseconds). A domain with a negative-value quota indicates that the - domain has infinite bandwidth, which means that it is not bandwidth - controlled. The value should be in range ``[1000, 18446744073709551]`` or - less than 0. A quota with value 0 means no value. You can use this feature - to ensure that all vCPUs run at the same speed. For example: - - .. code-block:: console - - $ openstack flavor set FLAVOR-NAME \ - --property quota:cpu_quota=10000 \ - --property quota:cpu_period=20000 - - In this example, an instance of ``FLAVOR-NAME`` can only consume a maximum - of 50% CPU of a physical CPU computing capability. - -.. _extra-specs-memory-limits: - -Memory limits - For VMware, you can configure the memory limits with control parameters. - - Use these optional parameters to limit the memory allocation, guarantee - minimum memory reservation, and to specify shares used in case of resource - contention: - - - ``memory_limit``: Specifies the upper limit for VMware machine memory - allocation in MB. The utilization of a virtual machine will not exceed this - limit, even if there are available resources. This is typically used to - ensure a consistent performance of virtual machines independent of - available resources. - - - ``memory_reservation``: Specifies the guaranteed minimum memory reservation - in MB for VMware. This means the specified amount of memory will definitely - be allocated to the machine. - - - ``memory_shares_level``: On VMware, specifies the allocation level. This - can be ``custom``, ``high``, ``normal`` or ``low``. If you choose - ``custom``, set the number of shares using ``memory_shares_share``. - - - ``memory_shares_share``: Specifies the number of shares allocated in the - event that ``custom`` is used. There is no unit for this value. It is a - relative measure based on the settings for other VMs. For example: - - .. code-block:: console - - $ openstack flavor set FLAVOR-NAME \ - --property quota:memory_shares_level=custom \ - --property quota:memory_shares_share=15 - -.. _extra-specs-disk-io-limits: - -Disk I/O limits - For VMware, you can configure the resource limits for disk with control - parameters. - - Use these optional parameters to limit the disk utilization, guarantee disk - allocation, and to specify shares used in case of resource contention. This - allows the VMware driver to enable disk allocations for the running instance. - - - ``disk_io_limit``: Specifies the upper limit for disk utilization in I/O - per second. The utilization of a virtual machine will not exceed this - limit, even if there are available resources. The default value is -1 which - indicates unlimited usage. - - - ``disk_io_reservation``: Specifies the guaranteed minimum disk allocation - in terms of Input/output Operations Per Second (IOPS). - - - ``disk_io_shares_level``: Specifies the allocation level. This can be - ``custom``, ``high``, ``normal`` or ``low``. If you choose custom, set the - number of shares using ``disk_io_shares_share``. - - - ``disk_io_shares_share``: Specifies the number of shares allocated in the - event that ``custom`` is used. When there is resource contention, this - value is used to determine the resource allocation. - - The example below sets the ``disk_io_reservation`` to 2000 IOPS. - - .. code-block:: console - - $ openstack flavor set FLAVOR-NAME \ - --property quota:disk_io_reservation=2000 - -.. _extra-specs-disk-tuning: - -Disk tuning - Using disk I/O quotas, you can set maximum disk write to 10 MB per second for - a VM user. For example: - - .. code-block:: console - - $ openstack flavor set FLAVOR-NAME \ - --property quota:disk_write_bytes_sec=10485760 - - The disk I/O options are: - - - ``disk_read_bytes_sec`` - - ``disk_read_iops_sec`` - - ``disk_write_bytes_sec`` - - ``disk_write_iops_sec`` - - ``disk_total_bytes_sec`` - - ``disk_total_iops_sec`` - -.. _extra-specs-bandwidth-io: - -Bandwidth I/O - The vif I/O options are: - - - ``vif_inbound_average`` - - ``vif_inbound_burst`` - - ``vif_inbound_peak`` - - ``vif_outbound_average`` - - ``vif_outbound_burst`` - - ``vif_outbound_peak`` - - Incoming and outgoing traffic can be shaped independently. The bandwidth - element can have at most, one inbound and at most, one outbound child - element. If you leave any of these child elements out, no quality of service - (QoS) is applied on that traffic direction. So, if you want to shape only the - network's incoming traffic, use inbound only (and vice versa). Each element - has one mandatory attribute average, which specifies the average bit rate on - the interface being shaped. - - There are also two optional attributes (integer): ``peak``, which specifies - the maximum rate at which a bridge can send data (kilobytes/second), and - ``burst``, the amount of bytes that can be burst at peak speed (kilobytes). - The rate is shared equally within domains connected to the network. - - The example below sets network traffic bandwidth limits for existing flavor - as follows: - - - Outbound traffic: - - - average: 262 Mbps (32768 kilobytes/second) - - - peak: 524 Mbps (65536 kilobytes/second) - - - burst: 65536 kilobytes - - - Inbound traffic: - - - average: 262 Mbps (32768 kilobytes/second) - - - peak: 524 Mbps (65536 kilobytes/second) - - - burst: 65536 kilobytes - - .. code-block:: console - - $ openstack flavor set FLAVOR-NAME \ - --property quota:vif_outbound_average=32768 \ - --property quota:vif_outbound_peak=65536 \ - --property quota:vif_outbound_burst=65536 \ - --property quota:vif_inbound_average=32768 \ - --property quota:vif_inbound_peak=65536 \ - --property quota:vif_inbound_burst=65536 - - .. note:: - - All the speed limit values in above example are specified in - kilobytes/second. And burst values are in kilobytes. Values were converted - using `Data rate units on Wikipedia - `_. - .. _extra-specs-hardware-video-ram: Hardware video RAM