Browse Source

docs: Add a resource limits guide

This is mostly regurgitated information from the current flavors guide
but we take the opportunity to significantly expand upon what we've
already stated here.

Change-Id: I9ad798427bbc6451fd920d6c08357d6e1eaa5136
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
changes/33/781433/2
Stephen Finucane 5 months ago
parent
commit
8528eaa602
  1. 1
      doc/source/admin/index.rst
  2. 312
      doc/source/admin/resource-limits.rst
  3. 209
      doc/source/user/flavors.rst

1
doc/source/admin/index.rst

@ -116,6 +116,7 @@ instance for these kind of workloads.
uefi
secure-boot
managing-resource-providers
resource-limits
Additional guides

312
doc/source/admin/resource-limits.rst

@ -0,0 +1,312 @@
===============
Resource Limits
===============
Nova supports configuring limits on individual resources including CPU, memory,
disk and network. These limits can be used to enforce basic Quality-of-Service
(QoS) policies on such resources.
.. note::
Hypervisor-enforced resource limits are distinct from API-enforced user and
project quotas. For information on the latter, refer to :doc:`quotas`.
.. warning::
This feature is poorly tested and poorly maintained. It may no longer work
as expected. Where possible, consider using the QoS policies provided by
other services, such as
:cinder-doc:`Cinder </admin/blockstorage-basic-volume-qos.html>` and
:neutron-doc:`Neutron </admin/config-qos.html>`.
Configuring resource limits
---------------------------
Resource quota enforcement support is specific to the virt driver in use on
compute hosts.
libvirt
~~~~~~~
The libvirt driver supports CPU, disk and VIF limits. Unfortunately all of
these work quite differently, as discussed below.
CPU limits
^^^^^^^^^^
Libvirt enforces CPU limits in terms of *shares* and *quotas*, configured
via :nova:extra-spec:`quota:cpu_shares` and :nova:extra-spec:`quota:cpu_period`
/ :nova:extra-spec:`quota:cpu_quota`, respectively. Both are implemented using
the `cgroups v1 cpu controller`__.
CPU shares are a proportional weighted share of total CPU resources relative to
other instances. It does not limit CPU usage if CPUs are not busy. There is no
unit and the value is purely relative to other instances, so an instance
configured with value of 2048 will get twice as much CPU time as a VM
configured with the value 1024. For example, to configure a CPU share of 1024
for a flavor:
.. code-block:: console
$ openstack flavor set $FLAVOR --property quota:cpu_shares=1024
The CPU quotas require both a period and quota. The CPU period specifies the
enforcement interval in microseconds, while the CPU quota specifies the maximum
allowed bandwidth in microseconds that the each vCPU of the instance can
consume. The CPU period must be in the range 1000 (1mS) to 1,000,000 (1s) or 0
(disabled). The CPU quota must be in the range 1000 (1mS) to 2^64 or 0
(disabled). Where the CPU quota exceeds the CPU period, this means the guest
vCPU process is able to consume multiple pCPUs worth of bandwidth. For example,
to limit each guest vCPU to 1 pCPU worth of runtime per period:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--property quota:cpu_period=1000 \
--property quota:cpu_quota=1000
To limit each guest vCPU to 2 pCPUs worth of runtime per period:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--property quota:cpu_period=1000 \
--property quota:cpu_quota=2000
Finally, to limit each guest vCPU to 0.5 pCPUs worth of runtime per period:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--property quota:cpu_period=1000 \
--property quota:cpu_quota=500
.. note::
Smaller periods will ensure a consistent latency response at the expense of
burst capacity.
CPU shares and CPU quotas can work hand-in-hand. For example, if two instances
were configured with :nova:extra-spec:`quota:cpu_shares`\ =1024 and
:nova:extra-spec:`quota:cpu_period`\ =100000 (100mS) for both, then configuring
both with a :nova:extra-spec:`quota:cpu_quota`\ =75000 (75mS) will result in
them sharing a host CPU equally, with both getting exactly 50mS of CPU time.
If instead only one instance gets :nova:extra-spec:`quota:cpu_quota`\ =75000
(75mS) while the other gets :nova:extra-spec:`quota:cpu_quota`\ =25000 (25mS),
then the first will get 3/4 of the time per period.
.. __: https://man7.org/linux/man-pages/man7/cgroups.7.html
Memory Limits
^^^^^^^^^^^^^
The libvirt driver does not support memory limits.
Disk I/O Limits
^^^^^^^^^^^^^^^
Libvirt enforces disk limits through maximum disk read, write and total bytes
per second, using the :nova:extra-spec:`quota:disk_read_bytes_sec`,
:nova:extra-spec:`quota:disk_write_bytes_sec` and
:nova:extra-spec:`quota:disk_total_bytes_sec` extra specs, respectively. It can
also enforce disk limits through maximum disk read, write and total I/O
operations per second, using the :nova:extra-spec:`quota:disk_read_iops_sec`,
:nova:extra-spec:`quota:disk_write_iops_sec` and
:nova:extra-spec:`quota:disk_total_iops_sec` extra specs, respectively. For
example, to set a maximum disk write of 10 MB/sec for a flavor:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--property quota:disk_write_bytes_sec=10485760
Network bandwidth limits
^^^^^^^^^^^^^^^^^^^^^^^^
.. warning::
These limits are enforced via libvirt and will only work where the network
is connect to the instance using a tap interface. It will not work for
things like :doc:`SR-IOV VFs <pci-passthrough>`.
:neutron-doc:`Neutron's QoS policies </admin/config-qos.html>` should be
preferred wherever possible.
Libvirt enforces network bandwidth limits through inbound and outbound average,
using the :nova:extra-spec:`quota:vif_inbound_average` and
:nova:extra-spec:`quota:vif_outbound_average` extra specs, respectively.
In addition, optional *peak* values, which specifies the maximum rate at which
a bridge can send data (kB/s), and *burst* values, which specifies the amount
of bytes that can be burst at peak speed (kilobytes), can be specified for both
inbound and outbound traffic, using the
:nova:extra-spec:`quota:vif_inbound_peak` /
:nova:extra-spec:`quota:vif_outbound_peak` and
:nova:extra-spec:`quota:vif_inbound_burst` /
:nova:extra-spec:`quota:vif_outbound_burst` extra specs, respectively.
For example, to configure **outbound** traffic to an average of 262 Mbit/s
(32768 kB/s), a peak of 524 Mbit/s, and burst of 65536 kilobytes:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--property quota:vif_outbound_average=32768 \
--property quota:vif_outbound_peak=65536 \
--property quota:vif_outbound_burst=65536
.. note::
The speed limit values in above example are specified in kilobytes/second,
whle the burst value is in kilobytes.
VMWare
~~~~~~
In contrast to libvirt, the VMWare virt driver enforces resource limits using
consistent terminology, specifically through relative allocation levels, hard
upper limits and minimum reservations configured via, for example, the
:nova:extra-spec:`quota:cpu_shares_level` /
:nova:extra-spec:`quota:cpu_shares_share`, :nova:extra-spec:`quota:cpu_limit`,
and :nova:extra-spec:`quota:cpu_reservation` extra specs, respectively.
Allocation levels can be specified using one of ``high``, ``normal``, ``low``,
or ``custom``. When ``custom`` is specified, the number of shares must be
specified using e.g. :nova:extra-spec:`quota:cpu_shares_share`. There is no
unit and the values are relative to other instances on the host. The upper
limits and reservations, by comparison, are measure in resource-specific units,
such as MHz for CPUs and will ensure that the instance never used more than or
gets less than the specified amount of the resource.
CPU limits
^^^^^^^^^^
CPU limits are configured via the :nova:extra-spec:`quota:cpu_shares_level` /
:nova:extra-spec:`quota:cpu_shares_share`, :nova:extra-spec:`quota:cpu_limit`,
and :nova:extra-spec:`quota:cpu_reservation` extra specs.
For example, to configure a CPU allocation level of ``custom`` with 1024
shares:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:cpu_shares_level=custom \
--quota:cpu_shares_share=1024
To configure a minimum CPU allocation of 1024 MHz and a maximum of 2048 MHz:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:cpu_reservation=1024 \
--quota:cpu_limit=2048
Memory limits
^^^^^^^^^^^^^
Memory limits are configured via the
:nova:extra-spec:`quota:memory_shares_level` /
:nova:extra-spec:`quota:memory_shares_share`,
:nova:extra-spec:`quota:memory_limit`, and
:nova:extra-spec:`quota:memory_reservation` extra specs.
For example, to configure a memory allocation level of ``custom`` with 1024
shares:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:memory_shares_level=custom \
--quota:memory_shares_share=1024
To configure a minimum memory allocation of 1024 MB and a maximum of 2048 MB:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:memory_reservation=1024 \
--quota:memory_limit=2048
Disk I/O limits
^^^^^^^^^^^^^^^
Disk I/O limits are configured via the
:nova:extra-spec:`quota:disk_io_shares_level` /
:nova:extra-spec:`quota:disk_io_shares_share`,
:nova:extra-spec:`quota:disk_io_limit`, and
:nova:extra-spec:`quota:disk_io_reservation` extra specs.
For example, to configure a disk I/O allocation level of ``custom`` with 1024
shares:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:disk_io_shares_level=custom \
--quota:disk_io_shares_share=1024
To configure a minimum disk I/O allocation of 1024 MB and a maximum of 2048 MB:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:disk_io_reservation=1024 \
--quota:disk_io_limit=2048
Network bandwidth limits
^^^^^^^^^^^^^^^^^^^^^^^^
Network bandwidth limits are configured via the
:nova:extra-spec:`quota:vif_shares_level` /
:nova:extra-spec:`quota:vif_shares_share`,
:nova:extra-spec:`quota:vif_limit`, and
:nova:extra-spec:`quota:vif_reservation` extra specs.
For example, to configure a network bandwidth allocation level of ``custom``
with 1024 shares:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:vif_shares_level=custom \
--quota:vif_shares_share=1024
To configure a minimum bandwidth allocation of 1024 Mbits/sec and a maximum of
2048 Mbits/sec:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--quota:vif_reservation=1024 \
--quota:vif_limit=2048
Hyper-V
~~~~~~~
CPU limits
^^^^^^^^^^
The Hyper-V driver does not support CPU limits.
Memory limits
^^^^^^^^^^^^^
The Hyper-V driver does not support memory limits.
Disk I/O limits
^^^^^^^^^^^^^^^
Hyper-V enforces disk limits through maximum total bytes and total I/O
operations per second, using the :nova:extra-spec:`quota:disk_total_bytes_sec`
and :nova:extra-spec:`quota:disk_total_iops_sec` extra specs, respectively. For
example, to set a maximum disk read/write of 10 MB/sec for a flavor:
.. code-block:: console
$ openstack flavor set $FLAVOR \
--property quota:disk_total_bytes_sec=10485760
Network bandwidth limits
^^^^^^^^^^^^^^^^^^^^^^^^
The Hyper-V driver does not support network bandwidth limits.

209
doc/source/user/flavors.rst

@ -94,220 +94,11 @@ Description
Extra Specs
~~~~~~~~~~~
.. todo::
A lot of these need investigation - for example, I can find no reference to
the ``cpu_shares_level`` option outside of documentation and (possibly)
useless tests. We should assess which drivers each option actually apply to.
.. todo::
This is now documented in :doc:`/configuration/extra-specs`, so this should
be removed and the documentation moved to those specs.
.. _extra-specs-CPU-limits:
CPU limits
You can configure the CPU limits with control parameters. For example, to
configure the I/O limit, use:
.. code-block:: console
$ openstack flavor set FLAVOR-NAME \
--property quota:read_bytes_sec=10240000 \
--property quota:write_bytes_sec=10240000
Use these optional parameters to control weight shares, enforcement intervals
for runtime quotas, and a quota for maximum allowed bandwidth:
- ``cpu_shares``: Specifies the proportional weighted share for the domain.
If this element is omitted, the service defaults to the OS provided
defaults. There is no unit for the value; it is a relative measure based on
the setting of other VMs. For example, a VM configured with value 2048 gets
twice as much CPU time as a VM configured with value 1024.
- ``cpu_shares_level``: On VMware, specifies the allocation level. Can be
``custom``, ``high``, ``normal``, or ``low``. If you choose ``custom``, set
the number of shares using ``cpu_shares_share``.
- ``cpu_period``: Specifies the enforcement interval (unit: microseconds)
for QEMU and LXC hypervisors. Within a period, each VCPU of the domain is
not allowed to consume more than the quota worth of runtime. The value
should be in range ``[1000, 1000000]``. A period with value 0 means no
value.
- ``cpu_limit``: Specifies the upper limit for VMware machine CPU allocation
in MHz. This parameter ensures that a machine never uses more than the
defined amount of CPU time. It can be used to enforce a limit on the
machine's CPU performance.
- ``cpu_reservation``: Specifies the guaranteed minimum CPU reservation in
MHz for VMware. This means that if needed, the machine will definitely get
allocated the reserved amount of CPU cycles.
- ``cpu_quota``: Specifies the maximum allowed bandwidth (unit:
microseconds). A domain with a negative-value quota indicates that the
domain has infinite bandwidth, which means that it is not bandwidth
controlled. The value should be in range ``[1000, 18446744073709551]`` or
less than 0. A quota with value 0 means no value. You can use this feature
to ensure that all vCPUs run at the same speed. For example:
.. code-block:: console
$ openstack flavor set FLAVOR-NAME \
--property quota:cpu_quota=10000 \
--property quota:cpu_period=20000
In this example, an instance of ``FLAVOR-NAME`` can only consume a maximum
of 50% CPU of a physical CPU computing capability.
.. _extra-specs-memory-limits:
Memory limits
For VMware, you can configure the memory limits with control parameters.
Use these optional parameters to limit the memory allocation, guarantee
minimum memory reservation, and to specify shares used in case of resource
contention:
- ``memory_limit``: Specifies the upper limit for VMware machine memory
allocation in MB. The utilization of a virtual machine will not exceed this
limit, even if there are available resources. This is typically used to
ensure a consistent performance of virtual machines independent of
available resources.
- ``memory_reservation``: Specifies the guaranteed minimum memory reservation
in MB for VMware. This means the specified amount of memory will definitely
be allocated to the machine.
- ``memory_shares_level``: On VMware, specifies the allocation level. This
can be ``custom``, ``high``, ``normal`` or ``low``. If you choose
``custom``, set the number of shares using ``memory_shares_share``.
- ``memory_shares_share``: Specifies the number of shares allocated in the
event that ``custom`` is used. There is no unit for this value. It is a
relative measure based on the settings for other VMs. For example:
.. code-block:: console
$ openstack flavor set FLAVOR-NAME \
--property quota:memory_shares_level=custom \
--property quota:memory_shares_share=15
.. _extra-specs-disk-io-limits:
Disk I/O limits
For VMware, you can configure the resource limits for disk with control
parameters.
Use these optional parameters to limit the disk utilization, guarantee disk
allocation, and to specify shares used in case of resource contention. This
allows the VMware driver to enable disk allocations for the running instance.
- ``disk_io_limit``: Specifies the upper limit for disk utilization in I/O
per second. The utilization of a virtual machine will not exceed this
limit, even if there are available resources. The default value is -1 which
indicates unlimited usage.
- ``disk_io_reservation``: Specifies the guaranteed minimum disk allocation
in terms of Input/output Operations Per Second (IOPS).
- ``disk_io_shares_level``: Specifies the allocation level. This can be
``custom``, ``high``, ``normal`` or ``low``. If you choose custom, set the
number of shares using ``disk_io_shares_share``.
- ``disk_io_shares_share``: Specifies the number of shares allocated in the
event that ``custom`` is used. When there is resource contention, this
value is used to determine the resource allocation.
The example below sets the ``disk_io_reservation`` to 2000 IOPS.
.. code-block:: console
$ openstack flavor set FLAVOR-NAME \
--property quota:disk_io_reservation=2000
.. _extra-specs-disk-tuning:
Disk tuning
Using disk I/O quotas, you can set maximum disk write to 10 MB per second for
a VM user. For example:
.. code-block:: console
$ openstack flavor set FLAVOR-NAME \
--property quota:disk_write_bytes_sec=10485760
The disk I/O options are:
- ``disk_read_bytes_sec``
- ``disk_read_iops_sec``
- ``disk_write_bytes_sec``
- ``disk_write_iops_sec``
- ``disk_total_bytes_sec``
- ``disk_total_iops_sec``
.. _extra-specs-bandwidth-io:
Bandwidth I/O
The vif I/O options are:
- ``vif_inbound_average``
- ``vif_inbound_burst``
- ``vif_inbound_peak``
- ``vif_outbound_average``
- ``vif_outbound_burst``
- ``vif_outbound_peak``
Incoming and outgoing traffic can be shaped independently. The bandwidth
element can have at most, one inbound and at most, one outbound child
element. If you leave any of these child elements out, no quality of service
(QoS) is applied on that traffic direction. So, if you want to shape only the
network's incoming traffic, use inbound only (and vice versa). Each element
has one mandatory attribute average, which specifies the average bit rate on
the interface being shaped.
There are also two optional attributes (integer): ``peak``, which specifies
the maximum rate at which a bridge can send data (kilobytes/second), and
``burst``, the amount of bytes that can be burst at peak speed (kilobytes).
The rate is shared equally within domains connected to the network.
The example below sets network traffic bandwidth limits for existing flavor
as follows:
- Outbound traffic:
- average: 262 Mbps (32768 kilobytes/second)
- peak: 524 Mbps (65536 kilobytes/second)
- burst: 65536 kilobytes
- Inbound traffic:
- average: 262 Mbps (32768 kilobytes/second)
- peak: 524 Mbps (65536 kilobytes/second)
- burst: 65536 kilobytes
.. code-block:: console
$ openstack flavor set FLAVOR-NAME \
--property quota:vif_outbound_average=32768 \
--property quota:vif_outbound_peak=65536 \
--property quota:vif_outbound_burst=65536 \
--property quota:vif_inbound_average=32768 \
--property quota:vif_inbound_peak=65536 \
--property quota:vif_inbound_burst=65536
.. note::
All the speed limit values in above example are specified in
kilobytes/second. And burst values are in kilobytes. Values were converted
using `Data rate units on Wikipedia
<https://en.wikipedia.org/wiki/Data_rate_units>`_.
.. _extra-specs-hardware-video-ram:
Hardware video RAM

Loading…
Cancel
Save