openstack/nova
Code Issues Proposed changes

Merge "docs: Add more information about unified limits"

Browse Source
This commit is contained in:
Zuul 2024-05-23 19:48:39 +00:00 committed by Gerrit Code Review
commit bded279a00
2 changed files with 370 additions and 241 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

490
doc/source/admin/unified-limits.rst
View File

@ -5,221 +5,393 @@ Manage Unified Limits Quotas
.. note::
This section provides deployment information about the quota feature. For
end-user information about quotas, including information about the type of
quotas available, refer to the :doc:`user guide </user/unified-limits>`.
end-user information about quotas, including information about the type
of quotas available, refer to the :doc:`user guide
</user/unified-limits>`.
Since the Nova 28.0.0 (2023.2 Bobcat) release, it is recommended to use
Since the **Nova 28.0.0 (2023.2 Bobcat)** release, it is recommended to use
`Keystone unified limits`_ for Nova quota limits.
For information about legacy quota limits, see the :doc:`legacy quota
documentation </admin/quotas>`.
To enable unified limits quotas, set the quota driver in the Nova
configuration:
Quotas
------
To prevent system capacities from being exhausted without notification, you
can set up quotas. Quotas are operational limits. The number of servers
allowed for each project can be controlled so that cloud resources are
optimized, for example. Quotas can be enforced at both the global
(default) level and at the project level.
Unified limits
--------------
Unified limits is a modern quota system in which quota limits are centralized
in the Keystone identity service. There are three steps for quota enforcement
in this model:
#. Quota limits are retrieved by calling the `Keystone unified limits API`_
#. Quota usage is counted from the `Placement API service`_
#. Quota is enforced locally using the `oslo.limit`_ limit enforcement
library
In unified limits, the terminology is a bit different from legacy quotas:
* A **registered limit** is a global or default limit that applies to all
projects
* A **limit** is a project-scoped limit that applies to a particular project
Cloud operators will need to manage their quota limits in the Keystone service
by calling the API directly or by using the OpenStackClient (OSC) `registered
limit`_ and `limit`_ commands.
.. _Keystone unified limits:
https://docs.openstack.org/keystone/latest/admin/unified-limits.html
.. _Keystone unified limits API:
https://docs.openstack.org/api-ref/identity/v3/index.html#unified-limits
.. _Placement API service: https://docs.openstack.org/placement
.. _oslo.limit: https://docs.openstack.org/oslo.limit
.. _registered limit:
https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/registered-limit.html
.. _limit:
https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/limit.html
Roles
~~~~~
By default Keystone API policy, a user must have the following roles and
scopes in order to perform actions with unified limits.
.. list-table::
:header-rows: 1
:widths: 10 20 20
* - Action
- Role
- Scope
* - List registered limits
- ``*``
- ``*``
* - Get registered limit
- ``*``
- ``*``
* - Create registered limit
- ``admin``
- ``system=all``
* - Update registered limit
- ``admin``
- ``system=all``
* - Delete registered limit
- ``admin``
- ``system=all``
* - List limits
- ``*``
- ``*``
* - Get limit
- ``*``
- ``*``
* - Create limit
- ``admin``
- ``system=all``
* - Update limit
- ``admin``
- ``system=all``
* - Delete limit
- ``admin``
- ``system=all``
Configuration
-------------
To enable unified limits quotas, some Nova configuration of
the :program:`nova-api` and :program:`nova-conductor` services is necessary.
Set the quota driver to the ``nova.quota.UnifiedLimitsDriver``:
.. code-block:: ini
[quota]
driver = nova.quota.UnifiedLimitsDriver
Quota limits are set and retrieved for enforcement using the `Keystone API`_.
Add a configuration section for oslo.limit:
.. _Keystone unified limits: https://docs.openstack.org/keystone/latest/admin/unified-limits.html
.. _Keystone API: https://docs.openstack.org/api-ref/identity/v3/index.html#unified-limits
.. code-block:: ini
To prevent system capacities from being exhausted without notification, you can
set up quotas. Quotas are operational limits. For example, the number of
servers allowed for each project can be controlled so that cloud resources
are optimized. Quotas can be enforced at both the global (default) and the
project level.
[oslo_limit]
username = nova
user_domain_name = $SERVICE_DOMAIN_NAME
auth_url = $KEYSTONE_SERVICE_URI
auth_type = password
password = $SERVICE_PASSWORD
system_scope = all
endpoint_id = $SERVICE_ENDPOINT_ID
Resource usage is counted from the placement service with unified limits.
.. note::
The Nova service endpoint ID can be obtained by ``openstack endpoint
list --service nova -f value -c ID``
Ensure that the ``nova`` service user has the ``reader`` role with ``system``
scope:
.. code-block:: console
openstack role add --user nova --user-domain $SERVICE_DOMAIN_NAME \
--system all reader
Checking quota
--------------
Setting quota limits on resources
---------------------------------
When calculating limits for a given resource and project, the following checks
Any resource that can be requested in the cloud must have a registered limit
set. Quota checks on cloud resources that do not have registered limits will
continue to fail until registered limits are set because oslo.limit considers
an unregistered resource to have a limit of 0.
Types of quota
~~~~~~~~~~~~~~
Unified limit resource names for resources that are tracked as `resource
classes`_ in the Placement API service follow the naming pattern of the
``class:`` prefix followed by the name of the resource class. For example:
class:VCPU, class:PCPU, class:MEMORY_MB, class:DISK_GB, class:VGPU.
.. list-table::
:header-rows: 1
:widths: 10 40
* - Quota Name
- Description
* - class:VCPU
- Number of shared CPU cores (VCPUs) allowed per project
* - class:PCPU
- Number of dedicated CPU cores (PCPUs) allowed per project
* - servers
- Number of instances allowed per project
* - server_key_pairs
- Number of key pairs allowed per user
* - server_metadata_items
- Number of metadata items allowed per instance
* - class:MEMORY_MB
- Megabytes of instance ram allowed per project
* - server_groups
- Number of server groups per project
* - server_group_members
- Number of servers per server group
* - class:DISK_GB
- Gigabytes of instance disk allowed per project
* - class:$RESOURCE_CLASS
- Any resource class in the Placement API service can have a quota limit
specified for it (example: class:VGPU)
.. _resource classes: https://docs.openstack.org/os-resource-classes/latest
OpenStack CLI commands
~~~~~~~~~~~~~~~~~~~~~~
For full OpenStackClient documentation, see
https://docs.openstack.org/python-openstackclient/latest/index.html.
Registered Limits
^^^^^^^^^^^^^^^^^
To list default limits for Nova:
.. code-block:: console
openstack registered limit list --service nova
To show details about a default limit:
.. code-block:: console
openstack registered limit show $REGISTERED_LIMIT_ID
To create a default limit:
.. code-block:: console
openstack registered limit create --service nova --default-limit $LIMIT \
$RESOURCE
To update a default limit:
.. code-block:: console
openstack registered limit set --resource-name $RESOURCE \
--default-limit $LIMIT $REGISTERED_LIMIT_ID
To delete a default limit:
.. code-block:: console
openstack registered limit delete $REGISTERED_LIMIT_ID
Limits
^^^^^^
To list project limits for Nova:
.. code-block:: console
openstack limit list --service nova
To list limits for a particular project:
.. code-block:: console
openstack limit list --service nova --project $PROJECT_ID
To show details about a project limit:
.. code-block:: console
openstack limit show $LIMIT_ID
To create a project limit:
.. code-block:: console
openstack limit create --service nova --project $PROJECT_ID \
--resource-limit $LIMIT $RESOURCE
To update a project limit:
.. code-block:: console
openstack limit set --resource-name $RESOURCE --resource-limit $LIMIT \
$LIMIT_ID
To delete a project limit:
.. code-block:: console
openstack limit delete $LIMIT_ID
Quota enforcement
-----------------
When enforcing limits for a given resource and project, the following checks
are made in order:
#. Project-specific limits
#. Limits (project-specific)
Depending on the resource, is there a project-specific limit on the resource
in Keystone limits? If so, use that as the limit. You can create these
resource limits using:
Depending on the resource, is there a project-specific limit on the
resource in Keystone limits? If so, use that as the limit. If not, proceed
to check the registered default limit.
.. code-block:: console
#. Registered limits (default)
$ openstack limit create --service nova --project <project> --resource-limit 5 servers
Depending on the resource, is there a default limit on the resource in
Keystone limits? If so, use that as the limit. If not, oslo.limit will
consider the limit as 0, the quota check will fail, and a quota limit
exceeded exception will be raised.
#. Default limits
.. warning::
Use the Keystone registered limit for the resource as the limit. You can
create these default limits using:
.. code-block:: console
$ openstack registered limit create --service nova --default-limit 5 servers
Every resource that can be requested in the cloud **must** at a minimum
have a registered limit set. Any resource that does **not** have a
registered limit set will fail quota enforcement because oslo.limit
considers an unregistered resource to have a limit of **0**.
Rechecking quota
~~~~~~~~~~~~~~~~
If :oslo.config:option:`quota.recheck_quota` is True (this is the default),
If :oslo.config:option:`quota.recheck_quota` = True (this is the default),
Nova will perform a second quota check after allocating resources. The first
quota check is performed before resources are allocated. Rechecking quota
ensures that quota limits are strictly enforced and prevents any possibility of
resource allocation going over the quota limit in the event of racing parallel
API requests.
ensures that quota limits are strictly enforced and prevents any possibility
of resource allocation going over the quota limit in the event of racing
parallel API requests.
It can be disabled by setting :oslo.config:option:`quota.recheck_quota` to
It can be disabled by setting :oslo.config:option:`quota.recheck_quota` =
False if strict quota enforcement is not important to the operator.
Quota usage from placement
Quota usage from Placement
--------------------------
With unified limits quotas, it is required that quota resource usage is counted
from placement. As such, the
:oslo.config:option:`quota.count_usage_from_placement` configuration option is
ignored when :oslo.config:option:`quota.driver` is set to
With unified limits quotas, it is required that quota resource usage is
counted from the Placement API service. As such,
the :oslo.config:option:`quota.count_usage_from_placement` configuration
option is ignored when :oslo.config:option:`quota.driver` is set to
``nova.quota.UnifiedLimitsDriver``.
There are some things to note when quota resource usage is counted from
placement:
There are some things to note when quota resource usage is counted from the
Placement API service:
* Counted usage will not be accurate in an environment where multiple Nova
deployments are sharing a placement deployment because currently placement
deployments are sharing a Placement deployment because currently Placement
has no way of partitioning resource providers between different Nova
deployments. Operators who are running multiple Nova deployments that share a
placement deployment should not use the ``nova.quota.UnifiedLimitsDriver``.
deployments. Operators who are running multiple Nova deployments that share
a Placement deployment should not use the ``nova.quota.UnifiedLimitsDriver``.
* Behavior will be different for resizes. During a resize, resource allocations
are held on both the source and destination (even on the same host, see
https://bugs.launchpad.net/nova/+bug/1790204) until the resize is confirmed
or reverted. Quota usage will be inflated for servers in this state.
* Behavior will be different for resizes. During a resize, resource
allocations are held on both the source and destination (even on the same
host, see https://bugs.launchpad.net/nova/+bug/1790204) until the resize is
confirmed or reverted. Quota usage will be inflated for servers in this
state.
* The ``populate_queued_for_delete`` and ``populate_user_id`` online data
migrations must be completed before usage can be counted from placement.
migrations must be completed before usage can be counted from Placement.
Until the data migration is complete, the system will fall back to legacy
quota usage counting from cell databases depending on the result of an EXISTS
database query during each quota check. Use
``nova-manage db online_data_migrations`` to run online data migrations.
quota usage counting from cell databases depending on the result of an
EXISTS database query during each quota check. Use ``nova-manage db
online_data_migrations`` to run online data migrations.
* Behavior will be different for unscheduled servers in ``ERROR`` state. A
server in ``ERROR`` state that has never been scheduled to a compute host
will not have placement allocations, so it will not consume quota usage for
will not have Placement allocations, so it will not consume quota usage for
cores and ram.
* Behavior will be different for servers in ``SHELVED_OFFLOADED`` state. A
server in ``SHELVED_OFFLOADED`` state will not have placement allocations, so
it will not consume quota usage for cores and ram. Note that because of this,
it will be possible for a request to unshelve a server to be rejected if the
user does not have enough quota available to support the cores and ram needed
by the server to be unshelved.
Configuration
-------------
View and update default quota values
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To list all default quotas for a project, run:
.. code-block:: console
$ openstack registered limit list
.. note::
This lists default quotas for all services and not just nova.
To show details about a default limit, run:
.. code-block:: console
$ openstack registered limit show <registered-limit-id>
To create a default quota limit, run:
.. code-block:: console
$ openstack registered limit create --service nova --default-limit <value> <resource-name>
.. note::
Creating or updating registered limits requires a system-scoped
authorization token by default. See the `Keystone tokens documentation`_ for
more information.
.. _Keystone tokens documentation: https://docs.openstack.org/keystone/latest/admin/tokens-overview.html#operation_create_system_token
To update a default quota value, run:
.. code-block:: console
$ openstack registered limit set --default-limit <value> <registered-limit-id>
To delete a default quota limit, run:
.. code-block:: console
$ openstack registered limit delete <registered-limit-id> [<registered-limit-id> ...]
View and update quota values for a project
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To list quotas for a project, run:
.. code-block:: console
$ openstack limit list --project <project>
.. note::
This lists project quotas for all services and not just nova.
To list quotas for all projects, you must have a system-scoped authorization
token and run:
.. code-block:: console
$ openstack limit list
To show details about a quota limit, run:
.. code-block:: console
$ openstack limit show <limit-id>
To create a quota limit for a project, run:
.. code-block:: console
$ openstack limit create --service nova --project <project> --resource-limit <value> <resource-name>
To update quotas for a project, run:
.. code-block:: console
$ openstack limit set --resource-limit <value> <limit-id>
To delete quotas for a project, run:
.. code-block:: console
$ openstack limit delete <limit-id> [<limit-id> ...]
server in ``SHELVED_OFFLOADED`` state will not have Placement allocations,
so it will not consume quota usage for cores and ram. Note that because of
this, it will be possible for a request to unshelve a server to be rejected
if the user does not have enough quota available to support the cores and
ram needed by the server to be unshelved.
Migration to unified limits quotas
----------------------------------
There is a `nova-manage limits migrate_to_unified_limits`_ command available to
help with moving from legacy Nova database quotas to Keystone unified limits
quotas. The command will read quota limits from the Nova database and call the
Keystone API to create the corresponding unified limits. Per-user quota limits
will **not** be copied into Keystone because per-user quotas are not supported
in unified limits.
There is a `nova-manage`_ command available to help with moving from legacy
Nova database quotas to Keystone unified limits quotas. The command will read
quota limits from the Nova database and call the Keystone API to create the
corresponding unified limits.
.. _nova-manage limits migrate_to_unified_limits: https://docs.openstack.org/nova/latest/cli/nova-manage.html#limits-migrate-to-unified-limits
.. code-block:: console
$ nova-manage limits migrate_to_unified_limits -h
usage: nova-manage limits migrate_to_unified_limits
[-h] [--project-id <project-id>] [--region-id <region-id>] [--verbose]
[--dry-run]
Copy quota limits from the Nova API database to Keystone.
options:
-h, --help show this help message and exit
--project-id <project-id>
Project ID for which to migrate quota limits
--region-id <region-id>
Region ID for which to migrate quota limits
--verbose Provide verbose output during execution.
--dry-run Show what limits would be created without actually
creating them.
.. important::
Per-user quota limits will **not** be copied into Keystone because per-user
quotas are not supported in unified limits.
.. _nova-manage: https://docs.openstack.org/nova/latest/cli/nova-manage.html#limits-migrate-to-unified-limits

121
doc/source/user/unified-limits.rst
View File

@ -2,7 +2,7 @@
Unified Limits Quotas
=====================
Since the Nova 28.0.0 (2023.2 Bobcat) release, it is recommended to use
Since the **Nova 28.0.0 (2023.2 Bobcat)** release, it is recommended to use
`Keystone unified limits`_ for Nova quota limits.
For information about legacy quota limits, see the :doc:`legacy quota
@ -11,19 +11,19 @@ documentation </user/quotas>`.
Nova uses a quota system for setting limits on resources such as number of
instances or amount of CPU that a specific project or user can use.
Quota limits are set by admin and retrieved for enforcement using the `Keystone
API`_.
Quota limits are set by admin and retrieved for enforcement using the
`Keystone unified limits API`_.
.. _Keystone unified limits: https://docs.openstack.org/keystone/latest/admin/unified-limits.html
.. _Keystone API: https://docs.openstack.org/api-ref/identity/v3/index.html#unified-limits
.. _Keystone unified limits API: https://docs.openstack.org/api-ref/identity/v3/index.html#unified-limits
Types of quota
--------------
Unified limit resource names for resources that are tracked as `resource
classes`_ in the placement service follow the naming pattern of the ``class:``
prefix followed by the name of the resource class. For example: class:VCPU,
class:PCPU, class:MEMORY_MB, class:DISK_GB, class:VGPU.
classes`_ in the Placement API service follow the naming pattern of the
``class:`` prefix followed by the name of the resource class. For example:
class:VCPU, class:PCPU, class:MEMORY_MB, class:DISK_GB, class:VGPU.
.. list-table::
:header-rows: 1
@ -32,113 +32,64 @@ class:PCPU, class:MEMORY_MB, class:DISK_GB, class:VGPU.
* - Quota name
- Description
* - class:VCPU
- Number of shared CPU cores (VCPUs) allowed per project.
- Number of shared CPU cores (VCPUs) allowed per project
* - class:PCPU
- Number of dedicated CPU cores (PCPUs) allowed per project.
- Number of dedicated CPU cores (PCPUs) allowed per project
* - servers
- Number of instances allowed per project.
- Number of instances allowed per project
* - server_key_pairs
- Number of key pairs allowed per user.
- Number of key pairs allowed per user
* - server_metadata_items
- Number of metadata items allowed per instance.
- Number of metadata items allowed per instance
* - class:MEMORY_MB
- Megabytes of instance ram allowed per project.
- Megabytes of instance ram allowed per project
* - server_groups
- Number of server groups per project.
- Number of server groups per project
* - server_group_members
- Number of servers per server group.
- Number of servers per server group
* - class:DISK_GB
- Gigabytes of instance disk allowed per project.
* - class:<any resource class in the placement service>
- Any resource class in the placement service that is allocated by Nova
can have a quota limit specified for it. Example: class:VGPU.
- Gigabytes of instance disk allowed per project
* - class:$RESOURCE_CLASS
- Any resource class in the Placement API service can have a quota limit
specified for it (example: class:VGPU)
.. _resource classes: https://docs.openstack.org/os-resource-classes/latest
The following quotas were previously available but were removed in microversion
2.36 as they proxied information available from the networking service.
.. list-table::
:header-rows: 1
:widths: 10 40
OpenStack CLI commands
----------------------
* - Quota name
- Description
* - fixed_ips
- Number of fixed IP addresses allowed per project. This number
must be equal to or greater than the number of allowed
instances.
* - floating_ips
- Number of floating IP addresses allowed per project.
* - networks
- Number of networks allowed per project (no longer used).
* - security_groups
- Number of security groups per project.
* - security_group_rules
- Number of security group rules per project.
For full OpenStackClient documentation, see
https://docs.openstack.org/python-openstackclient/latest/index.html.
Similarly, the following quotas were previously available but were removed in
microversion 2.57 as the personality files feature was deprecated.
.. list-table::
:header-rows: 1
:widths: 10 40
* - Quota name
- Description
* - injected_files
- Number of injected files allowed per project.
* - injected_file_content_bytes
- Number of content bytes allowed per injected file.
* - injected_file_path_bytes
- Length of injected file path.
Usage
-----
Project quotas
~~~~~~~~~~~~~~
To list all default quotas for projects, run:
To list default limits for Nova:
.. code-block:: console
$ openstack registered limit list
.. note::
This lists default quotas for all services and not just nova.
openstack registered limit list --service nova
For example:
.. code-block:: console
$ openstack registered limit list
$ openstack registered limit list --service nova
+----------------------------------+----------------------------------+------------------------------------+---------------+-------------+-----------+
| ID | Service ID | Resource Name | Default Limit | Description | Region ID |
+----------------------------------+----------------------------------+------------------------------------+---------------+-------------+-----------+
| eeee406035fc4a7892c6fc6fcc76e61a | b5d97619e6354c03be50c6b1589e6547 | image_size_total | 1000 | None | RegionOne |
| b99d35b1b7b74cd0a26a6311963328af | b5d97619e6354c03be50c6b1589e6547 | image_stage_total | 1000 | None | RegionOne |
| 70945789dad24082bc2a8cdbf53a5091 | b5d97619e6354c03be50c6b1589e6547 | image_count_total | 100 | None | RegionOne |
| 908d86c0ac3f46d0bb45ed9194710ea7 | b5d97619e6354c03be50c6b1589e6547 | image_count_uploading | 100 | None | RegionOne |
| be6dfeebb7c340e8b93b602d41fbff9b | 8b22bf8a66fa4524a522b2a21865bbf2 | servers | 10 | None | None |
| 8a658096236549788e61f4fcbd5a4a12 | 8b22bf8a66fa4524a522b2a21865bbf2 | class:VCPU | 20 | None | None |
| 63890db7d6a14401ba55e7f7022b95d0 | 8b22bf8a66fa4524a522b2a21865bbf2 | class:MEMORY_MB | 51200 | None | None |
| 221ba1c19d2c4272952663828d659013 | 8b22bf8a66fa4524a522b2a21865bbf2 | server_metadata_items | 128 | None | None |
| 8e61cabd2e854a11bdd2cf94efd702d1 | 8b22bf8a66fa4524a522b2a21865bbf2 | server_injected_files | 5 | None | None |
| 3d259390f70e4e9b88ecb2b0fa075f9b | 8b22bf8a66fa4524a522b2a21865bbf2 | server_injected_file_content_bytes | 10240 | None | None |
| a12e10b991cc4bdd8e6ff30f6e6c15ac | 8b22bf8a66fa4524a522b2a21865bbf2 | server_injected_file_path_bytes | 255 | None | None |
| a32a9080be6b4a5481c16a91fe329e6f | 8b22bf8a66fa4524a522b2a21865bbf2 | server_key_pairs | 100 | None | None |
| 86408bb7a0e542b18404ec7d348da820 | 8b22bf8a66fa4524a522b2a21865bbf2 | server_groups | 10 | None | None |
| 17c4552c5aad4afca4813f37530fc897 | 8b22bf8a66fa4524a522b2a21865bbf2 | server_group_members | 10 | None | None |
+----------------------------------+----------------------------------+------------------------------------+---------------+-------------+-----------+
To show details about a default limit, run:
To show details about a default limit:
.. code-block:: console
$ openstack registered limit show <registered-limit-id>
openstack registered limit show $REGISTERED_LIMIT_ID
For example:
@ -156,28 +107,34 @@ For example:
| service_id | 8b22bf8a66fa4524a522b2a21865bbf2 |
+---------------+----------------------------------+
To list the currently set quota values for your project, run:
To list project limits for Nova:
.. code-block:: console
$ openstack limit list
openstack limit list --service nova
For example:
.. code-block:: console
$ openstack limit list
$ openstack limit list --service nova
+----------------------------------+----------------------------------+----------------------------------+---------------+----------------+-------------+-----------+
| ID | Project ID | Service ID | Resource Name | Resource Limit | Description | Region ID |
+----------------------------------+----------------------------------+----------------------------------+---------------+----------------+-------------+-----------+
| 8b3364b2241e4090aaaa49355c7a5b56 | 5cd3281595a9497ba87209701cd9f3f2 | 8b22bf8a66fa4524a522b2a21865bbf2 | class:VCPU | 5 | None | None |
+----------------------------------+----------------------------------+----------------------------------+---------------+----------------+-------------+-----------+
To show details about a quota limimt, run:
To list limits for a particular project:
.. code-block:: console
$ openstack limit show <limit-id>
openstack limit list --service nova --project $PROJECT_ID
To show details about a project limit:
.. code-block:: console
openstack limit show $LIMIT_ID
For example: