diff --git a/doc/source/admin/unified-limits.rst b/doc/source/admin/unified-limits.rst index 8f762f3c8af2..f1714914f769 100644 --- a/doc/source/admin/unified-limits.rst +++ b/doc/source/admin/unified-limits.rst @@ -5,221 +5,393 @@ Manage Unified Limits Quotas .. note:: This section provides deployment information about the quota feature. For - end-user information about quotas, including information about the type of - quotas available, refer to the :doc:`user guide `. + end-user information about quotas, including information about the type + of quotas available, refer to the :doc:`user guide + `. -Since the Nova 28.0.0 (2023.2 Bobcat) release, it is recommended to use +Since the **Nova 28.0.0 (2023.2 Bobcat)** release, it is recommended to use `Keystone unified limits`_ for Nova quota limits. For information about legacy quota limits, see the :doc:`legacy quota documentation `. -To enable unified limits quotas, set the quota driver in the Nova -configuration: + +Quotas +------ + +To prevent system capacities from being exhausted without notification, you +can set up quotas. Quotas are operational limits. The number of servers +allowed for each project can be controlled so that cloud resources are +optimized, for example. Quotas can be enforced at both the global +(default) level and at the project level. + + +Unified limits +-------------- + +Unified limits is a modern quota system in which quota limits are centralized +in the Keystone identity service. There are three steps for quota enforcement +in this model: + +#. Quota limits are retrieved by calling the `Keystone unified limits API`_ + +#. Quota usage is counted from the `Placement API service`_ + +#. Quota is enforced locally using the `oslo.limit`_ limit enforcement + library + +In unified limits, the terminology is a bit different from legacy quotas: + +* A **registered limit** is a global or default limit that applies to all + projects + +* A **limit** is a project-scoped limit that applies to a particular project + +Cloud operators will need to manage their quota limits in the Keystone service +by calling the API directly or by using the OpenStackClient (OSC) `registered +limit`_ and `limit`_ commands. + +.. _Keystone unified limits: + https://docs.openstack.org/keystone/latest/admin/unified-limits.html +.. _Keystone unified limits API: + https://docs.openstack.org/api-ref/identity/v3/index.html#unified-limits +.. _Placement API service: https://docs.openstack.org/placement +.. _oslo.limit: https://docs.openstack.org/oslo.limit +.. _registered limit: + https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/registered-limit.html +.. _limit: + https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/limit.html + + +Roles +~~~~~ + +By default Keystone API policy, a user must have the following roles and +scopes in order to perform actions with unified limits. + +.. list-table:: + :header-rows: 1 + :widths: 10 20 20 + + * - Action + - Role + - Scope + * - List registered limits + - ``*`` + - ``*`` + * - Get registered limit + - ``*`` + - ``*`` + * - Create registered limit + - ``admin`` + - ``system=all`` + * - Update registered limit + - ``admin`` + - ``system=all`` + * - Delete registered limit + - ``admin`` + - ``system=all`` + * - List limits + - ``*`` + - ``*`` + * - Get limit + - ``*`` + - ``*`` + * - Create limit + - ``admin`` + - ``system=all`` + * - Update limit + - ``admin`` + - ``system=all`` + * - Delete limit + - ``admin`` + - ``system=all`` + + +Configuration +------------- + +To enable unified limits quotas, some Nova configuration of +the :program:`nova-api` and :program:`nova-conductor` services is necessary. + +Set the quota driver to the ``nova.quota.UnifiedLimitsDriver``: .. code-block:: ini [quota] driver = nova.quota.UnifiedLimitsDriver -Quota limits are set and retrieved for enforcement using the `Keystone API`_. +Add a configuration section for oslo.limit: -.. _Keystone unified limits: https://docs.openstack.org/keystone/latest/admin/unified-limits.html -.. _Keystone API: https://docs.openstack.org/api-ref/identity/v3/index.html#unified-limits +.. code-block:: ini -To prevent system capacities from being exhausted without notification, you can -set up quotas. Quotas are operational limits. For example, the number of -servers allowed for each project can be controlled so that cloud resources -are optimized. Quotas can be enforced at both the global (default) and the -project level. + [oslo_limit] + username = nova + user_domain_name = $SERVICE_DOMAIN_NAME + auth_url = $KEYSTONE_SERVICE_URI + auth_type = password + password = $SERVICE_PASSWORD + system_scope = all + endpoint_id = $SERVICE_ENDPOINT_ID -Resource usage is counted from the placement service with unified limits. +.. note:: + + The Nova service endpoint ID can be obtained by ``openstack endpoint + list --service nova -f value -c ID`` + +Ensure that the ``nova`` service user has the ``reader`` role with ``system`` +scope: + +.. code-block:: console + + openstack role add --user nova --user-domain $SERVICE_DOMAIN_NAME \ + --system all reader -Checking quota --------------- +Setting quota limits on resources +--------------------------------- -When calculating limits for a given resource and project, the following checks +Any resource that can be requested in the cloud must have a registered limit +set. Quota checks on cloud resources that do not have registered limits will +continue to fail until registered limits are set because oslo.limit considers +an unregistered resource to have a limit of 0. + +Types of quota +~~~~~~~~~~~~~~ + +Unified limit resource names for resources that are tracked as `resource +classes`_ in the Placement API service follow the naming pattern of the +``class:`` prefix followed by the name of the resource class. For example: +class:VCPU, class:PCPU, class:MEMORY_MB, class:DISK_GB, class:VGPU. + +.. list-table:: + :header-rows: 1 + :widths: 10 40 + + * - Quota Name + - Description + * - class:VCPU + - Number of shared CPU cores (VCPUs) allowed per project + * - class:PCPU + - Number of dedicated CPU cores (PCPUs) allowed per project + * - servers + - Number of instances allowed per project + * - server_key_pairs + - Number of key pairs allowed per user + * - server_metadata_items + - Number of metadata items allowed per instance + * - class:MEMORY_MB + - Megabytes of instance ram allowed per project + * - server_groups + - Number of server groups per project + * - server_group_members + - Number of servers per server group + * - class:DISK_GB + - Gigabytes of instance disk allowed per project + * - class:$RESOURCE_CLASS + - Any resource class in the Placement API service can have a quota limit + specified for it (example: class:VGPU) + +.. _resource classes: https://docs.openstack.org/os-resource-classes/latest + +OpenStack CLI commands +~~~~~~~~~~~~~~~~~~~~~~ + +For full OpenStackClient documentation, see +https://docs.openstack.org/python-openstackclient/latest/index.html. + +Registered Limits +^^^^^^^^^^^^^^^^^ + +To list default limits for Nova: + +.. code-block:: console + + openstack registered limit list --service nova + +To show details about a default limit: + +.. code-block:: console + + openstack registered limit show $REGISTERED_LIMIT_ID + +To create a default limit: + +.. code-block:: console + + openstack registered limit create --service nova --default-limit $LIMIT \ + $RESOURCE + +To update a default limit: + +.. code-block:: console + + openstack registered limit set --resource-name $RESOURCE \ + --default-limit $LIMIT $REGISTERED_LIMIT_ID + +To delete a default limit: + +.. code-block:: console + + openstack registered limit delete $REGISTERED_LIMIT_ID + +Limits +^^^^^^ + +To list project limits for Nova: + +.. code-block:: console + + openstack limit list --service nova + +To list limits for a particular project: + +.. code-block:: console + + openstack limit list --service nova --project $PROJECT_ID + +To show details about a project limit: + +.. code-block:: console + + openstack limit show $LIMIT_ID + +To create a project limit: + +.. code-block:: console + + openstack limit create --service nova --project $PROJECT_ID \ + --resource-limit $LIMIT $RESOURCE + +To update a project limit: + +.. code-block:: console + + openstack limit set --resource-name $RESOURCE --resource-limit $LIMIT \ + $LIMIT_ID + +To delete a project limit: + +.. code-block:: console + + openstack limit delete $LIMIT_ID + + +Quota enforcement +----------------- + +When enforcing limits for a given resource and project, the following checks are made in order: -#. Project-specific limits +#. Limits (project-specific) - Depending on the resource, is there a project-specific limit on the resource - in Keystone limits? If so, use that as the limit. You can create these - resource limits using: + Depending on the resource, is there a project-specific limit on the + resource in Keystone limits? If so, use that as the limit. If not, proceed + to check the registered default limit. - .. code-block:: console +#. Registered limits (default) - $ openstack limit create --service nova --project --resource-limit 5 servers + Depending on the resource, is there a default limit on the resource in + Keystone limits? If so, use that as the limit. If not, oslo.limit will + consider the limit as 0, the quota check will fail, and a quota limit + exceeded exception will be raised. -#. Default limits +.. warning:: - Use the Keystone registered limit for the resource as the limit. You can - create these default limits using: - - .. code-block:: console - - $ openstack registered limit create --service nova --default-limit 5 servers + Every resource that can be requested in the cloud **must** at a minimum + have a registered limit set. Any resource that does **not** have a + registered limit set will fail quota enforcement because oslo.limit + considers an unregistered resource to have a limit of **0**. Rechecking quota ~~~~~~~~~~~~~~~~ -If :oslo.config:option:`quota.recheck_quota` is True (this is the default), +If :oslo.config:option:`quota.recheck_quota` = True (this is the default), Nova will perform a second quota check after allocating resources. The first quota check is performed before resources are allocated. Rechecking quota -ensures that quota limits are strictly enforced and prevents any possibility of -resource allocation going over the quota limit in the event of racing parallel -API requests. +ensures that quota limits are strictly enforced and prevents any possibility +of resource allocation going over the quota limit in the event of racing +parallel API requests. -It can be disabled by setting :oslo.config:option:`quota.recheck_quota` to +It can be disabled by setting :oslo.config:option:`quota.recheck_quota` = False if strict quota enforcement is not important to the operator. -Quota usage from placement +Quota usage from Placement -------------------------- -With unified limits quotas, it is required that quota resource usage is counted -from placement. As such, the -:oslo.config:option:`quota.count_usage_from_placement` configuration option is -ignored when :oslo.config:option:`quota.driver` is set to +With unified limits quotas, it is required that quota resource usage is +counted from the Placement API service. As such, +the :oslo.config:option:`quota.count_usage_from_placement` configuration +option is ignored when :oslo.config:option:`quota.driver` is set to ``nova.quota.UnifiedLimitsDriver``. -There are some things to note when quota resource usage is counted from -placement: +There are some things to note when quota resource usage is counted from the +Placement API service: * Counted usage will not be accurate in an environment where multiple Nova - deployments are sharing a placement deployment because currently placement + deployments are sharing a Placement deployment because currently Placement has no way of partitioning resource providers between different Nova - deployments. Operators who are running multiple Nova deployments that share a - placement deployment should not use the ``nova.quota.UnifiedLimitsDriver``. + deployments. Operators who are running multiple Nova deployments that share + a Placement deployment should not use the ``nova.quota.UnifiedLimitsDriver``. -* Behavior will be different for resizes. During a resize, resource allocations - are held on both the source and destination (even on the same host, see - https://bugs.launchpad.net/nova/+bug/1790204) until the resize is confirmed - or reverted. Quota usage will be inflated for servers in this state. +* Behavior will be different for resizes. During a resize, resource + allocations are held on both the source and destination (even on the same + host, see https://bugs.launchpad.net/nova/+bug/1790204) until the resize is + confirmed or reverted. Quota usage will be inflated for servers in this + state. * The ``populate_queued_for_delete`` and ``populate_user_id`` online data - migrations must be completed before usage can be counted from placement. + migrations must be completed before usage can be counted from Placement. Until the data migration is complete, the system will fall back to legacy - quota usage counting from cell databases depending on the result of an EXISTS - database query during each quota check. Use - ``nova-manage db online_data_migrations`` to run online data migrations. + quota usage counting from cell databases depending on the result of an + EXISTS database query during each quota check. Use ``nova-manage db + online_data_migrations`` to run online data migrations. * Behavior will be different for unscheduled servers in ``ERROR`` state. A server in ``ERROR`` state that has never been scheduled to a compute host - will not have placement allocations, so it will not consume quota usage for + will not have Placement allocations, so it will not consume quota usage for cores and ram. * Behavior will be different for servers in ``SHELVED_OFFLOADED`` state. A - server in ``SHELVED_OFFLOADED`` state will not have placement allocations, so - it will not consume quota usage for cores and ram. Note that because of this, - it will be possible for a request to unshelve a server to be rejected if the - user does not have enough quota available to support the cores and ram needed - by the server to be unshelved. - - -Configuration -------------- - -View and update default quota values -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To list all default quotas for a project, run: - -.. code-block:: console - - $ openstack registered limit list - -.. note:: - - This lists default quotas for all services and not just nova. - -To show details about a default limit, run: - -.. code-block:: console - - $ openstack registered limit show - -To create a default quota limit, run: - -.. code-block:: console - - $ openstack registered limit create --service nova --default-limit - -.. note:: - - Creating or updating registered limits requires a system-scoped - authorization token by default. See the `Keystone tokens documentation`_ for - more information. - - .. _Keystone tokens documentation: https://docs.openstack.org/keystone/latest/admin/tokens-overview.html#operation_create_system_token - -To update a default quota value, run: - -.. code-block:: console - - $ openstack registered limit set --default-limit - -To delete a default quota limit, run: - -.. code-block:: console - - $ openstack registered limit delete [ ...] - -View and update quota values for a project -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To list quotas for a project, run: - -.. code-block:: console - - $ openstack limit list --project - -.. note:: - - This lists project quotas for all services and not just nova. - -To list quotas for all projects, you must have a system-scoped authorization -token and run: - -.. code-block:: console - - $ openstack limit list - -To show details about a quota limit, run: - -.. code-block:: console - - $ openstack limit show - -To create a quota limit for a project, run: - -.. code-block:: console - - $ openstack limit create --service nova --project --resource-limit - -To update quotas for a project, run: - -.. code-block:: console - - $ openstack limit set --resource-limit - -To delete quotas for a project, run: - -.. code-block:: console - - $ openstack limit delete [ ...] + server in ``SHELVED_OFFLOADED`` state will not have Placement allocations, + so it will not consume quota usage for cores and ram. Note that because of + this, it will be possible for a request to unshelve a server to be rejected + if the user does not have enough quota available to support the cores and + ram needed by the server to be unshelved. Migration to unified limits quotas ---------------------------------- -There is a `nova-manage limits migrate_to_unified_limits`_ command available to -help with moving from legacy Nova database quotas to Keystone unified limits -quotas. The command will read quota limits from the Nova database and call the -Keystone API to create the corresponding unified limits. Per-user quota limits -will **not** be copied into Keystone because per-user quotas are not supported -in unified limits. +There is a `nova-manage`_ command available to help with moving from legacy +Nova database quotas to Keystone unified limits quotas. The command will read +quota limits from the Nova database and call the Keystone API to create the +corresponding unified limits. -.. _nova-manage limits migrate_to_unified_limits: https://docs.openstack.org/nova/latest/cli/nova-manage.html#limits-migrate-to-unified-limits +.. code-block:: console + + $ nova-manage limits migrate_to_unified_limits -h + usage: nova-manage limits migrate_to_unified_limits + [-h] [--project-id ] [--region-id ] [--verbose] + [--dry-run] + + Copy quota limits from the Nova API database to Keystone. + + options: + -h, --help show this help message and exit + --project-id + Project ID for which to migrate quota limits + --region-id + Region ID for which to migrate quota limits + --verbose Provide verbose output during execution. + --dry-run Show what limits would be created without actually + creating them. + +.. important:: + + Per-user quota limits will **not** be copied into Keystone because per-user + quotas are not supported in unified limits. + +.. _nova-manage: https://docs.openstack.org/nova/latest/cli/nova-manage.html#limits-migrate-to-unified-limits diff --git a/doc/source/user/unified-limits.rst b/doc/source/user/unified-limits.rst index 7934dbb9122c..ff6f739b30d4 100644 --- a/doc/source/user/unified-limits.rst +++ b/doc/source/user/unified-limits.rst @@ -2,7 +2,7 @@ Unified Limits Quotas ===================== -Since the Nova 28.0.0 (2023.2 Bobcat) release, it is recommended to use +Since the **Nova 28.0.0 (2023.2 Bobcat)** release, it is recommended to use `Keystone unified limits`_ for Nova quota limits. For information about legacy quota limits, see the :doc:`legacy quota @@ -11,19 +11,19 @@ documentation `. Nova uses a quota system for setting limits on resources such as number of instances or amount of CPU that a specific project or user can use. -Quota limits are set by admin and retrieved for enforcement using the `Keystone -API`_. +Quota limits are set by admin and retrieved for enforcement using the +`Keystone unified limits API`_. .. _Keystone unified limits: https://docs.openstack.org/keystone/latest/admin/unified-limits.html -.. _Keystone API: https://docs.openstack.org/api-ref/identity/v3/index.html#unified-limits +.. _Keystone unified limits API: https://docs.openstack.org/api-ref/identity/v3/index.html#unified-limits Types of quota -------------- Unified limit resource names for resources that are tracked as `resource -classes`_ in the placement service follow the naming pattern of the ``class:`` -prefix followed by the name of the resource class. For example: class:VCPU, -class:PCPU, class:MEMORY_MB, class:DISK_GB, class:VGPU. +classes`_ in the Placement API service follow the naming pattern of the +``class:`` prefix followed by the name of the resource class. For example: +class:VCPU, class:PCPU, class:MEMORY_MB, class:DISK_GB, class:VGPU. .. list-table:: :header-rows: 1 @@ -32,113 +32,64 @@ class:PCPU, class:MEMORY_MB, class:DISK_GB, class:VGPU. * - Quota name - Description * - class:VCPU - - Number of shared CPU cores (VCPUs) allowed per project. + - Number of shared CPU cores (VCPUs) allowed per project * - class:PCPU - - Number of dedicated CPU cores (PCPUs) allowed per project. + - Number of dedicated CPU cores (PCPUs) allowed per project * - servers - - Number of instances allowed per project. + - Number of instances allowed per project * - server_key_pairs - - Number of key pairs allowed per user. + - Number of key pairs allowed per user * - server_metadata_items - - Number of metadata items allowed per instance. + - Number of metadata items allowed per instance * - class:MEMORY_MB - - Megabytes of instance ram allowed per project. + - Megabytes of instance ram allowed per project * - server_groups - - Number of server groups per project. + - Number of server groups per project * - server_group_members - - Number of servers per server group. + - Number of servers per server group * - class:DISK_GB - - Gigabytes of instance disk allowed per project. - * - class: - - Any resource class in the placement service that is allocated by Nova - can have a quota limit specified for it. Example: class:VGPU. + - Gigabytes of instance disk allowed per project + * - class:$RESOURCE_CLASS + - Any resource class in the Placement API service can have a quota limit + specified for it (example: class:VGPU) .. _resource classes: https://docs.openstack.org/os-resource-classes/latest -The following quotas were previously available but were removed in microversion -2.36 as they proxied information available from the networking service. -.. list-table:: - :header-rows: 1 - :widths: 10 40 +OpenStack CLI commands +---------------------- - * - Quota name - - Description - * - fixed_ips - - Number of fixed IP addresses allowed per project. This number - must be equal to or greater than the number of allowed - instances. - * - floating_ips - - Number of floating IP addresses allowed per project. - * - networks - - Number of networks allowed per project (no longer used). - * - security_groups - - Number of security groups per project. - * - security_group_rules - - Number of security group rules per project. +For full OpenStackClient documentation, see +https://docs.openstack.org/python-openstackclient/latest/index.html. -Similarly, the following quotas were previously available but were removed in -microversion 2.57 as the personality files feature was deprecated. - -.. list-table:: - :header-rows: 1 - :widths: 10 40 - - * - Quota name - - Description - * - injected_files - - Number of injected files allowed per project. - * - injected_file_content_bytes - - Number of content bytes allowed per injected file. - * - injected_file_path_bytes - - Length of injected file path. - - -Usage ------ - -Project quotas -~~~~~~~~~~~~~~ - -To list all default quotas for projects, run: +To list default limits for Nova: .. code-block:: console - $ openstack registered limit list - -.. note:: - - This lists default quotas for all services and not just nova. + openstack registered limit list --service nova For example: .. code-block:: console - $ openstack registered limit list + $ openstack registered limit list --service nova +----------------------------------+----------------------------------+------------------------------------+---------------+-------------+-----------+ | ID | Service ID | Resource Name | Default Limit | Description | Region ID | +----------------------------------+----------------------------------+------------------------------------+---------------+-------------+-----------+ - | eeee406035fc4a7892c6fc6fcc76e61a | b5d97619e6354c03be50c6b1589e6547 | image_size_total | 1000 | None | RegionOne | - | b99d35b1b7b74cd0a26a6311963328af | b5d97619e6354c03be50c6b1589e6547 | image_stage_total | 1000 | None | RegionOne | - | 70945789dad24082bc2a8cdbf53a5091 | b5d97619e6354c03be50c6b1589e6547 | image_count_total | 100 | None | RegionOne | - | 908d86c0ac3f46d0bb45ed9194710ea7 | b5d97619e6354c03be50c6b1589e6547 | image_count_uploading | 100 | None | RegionOne | | be6dfeebb7c340e8b93b602d41fbff9b | 8b22bf8a66fa4524a522b2a21865bbf2 | servers | 10 | None | None | | 8a658096236549788e61f4fcbd5a4a12 | 8b22bf8a66fa4524a522b2a21865bbf2 | class:VCPU | 20 | None | None | | 63890db7d6a14401ba55e7f7022b95d0 | 8b22bf8a66fa4524a522b2a21865bbf2 | class:MEMORY_MB | 51200 | None | None | | 221ba1c19d2c4272952663828d659013 | 8b22bf8a66fa4524a522b2a21865bbf2 | server_metadata_items | 128 | None | None | - | 8e61cabd2e854a11bdd2cf94efd702d1 | 8b22bf8a66fa4524a522b2a21865bbf2 | server_injected_files | 5 | None | None | - | 3d259390f70e4e9b88ecb2b0fa075f9b | 8b22bf8a66fa4524a522b2a21865bbf2 | server_injected_file_content_bytes | 10240 | None | None | - | a12e10b991cc4bdd8e6ff30f6e6c15ac | 8b22bf8a66fa4524a522b2a21865bbf2 | server_injected_file_path_bytes | 255 | None | None | | a32a9080be6b4a5481c16a91fe329e6f | 8b22bf8a66fa4524a522b2a21865bbf2 | server_key_pairs | 100 | None | None | | 86408bb7a0e542b18404ec7d348da820 | 8b22bf8a66fa4524a522b2a21865bbf2 | server_groups | 10 | None | None | | 17c4552c5aad4afca4813f37530fc897 | 8b22bf8a66fa4524a522b2a21865bbf2 | server_group_members | 10 | None | None | +----------------------------------+----------------------------------+------------------------------------+---------------+-------------+-----------+ -To show details about a default limit, run: +To show details about a default limit: .. code-block:: console - $ openstack registered limit show + openstack registered limit show $REGISTERED_LIMIT_ID For example: @@ -156,28 +107,34 @@ For example: | service_id | 8b22bf8a66fa4524a522b2a21865bbf2 | +---------------+----------------------------------+ -To list the currently set quota values for your project, run: +To list project limits for Nova: .. code-block:: console - $ openstack limit list + openstack limit list --service nova For example: .. code-block:: console - $ openstack limit list + $ openstack limit list --service nova +----------------------------------+----------------------------------+----------------------------------+---------------+----------------+-------------+-----------+ | ID | Project ID | Service ID | Resource Name | Resource Limit | Description | Region ID | +----------------------------------+----------------------------------+----------------------------------+---------------+----------------+-------------+-----------+ | 8b3364b2241e4090aaaa49355c7a5b56 | 5cd3281595a9497ba87209701cd9f3f2 | 8b22bf8a66fa4524a522b2a21865bbf2 | class:VCPU | 5 | None | None | +----------------------------------+----------------------------------+----------------------------------+---------------+----------------+-------------+-----------+ -To show details about a quota limimt, run: +To list limits for a particular project: .. code-block:: console - $ openstack limit show + openstack limit list --service nova --project $PROJECT_ID + +To show details about a project limit: + +.. code-block:: console + + openstack limit show $LIMIT_ID For example: