Re-propose spec for refresh-quotas-usage
This was approved for Liberty and Mitaka but code was never submitted. This was something that many operators expressed interest in before so it seems valuable and straight-forward to try again on this in Newton. There are two changes from the original Liberty spec: 1. --user is optional with the quota_usage_refresh command. 2. The testing section is updated to point out that unit and functional tests would be provided. Previously-approved: Mitaka Spec for blueprint refresh-quotas-usage Originally authored by: Romain Hardouin <romain.hardouin@cloudwatt.com> Change-Id: Ie71cc6836b1a291c07df3db913ea8655420819ba
This commit is contained in:
Chuck Carmack
322
specs/newton/approved/refresh-quotas-usage.rst
Normal file
322
specs/newton/approved/refresh-quotas-usage.rst
Normal file
@@ -0,0 +1,322 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==========================================
|
||||
Refresh quotas usage
|
||||
==========================================
|
||||
|
||||
https://blueprints.launchpad.net/nova/+spec/refresh-quotas-usage
|
||||
|
||||
For some reasons [*]_, the quotas usage can be out of sync.
|
||||
When a quota is wrongfully reached, a user cannot launch new VMs anymore.
|
||||
This "refresh" feature allows operators to quickly unblock users without
|
||||
manually running queries against the database or temporarily increase the
|
||||
quota.
|
||||
|
||||
.. [*] It seems that there are several root causes and there is no procedure
|
||||
to reproduce bugs. Although these root causes will eventually be
|
||||
identified, we cannot guarantee that some bugs will not occur again.
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Quotas usage can be out of sync in quota_usages table.
|
||||
The number of used resources may not reflect the actual use.
|
||||
For instance we can see 7 cores used while the actual use is only 4.
|
||||
|
||||
Use Cases
|
||||
----------
|
||||
|
||||
An end user can be blocked if a quota is wrongfully reached for a resource
|
||||
type.
|
||||
|
||||
If a refresh quotas usage feature is implemented in Nova an operator can
|
||||
correct the usage without running a SQL query directly on the database.
|
||||
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
Currently, the "refresh quotas usage" feature is hidden inside SQLAlchemy
|
||||
implementation.
|
||||
As described in "Alternatives" paragraph, the function
|
||||
``nova.db.sqlalchemy.api.quota_reserve()`` can refresh the database under
|
||||
some circumstances.
|
||||
|
||||
The change consists to:
|
||||
|
||||
* Create a function in the DB API to refresh quotas usage:
|
||||
``nova.db.api.quota_usage_refresh()``
|
||||
|
||||
* Do an "extract function" refactoring on
|
||||
``nova.db.sqlalchemy.api.quota_reserve()`` to implement the aforementioned
|
||||
``quota_usage_refresh()`` DB API. That is:
|
||||
``nova.db.sqlalchemy.api.quota_usage_refresh()``.
|
||||
|
||||
* Expose the ``nova.db.api.quota_usage_refresh()`` feature through a
|
||||
nova-manage command.
|
||||
|
||||
|
||||
The nova-manage command would look like:
|
||||
|
||||
::
|
||||
|
||||
$ nova-manage project quota_usage_refresh --project <Project name>
|
||||
[--user <User name>] [--key <Quota key>]
|
||||
|
||||
If ``--key`` is omitted, all quota resources are refreshed.
|
||||
|
||||
Specifying ``--user`` is optional since some resources are per-project quotas,
|
||||
like fixed_ips, floating_ips and networks. Similarly, the ``nova quota-update``
|
||||
command takes an optional user.
|
||||
|
||||
Example:
|
||||
|
||||
::
|
||||
|
||||
$ nova-manage project quota_usage_refresh --project demo --user john_doe
|
||||
--key cores
|
||||
|
||||
$ nova-manage project quota_usage_refresh
|
||||
--project f85aa788e8ee48fca3da27e0579d3597
|
||||
--key cores
|
||||
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
Another nova-manage command
|
||||
"""""""""""""""""""""""""""
|
||||
|
||||
Another nova-manage command can use the already implemented
|
||||
``nova.quota.usage_reset()`` function.
|
||||
|
||||
Here is the docstring of this function, note the "for a particular user":
|
||||
|
||||
::
|
||||
|
||||
Reset the usage records for a particular user on a list of
|
||||
resources. This will force that user's usage records to be
|
||||
refreshed the next time a reservation is made.
|
||||
|
||||
Note: this does not affect the currently outstanding
|
||||
reservations the user has; those reservations must be
|
||||
committed or rolled back (or expired).
|
||||
|
||||
:param context: The request context, for access checks.
|
||||
:param resources: A list of the resource names for which the
|
||||
usage must be reset.
|
||||
|
||||
|
||||
"Reset" means "set to -1 in database".
|
||||
|
||||
The resulting command would be:
|
||||
|
||||
::
|
||||
|
||||
$ nova-manage project quota_usage_reset --project <Project name>
|
||||
--user <User name> [--key <Quota key>]
|
||||
|
||||
If ``--key`` is omitted, all quota resources are reset.
|
||||
|
||||
Pros
|
||||
""""
|
||||
|
||||
Only ``nova/cmd/manage.py`` is modified.
|
||||
Unlike quota_usage_refresh, no changes are required at the database API level.
|
||||
|
||||
|
||||
Cons
|
||||
""""
|
||||
|
||||
The main difference with quota_usage_refresh is that the user won't see actual
|
||||
quota usages until the next reservation.
|
||||
|
||||
|
||||
Credits: This command is proposed by Joe Gordon (cf. patch set 2)
|
||||
|
||||
|
||||
Configure Nova
|
||||
""""""""""""""
|
||||
|
||||
We can enable quotas "auto refresh" in nova.conf thanks to:
|
||||
|
||||
* until_refresh (IntOpt) Count of reservations until usage is refreshed
|
||||
* max_age (IntOpt) Number of seconds between subsequent usage refreshes
|
||||
|
||||
These two settings (disabled by default) allow to refresh quotas usage but only
|
||||
during a quota reservation. The algorithm is:
|
||||
|
||||
1. Check quota
|
||||
2. until_refresh or max_age threshold reached?
|
||||
3. If yes: refresh
|
||||
|
||||
Let's take an example: a quota of 10 instances is set on a tenant.
|
||||
|
||||
The quota is wrongfully reached:
|
||||
|
||||
* nova absolute-limits shows totalInstancesUsed = 10
|
||||
* nova quota-show shows instances = 10
|
||||
|
||||
The actual instances number is 9.
|
||||
|
||||
When a user runs a ``nova boot`` he will get an error: "Quota exceeded".
|
||||
Many users will stop here and contact their support. Actually, a second
|
||||
``nova boot`` might succeed if the first one has refreshed the quotas usage
|
||||
(depending on until_refresh or max_age threshold).
|
||||
We would need to improve this behavior but it's off topic here.
|
||||
|
||||
Note that on Horizon a user will not able to spawn an instance (corresponding
|
||||
to the first ``nova boot``) because the button is disabled when a quota is
|
||||
reached.
|
||||
|
||||
To conclude:
|
||||
|
||||
* until_refresh or max_age need to be enabled but a cloud operator
|
||||
may not want to enable them if only few tenants encounter a bug on quotas
|
||||
usage.
|
||||
|
||||
* Even with these two settings enabled, we can't force a refresh.
|
||||
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
None
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Policy changes
|
||||
--------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
The feature hits the table quota_usages the same way
|
||||
``nova.db.sqlalchemy.api.quota_reserve()`` does when triggering a refresh.
|
||||
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
Other implementations of ``nova.db.api`` should implement
|
||||
``nova.db.api.quota_usage_refresh()``.
|
||||
|
||||
Handle nested projects?
|
||||
https://blueprints.launchpad.net/nova/+spec/nested-quota-driver-api
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
Chuck Carmack <carmack@us.ibm.com>
|
||||
|
||||
Other contributors:
|
||||
Romain Hardouin <romain.hardouin@cloudwatt.com>
|
||||
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
Not a big change, this BP can be submitted as a whole.
|
||||
|
||||
Two subtasks:
|
||||
|
||||
* Change the DB API
|
||||
* Implement the nova-manage command
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
In-tree unit and functional testing should be sufficient.
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
Document the new nova-manage command.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
* "nova quota statistics can be incorrect":
|
||||
https://bugs.launchpad.net/nova/+bug/1284424
|
||||
|
||||
* "Test job failes with FixedIpLimitExceeded with nova network":
|
||||
https://bugs.launchpad.net/nova/+bug/1353962
|
||||
|
||||
* "How to reset incorrect quota count?":
|
||||
https://ask.openstack.org/en/question/494/how-to-reset-incorrect-quota-count/
|
||||
|
||||
* "nova 'absolute-limits': [...] (they are wrong)"
|
||||
http://lists.openstack.org/pipermail/openstack/2014-November/010250.html
|
||||
|
||||
* "[...] usage now out-of-sync":
|
||||
https://ask.openstack.org/en/question/11943/deleted-vms-still-showing-in-nova-dashboard-usage-now-out-of-sync/
|
||||
|
||||
For information, on Horizon side:
|
||||
|
||||
* "absolute-limits sometimes returns negative value" :
|
||||
https://bugs.launchpad.net/nova/+bug/1370867
|
||||
|
||||
|
||||
History
|
||||
=======
|
||||
|
||||
.. list-table:: Revisions
|
||||
:header-rows: 1
|
||||
|
||||
* - Release Name
|
||||
- Description
|
||||
* - Newton
|
||||
- Re-proposed with No changes
|
||||
* - Mitaka
|
||||
- Re-proposed with 2 changes
|
||||
|
||||
1. --user is optional with the quota_usage_refresh command.
|
||||
2. The testing section is updated to point out that unit and functional tests
|
||||
|
||||
* - Liberty
|
||||
- Introduced
|
||||
Reference in New Issue
Block a user