Browse Source

Merge "New Quota driver ``DbQuotaNoLockDriver``"

Zuul 2 weeks ago
committed by Gerrit Code Review
11 changed files with 261 additions and 60 deletions
  1. +69
  2. +3
  3. +78
  4. +1
  5. +8
  6. +56
  7. +5
  8. +12
  9. +17
  10. +1
  11. +11

+ 69
- 23
doc/source/contributor/internals/quota.rst View File

@ -43,7 +43,7 @@ limits are currently not enforced on RPC interfaces listening on the AMQP
Plugin and ML2 drivers are not supposed to enforce quotas for resources they
manage. However, the subnet_allocation [#]_ extension is an exception and will
manage. However, the subnet_allocation [1]_ extension is an exception and will
be discussed below.
The quota management and enforcement mechanisms discussed here apply to every
@ -59,12 +59,14 @@ There are two main components in the Neutron quota system:
* The Quota Engine.
Both components rely on a quota driver. The neutron codebase currently defines
two quota drivers:
three quota drivers:
* neutron.db.quota.driver.DbQuotaDriver
* neutron.db.quota.driver_nolock.DbQuotaNoLockDriver (default)
* neutron.quota.ConfDriver
The latter driver is however deprecated.
The latter driver is however deprecated. The ``DbQuotaNoLockDriver`` is the
default quota driver, defined in the configuration option ``quota_driver``.
The Quota API extension handles quota management, whereas the Quota Engine
component handles quota enforcement. This API extension is loaded like any
@ -91,7 +93,7 @@ For a reservation to be successful, the total amount of resources requested,
plus the total amount of resources reserved, plus the total amount of resources
already stored in the database should not exceed the project's quota limit.
Finally, both quota management and enforcement rely on a "quota driver" [#]_,
Finally, both quota management and enforcement rely on a "quota driver" [2]_,
whose task is basically to perform database operations.
Quota Management
@ -100,14 +102,14 @@ Quota Management
The quota management component is fairly straightforward.
However, unlike the vast majority of Neutron extensions, it uses it own
controller class [#]_.
controller class [3]_.
This class does not implement the POST operation. List, get, update, and
delete operations are implemented by the usual index, show, update and
delete methods. These method simply call into the quota driver for either
fetching project quotas or updating them.
The _update_attributes method is called only once in the controller lifetime.
This method dynamically updates Neutron's resource attribute map [#]_ so that
This method dynamically updates Neutron's resource attribute map [4]_ so that
an attribute is added for every resource managed by the quota engine.
Request authorisation is performed in this controller, and only 'admin' users
are allowed to modify quotas for projects. As the neutron policy engine is not
@ -131,11 +133,18 @@ Resource Usage Info
Neutron has two ways of tracking resource usage info:
* CountableResource, where resource usage is calculated every time quotas
limits are enforced by counting rows in the resource table and reservations
for that resource.
* TrackedResource, which instead relies on a specific table tracking usage
data, and performs explicitly counting only when the data in this table are
not in sync with actual used and reserved resources.
limits are enforced by counting rows in the resource table or resources
tables and reservations for that resource.
* TrackedResource, depends on the selected driver:
* DbQuotaDriver: the resource usage relies on a specific table tracking
usage data, and performs explicitly counting only when the data in this
table are not in sync with actual used and reserved resources.
* DbQuotaNoLockDriver: the resource usage is counted directly from the
database table associated to the resource. In this new driver,
CountableResource and TrackedResource could look similar but
TrackedResource depends on one single database model (table) and the
resource count is done directly on this table only.
Another difference between CountableResource and TrackedResource is that the
former invokes a plugin method to count resources. CountableResource should be
@ -147,6 +156,9 @@ use CountableResource instances.
Resource creation is performed by the create_resource_instance factory method
in the neutron.quota.resource module.
DbQuotaDriver description
From a performance perspective, having a table tracking resource usage
has some advantages, albeit not fundamental. Indeed the time required for
executing queries to explicitly count objects will increase with the number of
@ -183,15 +195,48 @@ caveats' section, it is more reliable than solutions such as:
* Having a periodic task synchronising quota usage data with actual data in
the Neutron DB.
Finally, regardless of whether CountableResource or TrackedResource is used,
the quota engine always invokes its count() method to retrieve resource usage.
DbQuotaNoLockDriver description
The strategy of this quota driver is the opposite to ``DbQuotaDriver``.
Instead of tracking the usage quota of each resource in a specific table,
this driver retrieves the used resources directly form the database.
Each TrackedResource is linked to a database table that stores the tracked
resources. This driver claims that a trivial query on the resource table,
filtering by project ID, is faster than attending to the DB events and tracking
the quota usage in an independent table.
This driver relays on the database engine transactionality isolation. Each
time a new resource is requested, the quota driver opens a database transaction
* Clean up the expired reservations. The amount of expired reservations is
always limited because of the short timeout set (2 minutes).
* Retrieve the used resources for a specific project. This query retrieves
only the "project_id" column of the resource to avoid backref requests; that
limits the scope of the query and speeds up it.
* Retrieve the reserved resources, created by other concurrent operations.
* If there is enough quota, create a new reservation register.
Those operations, executed in the same transaction, are fast enough to avoid
another concurrent resource reservation, exceeding the available quota. At the
same time, this driver does not create a lock per resource and project ID,
allowing concurrent requests that won't be blocked by the resource lock.
Because the quota reservation process, described before, is a fast operation,
the chances of overcommiting resources over the quota limits are low. Neutron
does not enforce quota in such way that a quota limit violation could never
occur [5]_.
Regardless of whether CountableResource or TrackedResource is used, the quota
engine always invokes its count() method to retrieve resource usage.
Therefore, from the perspective of the Quota engine there is absolutely no
difference between CountableResource and TrackedResource.
Quota Enforcement
Quota Enforcement in DbQuotaDriver
Before dispatching a request to the plugin, the Neutron 'base' controller [#]_
Before dispatching a request to the plugin, the Neutron 'base' controller [6]_
attempts to make a reservation for requested resource(s).
Reservations are made by calling the make_reservation method in
@ -228,7 +273,7 @@ While non-locking approaches are possible, it has been found out that, since
a non-locking algorithms increases the chances of collision, the cost of
handling a DBDeadlock is still lower than the cost of retrying the operation
when a collision is detected. A study in this direction was conducted for
IP allocation operations, but the same principles apply here as well [#]_.
IP allocation operations, but the same principles apply here as well [7]_.
Nevertheless, moving away for DB-level locks is something that must happen
for quota enforcement in the future.
@ -345,9 +390,10 @@ Please be aware of the following limitations of the quota enforcement engine:
.. [#] Subnet allocation extension:
.. [#] DB Quota driver class:
.. [#] Quota API extension controller:
.. [#] Neutron resource attribute map:
.. [#] Base controller class:
.. [#]
.. [1] Subnet allocation extension:
.. [2] DB Quota driver class:
.. [3] Quota API extension controller:
.. [4] Neutron resource attribute map:
.. [5] Quota limit exceeded:
.. [6] Base controller class:
.. [7]

+ 3
- 2
neutron/conf/ View File

@ -18,8 +18,9 @@ from oslo_config import cfg
from neutron._i18n import _
QUOTA_DB_MODULE = 'neutron.db.quota.driver'
QUOTA_DB_MODULE = 'neutron.db.quota.driver_nolock'
QUOTA_CONF_DRIVER = 'neutron.quota.ConfDriver'

+ 78
- 0
neutron/db/quota/ View File

@ -0,0 +1,78 @@
# Copyright (c) 2021 Red Hat Inc.
# All Rights Reserved.
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from neutron_lib.db import api as db_api
from neutron_lib import exceptions
from oslo_log import log
from neutron.db.quota import api as quota_api
from neutron.db.quota import driver as quota_driver
LOG = log.getLogger(__name__)
class DbQuotaNoLockDriver(quota_driver.DbQuotaDriver):
"""Driver to enforce quotas and retrieve quota information
This driver does not use a (resource, project_id) lock but relays on the
simplicity of the calculation method, that is executed in a single database
transaction. The method (1) counts the number of created resources and (2)
the number of resource reservations. If the requested number of resources
do not exceeds the quota, a new reservation register is created.
This calculation method does not guarantee the quota enforcement if, for
example, the database isolation level is read committed; two transactions
can count the same number of resources and reservations, committing both
a new reservation exceeding the quota. But the goal of this reservation
method is to be fast enough to avoid the concurrency when counting the
resources while not blocking concurrent API operations.
def make_reservation(self, context, project_id, resources, deltas, plugin):
resources_over_limit = []
with db_api.CONTEXT_WRITER.using(context):
# Filter out unlimited resources.
limits = self.get_tenant_quotas(context, resources, project_id)
unlimited_resources = set([resource for (resource, limit) in
limits.items() if limit < 0])
requested_resources = (set(deltas.keys()) - unlimited_resources)
# Delete expired reservations before counting valid ones. This
# operation is fast and by calling it before making any
# reservation, we ensure the freshness of the reservations.
# Count the number of (1) used and (2) reserved resources for this
# project_id. If any resource limit is exceeded, raise exception.
for resource_name in requested_resources:
tracked_resource = resources.get(resource_name)
if not tracked_resource:
used_and_reserved = tracked_resource.count(
context, None, project_id, count_db_registers=True)
resource_num = deltas[resource_name]
if limits[resource_name] < (used_and_reserved + resource_num):
if resources_over_limit:
raise exceptions.OverQuota(overs=sorted(resources_over_limit))
return quota_api.create_reservation(context, project_id, deltas)
def cancel_reservation(self, context, reservation_id):
quota_api.remove_reservation(context, reservation_id, set_dirty=False)

+ 1
- 1
neutron/extensions/ View File

@ -38,7 +38,7 @@ DEFAULT_QUOTAS_ACTION = 'default'
DB_QUOTA_DRIVER = 'neutron.db.quota.driver.DbQuotaDriver'
DB_QUOTA_DRIVER = cfg.CONF.QUOTAS.quota_driver

+ 8
- 3
neutron/extensions/ View File

@ -23,6 +23,8 @@ from oslo_config import cfg
from neutron._i18n import _
from neutron.api import extensions
from neutron.api.v2 import resource
from neutron.db.quota import driver
from neutron.db.quota import driver_nolock
from neutron.extensions import quotasv2
from neutron.quota import resource_registry
@ -32,7 +34,11 @@ RESOURCE_NAME = 'quota'
QUOTA_DRIVER = cfg.CONF.QUOTAS.quota_driver
DB_QUOTA_DRIVER = 'neutron.db.quota.driver.DbQuotaDriver'
DB_QUOTA_DRIVERS = tuple('.'.join([klass.__module__, klass.__name__])
for klass in (driver.DbQuotaDriver,
@ -61,8 +67,7 @@ class Quotasv2_detail(api_extensions.ExtensionDescriptor):
# Ensure new extension is not loaded with old conf driver.
ALIAS, lambda: QUOTA_DRIVER in DB_QUOTA_DRIVERS, plugin_agnostic=True)
def get_name(cls):

+ 56
- 8
neutron/quota/ View File

@ -12,6 +12,8 @@
# License for the specific language governing permissions and limitations
# under the License.
import abc
from neutron_lib.db import api as db_api
from neutron_lib.plugins import constants
from neutron_lib.plugins import directory
@ -22,6 +24,8 @@ from sqlalchemy import exc as sql_exc
from sqlalchemy.orm import session as se
from neutron._i18n import _
from neutron.common import utils as n_utils
from neutron.conf import quota as quota_conf
from neutron.db.quota import api as quota_api
LOG = log.getLogger(__name__)
@ -55,7 +59,7 @@ def _count_resource(context, collection_name, tenant_id):
_('No plugins that support counting %s found.') % collection_name)
class BaseResource(object):
class BaseResource(object, metaclass=abc.ABCMeta):
"""Describe a single resource for quota checking."""
def __init__(self, name, flag, plural_name=None):
@ -98,6 +102,7 @@ class BaseResource(object):
return max(value, -1)
def dirty(self):
"""Return the current state of the Resource instance.
@ -106,6 +111,10 @@ class BaseResource(object):
does not track usage.
def count(self, context, plugin, project_id, **kwargs):
"""Return the total count of this resource"""
class CountableResource(BaseResource):
"""Describe a resource where the counts are determined by a function."""
@ -143,9 +152,13 @@ class CountableResource(BaseResource):
name, flag=flag, plural_name=plural_name)
self._count_func = count
def count(self, context, plugin, tenant_id, **kwargs):
def dirty(self):
def count(self, context, plugin, project_id, **kwargs):
# NOTE(ihrachys) _count_resource doesn't receive plugin
return self._count_func(context, self.plural_name, tenant_id)
return self._count_func(context, self.plural_name, project_id)
class TrackedResource(BaseResource):
@ -184,13 +197,21 @@ class TrackedResource(BaseResource):
self._model_class = model_class
self._dirty_tenants = set()
self._out_of_sync_tenants = set()
# NOTE(ralonsoh): "DbQuotaNoLockDriver" driver does not need to track
# the DB events or resync the resource quota usage.
if cfg.CONF.QUOTAS.quota_driver == quota_conf.QUOTA_DB_DRIVER:
self._track_resource_events = False
self._track_resource_events = True
def dirty(self):
if not self._track_resource_events:
return self._dirty_tenants
def mark_dirty(self, context):
if not self._dirty_tenants:
if not self._dirty_tenants or not self._track_resource_events:
with db_api.CONTEXT_WRITER.using(context):
# It is not necessary to protect this operation with a lock.
@ -236,7 +257,8 @@ class TrackedResource(BaseResource):
return usage_info
def resync(self, context, tenant_id):
if tenant_id not in self._out_of_sync_tenants:
if (tenant_id not in self._out_of_sync_tenants or
not self._track_resource_events):
LOG.debug(("Synchronizing usage tracker for tenant:%(tenant_id)s on "
@ -302,15 +324,37 @@ class TrackedResource(BaseResource):
reserved = reservations.get(, 0)
return reserved
def count(self, context, _plugin, tenant_id, resync_usage=True):
def count(self, context, _plugin, tenant_id, resync_usage=True,
"""Return the count of the resource.
The _plugin parameter is unused but kept for
compatibility with the signature of the count method for
CountableResource instances.
return (self.count_used(context, tenant_id, resync_usage) +
self.count_reserved(context, tenant_id))
if count_db_registers:
count = self._count_db_registers(context, tenant_id)
count = self.count_used(context, tenant_id, resync_usage)
return count + self.count_reserved(context, tenant_id)
def _count_db_registers(self, context, project_id):
"""Return the existing resources (self._model_class) in a project.
The query executed must be as fast as possible. To avoid retrieving all
model backref relationship columns, only "project_id" is requested
(this column always exists in the DB model because is used in the
# TODO(ralonsoh): declare the OVO class instead the DB model and use
# ``NeutronDbObject.count`` with the needed filters and fields to
# retrieve ("project_id").
admin_context = n_utils.get_elevated_context(context)
with db_api.CONTEXT_READER.using(admin_context):
query = admin_context.session.query(self._model_class.project_id)
query = query.filter(self._model_class.project_id == project_id)
return query.count()
def _except_bulk_delete(self, delete_context):
if delete_context.mapper.class_ == self._model_class:
@ -321,12 +365,16 @@ class TrackedResource(BaseResource):
def register_events(self):
if not self._track_resource_events:
listen = db_api.sqla_listen
listen(self._model_class, 'after_insert', self._db_event_handler)
listen(self._model_class, 'after_delete', self._db_event_handler)
listen(se.Session, 'after_bulk_delete', self._except_bulk_delete)
def unregister_events(self):
if not self._track_resource_events:
db_api.sqla_remove(self._model_class, 'after_insert',

+ 5
- 1
neutron/tests/unit/db/ View File

@ -51,6 +51,7 @@ from neutron.common import ipv6_utils
from neutron.common import test_lib
from neutron.common import utils
from neutron.conf import policies
from neutron.conf import quota as quota_conf
from neutron.db import db_base_plugin_common
from neutron.db import ipam_backend_mixin
from neutron.db.models import l3 as l3_models
@ -110,7 +111,10 @@ class NeutronDbPluginV2TestCase(testlib_api.WebTestCase):
def setUp(self, plugin=None, service_plugins=None,
quota_conf.register_quota_opts(quota_conf.core_quota_opts, cfg.CONF)
'quota_driver', 'neutron.db.quota.driver.DbQuotaDriver',
super(NeutronDbPluginV2TestCase, self).setUp()
cfg.CONF.set_override('notify_nova_on_port_status_changes', False)
cfg.CONF.set_override('allow_overlapping_ips', True)

+ 12
- 16
neutron/tests/unit/extensions/ View File

@ -30,6 +30,7 @@ from neutron.api.v2 import router
from neutron.common import config
from neutron.conf import quota as qconf
from neutron.db.quota import driver
from neutron.db.quota import driver_nolock
from neutron import quota
from neutron.quota import resource_registry
from neutron.tests import base
@ -504,29 +505,24 @@ class TestDbQuotaDriver(base.BaseTestCase):
class TestQuotaDriverLoad(base.BaseTestCase):
def setUp(self):
super(TestQuotaDriverLoad, self).setUp()
# Make sure QuotaEngine is reinitialized in each test.
quota.QUOTAS._driver = None
def _test_quota_driver(self, cfg_driver, loaded_driver,
quota.QUOTAS._driver = None
cfg.CONF.set_override('quota_driver', cfg_driver, group='QUOTAS')
with mock.patch.dict(sys.modules, {}):
if (not with_quota_db_module and
'neutron.db.quota.driver' in sys.modules):
del sys.modules['neutron.db.quota.driver']
quota.QUOTA_DB_MODULE in sys.modules):
del sys.modules[quota.QUOTA_DB_MODULE]
driver = quota.QUOTAS.get_driver()
self.assertEqual(loaded_driver, driver.__class__.__name__)
def test_quota_db_driver_with_quotas_table(self):
'DbQuotaDriver', True)
def test_quota_db_driver_fallback_conf_driver(self):
'ConfDriver', False)
def test_quota_driver_load(self):
for klass in (quota.ConfDriver, driver.DbQuotaDriver,
'.'.join([klass.__module__, klass.__name__]),
klass.__name__, True)
def test_quota_conf_driver(self):
'ConfDriver', True)
def test_quota_driver_fallback_conf_driver(self):
self._test_quota_driver(quota.QUOTA_DB_DRIVER, 'ConfDriver', False)

+ 17
- 6
neutron/tests/unit/quota/ View File

@ -30,34 +30,44 @@ from neutron.tests.unit import testlib_api
DB_PLUGIN_KLASS = 'neutron.db.db_base_plugin_v2.NeutronDbPluginV2'
QUOTA_DRIVER = 'neutron.db.quota.DbQuotaDriver'
meh_quota_flag = 'quota_meh'
meh_quota_opts = [cfg.IntOpt(meh_quota_flag, default=99)]
class _BaseResource(resource.BaseResource):
def dirty(self):
return False
def count(self, context, plugin, project_id, **kwargs):
class TestResource(base.DietTestCase):
"""Unit tests for neutron.quota.resource.BaseResource"""
def test_create_resource_without_plural_name(self):
res = resource.BaseResource('foo', None)
res = _BaseResource('foo', None)
self.assertEqual('foos', res.plural_name)
res = resource.BaseResource('foy', None)
res = _BaseResource('foy', None)
self.assertEqual('foies', res.plural_name)
def test_create_resource_with_plural_name(self):
res = resource.BaseResource('foo', None,
res = _BaseResource('foo', None, plural_name='foopsies')
self.assertEqual('foopsies', res.plural_name)
def test_resource_default_value(self):
res = resource.BaseResource('foo', 'foo_quota')
res = _BaseResource('foo', 'foo_quota')
with mock.patch('oslo_config.cfg.CONF') as mock_cfg:
mock_cfg.QUOTAS.foo_quota = 99
self.assertEqual(99, res.default)
def test_resource_negative_default_value(self):
res = resource.BaseResource('foo', 'foo_quota')
res = _BaseResource('foo', 'foo_quota')
with mock.patch('oslo_config.cfg.CONF') as mock_cfg:
mock_cfg.QUOTAS.foo_quota = -99
self.assertEqual(-1, res.default)
@ -94,6 +104,7 @@ class TestTrackedResource(testlib_api.SqlTestCase):
def setUp(self):
cfg.CONF.set_override('quota_driver', QUOTA_DRIVER, group='QUOTAS')
super(TestTrackedResource, self).setUp()
self.resource = 'meh'

+ 1
- 0
neutron/tests/unit/quota/ View File

@ -160,6 +160,7 @@ class TestAuxiliaryFunctions(base.DietTestCase):
'TrackedResource.mark_dirty') as mock_mark_dirty:
self.registry.set_tracked_resource('meh', test_quota.MehModel)
self.registry.resources['meh']._track_resource_events = True
res = self.registry.get_resource('meh')
# This ensures dirty is true

+ 11
- 0
releasenotes/notes/new-quota-driver-DbQuotaNoLockDriver-5f8a44915ec16a1b.yaml View File

@ -0,0 +1,11 @@
- |
A new quota driver is added: ``DbQuotaNoLockDriver``. This driver, unlike
``DbQuotaDriver``, does not create a unique lock per (resource,
project_id). That may lead to a database deadlock state if the number of
server requests exceeds the number of resolved resource creations, as
described in `LP#1926787 <>`_.
This driver relays on the database transactionality isolation and counts
the number of used and reserved resources and, if available, creates the
new resource reservations in one single database transaction.