New Quota driver `DbQuotaNoLockDriver`

This new quota driver, ``DbQuotaNoLockDriver``, does not create a lock
per (resource, project_id) but retrieves the instant (resource,
project_id) usage and the current (resource, project_id) reservations.
If the requested number of resources fit the available quota, a new
``Reservation`` register is created with the amount of units requested.

All those operations are done inside a DB transaction context. That
means the amount of resources and reservations is guaranteed inside
this transaction (depending on the DB backend isolation level defined)
and the new reservation created will not clash with other DB transation.
That will guarantee the number of resources and instant reservations
never exceed the quota limits defined for this (resource, project_id).

NOTES:
- This change tries to be as unobtrusive as possible. The new driver
  uses the same ``DbQuotaDriver`` dabatase tables (except for
  ``QuotaUsage``) and the same Quota engine API, located in
  ``neutron.quota``. However, the Quota engine resources implements some
  particular API actions like "dirty", that are not used in the new
  driver.
- The Pecan Quota enforcement hooks,
  ``neutron.pecan_wgsi.hooks.quota_enforcement``, execute actions like
  "resync", "mark_resources_dirty" or "set_resources_dirty", that has
  no meaning in the new driver.
- The isolation between the Quota engine and the Pecan hook, and the
  driver itself is not clearly defined. A refactor of the Quota engine,
  Quota service, Quota drivers and a common API between the driver and
  the engine is needed.
- If ``DbQuotaDriver`` is deprecated, ``CountableResource`` and
  ``TrackedResource`` will be joined in a single class. This resource
  class will have a count method (countable) or a hard dependency on a
  database table (tracked resource). The only difference will be the
  "count" method implementation.

Closes-Bug: #1926787

Change-Id: I4f98c6fcd781459fd7150aff426d19c7fdfa98c1

doc/source/contributor/internals/quota.rst

@@ -43,7 +43,7 @@ limits are currently not enforced on RPC interfaces listening on the AMQP
 bus.
 
 Plugin and ML2 drivers are not supposed to enforce quotas for resources they
-manage. However, the subnet_allocation [#]_ extension is an exception and will
+manage. However, the subnet_allocation [1]_ extension is an exception and will
 be discussed below.
 
 The quota management and enforcement mechanisms discussed here apply to every
@@ -59,12 +59,14 @@ There are two main components in the Neutron quota system:
  * The Quota Engine.
 
 Both components rely on a quota driver. The neutron codebase currently defines
-two quota drivers:
+three quota drivers:
 
  * neutron.db.quota.driver.DbQuotaDriver
+ * neutron.db.quota.driver_nolock.DbQuotaNoLockDriver (default)
  * neutron.quota.ConfDriver
 
-The latter driver is however deprecated.
+The latter driver is however deprecated. The ``DbQuotaNoLockDriver`` is the
+default quota driver, defined in the configuration option ``quota_driver``.
 
 The Quota API extension handles quota management, whereas the Quota Engine
 component handles quota enforcement. This API extension is loaded like any
@@ -91,7 +93,7 @@ For a reservation to be successful, the total amount of resources requested,
 plus the total amount of resources reserved, plus the total amount of resources
 already stored in the database should not exceed the project's quota limit.
 
-Finally, both quota management and enforcement rely on a "quota driver" [#]_,
+Finally, both quota management and enforcement rely on a "quota driver" [2]_,
 whose task is basically to perform database operations.
 
 Quota Management
@@ -100,14 +102,14 @@ Quota Management
 The quota management component is fairly straightforward.
 
 However, unlike the vast majority of Neutron extensions, it uses it own
-controller class [#]_.
+controller class [3]_.
 This class does not implement the POST operation. List, get, update, and
 delete operations are implemented by the usual index, show, update and
 delete methods. These method simply call into the quota driver for either
 fetching project quotas or updating them.
 
 The _update_attributes method is called only once in the controller lifetime.
-This method dynamically updates Neutron's resource attribute map [#]_ so that
+This method dynamically updates Neutron's resource attribute map [4]_ so that
 an attribute is added for every resource managed by the quota engine.
 Request authorisation is performed in this controller, and only 'admin' users
 are allowed to modify quotas for projects. As the neutron policy engine is not
@@ -131,11 +133,18 @@ Resource Usage Info
 Neutron has two ways of tracking resource usage info:
 
  * CountableResource, where resource usage is calculated every time quotas
-   limits are enforced by counting rows in the resource table and reservations
+   limits are enforced by counting rows in the resource table or resources
-   for that resource.
+   tables and reservations for that resource.
- * TrackedResource, which instead relies on a specific table tracking usage
+ * TrackedResource, depends on the selected driver:
-   data, and performs explicitly counting only when the data in this table are
+
-   not in sync with actual used and reserved resources.
+   * DbQuotaDriver: the resource usage relies on a specific table tracking
+     usage data, and performs explicitly counting only when the data in this
+     table are not in sync with actual used and reserved resources.
+   * DbQuotaNoLockDriver: the resource usage is counted directly from the
+     database table associated to the resource. In this new driver,
+     CountableResource and TrackedResource could look similar but
+     TrackedResource depends on one single database model (table) and the
+     resource count is done directly on this table only.
 
 Another difference between CountableResource and TrackedResource is that the
 former invokes a plugin method to count resources. CountableResource should be
@@ -147,6 +156,9 @@ use CountableResource instances.
 Resource creation is performed by the create_resource_instance factory method
 in the neutron.quota.resource module.
 
+DbQuotaDriver description
+-------------------------
+
 From a performance perspective, having a table tracking resource usage
 has some advantages, albeit not fundamental. Indeed the time required for
 executing queries to explicitly count objects will increase with the number of
@@ -183,15 +195,48 @@ caveats' section, it is more reliable than solutions such as:
  * Having a periodic task synchronising quota usage data with actual data in
    the Neutron DB.
 
-Finally, regardless of whether CountableResource or TrackedResource is used,
+
-the quota engine always invokes its count() method to retrieve resource usage.
+DbQuotaNoLockDriver description
+-------------------------------
+
+The strategy of this quota driver is the opposite to ``DbQuotaDriver``.
+Instead of tracking the usage quota of each resource in a specific table,
+this driver retrieves the used resources directly form the database.
+Each TrackedResource is linked to a database table that stores the tracked
+resources. This driver claims that a trivial query on the resource table,
+filtering by project ID, is faster than attending to the DB events and tracking
+the quota usage in an independent table.
+
+This driver relays on the database engine transactionality isolation. Each
+time a new resource is requested, the quota driver opens a database transaction
+to:
+
+ * Clean up the expired reservations. The amount of expired reservations is
+   always limited because of the short timeout set (2 minutes).
+ * Retrieve the used resources for a specific project. This query retrieves
+   only the "project_id" column of the resource to avoid backref requests; that
+   limits the scope of the query and speeds up it.
+ * Retrieve the reserved resources, created by other concurrent operations.
+ * If there is enough quota, create a new reservation register.
+
+Those operations, executed in the same transaction, are fast enough to avoid
+another concurrent resource reservation, exceeding the available quota. At the
+same time, this driver does not create a lock per resource and project ID,
+allowing concurrent requests that won't be blocked by the resource lock.
+Because the quota reservation process, described before, is a fast operation,
+the chances of overcommiting resources over the quota limits are low. Neutron
+does not enforce quota in such way that a quota limit violation could never
+occur [5]_.
+
+Regardless of whether CountableResource or TrackedResource is used, the quota
+engine always invokes its count() method to retrieve resource usage.
 Therefore, from the perspective of the Quota engine there is absolutely no
 difference between CountableResource and TrackedResource.
 
-Quota Enforcement
+Quota Enforcement in DbQuotaDriver
------------------
+----------------------------------
 
-Before dispatching a request to the plugin, the Neutron 'base' controller [#]_
+Before dispatching a request to the plugin, the Neutron 'base' controller [6]_
 attempts to make a reservation for requested resource(s).
 Reservations are made by calling the make_reservation method in
 neutron.quota.QuotaEngine.
@@ -228,7 +273,7 @@ While non-locking approaches are possible, it has been found out that, since
 a non-locking algorithms increases the chances of collision, the cost of
 handling a DBDeadlock is still lower than the cost of retrying the operation
 when a collision is detected. A study in this direction was conducted for
-IP allocation operations, but the same principles apply here as well [#]_.
+IP allocation operations, but the same principles apply here as well [7]_.
 Nevertheless, moving away for DB-level locks is something that must happen
 for quota enforcement in the future.
 
@@ -345,9 +390,10 @@ Please be aware of the following limitations of the quota enforcement engine:
 References
 ----------
 
-.. [#] Subnet allocation extension: http://opendev.org/openstack/neutron/tree/neutron/extensions/subnetallocation.py
+.. [1] Subnet allocation extension: http://opendev.org/openstack/neutron/tree/neutron/extensions/subnetallocation.py
-.. [#] DB Quota driver class: http://opendev.org/openstack/neutron/tree/neutron/db/quota/driver.py#n30
+.. [2] DB Quota driver class: http://opendev.org/openstack/neutron/tree/neutron/db/quota/driver.py#n30
-.. [#] Quota API extension controller: http://opendev.org/openstack/neutron/tree/neutron/extensions/quotasv2.py#n40
+.. [3] Quota API extension controller: http://opendev.org/openstack/neutron/tree/neutron/extensions/quotasv2.py#n40
-.. [#] Neutron resource attribute map: http://opendev.org/openstack/neutron/tree/neutron/api/v2/attributes.py#n639
+.. [4] Neutron resource attribute map: http://opendev.org/openstack/neutron/tree/neutron/api/v2/attributes.py#n639
-.. [#] Base controller class: http://opendev.org/openstack/neutron/tree/neutron/api/v2/base.py#n50
+.. [5] Quota limit exceeded: https://bugs.launchpad.net/neutron/+bug/1862050/
-.. [#] http://lists.openstack.org/pipermail/openstack-dev/2015-February/057534.html
+.. [6] Base controller class: http://opendev.org/openstack/neutron/tree/neutron/api/v2/base.py#n50
+.. [7] http://lists.openstack.org/pipermail/openstack-dev/2015-February/057534.html

neutron/conf/quota.py

@@ -18,8 +18,9 @@ from oslo_config import cfg
 
 from neutron._i18n import _
 
-QUOTA_DB_MODULE = 'neutron.db.quota.driver'
+
-QUOTA_DB_DRIVER = '%s.DbQuotaDriver' % QUOTA_DB_MODULE
+QUOTA_DB_MODULE = 'neutron.db.quota.driver_nolock'
+QUOTA_DB_DRIVER = QUOTA_DB_MODULE + '.DbQuotaNoLockDriver'
 QUOTA_CONF_DRIVER = 'neutron.quota.ConfDriver'
 QUOTAS_CFG_GROUP = 'QUOTAS'
 

neutron/db/quota/driver_nolock.py

@@ -0,0 +1,78 @@
+# Copyright (c) 2021 Red Hat Inc.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from neutron_lib.db import api as db_api
+from neutron_lib import exceptions
+from oslo_log import log
+
+from neutron.db.quota import api as quota_api
+from neutron.db.quota import driver as quota_driver
+
+
+LOG = log.getLogger(__name__)
+
+
+class DbQuotaNoLockDriver(quota_driver.DbQuotaDriver):
+    """Driver to enforce quotas and retrieve quota information
+
+    This driver does not use a (resource, project_id) lock but relays on the
+    simplicity of the calculation method, that is executed in a single database
+    transaction. The method (1) counts the number of created resources and (2)
+    the number of resource reservations. If the requested number of resources
+    do not exceeds the quota, a new reservation register is created.
+
+    This calculation method does not guarantee the quota enforcement if, for
+    example, the database isolation level is read committed; two transactions
+    can count the same number of resources and reservations, committing both
+    a new reservation exceeding the quota. But the goal of this reservation
+    method is to be fast enough to avoid the concurrency when counting the
+    resources while not blocking concurrent API operations.
+    """
+    @db_api.retry_if_session_inactive()
+    def make_reservation(self, context, project_id, resources, deltas, plugin):
+        resources_over_limit = []
+        with db_api.CONTEXT_WRITER.using(context):
+            # Filter out unlimited resources.
+            limits = self.get_tenant_quotas(context, resources, project_id)
+            unlimited_resources = set([resource for (resource, limit) in
+                                       limits.items() if limit < 0])
+            requested_resources = (set(deltas.keys()) - unlimited_resources)
+
+            # Delete expired reservations before counting valid ones. This
+            # operation is fast and by calling it before making any
+            # reservation, we ensure the freshness of the reservations.
+            quota_api.remove_expired_reservations(context,
+                                                  tenant_id=project_id)
+
+            # Count the number of (1) used and (2) reserved resources for this
+            # project_id. If any resource limit is exceeded, raise exception.
+            for resource_name in requested_resources:
+                tracked_resource = resources.get(resource_name)
+                if not tracked_resource:
+                    continue
+
+                used_and_reserved = tracked_resource.count(
+                    context, None, project_id, count_db_registers=True)
+                resource_num = deltas[resource_name]
+                if limits[resource_name] < (used_and_reserved + resource_num):
+                    resources_over_limit.append(resource_name)
+
+            if resources_over_limit:
+                raise exceptions.OverQuota(overs=sorted(resources_over_limit))
+
+            return quota_api.create_reservation(context, project_id, deltas)
+
+    def cancel_reservation(self, context, reservation_id):
+        quota_api.remove_reservation(context, reservation_id, set_dirty=False)

neutron/extensions/quotasv2.py

@@ -38,7 +38,7 @@ DEFAULT_QUOTAS_ACTION = 'default'
 RESOURCE_NAME = 'quota'
 RESOURCE_COLLECTION = RESOURCE_NAME + "s"
 QUOTAS = quota.QUOTAS
-DB_QUOTA_DRIVER = 'neutron.db.quota.driver.DbQuotaDriver'
+DB_QUOTA_DRIVER = cfg.CONF.QUOTAS.quota_driver
 EXTENDED_ATTRIBUTES_2_0 = {
     RESOURCE_COLLECTION: {}
 }

neutron/extensions/quotasv2_detail.py

@@ -23,6 +23,8 @@ from oslo_config import cfg
 from neutron._i18n import _
 from neutron.api import extensions
 from neutron.api.v2 import resource
+from neutron.db.quota import driver
+from neutron.db.quota import driver_nolock
 from neutron.extensions import quotasv2
 from neutron.quota import resource_registry
 
@@ -32,7 +34,11 @@ RESOURCE_NAME = 'quota'
 ALIAS = RESOURCE_NAME + '_' + DETAIL_QUOTAS_ACTION
 QUOTA_DRIVER = cfg.CONF.QUOTAS.quota_driver
 RESOURCE_COLLECTION = RESOURCE_NAME + "s"
-DB_QUOTA_DRIVER = 'neutron.db.quota.driver.DbQuotaDriver'
+DB_QUOTA_DRIVERS = tuple('.'.join([klass.__module__, klass.__name__])
+                         for klass in (driver.DbQuotaDriver,
+                                       driver_nolock.DbQuotaNoLockDriver,
+                                       )
+                         )
 EXTENDED_ATTRIBUTES_2_0 = {
     RESOURCE_COLLECTION: {}
 }
@@ -61,8 +67,7 @@ class Quotasv2_detail(api_extensions.ExtensionDescriptor):
 
     # Ensure new extension is not loaded with old conf driver.
     extensions.register_custom_supported_check(
-        ALIAS, lambda: QUOTA_DRIVER == DB_QUOTA_DRIVER,
+        ALIAS, lambda: QUOTA_DRIVER in DB_QUOTA_DRIVERS, plugin_agnostic=True)
-        plugin_agnostic=True)
 
     @classmethod
     def get_name(cls):

neutron/quota/resource.py

@@ -12,6 +12,8 @@
 #    License for the specific language governing permissions and limitations
 #    under the License.
 
+import abc
+
 from neutron_lib.db import api as db_api
 from neutron_lib.plugins import constants
 from neutron_lib.plugins import directory
@@ -22,6 +24,8 @@ from sqlalchemy import exc as sql_exc
 from sqlalchemy.orm import session as se
 
 from neutron._i18n import _
+from neutron.common import utils as n_utils
+from neutron.conf import quota as quota_conf
 from neutron.db.quota import api as quota_api
 
 LOG = log.getLogger(__name__)
@@ -55,7 +59,7 @@ def _count_resource(context, collection_name, tenant_id):
         _('No plugins that support counting %s found.') % collection_name)
 
 
-class BaseResource(object):
+class BaseResource(object, metaclass=abc.ABCMeta):
     """Describe a single resource for quota checking."""
 
     def __init__(self, name, flag, plural_name=None):
@@ -98,6 +102,7 @@ class BaseResource(object):
         return max(value, -1)
 
     @property
+    @abc.abstractmethod
     def dirty(self):
         """Return the current state of the Resource instance.
 
@@ -106,6 +111,10 @@ class BaseResource(object):
                   does not track usage.
         """
 
+    @abc.abstractmethod
+    def count(self, context, plugin, project_id, **kwargs):
+        """Return the total count of this resource"""
+
 
 class CountableResource(BaseResource):
     """Describe a resource where the counts are determined by a function."""
@@ -143,9 +152,13 @@ class CountableResource(BaseResource):
             name, flag=flag, plural_name=plural_name)
         self._count_func = count
 
-    def count(self, context, plugin, tenant_id, **kwargs):
+    @property
+    def dirty(self):
+        return
+
+    def count(self, context, plugin, project_id, **kwargs):
         # NOTE(ihrachys) _count_resource doesn't receive plugin
-        return self._count_func(context, self.plural_name, tenant_id)
+        return self._count_func(context, self.plural_name, project_id)
 
 
 class TrackedResource(BaseResource):
@@ -184,13 +197,21 @@ class TrackedResource(BaseResource):
         self._model_class = model_class
         self._dirty_tenants = set()
         self._out_of_sync_tenants = set()
+        # NOTE(ralonsoh): "DbQuotaNoLockDriver" driver does not need to track
+        # the DB events or resync the resource quota usage.
+        if cfg.CONF.QUOTAS.quota_driver == quota_conf.QUOTA_DB_DRIVER:
+            self._track_resource_events = False
+        else:
+            self._track_resource_events = True
 
     @property
     def dirty(self):
+        if not self._track_resource_events:
+            return
         return self._dirty_tenants
 
     def mark_dirty(self, context):
-        if not self._dirty_tenants:
+        if not self._dirty_tenants or not self._track_resource_events:
             return
         with db_api.CONTEXT_WRITER.using(context):
             # It is not necessary to protect this operation with a lock.
@@ -236,7 +257,8 @@ class TrackedResource(BaseResource):
         return usage_info
 
     def resync(self, context, tenant_id):
-        if tenant_id not in self._out_of_sync_tenants:
+        if (tenant_id not in self._out_of_sync_tenants or
+                not self._track_resource_events):
             return
         LOG.debug(("Synchronizing usage tracker for tenant:%(tenant_id)s on "
                    "resource:%(resource)s"),
@@ -302,15 +324,37 @@ class TrackedResource(BaseResource):
         reserved = reservations.get(self.name, 0)
         return reserved
 
-    def count(self, context, _plugin, tenant_id, resync_usage=True):
+    def count(self, context, _plugin, tenant_id, resync_usage=True,
+              count_db_registers=False):
         """Return the count of the resource.
 
         The _plugin parameter is unused but kept for
         compatibility with the signature of the count method for
         CountableResource instances.
         """
-        return (self.count_used(context, tenant_id, resync_usage) +
+        if count_db_registers:
-                self.count_reserved(context, tenant_id))
+            count = self._count_db_registers(context, tenant_id)
+        else:
+            count = self.count_used(context, tenant_id, resync_usage)
+
+        return count + self.count_reserved(context, tenant_id)
+
+    def _count_db_registers(self, context, project_id):
+        """Return the existing resources (self._model_class) in a project.
+
+        The query executed must be as fast as possible. To avoid retrieving all
+        model backref relationship columns, only "project_id" is requested
+        (this column always exists in the DB model because is used in the
+        filter).
+        """
+        # TODO(ralonsoh): declare the OVO class instead the DB model and use
+        # ``NeutronDbObject.count`` with the needed filters and fields to
+        # retrieve ("project_id").
+        admin_context = n_utils.get_elevated_context(context)
+        with db_api.CONTEXT_READER.using(admin_context):
+            query = admin_context.session.query(self._model_class.project_id)
+            query = query.filter(self._model_class.project_id == project_id)
+            return query.count()
 
     def _except_bulk_delete(self, delete_context):
         if delete_context.mapper.class_ == self._model_class:
@@ -321,12 +365,16 @@ class TrackedResource(BaseResource):
                                self._model_class)
 
     def register_events(self):
+        if not self._track_resource_events:
+            return
         listen = db_api.sqla_listen
         listen(self._model_class, 'after_insert', self._db_event_handler)
         listen(self._model_class, 'after_delete', self._db_event_handler)
         listen(se.Session, 'after_bulk_delete', self._except_bulk_delete)
 
     def unregister_events(self):
+        if not self._track_resource_events:
+            return
         try:
             db_api.sqla_remove(self._model_class, 'after_insert',
                                self._db_event_handler)

neutron/tests/unit/db/test_db_base_plugin_v2.py

@@ -51,6 +51,7 @@ from neutron.common import ipv6_utils
 from neutron.common import test_lib
 from neutron.common import utils
 from neutron.conf import policies
+from neutron.conf import quota as quota_conf
 from neutron.db import db_base_plugin_common
 from neutron.db import ipam_backend_mixin
 from neutron.db.models import l3 as l3_models
@@ -110,7 +111,10 @@ class NeutronDbPluginV2TestCase(testlib_api.WebTestCase):
 
     def setUp(self, plugin=None, service_plugins=None,
               ext_mgr=None):
-
+        quota_conf.register_quota_opts(quota_conf.core_quota_opts, cfg.CONF)
+        cfg.CONF.set_override(
+            'quota_driver', 'neutron.db.quota.driver.DbQuotaDriver',
+            group=quota_conf.QUOTAS_CFG_GROUP)
         super(NeutronDbPluginV2TestCase, self).setUp()
         cfg.CONF.set_override('notify_nova_on_port_status_changes', False)
         cfg.CONF.set_override('allow_overlapping_ips', True)

neutron/tests/unit/extensions/test_quotasv2.py

@@ -30,6 +30,7 @@ from neutron.api.v2 import router
 from neutron.common import config
 from neutron.conf import quota as qconf
 from neutron.db.quota import driver
+from neutron.db.quota import driver_nolock
 from neutron import quota
 from neutron.quota import resource_registry
 from neutron.tests import base
@@ -504,29 +505,24 @@ class TestDbQuotaDriver(base.BaseTestCase):
 
 
 class TestQuotaDriverLoad(base.BaseTestCase):
-    def setUp(self):
-        super(TestQuotaDriverLoad, self).setUp()
-        # Make sure QuotaEngine is reinitialized in each test.
-        quota.QUOTAS._driver = None
 
     def _test_quota_driver(self, cfg_driver, loaded_driver,
                            with_quota_db_module=True):
+        quota.QUOTAS._driver = None
         cfg.CONF.set_override('quota_driver', cfg_driver, group='QUOTAS')
         with mock.patch.dict(sys.modules, {}):
             if (not with_quota_db_module and
-                    'neutron.db.quota.driver' in sys.modules):
+                    quota.QUOTA_DB_MODULE in sys.modules):
-                del sys.modules['neutron.db.quota.driver']
+                del sys.modules[quota.QUOTA_DB_MODULE]
             driver = quota.QUOTAS.get_driver()
             self.assertEqual(loaded_driver, driver.__class__.__name__)
 
-    def test_quota_db_driver_with_quotas_table(self):
+    def test_quota_driver_load(self):
-        self._test_quota_driver('neutron.db.quota.driver.DbQuotaDriver',
+        for klass in (quota.ConfDriver, driver.DbQuotaDriver,
-                                'DbQuotaDriver', True)
+                      driver_nolock.DbQuotaNoLockDriver):
+            self._test_quota_driver(
+                '.'.join([klass.__module__, klass.__name__]),
+                klass.__name__, True)
 
-    def test_quota_db_driver_fallback_conf_driver(self):
+    def test_quota_driver_fallback_conf_driver(self):
-        self._test_quota_driver('neutron.db.quota.driver.DbQuotaDriver',
+        self._test_quota_driver(quota.QUOTA_DB_DRIVER, 'ConfDriver', False)
-                                'ConfDriver', False)
-
-    def test_quota_conf_driver(self):
-        self._test_quota_driver('neutron.quota.ConfDriver',
-                                'ConfDriver', True)

neutron/tests/unit/quota/test_resource.py

@@ -30,34 +30,44 @@ from neutron.tests.unit import testlib_api
 
 
 DB_PLUGIN_KLASS = 'neutron.db.db_base_plugin_v2.NeutronDbPluginV2'
+QUOTA_DRIVER = 'neutron.db.quota.DbQuotaDriver'
 
 
 meh_quota_flag = 'quota_meh'
 meh_quota_opts = [cfg.IntOpt(meh_quota_flag, default=99)]
 
 
+class _BaseResource(resource.BaseResource):
+
+    @property
+    def dirty(self):
+        return False
+
+    def count(self, context, plugin, project_id, **kwargs):
+        pass
+
+
 class TestResource(base.DietTestCase):
     """Unit tests for neutron.quota.resource.BaseResource"""
 
     def test_create_resource_without_plural_name(self):
-        res = resource.BaseResource('foo', None)
+        res = _BaseResource('foo', None)
         self.assertEqual('foos', res.plural_name)
-        res = resource.BaseResource('foy', None)
+        res = _BaseResource('foy', None)
         self.assertEqual('foies', res.plural_name)
 
     def test_create_resource_with_plural_name(self):
-        res = resource.BaseResource('foo', None,
+        res = _BaseResource('foo', None, plural_name='foopsies')
-                                    plural_name='foopsies')
         self.assertEqual('foopsies', res.plural_name)
 
     def test_resource_default_value(self):
-        res = resource.BaseResource('foo', 'foo_quota')
+        res = _BaseResource('foo', 'foo_quota')
         with mock.patch('oslo_config.cfg.CONF') as mock_cfg:
             mock_cfg.QUOTAS.foo_quota = 99
             self.assertEqual(99, res.default)
 
     def test_resource_negative_default_value(self):
-        res = resource.BaseResource('foo', 'foo_quota')
+        res = _BaseResource('foo', 'foo_quota')
         with mock.patch('oslo_config.cfg.CONF') as mock_cfg:
             mock_cfg.QUOTAS.foo_quota = -99
             self.assertEqual(-1, res.default)
@@ -94,6 +104,7 @@ class TestTrackedResource(testlib_api.SqlTestCase):
                 session.add(item)
 
     def setUp(self):
+        cfg.CONF.set_override('quota_driver', QUOTA_DRIVER, group='QUOTAS')
         super(TestTrackedResource, self).setUp()
         self.setup_coreplugin(DB_PLUGIN_KLASS)
         self.resource = 'meh'

neutron/tests/unit/quota/test_resource_registry.py

@@ -160,6 +160,7 @@ class TestAuxiliaryFunctions(base.DietTestCase):
                         'TrackedResource.mark_dirty') as mock_mark_dirty:
             self.registry.set_tracked_resource('meh', test_quota.MehModel)
             self.registry.register_resource_by_name('meh')
+            self.registry.resources['meh']._track_resource_events = True
             res = self.registry.get_resource('meh')
             # This ensures dirty is true
             res._dirty_tenants.add('tenant_id')

releasenotes/notes/new-quota-driver-DbQuotaNoLockDriver-5f8a44915ec16a1b.yaml

@@ -0,0 +1,11 @@
+---
+features:
+  - |
+    A new quota driver is added: ``DbQuotaNoLockDriver``. This driver, unlike
+    ``DbQuotaDriver``, does not create a unique lock per (resource,
+    project_id). That may lead to a database deadlock state if the number of
+    server requests exceeds the number of resolved resource creations, as
+    described in `LP#1926787 <https://bugs.launchpad.net/neutron/+bug/1926787>`_.
+    This driver relays on the database transactionality isolation and counts
+    the number of used and reserved resources and, if available, creates the
+    new resource reservations in one single database transaction.