Now that we are python3 only, we should move to using the built
in version of mock that supports all of our testing needs and
remove the dependency on the "mock" package.
This patch moves all references to "import mock" to
"from unittest import mock". It also cleans up some new line
Fixed an inconsistency in the OVSBridge.deferred() definition
as it needs to also have an *args argument.
Fixed an issue where an l3-agent test was mocking
functools.partial, causing a python3.8 failure.
Unit tests only, removing from tests/base.py affects
functional tests which need additional work.
The get_reader_session and get_writer_session functions are available
in neutron-lib. This patch consumes them by removing the functions
from neutron and using lib's instead.
It's of no guarantee that core plugin implements counter/getter function
for a CountableResource. Instead of just trying core plugin, try every
plugin registered in the directory.
To retain backwards compatibility, we also make sure that core plugin is
Extend existing quota api to report a quota set. The quota set
will contain a set of resources and its corresponding reservation,
limits and in_use count for each tenant.
DocImpact:Documentation describing the new API as well as the new
information that it exposes.
Co-Authored-By: Prince Boateng<email@example.com>
This is achieved by switching the test class to BaseTestCase and using
setup_coreplugin. This makes later calls to neutron.objects.db.api
functions that extract the plugin from plugins directory to work.
The test class registers some sqlalchemy events for models not
registered in BASEV2, which later makes unregister_events fail because
of failure to compare those model classes with other registered model
classes (those that belong to proper model classes like Port). Before
the patch, the core plugin was not correctly registered in plugins
directory, and no code was actually loading the plugin, so no other
sqla events were in _REGISTERED_SQLA_EVENTS, which explains why
previously calls to unregister_events hasn't failed.
Good news is that the new parent class does clean up all events itself,
so we don't need to set the cleanup hook ourselves.
This switch also facilitated removal of some other preparatory steps
from the class that are already implemented in the new parent.
This patch introduces and integrates Oslo-Versioned Object for
ResourceDelta, Reservation, Quota and QuotaUsage model classes.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Co-Authored-By: Victor Morales <firstname.lastname@example.org>
Introduce get_reader_session() and get_writer_session()
and replace get_session() with them.
Mark get_session as depricated.
Stop using get_engine from legacy facade. Use writer engine for
places where it is required.
Partially-Implements blueprint: enginefacade-switch
The devref was suggesting to do exactly the wrong thing for cases where
we have SQLAlchemy event listeners waiting for deleted objects.
DELETE statements fail to trigger any kind of SQLAlchemy event
listeners and the majority of our delete operations are small
(<10 items at once) so the performance difference is negligible
(especially since we do it so infrequently).
If a resource was registered in the quota engine before
being set as a tracked resource, it would have been registered
as a CountableResource instead of a TrackedResource.
This would cause a failure later when the resources were
being unregistered because the quota engine thought it
should have been a TrackedResource and tried to call
unregister_events on it.
This patch just prevents resources from being registered
as both types. It also callse unregister_all_resources as
part of the standard test case cleanup since the vast majority
of tests are using a plugin that may register resources.
All occurences of ``tenant_id`` across the database are renamed
to ``project_id``. Both options are equally valid, but ``project_id``
To inform external users about the change, HasTenant class was
Partially-Implements: blueprint keystone-v3
As previously implemented, the TestTrackedResource class is designed
to inject random failures into the gate. It generates random numbers
within the range of 0..10000, and will fail if it generates duplicate
random numbers during its run.
This patch creates UUIDs instead of random numbers, and makes the
chance of an collision vanishingly small.
This patch deals with the lock wait timeout and the deadlock errors
observed under high concurrency (api_workers >= 4) with the pymysql
driver. It includes the following changes:
- Stop setting dirty status for resource usage when creating
reservation, as usage of reserved resources is not tracked anymore;
- Add a variable, increasing delay when retrying make_reservation
upon a DBDeadlock error in order to reduce the chances of further
- Enable transaction retry upon DBDeadlock errors for set_quota_usage;
- Do not resync quota usage while making reservation. This puts a lot
of stress on the database and is also wasteful since resource usage
is very likely to change again once the transaction is committed;
- Use autonested_transaction to simplify logic around when the
nested flag should be used.
Reservations have a transient nature: a reservation lifespan
typically begins and ends with a single request.
Therefore tracking reserved amounts for each tenant and resource
is not nearly as efficient as tracking resource usage.
Indeed it is fairly easy to verify that the overhead for tracking
reserved amounts is much greater than the one needed for counting
active reservations for each tenant and resource.
This patch removes the logic for tracking reservations, and
replaces it with an explicit count of active reservations.
Please note that this patch does not adjust accordingly the
ResourceUsage DB model. This will be done in a separate patch with
an expand migration; this should avoid most merge conflicts before
the final patch for restoring reservation logic merges.
Add the concept of resource reservation in neutron.
Usage tracking logic is also updated to support reservations.
Reservations are not however available with the now deprecated
configuration-based quota driver.
The base API controller will now use reservations to perform
quota checks rather than counting resource usage and then
invoking the limit_check routine.
The limit_check routine however has not been removed and
depreacated as a part of this patch. In order to ensure all
quota drivers expose a consistent interface, a
make_reservation method has been added to the configuration
based driver as well. This method simply performs "old-style"
limit checks by counting resource usage and then invoking
Implements blueprint better-quotas.
Introduce a 'plural_name' attribute in the BaseResource
class in the neutron.quota.resource module. This will
enable to specify the plural name of a resource when it's
entered in the quota registry.
This will come handy for managing reservations and also
removes the need for passing the collection name when
counting a resource.
Also, the name of the 'resources' parameter for the
_count_resource routine has ben changed to collection_name,
as the previous one was misleading, since it led to believe
it should be used for a collection, whereas it is simply
the name of a collection.
This patch introduces application logic support for tracking
Neutron resource usage data, thus introducing a different
way of enforcing quota limits, which now relies on records
tracking resource usage for each tenant.
When these records go "out-of-sync" with actual resource usage,
the quota usage table entry is marked "dirty".
And it will be resynchronized the next time a resource count is
requested. Changes in resource utilization are detected using
This patch however does not automatically enable resource usage
tracking for any plugin. Plugins must explicitly declare for which
resources they wish to enable tracking. To this aim, this patch
provides the @tracked_resources decorator.
As some operators might wish to not use resource usage tracking, this
patch adds the track_quota_usage configuration option. If set to
False, DB-level usage tracking will not be performed, and
resources will be counted as before, ie: invoking a plugin method.
Usage tracking is performed using a new resource type,
TrackedResource, which can work side by side with CountableResource,
the one Neutron used so far. To this aim a ResourceRegistry class
has been introduced for registering and managing resources. Most
of the resource management code previously embedded in the
QuotaEngine class is being moved to ResourceRegistry as a part of
Partially implements blueprint better-quota