Now that we are python3 only, we should move to using the built
in version of mock that supports all of our testing needs and
remove the dependency on the "mock" package.
This patch moves all references to "import mock" to
"from unittest import mock". It also cleans up some new line
inconsistency.
Fixed an inconsistency in the OVSBridge.deferred() definition
as it needs to also have an *args argument.
Fixed an issue where an l3-agent test was mocking
functools.partial, causing a python3.8 failure.
Unit tests only, removing from tests/base.py affects
functional tests which need additional work.
Change-Id: I40e8a8410840c3774c72ae1a8054574445d66ece
The get_reader_session and get_writer_session functions are available
in neutron-lib. This patch consumes them by removing the functions
from neutron and using lib's instead.
NeutronLibImpact
Change-Id: I4055172799a0d9a0ceb387b6d47e421e1c270ada
It's of no guarantee that core plugin implements counter/getter function
for a CountableResource. Instead of just trying core plugin, try every
plugin registered in the directory.
To retain backwards compatibility, we also make sure that core plugin is
checked first.
Change-Id: I5245e217e1f44281f85febbdfaf873321253dc5d
Closes-Bug: #1714769
Extend existing quota api to report a quota set. The quota set
will contain a set of resources and its corresponding reservation,
limits and in_use count for each tenant.
DocImpact:Documentation describing the new API as well as the new
information that it exposes.
APIImpact
Co-Authored-By: Prince Boateng<prince.a.owusu.boateng@intel.com>
Change-Id: Ief2a6a4d2d7085e2a9dcd901123bc4fe6ac7ca22
Related-bug: #1599488
This is achieved by switching the test class to BaseTestCase and using
setup_coreplugin. This makes later calls to neutron.objects.db.api
functions that extract the plugin from plugins directory to work.
The test class registers some sqlalchemy events for models not
registered in BASEV2, which later makes unregister_events fail because
of failure to compare those model classes with other registered model
classes (those that belong to proper model classes like Port). Before
the patch, the core plugin was not correctly registered in plugins
directory, and no code was actually loading the plugin, so no other
sqla events were in _REGISTERED_SQLA_EVENTS, which explains why
previously calls to unregister_events hasn't failed.
Good news is that the new parent class does clean up all events itself,
so we don't need to set the cleanup hook ourselves.
This switch also facilitated removal of some other preparatory steps
from the class that are already implemented in the new parent.
Closes-Bug: #1677636
Change-Id: Id462e532066ac3cd5e2f76c3670da7f7820c6581
This patch introduces and integrates Oslo-Versioned Object for
ResourceDelta, Reservation, Quota and QuotaUsage model classes.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Co-Authored-By: Victor Morales <victor.morales@intel.com>
Change-Id: Ic058e66e6780e1f6ff25a5b2bbe7294959905765
Introduce get_reader_session() and get_writer_session()
and replace get_session() with them.
Mark get_session as depricated.
Stop using get_engine from legacy facade. Use writer engine for
places where it is required.
Partially-Implements blueprint: enginefacade-switch
Change-Id: I28b741bfa27bf04cbe273586e6e3e00e14fbe683
The devref was suggesting to do exactly the wrong thing for cases where
we have SQLAlchemy event listeners waiting for deleted objects.
DELETE statements fail to trigger any kind of SQLAlchemy event
listeners and the majority of our delete operations are small
(<10 items at once) so the performance difference is negligible
(especially since we do it so infrequently).
Change-Id: Id142179418a7b94d8d9695871b3fcd5dcc64730c
Openstack common has a wrapper for generating uuids.We should
use that function when generating uuids for consistency.
Change-Id: Id13e01ba59f37a5be90db1fe778e9735f0e8b9b6
Closes-Bug: #1082248
If a resource was registered in the quota engine before
being set as a tracked resource, it would have been registered
as a CountableResource instead of a TrackedResource.
This would cause a failure later when the resources were
being unregistered because the quota engine thought it
should have been a TrackedResource and tried to call
unregister_events on it.
This patch just prevents resources from being registered
as both types. It also callse unregister_all_resources as
part of the standard test case cleanup since the vast majority
of tests are using a plugin that may register resources.
Closes-Bug: #1612222
Change-Id: I8a40f38d7c0e5aeca257ba62115fa9b02ad5aa93
All occurences of ``tenant_id`` across the database are renamed
to ``project_id``. Both options are equally valid, but ``project_id``
is preferred.
To inform external users about the change, HasTenant class was
deprecated.
UpgradeImpact
Partially-Implements: blueprint keystone-v3
Change-Id: I87a8ef342ccea004731ba0192b23a8e79bc382dc
I saw a patch set [1] that was running in to circular imports and
removing these. I thought it'd be better to remove them all in one
shot.
[1] https://review.openstack.org/#/c/261222/1/neutron/db/agents_db.py
Change-Id: If1a9ba3091ac99f01f73b98a7459358c3efd4dd5
Went through all of the docstrings in Neutron and did
some cleanup. I'm sure there are bunch more that I have missed.
Change-Id: Ib29d2de1c580880c89ed4fd069e1515d0977a3e7
As previously implemented, the TestTrackedResource class is designed
to inject random failures into the gate. It generates random numbers
within the range of 0..10000, and will fail if it generates duplicate
random numbers during its run.
This patch creates UUIDs instead of random numbers, and makes the
chance of an collision vanishingly small.
Change-Id: I0cf535d1c5a3995a50b506aafce10e983872dcb7
Closes-bug: #1494021
This patch deals with the lock wait timeout and the deadlock errors
observed under high concurrency (api_workers >= 4) with the pymysql
driver. It includes the following changes:
- Stop setting dirty status for resource usage when creating
reservation, as usage of reserved resources is not tracked anymore;
- Add a variable, increasing delay when retrying make_reservation
upon a DBDeadlock error in order to reduce the chances of further
collisions;
- Enable transaction retry upon DBDeadlock errors for set_quota_usage;
- Do not resync quota usage while making reservation. This puts a lot
of stress on the database and is also wasteful since resource usage
is very likely to change again once the transaction is committed;
- Use autonested_transaction to simplify logic around when the
nested flag should be used.
Change-Id: I7a335f9ebea3c0d6fee6e6b757554e045a66075c
Closes-Bug: #1486134
Related-Blueprint: better-quotas
Reservations have a transient nature: a reservation lifespan
typically begins and ends with a single request.
Therefore tracking reserved amounts for each tenant and resource
is not nearly as efficient as tracking resource usage.
Indeed it is fairly easy to verify that the overhead for tracking
reserved amounts is much greater than the one needed for counting
active reservations for each tenant and resource.
This patch removes the logic for tracking reservations, and
replaces it with an explicit count of active reservations.
Please note that this patch does not adjust accordingly the
ResourceUsage DB model. This will be done in a separate patch with
an expand migration; this should avoid most merge conflicts before
the final patch for restoring reservation logic merges.
Related-Blueprint: better-quotas
Change-Id: Ib5e3bd61c1bc0fc8a5d612dae5c1740a8834a980
Add the concept of resource reservation in neutron.
Usage tracking logic is also updated to support reservations.
Reservations are not however available with the now deprecated
configuration-based quota driver.
The base API controller will now use reservations to perform
quota checks rather than counting resource usage and then
invoking the limit_check routine.
The limit_check routine however has not been removed and
depreacated as a part of this patch. In order to ensure all
quota drivers expose a consistent interface, a
make_reservation method has been added to the configuration
based driver as well. This method simply performs "old-style"
limit checks by counting resource usage and then invoking
limit_check.
DocImpact
Implements blueprint better-quotas.
Change-Id: Ifea07f461def564884af5b291c8a56655a4d818b
Introduce a 'plural_name' attribute in the BaseResource
class in the neutron.quota.resource module. This will
enable to specify the plural name of a resource when it's
entered in the quota registry.
This will come handy for managing reservations and also
removes the need for passing the collection name when
counting a resource.
Also, the name of the 'resources' parameter for the
_count_resource routine has ben changed to collection_name,
as the previous one was misleading, since it led to believe
it should be used for a collection, whereas it is simply
the name of a collection.
Change-Id: I0b61ba1856e9552cc14574be28fec6c77fb8783c
Related-Blueprint: better-quotas
This patch introduces application logic support for tracking
Neutron resource usage data, thus introducing a different
way of enforcing quota limits, which now relies on records
tracking resource usage for each tenant.
When these records go "out-of-sync" with actual resource usage,
the quota usage table entry is marked "dirty".
And it will be resynchronized the next time a resource count is
requested. Changes in resource utilization are detected using
SQLAlchemy events.
This patch however does not automatically enable resource usage
tracking for any plugin. Plugins must explicitly declare for which
resources they wish to enable tracking. To this aim, this patch
provides the @tracked_resources decorator.
As some operators might wish to not use resource usage tracking, this
patch adds the track_quota_usage configuration option. If set to
False, DB-level usage tracking will not be performed, and
resources will be counted as before, ie: invoking a plugin method.
Usage tracking is performed using a new resource type,
TrackedResource, which can work side by side with CountableResource,
the one Neutron used so far. To this aim a ResourceRegistry class
has been introduced for registering and managing resources. Most
of the resource management code previously embedded in the
QuotaEngine class is being moved to ResourceRegistry as a part of
this patch.
Partially implements blueprint better-quota
Change-Id: If461399900973a38d7b82e0b3ac54fc052f09529
Brian Haley