When replacing a provider's set of aggregate associations, we were
issuing a call to:
DELETE resource_provider_aggregates WHERE resource_provider_id = $rp
and then a single call to:
INSERT INTO resource_provider_aggregates
SELECT $rp, aggs.id
FROM provider_aggregates AS aggs
WHERE aggs.uuid IN ($agg_uuids)
This patch changes the _set_aggregates() function in a few ways.
First, we grab the aggregate's internal ID value when creating new
aggregate records (or grabbing a provider's existing aggregate
associations). This eliminates the need for any join to
provider_aggregates in an INSERT/DELETE statement.
Second, instead of a multi-row INSERT .. SELECT statement, we do
single-shot INSERT ... VALUES statements, one for each added aggregate.
Third, we no longer DELETE all aggregate associations for the provider
in question. Instead, we issue single-shot DELETE statements for only
the aggregates that are being disassociated.
Finally, I've added a number of log debug statements so that we can have
a little more information if this particular patch does not fix the
deadlock issue described in the associated bug.
Change-Id: I87e765305017eae1424005f7d6f419f42a2f8370
Closes-bug: #1786703
This updates the EXPLANATION and sets the pinned version placeload
to the just release 0.3.0. This ought to hold us for a while. If
we need to do this again, we should probably switch to using
requirements files in some fashion, but I'm hoping we can avoid
that until later, potentially even after placement extraction
when we will have to moving and changing this anyway.
Change-Id: Ia3383c5dbbf8445254df774dc6ad23f2b9a3721e
The pirate on crack output of placeload can be confusing
so this change adds a prefix to the placement-perf.txt log
file so that it is somewhat more self-explanatory.
This change also pins the version of placeload because the
explanation is version dependent.
Change-Id: I055adb5f6004c93109b17db8313a7fef85538217
Xen UT failures seem to have cleared up, which may or may not have
anything to do with [1]. Remove the blacklist file and get these back
in the py3 jobs.
[1] https://review.openstack.org/#/c/591061/
Change-Id: Iacc1ae01f535caed64793d57b757ea07a8c46620
This change adds a post test hook to the nova-next job to report
timing of a query to GET /allocation_candidates when there are 1000
resource providers with the same inventory.
A summary of the work ends up in logs/placement-perf.txt
Change-Id: Idc446347cd8773f579b23c96235348d8e10ea3f6
About "PUT /resource_providers/{uuid}/aggregates"
in the Placement API reference,
the example of the response body in microversion 1.1 - 1.18 is missing.
So add it.
Change-Id: Ic3dc665124ae7927cfa20cf70a08c39b856b8961
Closes-Bug: #1786759
Somewhere in the past release, we started using extremely complex code
paths involving sharing providers, anchor providers, and nested resource
provider calculations when we absolutely don't need to do so.
There was a _has_provider_trees() function in the
nova/api/openstack/placement/objects/resource_provider.py file that used
to be used for top-level switching between a faster, simpler approach to
finding allocation candidates for a simple search of resources and
traits when no sharing providers and no nesting was used. That was
removed at some point and all code paths -- even for simple "get me
these amounts of these resources" when no trees or sharing providers are
present (which is the vast majority of OpenStack deployments) -- were
going through the complex tree-search-and-match queries and algorithms.
This patch changes that so that when there's a request for some
resources and there's no trees or sharing providers, we do the simple
code path. Hopefully this gets our performance for the simple, common
cases back to where we were pre-Rocky.
This change is a prerequisite for the following change which adds
debugging output to help diagnose which resource classes are running
out of inventory when GET /allocation_candidates returns 0 results.
That code is not possible without the changes here as they only
work if we can identify when a "simpler approach" is possible and
call that simpler code.
Related-Bug: #1786055
Partial-Bug: #1786519
Change-Id: I1fdbcdb7a1dd51e738924c8a30238237d7ac74e1
This patch adds a test for _get_provider_ids_matching()
to verify it works correctly with required traits.
Related-Bug: #1786519
Change-Id: I2512e361f5eaa4e60701be7c8bf57b2e0a02a146
Rather than spam the placement API logs with scope check
warnings from oslo.policy when placement isn't configured
for scope type enforcement, ignore those warnings.
Note that the same warnings filter in the placement WarningsFixture
is left intact because the configuration defaults setting code in
wsgi.py which allows the warning filter to be set in deploy.py,
is not run by the functional tests. In future changes this will
be resolved by unifying configuration handling.
Closes-Bug: #1786498
Related-Bug: #1784663
Change-Id: I34e4e550c9c31a654308e555210588156418f9e3
Since [1] was fixed we don't need to manually patch the mock lib to
avoid silently passing mistyped assert methods on mocks.
[1] https://bugs.python.org/issue21238
Change-Id: Iecf4dcf8e648c9191bf8846428683ec81812c026
Mention that if any nonexistent aggregate uuids are provided, those
aggregates will be created automatically first during the update call.
I think this is a good addition since in fact I was much confused how
the aggregates can be created in the first place when playing with
the placement APIs.
Change-Id: I2f86715a21784eb974b73fc3edc74eedd73ad48b
In the placement API reference, there is a separate section for
"microversions 1.19 -" for both List and Update resource provider
aggregates, so there's no need for the resource_provider_generation
field to mention that it's "New in version 1.19" in those sections.
Change-Id: I3fd02f21d4af11633c4aeb33925c279a1c3b4abd
Add description for nested resource providers
in allocation candidates, Placement API microversion 1.29
in the Placement API reference.
Change-Id: I22c2068ae860c1ef2cbdb0801e85a40acc2a097a
Implements: blueprint nested-resource-providers-allocation-candidates
We were calling ResourceProvider.get_by_uuid() inside
_build_provider_summaries() main loop over all providers involved in the
resulting allocation candidates. This results in a query per provider
involved, which is quite obviously not going to perform well. This patch
modifies the _build_provider_summaries() function to make a single call
to a new _provider_ids_from_rp_ids() function instead of multiple calls
to ResourceProvider.get_by_uuid().
Change-Id: I0e0a44e833afece0775ec712fbdf9fcf4eae7a93
Related-bug: #1786055
We already have a fully loaded resource provider object in the loop, so
we don't need to create another one. Doing so has a very large
performance impact, especially when there are many resource providers
in the collection of summaries (which will be true in a large and
sparsely used cloud).
The code which creates the summaries used here as a data source has the
same expensive use of get_by_uuid in a loop. That will be fixed in a
separate patch.
Existing functional tests cover this code.
Change-Id: I6068db78240c33a1dcefedc0c94e76740fd8d6e2
Partial-Bug: #1786055
Without this, you can successfully run a 'functional-py36' environment,
but no actual tests are run. Python 3.6 is common in many environments
these days, and we want devs to test locally. This helps.
Change-Id: I2c04b7a8b6ee06638c6ce92dc51e200fa2b13539
Due to a regression in I8624e194fe0173531c5aa2119c903e3c68b8c6cd, the
min_version on the parent_provider_uuid response parameter was missing
and is added back here. For sanity reasons, the variables are renamed
and substitution anchor usage is dropped.
To recap the microversions:
1.14
----
parent_provider_uuid and root_provider_uuid were added to the
responses for:
* GET /resource_providers (list)
* GET /resource_providers/{uuid} (show)
* PUT /resource_providers/{uuid} (update)
parent_provider_uuid was added to the request for:
* POST /resource_providers (create)
* PUT /resource_providers/{uuid} (update)
1.20
----
We started returning a response body from
POST /resource_providers (create) which included
parent_provider_uuid and root_provider_uuid but the parameters
are not versioned in that case since the entire response is
versioned for 1.20.
Change-Id: I6c9a1072fa2a59e408472f1a4b52b3a8c6a9b343
Closes-Bug: #1779700
Add 'versionadded' directives ("New in version Rocky.")
in microversion 1.28 and 1.29
in the placement API history document.
TrivialFix
Change-Id: If91da9d59938ecaca20f37aa25941d863299a88a
For code refactoring purpose in the placement granular fixture
setup, this patch substitutes common functions in test_base.py
for existing local functions of _create_providers(),
_add_inventory(), and _set_traits().
Change-Id: I76a3f7d7e446e0f3af379f83c9d8333279884c73
tempest-full-py3 job runs on nova check and gate pipeline
as part of 'integrated-gate-py35' template.
Unlike tempest-full or other job, nova job definition list on
project-config side[1] does not define the irrelevant-files for
tempest-full-py job which leads to run this job on doc/test file
only change also.
This commit defines the irrelevant-files for tempest-full-py3 on
nova's zuul.yaml.
Closes-Bug: #1785425
[1] 57907fbf04/zuul.d/projects.yaml (L9283)
Change-Id: I887177c078a53c53e84289a6b134d7ea357dfbef
