This reverts commit bd7d991309ea2bea5d175cb1f2710519936fd0c2 and bumps
the minimum version of oslo.db to 4.40.0, as that is the first version
of the library to include the renamed attribute.
Change-Id: Ic9e7864be3af7ef362cad5648dfc7bdecd104465
Related-Bug: #1788833
In commit 2d532963, all instances of 'async' were replaced with 'async_'
in preparation for Python 3.7. However, one of these should not have
been changed as it refers to an oslo.db object attribute. That attribute
has actually been renamed itself but that rename is only present from
oslo.db 4.40.0 [1]. Thankfully, an alias to the older name is provided
so we use that.
[1] https://github.com/openstack/oslo.db/commit/df6bf34
Change-Id: I1afd0ba34a9ebcb63edb91e880ef60580befb32e
Closes-Bug: #1788833
There was a eventlet.monkey_patch [1] when we launch a nova
process(like nova-api, nova-compute), but it's invalid under
the uwsgi mode.
But there are concurrency requirements in the api service, such
as, when listing instances cross multiple cells we're using
greenthreads and oslo.db does a time.sleep to allow switching
greenthreads [2].
So, in this patch we add the monkey_patch in the uwsgi
application setup and refactor the monkey patching to use common
code.
[1] https://github.com/openstack/nova/blob/233ea58/nova/cmd/__init__.py#L26
[2] https://github.com/openstack/oslo.db/blob/9c66959/oslo_db/sqlalchemy/engines.py#L51
Closes-bug: #1787331
Change-Id: Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56
This patch modifies the code paths for the non-granular request group
allocation candidates processing. It removes the giant multi-join SQL
query and replaces it with multiple calls to
_get_providers_with_resource(), logging the number of matched providers
for each resource class requested and filter (on required traits,
forbidden traits and aggregate memebership).
Here are some examples of the debug output:
- A request for three resources with no aggregate or trait filters:
found 7 providers with available 5 VCPU
found 9 providers with available 1024 MEMORY_MB
found 5 providers after filtering by previous result
found 8 providers with available 1500 DISK_GB
found 2 providers after filtering by previous result
- The same request, but with a required trait that nobody has, shorts
out quickly:
found 0 providers after applying required traits filter (['HW_CPU_X86_AVX2'])
- A request for one resource with aggregates and forbidden (but no
required) traits:
found 2 providers after applying aggregates filter ([['3ed8fb2f-4793-46ee-a55b-fdf42cb392ca']])
found 1 providers after applying forbidden traits filter ([u'CUSTOM_TWO', u'CUSTOM_THREE'])
found 3 providers with available 4 VCPU
found 1 providers after applying initial aggregate and trait filters
Co-authored-by: Eric Fried <efried@us.ibm.com>
Closes-Bug: #1786519
Change-Id: If9ddb8a6d2f03392f3cc11136c4a0b026212b95b
This makes the instance_list module support batching across cells
with a couple of different strategies, and with room to add more
in the future.
Before this change, an instance list with limit 1000 to a
deployment with 10 cells would generate a query to each cell
database with the same limit. Thus, that API request could end
up processing up to 10,000 instance records despite only
returning 1000 to the user (because of the limit).
This uses the batch functionality in the base code added in
Iaa4759822e70b39bd735104d03d4deec988d35a1
by providing a couple of strategies by which the batch size
per cell can be determined. These should provide a lot of gain
in the short term, and we can extend them with other strategies
as we identify some with additional benefits.
Closes-Bug: #1787977
Change-Id: Ie3a5f5dc49f8d9a4b96f1e97f8a6ea0b5738b768
Add a test that traverses all available placement URLs at the latest
microversion and tries to access them as non-admin. If something other
than a 403 response is given a failed test with a message like
method POST on route /resource_providers/{uuid}/inventories
is open for user, status: 404
is produced.
This works because we do authZ handling early in each handler, before
data processing and path parameter handling.
The method is pretty straightforward: traverse ROUTE_DECLARATIONS, visit
every url with each the declared methods, except the root version document,
and confirm a 403 response when the provided auth token is non-admin.
This has been created to avoid situations where a route is added without
policy like happened on https://review.openstack.org/#/c/576927/ . Until
recently we had a failover where any route not defined to have policy
would default to admin. That went away so now we need some test
automation to catch our forgetful humanness.
Change-Id: Id582886ec4b621b97d7cc7237b4670ad7bb12295
When creating resource provider with '--uuid' argument, nova
accept uuid without dash('-') too, which some time results in,
resource provider with same uuid i.e one with dash and one without.
This patch attempts to fix it by transforming dashless UUID into
dashed one before inserting it into the database.
Co-Authored-By: Chen <dstbtgagt@foxmail.com>
Change-Id: I2685eb65907adbd22b2d09264b110692e100eaf9
Closes-Bug: #1758057
Factor out some regexes used across multiple schemata into a new
nova.api.openstack.placement.schemas.common module.
The UUID part of this is in preparation for fixing the related bug.
Change-Id: If62bfeeb32d0ad77dd1205116ee4e5e844bb07e4
Related-Bug: #1758057
Without this change, tests can intermittently fail with NoSuchOptError
when a single process does not have the ConfigFixture running before
this test. This change ensure the opts are registered and defaulted.
Change-Id: I6a4873726e3e7fe0d4db3d1dea61309702b8f24b
Closes-Bug: #1788176
Without this change, tests can intermittently fail with NoSuchOptError
when a single process does not have other tests running prior to
gabbi tests. This change ensure the opts are registered and defaulted.
Change-Id: I1c7e347b6e788928bef96e32c3365d0fdc5ba00f
Related-Bug: #1786498
Closes-Bug: #1788176
ChanceScheduler is deprecated in Pike [1] and will be removed in a
subsequent release.
[1] https://review.openstack.org/#/c/492210/
Change-Id: I44f9c1cabf9fc64b1a6903236bc88f5ed8619e9e
This patch will replace all GB with GiB and MB with MiB
to have a consistent use of units in the compute API reference
and the placement API reference.
Co-Authored-By: Takashi Natsume <natsume.takashi@lab.ntt.co.jp>
Change-Id: Ie40413752b591b222ff29dbe975ddd7d10638eca
Closes-Bug: #1752340
When replacing a provider's set of aggregate associations, we were
issuing a call to:
DELETE resource_provider_aggregates WHERE resource_provider_id = $rp
and then a single call to:
INSERT INTO resource_provider_aggregates
SELECT $rp, aggs.id
FROM provider_aggregates AS aggs
WHERE aggs.uuid IN ($agg_uuids)
This patch changes the _set_aggregates() function in a few ways.
First, we grab the aggregate's internal ID value when creating new
aggregate records (or grabbing a provider's existing aggregate
associations). This eliminates the need for any join to
provider_aggregates in an INSERT/DELETE statement.
Second, instead of a multi-row INSERT .. SELECT statement, we do
single-shot INSERT ... VALUES statements, one for each added aggregate.
Third, we no longer DELETE all aggregate associations for the provider
in question. Instead, we issue single-shot DELETE statements for only
the aggregates that are being disassociated.
Finally, I've added a number of log debug statements so that we can have
a little more information if this particular patch does not fix the
deadlock issue described in the associated bug.
Change-Id: I87e765305017eae1424005f7d6f419f42a2f8370
Closes-bug: #1786703
Xen UT failures seem to have cleared up, which may or may not have
anything to do with [1]. Remove the blacklist file and get these back
in the py3 jobs.
[1] https://review.openstack.org/#/c/591061/
Change-Id: Iacc1ae01f535caed64793d57b757ea07a8c46620
This change adds a post test hook to the nova-next job to report
timing of a query to GET /allocation_candidates when there are 1000
resource providers with the same inventory.
A summary of the work ends up in logs/placement-perf.txt
Change-Id: Idc446347cd8773f579b23c96235348d8e10ea3f6
About "PUT /resource_providers/{uuid}/aggregates"
in the Placement API reference,
the example of the response body in microversion 1.1 - 1.18 is missing.
So add it.
Change-Id: Ic3dc665124ae7927cfa20cf70a08c39b856b8961
Closes-Bug: #1786759
Somewhere in the past release, we started using extremely complex code
paths involving sharing providers, anchor providers, and nested resource
provider calculations when we absolutely don't need to do so.
There was a _has_provider_trees() function in the
nova/api/openstack/placement/objects/resource_provider.py file that used
to be used for top-level switching between a faster, simpler approach to
finding allocation candidates for a simple search of resources and
traits when no sharing providers and no nesting was used. That was
removed at some point and all code paths -- even for simple "get me
these amounts of these resources" when no trees or sharing providers are
present (which is the vast majority of OpenStack deployments) -- were
going through the complex tree-search-and-match queries and algorithms.
This patch changes that so that when there's a request for some
resources and there's no trees or sharing providers, we do the simple
code path. Hopefully this gets our performance for the simple, common
cases back to where we were pre-Rocky.
This change is a prerequisite for the following change which adds
debugging output to help diagnose which resource classes are running
out of inventory when GET /allocation_candidates returns 0 results.
That code is not possible without the changes here as they only
work if we can identify when a "simpler approach" is possible and
call that simpler code.
Related-Bug: #1786055
Partial-Bug: #1786519
Change-Id: I1fdbcdb7a1dd51e738924c8a30238237d7ac74e1
This patch adds a test for _get_provider_ids_matching()
to verify it works correctly with required traits.
Related-Bug: #1786519
Change-Id: I2512e361f5eaa4e60701be7c8bf57b2e0a02a146
Rather than spam the placement API logs with scope check
warnings from oslo.policy when placement isn't configured
for scope type enforcement, ignore those warnings.
Note that the same warnings filter in the placement WarningsFixture
is left intact because the configuration defaults setting code in
wsgi.py which allows the warning filter to be set in deploy.py,
is not run by the functional tests. In future changes this will
be resolved by unifying configuration handling.
Closes-Bug: #1786498
Related-Bug: #1784663
Change-Id: I34e4e550c9c31a654308e555210588156418f9e3
Since [1] was fixed we don't need to manually patch the mock lib to
avoid silently passing mistyped assert methods on mocks.
[1] https://bugs.python.org/issue21238
Change-Id: Iecf4dcf8e648c9191bf8846428683ec81812c026
Mention that if any nonexistent aggregate uuids are provided, those
aggregates will be created automatically first during the update call.
I think this is a good addition since in fact I was much confused how
the aggregates can be created in the first place when playing with
the placement APIs.
Change-Id: I2f86715a21784eb974b73fc3edc74eedd73ad48b
In the placement API reference, there is a separate section for
"microversions 1.19 -" for both List and Update resource provider
aggregates, so there's no need for the resource_provider_generation
field to mention that it's "New in version 1.19" in those sections.
Change-Id: I3fd02f21d4af11633c4aeb33925c279a1c3b4abd
Add description for nested resource providers
in allocation candidates, Placement API microversion 1.29
in the Placement API reference.
Change-Id: I22c2068ae860c1ef2cbdb0801e85a40acc2a097a
Implements: blueprint nested-resource-providers-allocation-candidates