10618 Commits

Author SHA1 Message Date
Stephen Finucane
4df41cb4b3 Revert "Don't use '_TransactionContextManager._async'"
This reverts commit bd7d991309ea2bea5d175cb1f2710519936fd0c2 and bumps
the minimum version of oslo.db to 4.40.0, as that is the first version
of the library to include the renamed attribute.

Change-Id: Ic9e7864be3af7ef362cad5648dfc7bdecd104465
Related-Bug: #1788833
2018-08-28 17:18:51 +01:00
Stephen Finucane
37fb0708a5 Don't use '_TransactionContextManager._async'
In commit 2d532963, all instances of 'async' were replaced with 'async_'
in preparation for Python 3.7. However, one of these should not have
been changed as it refers to an oslo.db object attribute. That attribute
has actually been renamed itself but that rename is only present from
oslo.db 4.40.0 [1]. Thankfully, an alias to the older name is provided
so we use that.

[1] https://github.com/openstack/oslo.db/commit/df6bf34

Change-Id: I1afd0ba34a9ebcb63edb91e880ef60580befb32e
Closes-Bug: #1788833
2018-08-28 17:14:19 +01:00
Zuul
4f7c9b096a Merge "Make monkey patch work in uWSGI mode" 2018-08-28 15:38:04 +00:00
Zuul
931dc02d47 Merge "Make instance_list perform per-cell batching" 2018-08-28 04:55:17 +00:00
Yikun Jiang
b26f21df3d Make monkey patch work in uWSGI mode
There was a eventlet.monkey_patch [1] when we launch a nova
process(like nova-api, nova-compute), but it's invalid under
the uwsgi mode.

But there are concurrency requirements in the api service, such
as, when listing instances cross multiple cells we're using
greenthreads and oslo.db does a time.sleep to allow switching
greenthreads [2].

So, in this patch we add the monkey_patch in the uwsgi
application setup and refactor the monkey patching to use common
code.

[1] https://github.com/openstack/nova/blob/233ea58/nova/cmd/__init__.py#L26
[2] https://github.com/openstack/oslo.db/blob/9c66959/oslo_db/sqlalchemy/engines.py#L51

Closes-bug: #1787331

Change-Id: Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56
2018-08-28 09:49:01 +08:00
Zuul
ea73ab01f1 Merge "[placement] split gigantor SQL query, add logging" 2018-08-27 18:16:28 +00:00
Zuul
df776affc3 Merge "Fix create_resource_provider docstring" 2018-08-27 16:01:14 +00:00
Jay Pipes
10e7228707 [placement] split gigantor SQL query, add logging
This patch modifies the code paths for the non-granular request group
allocation candidates processing. It removes the giant multi-join SQL
query and replaces it with multiple calls to
_get_providers_with_resource(), logging the number of matched providers
for each resource class requested and filter (on required traits,
forbidden traits and aggregate memebership).

Here are some examples of the debug output:

- A request for three resources with no aggregate or trait filters:

 found 7 providers with available 5 VCPU
 found 9 providers with available 1024 MEMORY_MB
 found 5 providers after filtering by previous result
 found 8 providers with available 1500 DISK_GB
 found 2 providers after filtering by previous result

- The same request, but with a required trait that nobody has, shorts
  out quickly:

 found 0 providers after applying required traits filter (['HW_CPU_X86_AVX2'])

- A request for one resource with aggregates and forbidden (but no
  required) traits:

 found 2 providers after applying aggregates filter ([['3ed8fb2f-4793-46ee-a55b-fdf42cb392ca']])
 found 1 providers after applying forbidden traits filter ([u'CUSTOM_TWO', u'CUSTOM_THREE'])
 found 3 providers with available 4 VCPU
 found 1 providers after applying initial aggregate and trait filters

Co-authored-by: Eric Fried <efried@us.ibm.com>
Closes-Bug: #1786519
Change-Id: If9ddb8a6d2f03392f3cc11136c4a0b026212b95b
2018-08-27 13:50:22 +00:00
Dan Smith
a8f451a7e4 Make instance_list perform per-cell batching
This makes the instance_list module support batching across cells
with a couple of different strategies, and with room to add more
in the future.

Before this change, an instance list with limit 1000 to a
deployment with 10 cells would generate a query to each cell
database with the same limit. Thus, that API request could end
up processing up to 10,000 instance records despite only
returning 1000 to the user (because of the limit).

This uses the batch functionality in the base code added in
Iaa4759822e70b39bd735104d03d4deec988d35a1
by providing a couple of strategies by which the batch size
per cell can be determined. These should provide a lot of gain
in the short term, and we can extend them with other strategies
as we identify some with additional benefits.

Closes-Bug: #1787977
Change-Id: Ie3a5f5dc49f8d9a4b96f1e97f8a6ea0b5738b768
2018-08-27 06:44:32 -07:00
Zuul
09afc32669 Merge "[placement] Add functional test to verify presence of policy" 2018-08-23 21:30:55 +00:00
Zuul
cd2c9bd5c2 Merge "Normalize dashless 'resource provider create' uuid" 2018-08-23 21:16:05 +00:00
Zuul
5aa493b858 Merge "placement: use single-shot INSERT/DELETE agg" 2018-08-23 16:08:29 +00:00
Chen
675b111469 Fix create_resource_provider docstring
Success response varies for different microversions [1].

[1] https://developer.openstack.org/api-ref/placement/#create-resource-provider

Change-Id: Ic7f4ea6f97f6bd434cf7338dec9ce7db40a300a6
2018-08-23 23:27:21 +08:00
Zuul
e77a76ab16 Merge "Fix nits in resource_provider.py" 2018-08-23 15:12:16 +00:00
Zuul
42599ba099 Merge "[placement] Regex consts for placement schema" 2018-08-23 12:45:08 +00:00
Chris Dent
3a396fe5de [placement] Add functional test to verify presence of policy
Add a test that traverses all available placement URLs at the latest
microversion and tries to access them as non-admin. If something other
than a 403 response is given a failed test with a message like

    method POST on route /resource_providers/{uuid}/inventories
    is open for user, status: 404

is produced.

This works because we do authZ handling early in each handler, before
data processing and path parameter handling.

The method is pretty straightforward: traverse ROUTE_DECLARATIONS, visit
every url with each the declared methods, except the root version document,
and confirm a 403 response when the provided auth token is non-admin.

This has been created to avoid situations where a route is added without
policy like happened on https://review.openstack.org/#/c/576927/ . Until
recently we had a failover where any route not defined to have policy
would default to admin. That went away so now we need some test
automation to catch our forgetful humanness.

Change-Id: Id582886ec4b621b97d7cc7237b4670ad7bb12295
2018-08-23 10:27:53 +01:00
rajat29
6343f13e0c Normalize dashless 'resource provider create' uuid
When creating resource provider with '--uuid' argument, nova
accept uuid without dash('-') too, which some time results in,
resource provider with same uuid i.e one with dash and one without.

This patch attempts to fix it by transforming dashless UUID into
dashed one before inserting it into the database.

Co-Authored-By: Chen <dstbtgagt@foxmail.com>

Change-Id: I2685eb65907adbd22b2d09264b110692e100eaf9
Closes-Bug: #1758057
2018-08-23 11:20:42 +08:00
Zuul
bc97a5838a Merge "Remove ChanceScheduler" 2018-08-22 22:54:00 +00:00
Eric Fried
dd1c86abec [placement] Regex consts for placement schema
Factor out some regexes used across multiple schemata into a new
nova.api.openstack.placement.schemas.common module.

The UUID part of this is in preparation for fixing the related bug.

Change-Id: If62bfeeb32d0ad77dd1205116ee4e5e844bb07e4
Related-Bug: #1758057
2018-08-22 11:09:06 -05:00
Zuul
fecae3fa6a Merge "Making consistent used of GiB and MiB in API ref" 2018-08-22 04:03:01 +00:00
Zuul
f427c98cf5 Merge "Set policy_opt defaults in placement deploy unit test" 2018-08-21 17:39:37 +00:00
Zuul
6c8759de45 Merge "Set policy_opt defaults in placement gabbi fixture" 2018-08-21 17:36:56 +00:00
Chris Dent
094c609b87 Set policy_opt defaults in placement deploy unit test
Without this change, tests can intermittently fail with NoSuchOptError
when a single process does not have the ConfigFixture running before
this test. This change ensure the opts are registered and defaulted.

Change-Id: I6a4873726e3e7fe0d4db3d1dea61309702b8f24b
Closes-Bug: #1788176
2018-08-21 16:56:43 +01:00
Chris Dent
06ce5d5df6 Set policy_opt defaults in placement gabbi fixture
Without this change, tests can intermittently fail with NoSuchOptError
when a single process does not have other tests running prior to
gabbi tests. This change ensure the opts are registered and defaulted.

Change-Id: I1c7e347b6e788928bef96e32c3365d0fdc5ba00f
Related-Bug: #1786498
Closes-Bug: #1788176
2018-08-21 14:28:32 +01:00
zhangyangyang
a59e7dda5d Remove ChanceScheduler
ChanceScheduler is deprecated in Pike [1] and will be removed in a
subsequent release.

[1] https://review.openstack.org/#/c/492210/

Change-Id: I44f9c1cabf9fc64b1a6903236bc88f5ed8619e9e
2018-08-20 15:56:13 +01:00
Petersingh Anburaj
6108dc6a1d Making consistent used of GiB and MiB in API ref
This patch will replace all GB with GiB and MB with MiB
to have a consistent use of units in the compute API reference
and the placement API reference.

Co-Authored-By: Takashi Natsume <natsume.takashi@lab.ntt.co.jp>
Change-Id: Ie40413752b591b222ff29dbe975ddd7d10638eca
Closes-Bug: #1752340
2018-08-16 22:42:00 +00:00
Jay Pipes
4c2e11af67 placement: use single-shot INSERT/DELETE agg
When replacing a provider's set of aggregate associations, we were
issuing a call to:

 DELETE resource_provider_aggregates WHERE resource_provider_id = $rp

and then a single call to:

 INSERT INTO resource_provider_aggregates
 SELECT $rp, aggs.id
 FROM provider_aggregates AS aggs
 WHERE aggs.uuid IN ($agg_uuids)

This patch changes the _set_aggregates() function in a few ways.
First, we grab the aggregate's internal ID value when creating new
aggregate records (or grabbing a provider's existing aggregate
associations). This eliminates the need for any join to
provider_aggregates in an INSERT/DELETE statement.

Second, instead of a multi-row INSERT .. SELECT statement, we do
single-shot INSERT ... VALUES statements, one for each added aggregate.

Third, we no longer DELETE all aggregate associations for the provider
in question. Instead, we issue single-shot DELETE statements for only
the aggregates that are being disassociated.

Finally, I've added a number of log debug statements so that we can have
a little more information if this particular patch does not fix the
deadlock issue described in the associated bug.

Change-Id: I87e765305017eae1424005f7d6f419f42a2f8370
Closes-bug: #1786703
2018-08-16 18:04:40 -04:00
Zuul
fe4754f8bd Merge "Add placement perf info gathering hook to end of nova-next" 2018-08-15 21:52:02 +00:00
Zuul
0ed6c78835 Merge "Remove blacklisted py3 xen tests" 2018-08-15 07:10:21 +00:00
Eric Fried
38dfc8f56f Remove blacklisted py3 xen tests
Xen UT failures seem to have cleared up, which may or may not have
anything to do with [1]. Remove the blacklist file and get these back
in the py3 jobs.

[1] https://review.openstack.org/#/c/591061/

Change-Id: Iacc1ae01f535caed64793d57b757ea07a8c46620
2018-08-14 16:34:19 +00:00
Chris Dent
fc45edca78 Add placement perf info gathering hook to end of nova-next
This change adds a post test hook to the nova-next job to report
timing of a query to GET /allocation_candidates when there are 1000
resource providers with the same inventory.

A summary of the work ends up in logs/placement-perf.txt

Change-Id: Idc446347cd8773f579b23c96235348d8e10ea3f6
2018-08-14 15:42:08 +01:00
Zuul
2e912c6153 Merge "placement: ignore policy scope check failures if not enforcing scope" 2018-08-14 09:19:14 +00:00
Zuul
5aec2bd1aa Merge "[placement] api-ref: Add missing aggregates example" 2018-08-13 17:03:01 +00:00
Takashi NATSUME
01c682e8be [placement] api-ref: Add missing aggregates example
About "PUT /resource_providers/{uuid}/aggregates"
in the Placement API reference,
the example of the response body in microversion 1.1 - 1.18 is missing.
So add it.

Change-Id: Ic3dc665124ae7927cfa20cf70a08c39b856b8961
Closes-Bug: #1786759
2018-08-13 13:09:49 +00:00
Zuul
8e352717bd Merge "placement: use simple code paths when possible" 2018-08-13 04:05:19 +00:00
Zuul
f245b4a96e Merge "Test case for multiple forbidden traits" 2018-08-13 03:58:48 +00:00
Zuul
7d1190e6b8 Merge "Adds a test for _get_provider_ids_matching()" 2018-08-13 03:58:41 +00:00
Zuul
a7643e3bfa Merge "Remove patching the mock lib" 2018-08-10 21:00:44 +00:00
Jay Pipes
4f54c204dc placement: use simple code paths when possible
Somewhere in the past release, we started using extremely complex code
paths involving sharing providers, anchor providers, and nested resource
provider calculations when we absolutely don't need to do so.

There was a _has_provider_trees() function in the
nova/api/openstack/placement/objects/resource_provider.py file that used
to be used for top-level switching between a faster, simpler approach to
finding allocation candidates for a simple search of resources and
traits when no sharing providers and no nesting was used. That was
removed at some point and all code paths -- even for simple "get me
these amounts of these resources" when no trees or sharing providers are
present (which is the vast majority of OpenStack deployments) -- were
going through the complex tree-search-and-match queries and algorithms.

This patch changes that so that when there's a request for some
resources and there's no trees or sharing providers, we do the simple
code path. Hopefully this gets our performance for the simple, common
cases back to where we were pre-Rocky.

This change is a prerequisite for the following change which adds
debugging output to help diagnose which resource classes are running
out of inventory when GET /allocation_candidates returns 0 results.
That code is not possible without the changes here as they only
work if we can identify when a "simpler approach" is possible and
call that simpler code.

Related-Bug: #1786055
Partial-Bug: #1786519
Change-Id: I1fdbcdb7a1dd51e738924c8a30238237d7ac74e1
2018-08-10 16:19:42 -04:00
Eric Fried
27fd442fd3 Test case for multiple forbidden traits
Test case to expose a bug at [1] where symmetric difference (^) is being
used instead of difference (-).

[1] https://review.openstack.org/#/c/590041/6/nova/api/openstack/placement/objects/resource_provider.py@2911

Change-Id: I2e80fe75da1d1104d0562f087c74117fbe6b457b
2018-08-10 16:19:41 -04:00
Tetsuro Nakamura
4faaf492af Adds a test for _get_provider_ids_matching()
This patch adds a test for _get_provider_ids_matching()
to verify it works correctly with required traits.

Related-Bug: #1786519
Change-Id: I2512e361f5eaa4e60701be7c8bf57b2e0a02a146
2018-08-10 16:19:41 -04:00
Zuul
1286a4eb75 Merge "Add additional info to resource provider aggregates update API" 2018-08-10 19:59:39 +00:00
Zuul
91f107ea0a Merge "Nix 'new in 1.19' from 1.19 sections for rp aggs" 2018-08-10 15:35:42 +00:00
Matt Riedemann
685c9382f1 placement: ignore policy scope check failures if not enforcing scope
Rather than spam the placement API logs with scope check
warnings from oslo.policy when placement isn't configured
for scope type enforcement, ignore those warnings.

Note that the same warnings filter in the placement WarningsFixture
is left intact because the configuration defaults setting code in
wsgi.py which allows the warning filter to be set in deploy.py,
is not run by the functional tests. In future changes this will
be resolved by unifying configuration handling.

Closes-Bug: #1786498
Related-Bug: #1784663
Change-Id: I34e4e550c9c31a654308e555210588156418f9e3
2018-08-10 16:29:17 +01:00
Balazs Gibizer
1ea074730d Remove patching the mock lib
Since [1] was fixed we don't need to manually patch the mock lib to
avoid silently passing mistyped assert methods on mocks.

[1] https://bugs.python.org/issue21238

Change-Id: Iecf4dcf8e648c9191bf8846428683ec81812c026
2018-08-10 10:58:52 +02:00
Chen
e154ca668d Add additional info to resource provider aggregates update API
Mention that if any nonexistent aggregate uuids are provided, those
aggregates will be created automatically first during the update call.
I think this is a good addition since in fact I was much confused how
the aggregates can be created in the first place when playing with
the placement APIs.

Change-Id: I2f86715a21784eb974b73fc3edc74eedd73ad48b
2018-08-10 11:17:51 +08:00
Zuul
e4cfbc2e57 Merge "[placement] api-ref: add description for 1.29" 2018-08-09 19:37:22 +00:00
Eric Fried
ee9951c6b3 Nix 'new in 1.19' from 1.19 sections for rp aggs
In the placement API reference, there is a separate section for
"microversions 1.19 -" for both List and Update resource provider
aggregates, so there's no need for the resource_provider_generation
field to mention that it's "New in version 1.19" in those sections.

Change-Id: I3fd02f21d4af11633c4aeb33925c279a1c3b4abd
2018-08-09 10:23:37 -05:00
Takashi NATSUME
3809042678 [placement] api-ref: add description for 1.29
Add description for nested resource providers
in allocation candidates, Placement API microversion 1.29
in the Placement API reference.

Change-Id: I22c2068ae860c1ef2cbdb0801e85a40acc2a097a
Implements: blueprint nested-resource-providers-allocation-candidates
2018-08-09 02:56:30 +00:00
Zuul
024086330a Merge "api-ref: fix min_version for parent_provider_uuid in responses" 2018-08-08 23:57:47 +00:00