10643 Commits

Author SHA1 Message Date
EdLeafe
88c3b84730 Rename the 'nova' directories to 'placement'
This is the result of running the following two commands:
    git mv nova placement
    git mv etc/nova/ etc/placement

Change-Id: Ied6e04462ac9d7b582df1411d3045048639f127b
2018-09-04 10:31:22 -05:00
Eric Fried
68b66b6c05 Apply placement.rst change from Idf8997d5efdfdfca6
After we did the initial clone to seed the placement repository, an
almost-exclusively-nova change [1] merged which included a delta to a
document [2] that was included in the clone. This change just brings in
that one delta.

[1] Idf8997d5efdfdfca6967899a0882ffb9ecf96915
[2] https://review.openstack.org/#/c/585034/20/doc/source/user/placement.rst

Change-Id: I736afc3ec0f007283bb4cc8bd1a5c366e62c2165
2018-09-04 10:31:20 -05:00
Eric Fried
c1767dcb99 Fix race condition in reshaper handler
This applies nova change Ie21bddc186364db2b10af45546c35b9ad5a0208b,
which was missed when seeding the placement repository. That change's
original commit message follows:

Fix the race condition identified at [1] by indexing the `inventories`
argument to the resource_provider.reshape method by ResourceProvider
object rather than its UUID. That means we never have to do a fresh
lookup for the provider.

[1] https://review.openstack.org/#/c/585033/18/nova/api/openstack/placement/objects/resource_provider.py@4195

Change-Id: Ie21bddc186364db2b10af45546c35b9ad5a0208b
2018-09-04 10:31:02 -05:00
Chris Dent
9d28a9f849 Set up initial .zuul.yaml
Set up placement with a noop job for 'gate' and 'check' and non-voting
unit, functional and pep8 test for 'check'. These jobs will be change
to non-voting as they reach a working state, and additional jobs will
be added as required.

The noop jobs are required for any code to merge.

Also .gitreview is updated to point to the right repo.

Change-Id: I741a6b86eda12afb515b6207a4535e8b6e1ead44
2018-08-31 14:24:59 +00:00
EdLeafe
1a45f11a06 Removed the zuul config file 2018-08-30 22:39:21 +00:00
Zuul
e3b6de69ff Merge "reshaper gabbit: Nix comments re doubled max_unit" 2018-08-30 17:21:13 +00:00
Zuul
32b291d4d2 Merge "(Re)start caching scheduler after starting computes in tests" 2018-08-30 13:47:17 +00:00
Zuul
198fcebe86 Merge "reshaper: Look up provider if not in inventories" 2018-08-30 01:12:50 +00:00
Zuul
602576dcf7 Merge "Add trait query to placement perf check" 2018-08-30 01:12:35 +00:00
Zuul
c893f1f74a Merge "Add explanatory prefix to post_test_perf output" 2018-08-30 01:05:05 +00:00
Zuul
5b26f84f78 Merge "[placement] Make _ensure_aggregate context not independent" 2018-08-30 00:52:28 +00:00
Matt Riedemann
4f0faa713a (Re)start caching scheduler after starting computes in tests
This moves the scheduler restart logic from the heal_allocations
tests into a more generic location (like restart_compute_service)
so that we can re-use it in other functional tests that rely on
the caching scheduler. There are a couple of other tests that don't
need to restart the scheduler if we just move the start of the
scheduler to after we start the computes.

Change-Id: I7720fe4a3a0e537b7b356947317766597d4b47cf
Related-Bug: #1781648
2018-08-29 13:16:09 -04:00
Chris Dent
5cdc77379c [placement] Make _ensure_aggregate context not independent
The use of the independent context on _ensure_aggregate appears to
be unnecessary. It causes file-based uses of SQLite dbs to fail
(with database locked errors, as reported in the associated bug,
1789633) and thus may mask issues with other databases. Adding the
independent context manager was the result of a series of "throw
stuff at the wall and see what sticks" patches, but it looks now
that it is not required, and in some situations causes problems.

Runs through the gate show that the behavior it was fixing (as
described in bug 1786703) is not happening.

Change-Id: I1f325d55ec256db34a4c3bbd230dcd8a91bce542
Related-Bug: #1786703
Closes-Bug: #1789633
2018-08-29 16:30:00 +01:00
Zuul
9a99c6b9fe Merge "Document no content on POST /reshaper 204" 2018-08-29 09:22:16 +00:00
Zuul
663e44462d Merge "[placement] Add /reshaper handler for POST" 2018-08-29 09:22:10 +00:00
Zuul
b8e93e2e33 Merge "Mention (unused) RP generation in POST /allocs/{c}" 2018-08-29 07:57:21 +00:00
Zuul
4e69bac144 Merge "Revert "Don't use '_TransactionContextManager._async'"" 2018-08-28 23:34:37 +00:00
Zuul
87688b72e4 Merge "Don't use '_TransactionContextManager._async'" 2018-08-28 23:34:31 +00:00
Eric Fried
fa749b1a96 Mention (unused) RP generation in POST /allocs/{c}
The schema accepts the (resource provider) 'generation' key in the
resource provider section of the allocations dictionary in the POST
/allocations/{consumer_uuid} request payload. As with PUT
/allocations/{consumer_uuid}, it is ignored. This was missing from the
API reference.

Change-Id: I2586875a60aba7a566f6c1b61ded133b8749031c
2018-08-28 17:34:19 -05:00
Eric Fried
af19086bea reshaper gabbit: Nix comments re doubled max_unit
As noted in a review [1] on the reshaper series, the comments about
doubling max_unit were obsolete (fixed via [2]), and are hereby removed.

[1] https://review.openstack.org/#/c/576927/35/nova/tests/functional/api/openstack/placement/gabbits/reshaper.yaml@222
[2] I8ba378ff5eeaf6c9cca11c5874708a17d4640097

Change-Id: Iaf3df651a76705d0f64182c46c665f037ea51b68
2018-08-28 14:46:00 -05:00
Stephen Finucane
4df41cb4b3 Revert "Don't use '_TransactionContextManager._async'"
This reverts commit bd7d991309ea2bea5d175cb1f2710519936fd0c2 and bumps
the minimum version of oslo.db to 4.40.0, as that is the first version
of the library to include the renamed attribute.

Change-Id: Ic9e7864be3af7ef362cad5648dfc7bdecd104465
Related-Bug: #1788833
2018-08-28 17:18:51 +01:00
Stephen Finucane
37fb0708a5 Don't use '_TransactionContextManager._async'
In commit 2d532963, all instances of 'async' were replaced with 'async_'
in preparation for Python 3.7. However, one of these should not have
been changed as it refers to an oslo.db object attribute. That attribute
has actually been renamed itself but that rename is only present from
oslo.db 4.40.0 [1]. Thankfully, an alias to the older name is provided
so we use that.

[1] https://github.com/openstack/oslo.db/commit/df6bf34

Change-Id: I1afd0ba34a9ebcb63edb91e880ef60580befb32e
Closes-Bug: #1788833
2018-08-28 17:14:19 +01:00
Zuul
4f7c9b096a Merge "Make monkey patch work in uWSGI mode" 2018-08-28 15:38:04 +00:00
Zuul
931dc02d47 Merge "Make instance_list perform per-cell batching" 2018-08-28 04:55:17 +00:00
Yikun Jiang
b26f21df3d Make monkey patch work in uWSGI mode
There was a eventlet.monkey_patch [1] when we launch a nova
process(like nova-api, nova-compute), but it's invalid under
the uwsgi mode.

But there are concurrency requirements in the api service, such
as, when listing instances cross multiple cells we're using
greenthreads and oslo.db does a time.sleep to allow switching
greenthreads [2].

So, in this patch we add the monkey_patch in the uwsgi
application setup and refactor the monkey patching to use common
code.

[1] https://github.com/openstack/nova/blob/233ea58/nova/cmd/__init__.py#L26
[2] https://github.com/openstack/oslo.db/blob/9c66959/oslo_db/sqlalchemy/engines.py#L51

Closes-bug: #1787331

Change-Id: Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56
2018-08-28 09:49:01 +08:00
Zuul
ea73ab01f1 Merge "[placement] split gigantor SQL query, add logging" 2018-08-27 18:16:28 +00:00
Zuul
df776affc3 Merge "Fix create_resource_provider docstring" 2018-08-27 16:01:14 +00:00
Jay Pipes
10e7228707 [placement] split gigantor SQL query, add logging
This patch modifies the code paths for the non-granular request group
allocation candidates processing. It removes the giant multi-join SQL
query and replaces it with multiple calls to
_get_providers_with_resource(), logging the number of matched providers
for each resource class requested and filter (on required traits,
forbidden traits and aggregate memebership).

Here are some examples of the debug output:

- A request for three resources with no aggregate or trait filters:

 found 7 providers with available 5 VCPU
 found 9 providers with available 1024 MEMORY_MB
 found 5 providers after filtering by previous result
 found 8 providers with available 1500 DISK_GB
 found 2 providers after filtering by previous result

- The same request, but with a required trait that nobody has, shorts
  out quickly:

 found 0 providers after applying required traits filter (['HW_CPU_X86_AVX2'])

- A request for one resource with aggregates and forbidden (but no
  required) traits:

 found 2 providers after applying aggregates filter ([['3ed8fb2f-4793-46ee-a55b-fdf42cb392ca']])
 found 1 providers after applying forbidden traits filter ([u'CUSTOM_TWO', u'CUSTOM_THREE'])
 found 3 providers with available 4 VCPU
 found 1 providers after applying initial aggregate and trait filters

Co-authored-by: Eric Fried <efried@us.ibm.com>
Closes-Bug: #1786519
Change-Id: If9ddb8a6d2f03392f3cc11136c4a0b026212b95b
2018-08-27 13:50:22 +00:00
Dan Smith
a8f451a7e4 Make instance_list perform per-cell batching
This makes the instance_list module support batching across cells
with a couple of different strategies, and with room to add more
in the future.

Before this change, an instance list with limit 1000 to a
deployment with 10 cells would generate a query to each cell
database with the same limit. Thus, that API request could end
up processing up to 10,000 instance records despite only
returning 1000 to the user (because of the limit).

This uses the batch functionality in the base code added in
Iaa4759822e70b39bd735104d03d4deec988d35a1
by providing a couple of strategies by which the batch size
per cell can be determined. These should provide a lot of gain
in the short term, and we can extend them with other strategies
as we identify some with additional benefits.

Closes-Bug: #1787977
Change-Id: Ie3a5f5dc49f8d9a4b96f1e97f8a6ea0b5738b768
2018-08-27 06:44:32 -07:00
Eric Fried
0c14b13a51 Document no content on POST /reshaper 204
For consistency with other 204-returning operations in the placement API
reference, mention that POST /reshaper returns no body content on
success.

FUP from https://review.openstack.org/#/c/576927/35/placement-api-ref/source/reshaper.inc@45

Change-Id: I50dda0b161404d96dae35a3a96225f1ac4ec7309
2018-08-24 16:48:49 -05:00
Zuul
09afc32669 Merge "[placement] Add functional test to verify presence of policy" 2018-08-23 21:30:55 +00:00
Zuul
cd2c9bd5c2 Merge "Normalize dashless 'resource provider create' uuid" 2018-08-23 21:16:05 +00:00
Zuul
5aa493b858 Merge "placement: use single-shot INSERT/DELETE agg" 2018-08-23 16:08:29 +00:00
Chen
675b111469 Fix create_resource_provider docstring
Success response varies for different microversions [1].

[1] https://developer.openstack.org/api-ref/placement/#create-resource-provider

Change-Id: Ic7f4ea6f97f6bd434cf7338dec9ce7db40a300a6
2018-08-23 23:27:21 +08:00
Zuul
e77a76ab16 Merge "Fix nits in resource_provider.py" 2018-08-23 15:12:16 +00:00
Eric Fried
5d13fd63f3 reshaper: Look up provider if not in inventories
Per the referenced bug, we weren't accounting for the scenario where a
reshape operation was removing *all* inventories for a provider (which
could be fairly common). With this fix, we do a three-stage lookup of
the provider object: If it's not in the inventories, we look in the
allocations; if it's not in the allocations, we look it up in the
database.

Change-Id: I594bb64f87c61b7ffd39c19e0fd42c4c087a3a11
Closes-Bug: #1783130
2018-08-23 08:45:34 -05:00
Zuul
42599ba099 Merge "[placement] Regex consts for placement schema" 2018-08-23 12:45:08 +00:00
Chris Dent
3a396fe5de [placement] Add functional test to verify presence of policy
Add a test that traverses all available placement URLs at the latest
microversion and tries to access them as non-admin. If something other
than a 403 response is given a failed test with a message like

    method POST on route /resource_providers/{uuid}/inventories
    is open for user, status: 404

is produced.

This works because we do authZ handling early in each handler, before
data processing and path parameter handling.

The method is pretty straightforward: traverse ROUTE_DECLARATIONS, visit
every url with each the declared methods, except the root version document,
and confirm a 403 response when the provided auth token is non-admin.

This has been created to avoid situations where a route is added without
policy like happened on https://review.openstack.org/#/c/576927/ . Until
recently we had a failover where any route not defined to have policy
would default to admin. That went away so now we need some test
automation to catch our forgetful humanness.

Change-Id: Id582886ec4b621b97d7cc7237b4670ad7bb12295
2018-08-23 10:27:53 +01:00
rajat29
6343f13e0c Normalize dashless 'resource provider create' uuid
When creating resource provider with '--uuid' argument, nova
accept uuid without dash('-') too, which some time results in,
resource provider with same uuid i.e one with dash and one without.

This patch attempts to fix it by transforming dashless UUID into
dashed one before inserting it into the database.

Co-Authored-By: Chen <dstbtgagt@foxmail.com>

Change-Id: I2685eb65907adbd22b2d09264b110692e100eaf9
Closes-Bug: #1758057
2018-08-23 11:20:42 +08:00
Chris Dent
ff27187347 [placement] Add /reshaper handler for POST
/reshaper provides a way to atomically modify some allocations and
inventory in a single transaction, allowing operations like migrating
some inventory from a parent provider to a new child.

A fair amount of code is reused from handler/inventory.py, some
refactoring is in order before things get too far with that.

In handler/allocation.py some code is extracted to its own methods
so it can be reused from reshaper.py.

This is done as microversion 1.30.

A suite of gabbi tests is provided which attempt to cover various
failures including schema violations, generation conflicts, and
data conflicts.

api-ref, release notes and rest history are updated

Change-Id: I5b33ac3572bc3789878174ffc86ca42ae8035cfa
Partially-Implements: blueprint reshape-provider-tree
2018-08-23 00:36:17 +00:00
Zuul
bc97a5838a Merge "Remove ChanceScheduler" 2018-08-22 22:54:00 +00:00
Eric Fried
dd1c86abec [placement] Regex consts for placement schema
Factor out some regexes used across multiple schemata into a new
nova.api.openstack.placement.schemas.common module.

The UUID part of this is in preparation for fixing the related bug.

Change-Id: If62bfeeb32d0ad77dd1205116ee4e5e844bb07e4
Related-Bug: #1758057
2018-08-22 11:09:06 -05:00
Zuul
fecae3fa6a Merge "Making consistent used of GiB and MiB in API ref" 2018-08-22 04:03:01 +00:00
Zuul
f427c98cf5 Merge "Set policy_opt defaults in placement deploy unit test" 2018-08-21 17:39:37 +00:00
Zuul
6c8759de45 Merge "Set policy_opt defaults in placement gabbi fixture" 2018-08-21 17:36:56 +00:00
Chris Dent
094c609b87 Set policy_opt defaults in placement deploy unit test
Without this change, tests can intermittently fail with NoSuchOptError
when a single process does not have the ConfigFixture running before
this test. This change ensure the opts are registered and defaulted.

Change-Id: I6a4873726e3e7fe0d4db3d1dea61309702b8f24b
Closes-Bug: #1788176
2018-08-21 16:56:43 +01:00
Chris Dent
06ce5d5df6 Set policy_opt defaults in placement gabbi fixture
Without this change, tests can intermittently fail with NoSuchOptError
when a single process does not have other tests running prior to
gabbi tests. This change ensure the opts are registered and defaulted.

Change-Id: I1c7e347b6e788928bef96e32c3365d0fdc5ba00f
Related-Bug: #1786498
Closes-Bug: #1788176
2018-08-21 14:28:32 +01:00
zhangyangyang
a59e7dda5d Remove ChanceScheduler
ChanceScheduler is deprecated in Pike [1] and will be removed in a
subsequent release.

[1] https://review.openstack.org/#/c/492210/

Change-Id: I44f9c1cabf9fc64b1a6903236bc88f5ed8619e9e
2018-08-20 15:56:13 +01:00
Petersingh Anburaj
6108dc6a1d Making consistent used of GiB and MiB in API ref
This patch will replace all GB with GiB and MB with MiB
to have a consistent use of units in the compute API reference
and the placement API reference.

Co-Authored-By: Takashi Natsume <natsume.takashi@lab.ntt.co.jp>
Change-Id: Ie40413752b591b222ff29dbe975ddd7d10638eca
Closes-Bug: #1752340
2018-08-16 22:42:00 +00:00
Jay Pipes
4c2e11af67 placement: use single-shot INSERT/DELETE agg
When replacing a provider's set of aggregate associations, we were
issuing a call to:

 DELETE resource_provider_aggregates WHERE resource_provider_id = $rp

and then a single call to:

 INSERT INTO resource_provider_aggregates
 SELECT $rp, aggs.id
 FROM provider_aggregates AS aggs
 WHERE aggs.uuid IN ($agg_uuids)

This patch changes the _set_aggregates() function in a few ways.
First, we grab the aggregate's internal ID value when creating new
aggregate records (or grabbing a provider's existing aggregate
associations). This eliminates the need for any join to
provider_aggregates in an INSERT/DELETE statement.

Second, instead of a multi-row INSERT .. SELECT statement, we do
single-shot INSERT ... VALUES statements, one for each added aggregate.

Third, we no longer DELETE all aggregate associations for the provider
in question. Instead, we issue single-shot DELETE statements for only
the aggregates that are being disassociated.

Finally, I've added a number of log debug statements so that we can have
a little more information if this particular patch does not fix the
deadlock issue described in the associated bug.

Change-Id: I87e765305017eae1424005f7d6f419f42a2f8370
Closes-bug: #1786703
2018-08-16 18:04:40 -04:00