This is the result of running the following two commands:
git mv nova placement
git mv etc/nova/ etc/placement
Change-Id: Ied6e04462ac9d7b582df1411d3045048639f127b
After we did the initial clone to seed the placement repository, an
almost-exclusively-nova change [1] merged which included a delta to a
document [2] that was included in the clone. This change just brings in
that one delta.
[1] Idf8997d5efdfdfca6967899a0882ffb9ecf96915
[2] https://review.openstack.org/#/c/585034/20/doc/source/user/placement.rst
Change-Id: I736afc3ec0f007283bb4cc8bd1a5c366e62c2165
This applies nova change Ie21bddc186364db2b10af45546c35b9ad5a0208b,
which was missed when seeding the placement repository. That change's
original commit message follows:
Fix the race condition identified at [1] by indexing the `inventories`
argument to the resource_provider.reshape method by ResourceProvider
object rather than its UUID. That means we never have to do a fresh
lookup for the provider.
[1] https://review.openstack.org/#/c/585033/18/nova/api/openstack/placement/objects/resource_provider.py@4195
Change-Id: Ie21bddc186364db2b10af45546c35b9ad5a0208b
Set up placement with a noop job for 'gate' and 'check' and non-voting
unit, functional and pep8 test for 'check'. These jobs will be change
to non-voting as they reach a working state, and additional jobs will
be added as required.
The noop jobs are required for any code to merge.
Also .gitreview is updated to point to the right repo.
Change-Id: I741a6b86eda12afb515b6207a4535e8b6e1ead44
This moves the scheduler restart logic from the heal_allocations
tests into a more generic location (like restart_compute_service)
so that we can re-use it in other functional tests that rely on
the caching scheduler. There are a couple of other tests that don't
need to restart the scheduler if we just move the start of the
scheduler to after we start the computes.
Change-Id: I7720fe4a3a0e537b7b356947317766597d4b47cf
Related-Bug: #1781648
The use of the independent context on _ensure_aggregate appears to
be unnecessary. It causes file-based uses of SQLite dbs to fail
(with database locked errors, as reported in the associated bug,
1789633) and thus may mask issues with other databases. Adding the
independent context manager was the result of a series of "throw
stuff at the wall and see what sticks" patches, but it looks now
that it is not required, and in some situations causes problems.
Runs through the gate show that the behavior it was fixing (as
described in bug 1786703) is not happening.
Change-Id: I1f325d55ec256db34a4c3bbd230dcd8a91bce542
Related-Bug: #1786703
Closes-Bug: #1789633
The schema accepts the (resource provider) 'generation' key in the
resource provider section of the allocations dictionary in the POST
/allocations/{consumer_uuid} request payload. As with PUT
/allocations/{consumer_uuid}, it is ignored. This was missing from the
API reference.
Change-Id: I2586875a60aba7a566f6c1b61ded133b8749031c
This reverts commit bd7d991309ea2bea5d175cb1f2710519936fd0c2 and bumps
the minimum version of oslo.db to 4.40.0, as that is the first version
of the library to include the renamed attribute.
Change-Id: Ic9e7864be3af7ef362cad5648dfc7bdecd104465
Related-Bug: #1788833
In commit 2d532963, all instances of 'async' were replaced with 'async_'
in preparation for Python 3.7. However, one of these should not have
been changed as it refers to an oslo.db object attribute. That attribute
has actually been renamed itself but that rename is only present from
oslo.db 4.40.0 [1]. Thankfully, an alias to the older name is provided
so we use that.
[1] https://github.com/openstack/oslo.db/commit/df6bf34
Change-Id: I1afd0ba34a9ebcb63edb91e880ef60580befb32e
Closes-Bug: #1788833
There was a eventlet.monkey_patch [1] when we launch a nova
process(like nova-api, nova-compute), but it's invalid under
the uwsgi mode.
But there are concurrency requirements in the api service, such
as, when listing instances cross multiple cells we're using
greenthreads and oslo.db does a time.sleep to allow switching
greenthreads [2].
So, in this patch we add the monkey_patch in the uwsgi
application setup and refactor the monkey patching to use common
code.
[1] https://github.com/openstack/nova/blob/233ea58/nova/cmd/__init__.py#L26
[2] https://github.com/openstack/oslo.db/blob/9c66959/oslo_db/sqlalchemy/engines.py#L51
Closes-bug: #1787331
Change-Id: Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56
This patch modifies the code paths for the non-granular request group
allocation candidates processing. It removes the giant multi-join SQL
query and replaces it with multiple calls to
_get_providers_with_resource(), logging the number of matched providers
for each resource class requested and filter (on required traits,
forbidden traits and aggregate memebership).
Here are some examples of the debug output:
- A request for three resources with no aggregate or trait filters:
found 7 providers with available 5 VCPU
found 9 providers with available 1024 MEMORY_MB
found 5 providers after filtering by previous result
found 8 providers with available 1500 DISK_GB
found 2 providers after filtering by previous result
- The same request, but with a required trait that nobody has, shorts
out quickly:
found 0 providers after applying required traits filter (['HW_CPU_X86_AVX2'])
- A request for one resource with aggregates and forbidden (but no
required) traits:
found 2 providers after applying aggregates filter ([['3ed8fb2f-4793-46ee-a55b-fdf42cb392ca']])
found 1 providers after applying forbidden traits filter ([u'CUSTOM_TWO', u'CUSTOM_THREE'])
found 3 providers with available 4 VCPU
found 1 providers after applying initial aggregate and trait filters
Co-authored-by: Eric Fried <efried@us.ibm.com>
Closes-Bug: #1786519
Change-Id: If9ddb8a6d2f03392f3cc11136c4a0b026212b95b
This makes the instance_list module support batching across cells
with a couple of different strategies, and with room to add more
in the future.
Before this change, an instance list with limit 1000 to a
deployment with 10 cells would generate a query to each cell
database with the same limit. Thus, that API request could end
up processing up to 10,000 instance records despite only
returning 1000 to the user (because of the limit).
This uses the batch functionality in the base code added in
Iaa4759822e70b39bd735104d03d4deec988d35a1
by providing a couple of strategies by which the batch size
per cell can be determined. These should provide a lot of gain
in the short term, and we can extend them with other strategies
as we identify some with additional benefits.
Closes-Bug: #1787977
Change-Id: Ie3a5f5dc49f8d9a4b96f1e97f8a6ea0b5738b768
Per the referenced bug, we weren't accounting for the scenario where a
reshape operation was removing *all* inventories for a provider (which
could be fairly common). With this fix, we do a three-stage lookup of
the provider object: If it's not in the inventories, we look in the
allocations; if it's not in the allocations, we look it up in the
database.
Change-Id: I594bb64f87c61b7ffd39c19e0fd42c4c087a3a11
Closes-Bug: #1783130
Add a test that traverses all available placement URLs at the latest
microversion and tries to access them as non-admin. If something other
than a 403 response is given a failed test with a message like
method POST on route /resource_providers/{uuid}/inventories
is open for user, status: 404
is produced.
This works because we do authZ handling early in each handler, before
data processing and path parameter handling.
The method is pretty straightforward: traverse ROUTE_DECLARATIONS, visit
every url with each the declared methods, except the root version document,
and confirm a 403 response when the provided auth token is non-admin.
This has been created to avoid situations where a route is added without
policy like happened on https://review.openstack.org/#/c/576927/ . Until
recently we had a failover where any route not defined to have policy
would default to admin. That went away so now we need some test
automation to catch our forgetful humanness.
Change-Id: Id582886ec4b621b97d7cc7237b4670ad7bb12295
When creating resource provider with '--uuid' argument, nova
accept uuid without dash('-') too, which some time results in,
resource provider with same uuid i.e one with dash and one without.
This patch attempts to fix it by transforming dashless UUID into
dashed one before inserting it into the database.
Co-Authored-By: Chen <dstbtgagt@foxmail.com>
Change-Id: I2685eb65907adbd22b2d09264b110692e100eaf9
Closes-Bug: #1758057
/reshaper provides a way to atomically modify some allocations and
inventory in a single transaction, allowing operations like migrating
some inventory from a parent provider to a new child.
A fair amount of code is reused from handler/inventory.py, some
refactoring is in order before things get too far with that.
In handler/allocation.py some code is extracted to its own methods
so it can be reused from reshaper.py.
This is done as microversion 1.30.
A suite of gabbi tests is provided which attempt to cover various
failures including schema violations, generation conflicts, and
data conflicts.
api-ref, release notes and rest history are updated
Change-Id: I5b33ac3572bc3789878174ffc86ca42ae8035cfa
Partially-Implements: blueprint reshape-provider-tree
Factor out some regexes used across multiple schemata into a new
nova.api.openstack.placement.schemas.common module.
The UUID part of this is in preparation for fixing the related bug.
Change-Id: If62bfeeb32d0ad77dd1205116ee4e5e844bb07e4
Related-Bug: #1758057
Without this change, tests can intermittently fail with NoSuchOptError
when a single process does not have the ConfigFixture running before
this test. This change ensure the opts are registered and defaulted.
Change-Id: I6a4873726e3e7fe0d4db3d1dea61309702b8f24b
Closes-Bug: #1788176
Without this change, tests can intermittently fail with NoSuchOptError
when a single process does not have other tests running prior to
gabbi tests. This change ensure the opts are registered and defaulted.
Change-Id: I1c7e347b6e788928bef96e32c3365d0fdc5ba00f
Related-Bug: #1786498
Closes-Bug: #1788176
ChanceScheduler is deprecated in Pike [1] and will be removed in a
subsequent release.
[1] https://review.openstack.org/#/c/492210/
Change-Id: I44f9c1cabf9fc64b1a6903236bc88f5ed8619e9e
This patch will replace all GB with GiB and MB with MiB
to have a consistent use of units in the compute API reference
and the placement API reference.
Co-Authored-By: Takashi Natsume <natsume.takashi@lab.ntt.co.jp>
Change-Id: Ie40413752b591b222ff29dbe975ddd7d10638eca
Closes-Bug: #1752340
When replacing a provider's set of aggregate associations, we were
issuing a call to:
DELETE resource_provider_aggregates WHERE resource_provider_id = $rp
and then a single call to:
INSERT INTO resource_provider_aggregates
SELECT $rp, aggs.id
FROM provider_aggregates AS aggs
WHERE aggs.uuid IN ($agg_uuids)
This patch changes the _set_aggregates() function in a few ways.
First, we grab the aggregate's internal ID value when creating new
aggregate records (or grabbing a provider's existing aggregate
associations). This eliminates the need for any join to
provider_aggregates in an INSERT/DELETE statement.
Second, instead of a multi-row INSERT .. SELECT statement, we do
single-shot INSERT ... VALUES statements, one for each added aggregate.
Third, we no longer DELETE all aggregate associations for the provider
in question. Instead, we issue single-shot DELETE statements for only
the aggregates that are being disassociated.
Finally, I've added a number of log debug statements so that we can have
a little more information if this particular patch does not fix the
deadlock issue described in the associated bug.
Change-Id: I87e765305017eae1424005f7d6f419f42a2f8370
Closes-bug: #1786703