10644 Commits

Author SHA1 Message Date
Tetsuro Nakamura
7342f50224 Not use project table for user table
`GET /resource_provider/{uuid}/allocations` API didn't
return all the allocations made by multiple users.

This was because the placement wrongly used project table
for user table. This patch fixes it with the test case.

Change-Id: I7c808dec5de1204ced0d1f1b31d8398de8c51679
Closes-Bug: #1785382
2018-08-04 19:34:04 +09:00
Tetsuro Nakamura
c9ccab55c0 Adds a test for getting allocations API
`GET /resource_provider/{uuid}/allocations` API currently doesn't
return all the allocations made by multiple users.

This patch adds a test to describe this bug. The fix for this
is coming in a follow up.

Change-Id: I2b01e27922f11bef2defcb01fe415692de1578ea
Partial-Bug: #1785382
2018-08-04 19:20:30 +09:00
Zuul
7168fcf60d Merge "[placement] ensure_rc_cache only at start of process" 2018-08-04 01:23:51 +00:00
Zuul
14892515f8 Merge "[placement] Move resource_class_cache into placement hierarchy" 2018-08-03 19:17:02 +00:00
Chris Dent
d908d05bb2 [placement] ensure_rc_cache only at start of process
Call ensure_rc_cache from deploy, so that we only try it once per
process.

A small number of unit tests needed an adjustment to either mock
properly or call the ensure_rc_cache in advance of their work.

Change-Id: I7499bba6ac6b463d8da46e10469121e62ee52ed1
2018-08-03 19:53:39 +01:00
Chris Dent
44ce7ec464 [placement] Move resource_class_cache into placement hierarchy
Since only placement uses the resource_class_cache, move it
into the placement hierarchy.

Change-Id: I837d4b691f94af0487a66dacf5ae563e4423f6a3
2018-08-03 14:55:49 +01:00
Eric Fried
75ef938728 [placement] Debug log per granular request group
Add a debug log as suggested in [1]: for each granular request group, it
prints how many allocation requests matched. See sample output [2].

[1] http://lists.openstack.org/pipermail/openstack-dev/2018-August/132802.html
[2] http://paste.openstack.org/raw/727229/

Change-Id: I89d2892fabb6273bfa793f4a06dced80b68e93d5
2018-08-03 08:41:28 -05:00
Zuul
265256ea5b Merge "Scrub hw:cpu_model from API samples" 2018-08-03 13:19:28 +00:00
Yikun Jiang
02a247492b Fix nits in resource_provider.py
1. Change the comments on forbidden trait
2. We already change _get_provider_ids_matching result from "provider
IDs" to "tuples of (internal provider ID, root provider ID)" in the
I343a0cb19f4037ddde5c5fc96d0a053f699f5257
3. Remove out-of-date comments about "handling nested providers" from
039c94a6b98546573f34e0dc9c0a7cd6d84caa5f

trivialfix

Change-Id: I7b23ef4c06be8963e43e0c23ac6cc149ffd2ddae
2018-08-03 17:54:43 +08:00
Zuul
4dae067198 Merge "In Python3.7 async is a keyword [1]" 2018-08-03 00:58:06 +00:00
Matt Riedemann
aca2c764c0 Scrub hw:cpu_model from API samples
Nova does not support a "hw:cpu_model" flavor extra
spec. This was added as part of microversion 2.47 from
WindRiver where their Titanium Cloud product does support
a hw:cpu_model flavor extra spec. We shouldn't include
this in upstream documentation lest someone be confused
and think upstream nova supports it.

Change-Id: Ie646130731fde648ab84a423024ab59f55f5daab
2018-08-02 16:00:45 -04:00
Zuul
68116ea696 Merge "Improve NeutronFixture and remove unncessary stubbing" 2018-08-02 17:44:55 +00:00
Zuul
318837ff4b Merge "Fix a typo in comment in resource_provider.py" 2018-08-02 15:18:20 +00:00
Zuul
871b47f5f8 Merge "tox: Ensure reused envdirs share the same deps" 2018-08-02 14:59:54 +00:00
Balazs Gibizer
a2224f38d9 Improve NeutronFixture and remove unncessary stubbing
With a small improvement on NeutronFixture that allows creating more
than two ports we can let our tests to exercise and therefore cover
most of the neutronv2/api code as well. To do that this path removes a
lot of fake_network.set_stub_network_methods() calls from the tests.
The remaining calls are not that trivial to remove so those are left in
for a later patch.

The numa functional tests uses the libvirt fixture and during the guest
config xml generation it tries to instantiate OVOs from os_vif. To make
this work the libvirt fixture has to make sure that the
os_vif.initialize() is called as that call registers the OVOs.

Change-Id: I1dbccc2be6ba79bf267edac9208c80e187e6256a
2018-08-02 15:15:03 +02:00
Stephen Finucane
8424f21cfc tox: Ensure reused envdirs share the same deps
In commit df669caf, we modified tox.ini to reuse a shared directory for
environments with the same dependencies. The 'venv' directory was
chosen as this seemed like a suitably generic target. However, the
'venv' target actually uses a non-default values for 'deps' as doc
requirements are included to allow things like reno using this target.
This means switching between the 'venv' target and any of the other
other targets using this directory will result in the virtualenv being
rebuilt each time.

Resolve this by using an explicit directory for these shared
environments. A single setting of 'deps' for one of the targets is
removed to make this clear.

Change-Id: I239606f5f354a5536b20b1d31ee81d80b4572106
2018-08-02 11:31:41 +01:00
Kevin_Zheng
6f5407c16e Fix a typo in comment in resource_provider.py
There is a typo in comment, this comment is very
important for readers to understand what the
function is doing, so fix it.

Trivial-fix

Change-Id: I30f3a385565dc31651d136d80e499be181fa436e
2018-08-02 14:23:23 +08:00
Tetsuro Nakamura
c30d1c995b Refactor AllocationFixture in placement test
This patch refactors the allocation fixture used in some placement
api tests by substituting common functions in test_base.py for the
existing object management functions.

Change-Id: Ide6544d1cf9e1ed154b42075acdd7af986c3afe8
2018-08-02 14:46:10 +09:00
Tetsuro Nakamura
36f6569f63 Increase max_unit in placement test fixture
In the AllocationFixture setup, to avoid the limitation of the
max_unit that is set to the inventory, we were using a bit hacky
way creating multiple allocation objects that have the same resource
class, the same consumer, and the same resource provider.

Since this is not how it works in real cases, and this prevents us
from refactoring, this patch fixes it.

Change-Id: I8ba378ff5eeaf6c9cca11c5874708a17d4640097
2018-08-02 14:00:45 +09:00
Tetsuro Nakamura
38a2ca8663 Use common functions in NonSharedStorageFixture
For code refactoring purpose in the placement NonSharedStorageFixture
setup, this patch substitutes common functions in test_base.py for
existing native setup functions for creating providers and setting
inventories.

Change-Id: I312333ed8ecd51b9f3f6b818c33b3ef54703f997
2018-08-02 10:56:57 +09:00
Zuul
bcde4386fb Merge "Nix unused raise_if_custom_resource_class_pre_v1_1" 2018-08-01 20:18:14 +00:00
Zuul
f2bb5d62a9 Merge "Ensure the order of AllocationRequestResources" 2018-08-01 14:43:02 +00:00
Zuul
54a65893ae Merge "[placement] Use own set_middleware_defaults" 2018-08-01 14:36:02 +00:00
Zuul
5b474d1eb9 Merge "Convert 'placement_api_docs' into a Sphinx extension" 2018-08-01 13:45:35 +00:00
Zuul
d50a1cbb9a Merge "Fix comments in _anchors_for_sharing_providers and related test" 2018-08-01 13:45:28 +00:00
Zuul
5a6c13dbcc Merge "[placement] Use a simplified WarningsFixture" 2018-08-01 13:32:37 +00:00
Zuul
d3ef4092f5 Merge "[placement] Use a non-nova log capture fixture" 2018-08-01 13:32:31 +00:00
Zuul
391bca6db4 Merge "[placement] Use oslotest CaptureOutput fixture" 2018-08-01 13:31:38 +00:00
Zuul
b0570d8ddd Merge "Add placement.concurrent_udpate to generation pre-checks" 2018-08-01 12:19:49 +00:00
Zuul
2770af563b Merge "Define common variables for irrelevant-files" 2018-08-01 09:33:51 +00:00
Yikun Jiang
18ab82e881 Fix comments in _anchors_for_sharing_providers and related test
1. Remove "GROUP by" line in _anchors_for_sharing_providers, because
this has been removed in Ib1738fb4a4664aa7b78398655fd23159a54f5f69.
2. Add reminder note when we are sure all root_provider_id values are
NOT NULL.
3. Fix note in test_anchors_for_sharing_providers, s1 get r3 only via
agg3.

trivialfix

Change-Id: Id8bfd83db58366047267ff0eeb2930a19bddbf4e
2018-08-01 16:34:50 +08:00
Tetsuro Nakamura
31098c2d47 Ensure the order of AllocationRequestResources
Getting allocation candidates with sharing providers, placement
creates a list of AllocationRequestResources to get all the
possible combinations of resource providers in the same aggregate.

However, the order of the list was arbitrary, which could cause
a bug later in duplicate check of the combination.

This patch ensures that the list is ordered by the resource
class id.

Note:
  This bug is only exposed when it is tested with python3.6,
  where order-preserving aspect is added to the dict object.

Change-Id: I2e236fbbc3a4cfd3bd66d50198de643e06d62331
Closes-Bug: #1784577
2018-08-01 15:10:41 +09:00
Zuul
9d39e8a043 Merge "Don't poison Host._init_events if it's already mocked" 2018-07-31 18:28:54 +00:00
Zuul
9f343f5823 Merge "Remove redundant join in _anchors_for_sharing_providers" 2018-07-31 18:28:39 +00:00
Zuul
4ad99dea88 Merge "[placement] Retry allocation writes server side" 2018-07-31 18:28:30 +00:00
Zuul
3aa2e37592 Merge "[placement] api-ref: add traits parameter" 2018-07-31 15:13:22 +00:00
Matthew Booth
60ee18aa3b Don't poison Host._init_events if it's already mocked
PoisonFunctions poisons Host._init_events without checking if it's
already mocked. This creates a proper headscratcher when you try to
use FakeLibvirtFixture with ServersTestBase: if you use
FakeLibvirtFixture before calling super.setUp() PoisonFunctions will
poison the function we already mocked, if we call it in the other
order setUp() will fail when it tries to start nova-compute.

Change-Id: I6d8214c75a67c87ec5a7406f3cc2e8b923aa4d84
2018-07-31 12:17:15 +00:00
Zuul
1eb0d1146e Merge "Update install guide for placement database configuration" 2018-07-31 10:32:49 +00:00
Yikun Jiang
8f8dc5ffc9 Remove redundant join in _anchors_for_sharing_providers
There is a redundant join when we want to get id from
_anchors_for_sharing_providers. The last outerjoin is used to get the
rp.UUID according rp.id, if we set get_id=True, we no longer need this
outer join.

So, we remove the redundant join in this patch.

Change-Id: Ib5fc6e4efae29dd88ce92df834700d2121ed8076
Closes-bug: #1784604
2018-07-31 18:27:39 +08:00
Chris Dent
5d96a6029c [placement] Retry allocation writes server side
This change adds a fast retry loop around
AllocationList._set_allocations if a resource provider generation
conflict happens. It turns out that under high concurrency of allocation
claims being made on the same resource provider conflicts can be quite
common and client side retries are insufficient.

Because both consumer generation and resource provider generations had
raised the same exception there was no way to distinguish between the
two so a child of ConcurrentUpdateDetected has been created as
ResourceProviderConcurrentUpdateDetected. In the future this will allow
us to send different error codes to the client as well, but that change
is not done here.

When the conflict is detected, all the resource providers in the
AllocationList are reloaded and the list objects refreshed.

Logging is provided to indicate:

* at debug that a retry is going to happen
* at warning that all the retries failed and the client is going to
  see the conflict

The tests for this are a bit funky: Some mocks are used to cause the
conflicts, then the real actions after a couple of iterations.

Change-Id: Id614d609fc8f3ed2d2ff29a2b52143f53b3b1b9a
Closes-Bug: #1719933
2018-07-31 10:57:16 +01:00
Takashi NATSUME
54d57a2e88 [placement] api-ref: add traits parameter
In the "List allocation candidates" API,
the 'traits' parameter is missing.
So add it and update examples.

Change-Id: I4a307759f9e8fa80d003d92142f8e99b2c09c9fe
Partial-Bug: #1778670
2018-07-31 14:36:44 +09:00
Zuul
2334dc0bc0 Merge "Add description for placement 1.26" 2018-07-30 20:53:52 +00:00
Chris Dent
66556cf896 [placement] Use a simplified WarningsFixture
Use a WarningsFixture specific to placement that worries about fewer
warnings and is not dependent on nova.

blueprint: placement-extract

Change-Id: Idfcc6882d7fe5141dcc793f0409f75c51fd26234
2018-07-30 19:38:35 +01:00
Chris Dent
5e0fc1c41f [placement] Use a non-nova log capture fixture
We want to avoid using the nova StandardLogging fixture, to limit
imports from nova, but it has two useful features that we want:

* always capture
* if the chosen log level is more than DEBUG, format DEBUG message
  anyway, but don't output

blueprint: placement-extract

Change-Id: Iadd32c731ebfb5a62308a4d5f907a69f93590935
2018-07-30 19:38:22 +01:00
Chris Dent
ed43cb10d9 [placement] Use oslotest CaptureOutput fixture
Instead of the nova fixture OutputStreamCapture. They do effectively
the same thing and once placement is extracted we'd like to not
have duplication.

blueprint: placement-extract

Change-Id: I4636533b1262f819e34ea78cca33ad9f90a35702
2018-07-30 19:12:19 +01:00
Chris Dent
7bb4bd6f29 [placement] Use own set_middleware_defaults
The CORS middleware needs to have its defaults set to something
reaonable. Placement had been using some code in nova to do so.
This change duplicates that code into the placement WSGI app
which had been calling it.

Change-Id: I025a2d14c4126f68e199efa54329797576c50677
blueprint: placement-extract
2018-07-30 19:00:09 +01:00
Zuul
2d7fb828e1 Merge "Add additional functional tests for NUMA networks" 2018-07-30 17:55:13 +00:00
Zuul
7f2f183824 Merge "tox: Reuse envdirs" 2018-07-30 15:02:21 +00:00
Stephen Finucane
da31f67892 Add additional functional tests for NUMA networks
Cover the move and move-like operations through cold migration and
rebuild tests. This requires extensive filling of gaps in the
NeutronFixture and fake libvirt driver.

Part of blueprint numa-aware-vswitches

Change-Id: I18acbda74dd8ac4e9a5d439f1393765a9a6a60f3
Co-Authored-By: Sean Mooney <work@seanmooney.info>
Co-Authored-By: Dan Smith <dms@danplanet.com>
2018-07-30 07:46:57 -07:00
Takashi NATSUME
fbb0d23f79 Add description for placement 1.26
Add description for placement microversion 1.26
in the following APIs in the placement API reference.

* PUT /resource_providers/{uuid}/inventories
* PUT /resource_providers/{uuid}/inventories/{resource_class}

Up to microversion 1.25, the value of 'reserved' has to be less
than the value of 'total' in the request.
Starting from microversion 1.26, it has to be less than or
equal to the value of 'total'.

Change-Id: Id1857abf323c9776ab8fe957202848ff7ab3fbd2
Closes-Bug: #1783380
2018-07-30 22:55:30 +09:00