This changes gabbits/nested-perfload.yaml to create a tree of
providers based on one of the compute nodes in the NUMANetworkFixture
used in the functional tests. For the time being only one type of
compute node is created (of which there will be 1000 instances).
Room is left for future expansion as requirements expand.
The resulting hierarchy has 7 resource providers.
The allocation candidates query is:
GET /allocation_candidates?
resources=DISK_GB:10&
required=COMPUTE_VOLUME_MULTI_ATTACH&
resources_COMPUTE=VCPU:1,MEMORY_MB:256&
required_COMPUTE=CUSTOM_FOO&
resources_FPGA=FPGA:1&
group_policy=none&
same_subtree=_COMPUTE,_FPGA
This is a step in the right direction but is not yet a complete
exercising of all the nested functionality. It is, however, more
complex than prior, notably testing 'same_subtree'. We should
continue to iterate to get it doing more.
Change-Id: I67d8091b464cd7b875b37766f52818a5a2faa780
Story: 2005443
Task: 35669
While experimenting with expanding the nested perfload tests,
it became clear that the call to parallel was not working as
expected because the documents were misread. With help from
Tetsuro the correct incantation was determined so that we
use 50% of available CPUs.
This should leave some space for the database and the web
server.
Subsequent patches will add a more complicated nested structure.
Co-Authored-By: Tetsuro Nakamura <tetsuro.nakamura.bc@hco.ntt.co.jp>
Change-Id: Ie4809abc31212711b96f69e5f291104ae761059e
This was explored as part of discussion with zzzeek on various
ways to make a statement with a large in_ be more performant.
Batching the queries by chunking rp_ids did not help, but an
expanding bindparam did: it makes the surrounding method about
50% faster [1].
bindparam expanding was added in sqlalchemy 1.2.0 so we bump
the requirements here to the latest 1.2.x bugfix release,
1.2.19.
[1] With the caveat that profiling and benchmarking are very
different things, and in this case it is profiling results
which are being analysed.
Change-Id: Ic4e0cdd87f8f2d76b921059ac4bf16a838913abf
When making a nested provider GET /allocation_candidates request
that returns around 6500 providers in the providers summary section,
we make ~22000 ResourceProvider and ProviderSummary objects, 7500
AllocationRequest objects and 30000 ProviderSummaryResource objects.
In aggregate, creating and accessing these objects contributes a
measurable cpu time cost. Using __slots__ [1] can speed up attribute
access (I measured it, it does), so lets use it for the common
objects. There's no evidence (yet) of other objects being an issue
so we won't do this wholesale.
A unit test is updated to change a mock from a public ResourceProvider
method (set_aggregates) to a private function (_set_aggregates)
because one of the limitations of __slots__ is it makes the
slotted objects difficult to mock. In this case the test is still
testing the same thing.
Subsequent patches will explore why we need 22000 ProviderSummary
objects when we're only producing 6500 resulting summaries to see
if there is a cleanup that can be done there.
[1] https://docs.python.org/3/reference/datamodel.html#slots
Change-Id: I4acb16d13976d64d47f798ebbec626a73b79b4f5
It turns out that all the callers of methods that return ProviderIds
access the data using attributes that are already present on the object
provided by a sql alchemy RowProxy object (which is the result of
iterating a ResultProxy object, returned by sqla.execute()). Creating
a namedtuple is redundant and adds some processing cost [1].
However, the AnchorIds namedtuple used in the same file _is_ necessary
because it provides a hashable type required by the set that wraps
the collection of AnchorIds.
The other user of namedtuple, RPCandidate, in
placement/objects/rp_candidates.py has a specific interface for which
a namedtuple is correct.
[1] provider_ids_from_rp_ids was identified as being a major contributor
when doing performance profiling. Most of the cost is executing with the
database, but the tuple creation is a factor.
Change-Id: I1e1522eae7b4360cf57c34164612ae228fad14a3
The integrated-gate-placement adds the tempest-integrated-placement
and grenade-py3 jobs. tempest-integrated-placement runs a subset of
tempest tests which are relevant for placement, e.g. it does not
run keystone tests.
The integrated-gate-py3 template is replaced by this change. It
specified the tempest-full-py3, meaning many tests not relevant
to placement were run.
See the depends-on for more information on the rationale.
Depends-On: https://review.opendev.org/669309
Change-Id: I509ded97085fbae5a0cc55fe66f6e30e638b72bc
The Trait and ResourceClass db objects have the same essential
structure and are used throughout the code in similar ways:
* turn a name into an id
* turn an id into a name
* get an unfiltered list of names
* make a mapping of ids to names
In I409a5e819a72d64e66ee390e4528da0c503d8d05 we made the resource
class cache request specific. Doing that work made it pretty clear
we could have a similar cache for traits and as a result visit the
traits db fewer times per request.
The implementation is straightforward: make an _AttributeCache super
class that is a parent to ResourceClassCache. Create TraitCache as
a sibling. The sole difference is the table used for the data authority
and the exception raised when an attribute is not found.
The new super class has been refactored to use private attributes and
methods.
A 'get_all' method is added to the cache to list the full collection
of dicts it contains. That can be be directly transformed into Trait
and ResourceClass objects. The order of the results of this method
are not predictable, and sorting them would add cost for no benefit,
so gabbi tests which had previously relied on the ordered of returned
resource classes have been removed.
From the API, listing traits and resource classes (without filters) now
uses the cache instead of going to the db. Where filters (in traits) are
required, the db is accessed.
The research_context turns lists of trait names into id, name maps for
required and forbidden traits.
Further, anywhere the traits table was joined to create a name of an id,
the cache is used instead. This allows to drop some joins and operate
fully in-process and in-RAM. No additional loops are added to make this
happen: the translation is done in existing loops.
The upshot of these changes is that unless there is a write operation on
a trait or resource class, both tables are scanned at most once in any
request. And when they are scanned it is to list their entire contents.
As noted in the _AttributeCache docstring there are restrictions
on what kinds of entities can use the cache and some necessary
precautions.
Change-Id: Ia19ea2b4ecdde25323579edf60ad6269d05e75a2
Following on from [1]:
- Since we're only ever calling _fully_qualified_name with module-level
functions, remove the ability to call it with a class (forcing such
usages to either use functions or modify this util).
- Consolidate a redundant invocation of _fully_qualified_name from
version_handler.
[1] Id8f96801116818dda6c68c301619aee185e8fbff
Change-Id: I205a7fd338a0db420b891341c3c40b10f3199fd3
The docs had been using a reference within the nova docs, which
is similar to, but less generic than the API-SIG doc.
Change-Id: I7087e311209716a2907575573b9c018c352ce63d
The api documentation is now published on docs.openstack.org instead
of developer.openstack.org. Update all links that are changed to the
new location.
Note that redirects will be set up as well but let's point now to the
new location.
For details, see:
http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007828.html
Change-Id: Iccddd4356b5d7c28977ad954654d0875972dae97
As part of Train community goal 'Support IPv6-Only Deployments and Testing'[1],
Tempest has defined the new job 'tempest-ipv6-only'(adding
in Depends-On patch) which will deploy services on IPv6 and run smoke
tests and IPv6 related tests present in Tempest.
This job will be part of Nova, Neutron, Cinder, Keystone, Glance, Swift
gate.
Verification structure will be:
- 'devstack-IPv6' deploy the service on IPv6
- 'devstack-tempest-ipv6' run will verify the IPv6-only setting and listen address
- 'tempest-ipv6-only' will run the smoke + IPv6 related test case.
This commit adds the new job 'tempest-ipv6-only' run on gate.
Story: #2005477
Task: #35919
Depends-On: https://review.opendev.org/#/c/671231/
[1] https://governance.openstack.org/tc/goals/train/ipv6-support-and-testing.html
Change-Id: Iff0e8d91048a4644cca4fd9242535b940b721d3b
os-resource-classes 0.5.0 was recently released and upper
constraints were bumped. We have decided to always pin placement
to the latest versions of os-resource-classes and os-traits.
This patch makes it so, again.
Change-Id: Idd4920885b10f0b21b2b67d9526f1048b06f7db5
Experimentation led to the discovery (as described in the story
noted below) that the global RC_CACHE is not safe in a multi-process
environment, which is common for a web service designed to scale
horizontally.
In rare circumstances it is possible for a custom resource class
to be deleted in one process but still appear to exist in another.
For many situations this wouldn't really matter, but there are
cases, even more rare, where it would be possible to write allocations
or resource provider inventory using the wrong resource class
id.
On the related story, a variety of options were discussed to fix
this. Reading through the code this one (which is option 2) was
the only one that proved workable in a relatively straightforward
fashion: Have a per request cache.
To that end, when a RequestContext is created (per request) the
resource class table is scanned to create a cache. Because the
context is local to this request, we no longer need to do any
locking around the cache, either when we create it or when we clear
it: The caller is linear.
The cost of this is that now every single request starts with a
scan of the resource class table. This isn't horrible: if we
had no cache at all we'd be reading rows from that table multiple
times throughout any request (read or write).
We should probably do some performance analysis to see what the
impact of this might be. The perfload jobs may be able to give
a limited sense of what the impact is, but profiling will be
required for accuracy.
It is the case that the functional tests seem a bit slower because
of that additional db query.
Change-Id: I409a5e819a72d64e66ee390e4528da0c503d8d05
Story: 2006232
Task: 35833
Placement can run just fine on OSX but running the tests can
be a bit cumbersome since dev libraries for ssl, postresql,
and mysql are required (directly or indirectly) by
test-requirements.txt, even though for non-gate testing we don't
usually need them, and dealing with dynamic libraries is a
frequent source of pain and discomfort on OSX.
This change says "don't import SQL drivers for mysql and postgresql
when testing on darwin".
Change-Id: Ic0ff55f00e0312251207fd079cf7b7e98d950c87
_fully_qualified_name was copied from Twisted. The original
version does more than Placement needs. We only need to
handle module-level classes and functions (and really only
functions). No nested of those, nor object methods in any
form.
This is because we only use the method for finding the
right microversioned handler of a method with the same
name and placement HTTP handlers are universally functions.
The previous version of the method had code that was never
called. Here, that dead code is removed and the remainder
is simplified.
If it is called incorrectly there will be an (intentionally)
untrapped TypeError during running of the tests.
N.B.: This was discovered while reviewing coverage data.
Change-Id: Id8f96801116818dda6c68c301619aee185e8fbff
The API reference for POST /allocations [1] and PUT
/allocations/{consumer_uuid} [2] specifically mentions that you can get
a 409 on provider/inventory conflict. In microversion 1.28, it also
became possible to get a 409 on an allocation (consumer generation)
conflict.
In the process of adding that information, it became evident that we
weren't doing a good job explaining the whole generation thing in
general, so this commit also adds a descriptive section to the front
matter of the API reference.
Links are included from the updated descriptions for the two affected
allocations operations. Future commits can add links from other
appropriate sections of the reference (e.g. the parameters.yaml entries
for resource provider and consumer generation fields). Future commits
could also enhance the descriptions of error codes for other operations
to (at least) the level of detail at which these have ended up.
[1] https://developer.openstack.org/api-ref/placement/?expanded=manage-allocations-detail#manage-allocations
[2] https://developer.openstack.org/api-ref/placement/?expanded=update-allocations-detail#update-allocations
Change-Id: I42e76785e0fe456b107fe843dbb242f2c5f5b9f7
Story: #2006180
Task: #35705
This commit documents a new error code placement.query.missing_value,
which was added via [1] but missed in the summary table in the API
reference [2].
Story: #2006194
Task: #35776
[1] 8395e3f099022d8506ed5e6624582ec03e46c3b9
[2] https://developer.openstack.org/api-ref/placement/#errors
Change-Id: I36ee636946c2e8b2c4b07b0449460bd31960f637
This patch adds usage documents for the new `same_subtree` queryparam
in `Modeling with Provider Trees` page.
Change-Id: I52a48692aa7069eb16ce4f078cc671fe38728729
Story: 2005575
Task: 30784
This commit documents new error codes placement.query.duplicate_key and
placement.query.bad_value, which were were added via [1] but missed in
the summary table in the API reference [2].
Story: #2006194
Task: #35776
[1] I76cad83248920fa71da122711f1f763c4ebdb1ba
[2] https://developer.openstack.org/api-ref/placement/#errors
Change-Id: I024eaa38c6574f5847d585c83d527e3374031105
This patch fixes incorrect usages of fixed articles changing them to
indefinite articles in same_subtree documentation.
Change-Id: I6ba2e2f13400b4c2ae9a44cad7a1fc9a9e39b41d
Story: 2005575
Task: 30784
This patch fixes some comments in code:
- Now suffixes can be string in _merge_candidates()
- Remove old doc for `allow_forbidden` parameter in
RequestGroup.dict_from_request
- Delete unnecessary space in ResourceClassCache
Change-Id: I6ffc19eea84b45dd452bc61d2a2007b6572719ad
A new same_subtree query parameter will be accepted. The value is
a comma-separated list of request group suffix strings $S. Each must
exactly match a suffix on a granular group somewhere else in the
request. Importantly, the identified request groups need not have
a resources$S.
If this is provided, at least one of the resource providers satisfying
the specified request group must be an ancestor of the rest.
The same_subtree query parameter can be repeated and each repeat group
is treated independently.
Co-Authored-By: Chris Dent <cdent@anticdent.org>
Change-Id: I7fdeac24606359d37f1a7405d22c5797840e1a9e
Story: 2005575
Task: 30784
We have an 'api-ref' docs target - use that instead.
Change-Id: Ie94add90b507d631a74f4617de7feba56fb0c646
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Use whereto to test that the redirect rules do what we expect.
Change-Id: Id3a8f3b9f372ebe9176f1df917f7a6aac30a8e92
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
