Re-propose nested resource providers spec
The only change from the Pike version is that I've added wording that the call to `GET /allocation_candidates` will also need to be modified to include the root filter. I've also updated the History section. Previously-approved: Ocata Previously-approved: Pike Blueprint: nested-resource-providers Change-Id: I6b5531a0fe8c1aa5056d25ae47901b05e241eb9e
This commit is contained in:
Ed Leafe
281
specs/queens/approved/nested-resource-providers.rst
Normal file
281
specs/queens/approved/nested-resource-providers.rst
Normal file
@@ -0,0 +1,281 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
=========================
|
||||||
|
Nested Resource Providers
|
||||||
|
=========================
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/nova/+spec/nested-resource-providers
|
||||||
|
|
||||||
|
We propose changing the database schema, object model and REST API of resource
|
||||||
|
providers to allow a hierarchical relationship among different resource
|
||||||
|
providers to be represented.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
With the addition of the new placement API, we now have a new way to account
|
||||||
|
for quantitative resources in the system. Resource providers contain
|
||||||
|
inventories of various resource classes. These inventories are simple integer
|
||||||
|
amounts and, along with the concept of allocation records, are designed to
|
||||||
|
answer the questions:
|
||||||
|
|
||||||
|
* "how many of a type of resource does this provider have available?"
|
||||||
|
* "how much of a type of resource is being consumed in the system?"
|
||||||
|
* "what level of over-commit does each provider expose for each type of
|
||||||
|
resource?"
|
||||||
|
|
||||||
|
In the initial version of the resource provider schema in the placement API, we
|
||||||
|
stuck with a simple world-view that resource providers could be related to each
|
||||||
|
other only via an aggregate relationship. In other words, a resource provider
|
||||||
|
"X" may provide shared resources to a set of other resource providers "S" if
|
||||||
|
and only if "X" was associated with an aggregate "A" that all members of "S"
|
||||||
|
were also associated with.
|
||||||
|
|
||||||
|
This relationship works perfectly fine for things like shared storage or IP
|
||||||
|
pools. However, certain classes of resource require a more parent->child
|
||||||
|
relationship than a many-to-many relationship that the aggregate association
|
||||||
|
offers. Two examples of where a parent->child relationship is more appropriate
|
||||||
|
are when handling VCPU/MEMORY_MB resources on NUMA nodes on a compute host and
|
||||||
|
when handling SRIOV_NET_VF resources for NICs on a compute host.
|
||||||
|
|
||||||
|
In the case of NUMA nodes, the system must be able to track how many VCPU and
|
||||||
|
MEMORY_MB have been allocated from each individual NUMA node on the host.
|
||||||
|
Allocating memory to a guest and having that memory span address space across
|
||||||
|
two banks of DIMMs attached to different NUMA nodes results in sub-optimal
|
||||||
|
performance, and for certain high-performance guest workloads this penalty is
|
||||||
|
not acceptable.
|
||||||
|
|
||||||
|
Another example is the SRIOV_NET_VF resource class, which is provided by
|
||||||
|
SRIOV-enabled network interface cards. In the case of multiple SRIOV-enabled
|
||||||
|
NICs on a compute host, different qualitative traits may be tagged to each NIC.
|
||||||
|
For example, the NIC called enp2s0 might have a trait "CUSTOM_PHYSNET_PUBLIC"
|
||||||
|
indicating that the NIC is attached to a physical network called "public". The
|
||||||
|
NIC enp2s1 might have a trait "CUSTOM_PHYSNET_PRIVATE" that indicates the NIC
|
||||||
|
is attached to the physical network called "Intranet". We need a way of
|
||||||
|
representing that these NICs each provide SRIOV_NET_VF resources but those
|
||||||
|
virtual functions are associated with different physical networks. In the
|
||||||
|
resource providers data modeling, the entity which is associated with
|
||||||
|
qualitative traits is the **resource provider** object. Therefore, we require a
|
||||||
|
way of representing that the SRIOV-enabled NICs are themselves resource
|
||||||
|
providers with inventories of SRIOV_NET_VF resources. Those resource providers
|
||||||
|
are contained on a compute host which is a resource provider that has inventory
|
||||||
|
records for *other* types of resources such as VCPU, MEMORY_MB or DISK_GB.
|
||||||
|
|
||||||
|
This spec proposes that nested resource providers be created to allow for
|
||||||
|
distinguishing details of complex components of some resource providers. During
|
||||||
|
review the question came up about "rolling up" amounts of these nested
|
||||||
|
providers to the root level. Imagine this scenario: I have a NIC with two PFs,
|
||||||
|
each of which has only 1 VF available, and I get a request for 2 VFs without
|
||||||
|
any traits to distinguish them. Since there is no single resource provider that
|
||||||
|
can satisfy this request, it will not select this root provider, even though
|
||||||
|
the root provider "owns" 2 VFs. This spec does not propose any sort of "rolling
|
||||||
|
up" of inventory, but this may be something to consider in the future. If it is
|
||||||
|
an idea that has support, another BP/spec can be created then to add this
|
||||||
|
behavior.
|
||||||
|
|
||||||
|
Use Cases
|
||||||
|
---------
|
||||||
|
|
||||||
|
As an NFV cloud operator, I wish to request that my VNF workload needs an SRIOV
|
||||||
|
virtual function on a NIC that is tagged to the physical network "public" and I
|
||||||
|
want to be able to view the resource consumption of SRIOV virtual functions on
|
||||||
|
a per-physical-network basis.
|
||||||
|
|
||||||
|
As an NFV cloud operator, I wish to ensure that the memory and vCPU assigned to
|
||||||
|
my workload is local to a particular NUMA topology and that those resources are
|
||||||
|
represented in unique inventories per NUMA node and reported as separate
|
||||||
|
allocations.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
We will add two new attributes to the resource provider data model:
|
||||||
|
|
||||||
|
* `parent_provider_uuid`: Indicates the UUID of the immediate parent provider.
|
||||||
|
This will be None for the vast majority of providers, and for nested resource
|
||||||
|
providers, this will most likely be the compute host's UUID. To be clear,
|
||||||
|
a resource provider can have 0 or 1 parents. We will not support multiple
|
||||||
|
parents for a resource provider.
|
||||||
|
* `root_provider_uuid`: Indicates the UUID of the resource provider that is at
|
||||||
|
the "root" of the tree of providers. This field allows us to implement
|
||||||
|
efficient tree-access queries and avoid use of recursive queries to follow
|
||||||
|
child->parent relations.
|
||||||
|
|
||||||
|
A new microversion will be added to the placement REST API that adds the above
|
||||||
|
attributes to the appropriate request and response payloads.
|
||||||
|
|
||||||
|
The scheduler reporting client shall be modified to track NUMA nodes and
|
||||||
|
SRIOV-enabled NICs as child resource providers to a parent compute host
|
||||||
|
resource provider.
|
||||||
|
|
||||||
|
The `VCPU` and `MEMORY_MB` resource classes will continue to be inventoried on
|
||||||
|
the parent resource provider (i.e the compute node resource provider) and not
|
||||||
|
the NUMA node child providers. The NUMA node child providers will have
|
||||||
|
inventory records populated for the `NUMA_CORE`, `NUMA_THREAD` and
|
||||||
|
`NUMA_MEMORY_MB` resource classes. When a boot request is received, the Nova
|
||||||
|
API service will need to determine whether the request (flavor and image)
|
||||||
|
specifies a particular NUMA topology and, if so, construct the request to the
|
||||||
|
placement service for the appropriate `NUMA_XXX` resources. This is currently
|
||||||
|
out of scope for this spec. This spec is only about the inventorying of the
|
||||||
|
various child providers with appropriate resource classes.
|
||||||
|
|
||||||
|
On the CPU-pinning side of the equation, we do not plan to allow a compute node
|
||||||
|
to serve as *either* a general-purpose compute node *or* as a target for
|
||||||
|
NUMA-specific (pinned) workloads. A compute node will be either a target for
|
||||||
|
pinned workloads or it will be a target for generic (floating CPU) workloads.
|
||||||
|
It is not yet clear what we will use to indicate that a compute node targets
|
||||||
|
floating workloads or not. Initial thoughts were to use the
|
||||||
|
pci_passthrough_whitelist CONF option to determine this however this still
|
||||||
|
needs to be debated.
|
||||||
|
|
||||||
|
This spec will simply ensure that if a virt driver returns a NUMATopology
|
||||||
|
object in the result of its get_available_resource() call, then we will create
|
||||||
|
child resource providers representing those NUMA nodes. Similarly, if the PCI
|
||||||
|
device manager returns a set of SR-IOV physical functions on the compute host,
|
||||||
|
we will create child resource provider records for those SR-IOV PFs.
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
We could try hiding the `root_provider_uuid` attribute from the GET
|
||||||
|
/resource-provider[s] REST API response payload to reduce complexity of the
|
||||||
|
API. We will still, however, need a REST API call that "gets all resource
|
||||||
|
providers in a tree" where the user would pass a UUID and we'd look up all
|
||||||
|
resource providers having that UUID as their root provider UUID.
|
||||||
|
|
||||||
|
Instead of having a concept of nested resource providers, we could force
|
||||||
|
deployers to create custom resource classes for every permutation of physical
|
||||||
|
network trait. For instance, assuming the example above, the operator would
|
||||||
|
need to create an SRIOV_NET_VF_PUBLIC_NET and a SRIOV_NET_VF_INTRANET_NET
|
||||||
|
custom resource class and then manually set the inventory of the compute node
|
||||||
|
resource provider to an amount of VFs each PF exposed. The problem with this
|
||||||
|
approach is two-fold. First, we no longer have any standardization on the
|
||||||
|
SRIOV_NET_VF resource class. Secondly, we are coupling the qualitative and
|
||||||
|
quantitative aspects of a provider together again, which is part of the problem
|
||||||
|
with the existing Nova codebase and why it has been hard to standardize the
|
||||||
|
tracking and scheduling of resources in the first place.
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
Two new fields will be added to the `resource_providers` DB table:
|
||||||
|
|
||||||
|
* `root_provider_uuid`: This will be populated using an online data migration
|
||||||
|
that sets `root_provider_uuid` to the value of the `resource_providers.uuid`
|
||||||
|
field for all existing resource providers.
|
||||||
|
* `parent_provider_uuid`: This will be a NULLable field and default to NULL
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
`root_provider_uuid` and `parent_provider_uuid` fields will be added to the
|
||||||
|
corresponding request and response payloads of appropriate placement REST APIs.
|
||||||
|
|
||||||
|
The `GET /resource_providers` call will get a new filter on `root={uuid}` that,
|
||||||
|
when present, will return all resource provider records, inclusive of the root,
|
||||||
|
having a `root_provider_uuid` equal to `{uuid}`.
|
||||||
|
|
||||||
|
The filter parameter `root={uuid}` will *not* be added to
|
||||||
|
`GET /allocation_candidates`, as this call is for a specific use case for the
|
||||||
|
Nova scheduler, and there is no use case for it.
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Notifications impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
None. The setting and getting of provider tree information will be entirely
|
||||||
|
handled in the `nova-compute` worker with no changes needed by the deployer.
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
jaypipes
|
||||||
|
|
||||||
|
Other contributors:
|
||||||
|
cdent
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
* Add DB schema and object model changes
|
||||||
|
* Add REST API microversion adding new attributes for resource providers and
|
||||||
|
allocation candidates
|
||||||
|
* Add REST API microversion adding new `root={uuid}` filter on `GET
|
||||||
|
/resource_providers`
|
||||||
|
* Add code in scheduler reporting client to track NUMA nodes as child resource
|
||||||
|
providers on the parent compute host resource provider
|
||||||
|
* Add code in scheduler reporting client to track SRIOV PFs as child resource
|
||||||
|
providers on the parent compute host resource provider
|
||||||
|
|
||||||
|
Please note that not all of this spec is expected to be implemented in a single
|
||||||
|
release cycle. At the Queens PTG we agreed that fully suppporting NUMA will
|
||||||
|
probably have to be deferred to the next release.
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
Most of the focus will be on functional tests for the DB/server and the REST
|
||||||
|
API with new functional tests added for the specific NUMA and SRIOV PF child
|
||||||
|
provider scenarios described in this spec.
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
Some devref content should be written.
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
http://etherpad.openstack.org/p/nested-resource-providers
|
||||||
|
|
||||||
|
History
|
||||||
|
=======
|
||||||
|
|
||||||
|
.. list-table:: Revisions
|
||||||
|
:header-rows: 1
|
||||||
|
|
||||||
|
* - Release Name
|
||||||
|
- Description
|
||||||
|
* - Ocata
|
||||||
|
- Introduced
|
||||||
|
* - Pike
|
||||||
|
- Re-proposed
|
||||||
|
* - Queens
|
||||||
|
- Re-proposed
|
||||||
Reference in New Issue
Block a user