nova-specs/specs/rocky/approved/update-provider-tree.rst

Resource tracker allows virt driver to update provider tree

https://blueprints.launchpad.net/nova/+spec/update-provider-tree

In the movement towards using placement for scheduling and resource management, the virt driver method get_available_resource was initially superseded by get_inventory, whereby the driver could specify its inventory in terms understood by placement. In Queens, a get_traits driver method was added. But get_inventory is limited to expressing only inventory (not traits or aggregates). And both of these methods are limited to the resource provider corresponding to the compute node.

Recent developments such as Nested Resource Providers necessitate the ability for the virt driver to have deeper control over what the resource tracker configures in placement on behalf of the compute node. This blueprint proposes a new virt driver method, update_provider_tree, and its method of consumption by the resource tracker, allowing full control over the placement representation of the compute node and its associated providers and metadata.

Problem description

Existing virt driver methods are limited in their ability to express resource provider information.

Use Cases

As a virt driver developer, I wish to be able to model my compute node and associated entities as any combination of provider trees and sharing providers, along with inventories, traits, and aggregate associations for those providers.

Proposed change

ComputeDriver.update_provider_tree

ComputeDriver.update_provider_tree is introduced. It accepts two parameters:

• A nova.compute.provider_tree.ProviderTree object representing all the providers in the tree associated with the compute node, and any sharing providers (those with the MISC_SHARES_VIA_AGGREGATE trait) associated via aggregate with any of those providers (but not their tree- or aggregate-associated providers), as currently known by placement. This object is fully owned by the update_provider_tree method, and can therefore be modified without locking/concurrency considerations. Note, however, that it may contain providers not directly owned/controlled by the compute host. Care must be taken not to remove or modify such providers inadvertently.
• String name of the compute node (i.e. ComputeNode.hypervisor_hostname) for which the caller is updating providers and inventory. Drivers may use this to help identify the compute node provider in the ProviderTree. Drivers managing more than one node (e.g. ironic) may also use it as a cue to indicate which node is being updated.

The virt driver is expected to update the ProviderTree object with current resource provider and inventory information. When the method returns, the ProviderTree should represent the correct hierarchy of nested resource providers associated with this compute node, as well as the inventory, aggregates, and traits associated with those resource providers.

Note

Despite the name, a ProviderTree instance may in fact contain more than one tree. For purposes of this specification, the ProviderTree passed to update_provider_tree will contain:

• the entire tree associated with the compute node; and
• any sharing providers (those with the MISC_SHARES_VIA_AGGREGATE trait) which are associated via aggregate with any of the providers in the compute node's tree. The sharing providers will be presented as lone roots in the ProviderTree, even if they happen to be part of a tree themselves.

Consider the example below. SSP is a shared storage provider and BW1 and BW2 are shared bandwidth providers; all three have the MISC_SHARES_VIA_AGGREGATE trait:

CN1                 SHR_ROOT               CN2
/   \       agg1    /   /\     agg1        /   \
NUMA1     NUMA2--------SSP--/--\-----------NUMA1     NUMA2
/     \   /    \            /    \         /     \   /    \
PF1    PF2 PF3   PF4--------BW1   BW2------PF1    PF2 PF3   PF4
            agg2             agg3

When update_provider_tree is invoked for CN1, it is passed a ProviderTree containing:

CN1 (root)
/   \       agg1
NUMA1     NUMA2-------SSP (root)
/     \   /    \
PF1    PF2 PF3   PF4------BW1 (root)
            agg2

This method supersedes get_inventory and get_traits: if this method is implemented, neither get_inventory nor get_traits is used.

Driver implementations of update_provider_tree are expected to use public ProviderTree methods to effect changes to the provider tree passed in. Some of the methods which may be useful are as follows:

• new_root: Add a new root provider to the tree.
• new_child: Add a new child under an existing provider.
• data: Access information (name, UUID, parent, inventory, traits, aggregates) about a provider in the tree.
• remove: Remove a provider and its descendants from the tree. Use caution in multiple-ownership scenarios.
• update_inventory: Set the inventory for a provider.
• update_traits: Set the traits for a provider.
• update_aggregates: Set the aggregates for a provider.

Note

There is no supported mechanism for update_provider_tree to effect changes to allocations. This is intentional: in Nova, allocations are managed exclusively outside of virt. (Usually by the scheduler; sometimes - e.g. for migrations - by the conductor.)

Porting from get_inventory

Virt driver developers wishing to move from get_inventory to update_provider_tree should use the ProviderTree.update_inventory method, specifying the compute node as the provider and the same inventory as returned by get_inventory. For example:

def get_inventory(self, nodename):
    inv_data = {
        'VCPU': { ... },
        'MEMORY_MB': { ... },
        'DISK_GB': { ... },
    }
    return inv_data

would become:

def update_provider_tree(self, provider_tree, nodename):
    inv_data = {
        'VCPU': { ... },
        'MEMORY_MB': { ... },
        'DISK_GB': { ... },
    }
    provider_tree.update_inventory(nodename, inv_data)

Porting from get_traits

To replace get_traits, developers should use the ProviderTree.update_traits method, specifying the compute node as the provider and the same traits as returned by get_traits. For example:

def get_traits(self, nodename):
    traits = ['HW_CPU_X86_AVX', 'HW_CPU_X86_AVX2', 'CUSTOM_GOLD']
    return traits

would become:

def update_provider_tree(self, provider_tree, nodename):
    traits = ['HW_CPU_X86_AVX', 'HW_CPU_X86_AVX2', 'CUSTOM_GOLD']
    provider_tree.update_traits(nodename, traits)

SchedulerReportClient.update_from_provider_tree

This is the report client method responsible for accepting the ProviderTree as modified by the virt driver via update_provider_tree and making the necessary placement API calls to ensure that the representation in the placement service matches it. In particular:

• Providers removed by update_provider_tree are removed from placement.
• Providers added by update_provider_tree are created in placement.
• If inventories, traits, or aggregates were changed for any providers by update_provider_tree, those changes are flushed back to placement.

Note

In multiple-ownership scenarios, virt drivers should be careful not to remove or modify providers not owned by the compute host.

ResourceTracker._update

This is where the virt driver is asked to report on compute resources. It is where, for example, the call to get_inventory was added to supersede the data returned by get_available_resource if get_inventory is implemented. Here we add another level to allow update_provider_tree to supersede get_inventory. The logic changes from:

try:
    ComputeDriver.get_inventory()
except NotImplementedError:
    SchedulerReportClient.update_compute_node()

try:
    ComputeDriver.get_traits()
except NotImplementedError:
    pass

to:

try:
    ComputeDriver.update_provider_tree()
    SchedulerReportClient.update_from_provider_tree()
except NotImplementedError:
    try:
        ComputeDriver.get_inventory()
    except NotImplementedError:
        SchedulerReportClient.update_compute_node()

    try:
        ComputeDriver.get_traits()
    except NotImplementedError:
        pass

Alternatives

• Continue to provide piecemeal methods in the spirit of get_inventory and get_traits. The proposed solution can subsume the functionality of both of those methods and more, but it can also grow along with placement and Nova's use thereof.
• Allow virt drivers direct control over placement. While we can't stop out-of-tree drivers from doing this, it has been discussed and decided that in-tree drivers should be funneled through the choke point of the SchedulerReportClient for actual placement API communication.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

No direct impact. This change, followed by virt drivers implementing update_provider_tree, followed by virt drivers extending their resource provider models, will ultimately allow operators to exert more power over scheduling operations.

Performance Impact

This change increases the amount of traffic to the placement service, which has the potential to affect performance. However, there is as yet no evidence that doing lots of placement calls is "expensive" relative to the other processing occurring in these code paths. The intent is to mitigate such impact if and when it is demonstrated to be problematic.

One mitigation strategy, already largely implemented, is caching the placement representation locally via a separate ProviderTree instance maintained in the SchedulerReportClient. The specifics are outside the scope of this document. However, the existing code in this area is inconsistent and needs to be codified in a separate specification so we can work towards consistency.

Other deployer impact

None

Developer impact

See above.

Upgrade impact

None

Implementation

Assignee(s)

Primary assignee:

efried

Work Items

The code for this has been completed. Some of it merged in Queens, including:

• https://review.openstack.org/#/c/521187/ introduces the update_provider_tree method in the ComputeDriver base class.
• https://review.openstack.org/#/c/533821/ implements the update_from_provider_tree method in the report client.
• https://review.openstack.org/#/c/520246/ implements the changes in the resource tracker to use the above.

These changes were developed under the Nested Resource Providers blueprint.

Dependencies

None (all dependencies have merged in Queens).

Continuing development of such features as Nested Resource Providers, Granular Resource Requests, and shared resource providers will expand the range of things driver developers can do through their implementation of update_provider_tree.

Testing

Extensive functional testing is included in addition to unit tests.

Documentation Impact

None

References

• Nested Resource Providers spec
• Support Traits in Allocation Candidates spec
• Support traits in the Ironic driver spec (get_traits)
• Granular Resource Requests spec

History

Revisions

Release Name	Description
Queens	Code finished and mostly merged.
Rocky	Figured we really ought to have something written down, so proposed an actual blueprint and this spec.