Add resource object models
Specification for describing resource amounts and usage records using a generalized class interface. Change-Id: I37b8a1c8b0bbf73bbcf2ee2c2d29018039eb1a9a Blueprint: resource-objects
This commit is contained in:
Jay Pipes
389
specs/kilo/approved/resource-objects.rst
Normal file
389
specs/kilo/approved/resource-objects.rst
Normal file
@@ -0,0 +1,389 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
==========================
|
||||||
|
Model resources as objects
|
||||||
|
==========================
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/nova/+spec/resource-objects
|
||||||
|
|
||||||
|
Adds model objects to represent the resources that may be requested
|
||||||
|
for an instance.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
In Nova, we have a very loose way of modeling the resources that are
|
||||||
|
consumed by virtual machine instances and provided by compute nodes.
|
||||||
|
The Flavor object has a number of static fields that correspond to amounts
|
||||||
|
of simple resources like CPU, RAM, and local disk. We use dictionaries
|
||||||
|
of key/value pairs and JSON-serialized BLOBs of data to model other types
|
||||||
|
of resources, like PCIe devices or NUMA cell layouts.
|
||||||
|
|
||||||
|
The resource tracker on the compute node keeps track of the collection of
|
||||||
|
resources that are consumed on the node. The `ResourceTracker.old_resources`
|
||||||
|
attribute is a dictionary with a bunch of random nested dictionaries. Some of
|
||||||
|
these nested dictionaries include the 'stats' dict for the extensible resource
|
||||||
|
tracker, various 'pci_devices', 'pci_stats', and 'pci_passthrough_devices'
|
||||||
|
things, a 'numa_topology' blob that stores a JSON-serialized representation of
|
||||||
|
an object in `nova.virt.hardware`, and a 'metrics' dictionary with completely
|
||||||
|
unstructured and undocumented key/value pairs. In addition to these
|
||||||
|
unstructured nested dictionaries, the `ResourceTracker.old_resources`
|
||||||
|
dictionary contains top-level keys including some that match the simple
|
||||||
|
resource types that a Flavor object exposes:
|
||||||
|
|
||||||
|
- `local_gb_used`: Amount of disk in GB used on the compute node
|
||||||
|
- `local_gb`: Total GB of local disk capacity the compute node provides
|
||||||
|
- `free_disk_gb`: Calculated amount of disk the compute node has available
|
||||||
|
- `vcpus_used`: Number of vCPUs consumed on the compute node
|
||||||
|
- `vcpus`: Total number of vCPUs the compute node provides
|
||||||
|
- `free_vcpus`: Calculated number of vCPUs the compute node has available
|
||||||
|
- `memory_mb_used`: Amount of RAM in MB used on the compute node
|
||||||
|
- `memory_mb`: Total MB of RAM capacity the compute node provides
|
||||||
|
- `free_ram_mb`: Calculated amount of RAM the compute node has available
|
||||||
|
- `running_vms`: Number of virtual machine instances running on the node
|
||||||
|
- `current_workload`: Some calculated value of the workload on the node
|
||||||
|
|
||||||
|
Unfortunately, none of the above is documented in the code, and in order to add
|
||||||
|
new features to the scheduler, people just continue to add these free-form keys
|
||||||
|
and nested dictionaries to the dictionary. This makes communicating actual
|
||||||
|
usage amounts to the scheduler error-prone: the resource tracker calls the
|
||||||
|
`scheduler_client.update_resource_stats()` method, passing in this
|
||||||
|
unstructured, unversioned dictionary of information as-is. This means the
|
||||||
|
scheduler interface is incredibly fragile since the interface can be altered on
|
||||||
|
a whim by any developer who decides to add a new key to the free-form
|
||||||
|
dictionary of resources. Typos in resource dictionary keys can be very easy to
|
||||||
|
miss in code reviews, and frankly, there is virtually no functional testing for
|
||||||
|
a lot of the edge case code in the resource tracker around the extensible
|
||||||
|
resource tracker.
|
||||||
|
|
||||||
|
In addition to the problem of fragile interfaces, the free-form nature of
|
||||||
|
the resources dictionary has meant that different resources are tracked
|
||||||
|
in different ways. PCI resources are tracked one way, NUMA topology usage
|
||||||
|
is tracked in a different way, CPU/RAM/disk is tracked differently, and
|
||||||
|
any resources modeled in the complete free-for-all of the extensible
|
||||||
|
resource tracker are tracked in an entirely different way, using plugins
|
||||||
|
that modify a supplied 'stats' nested dictionary.
|
||||||
|
|
||||||
|
An example of the mess this has created in the resource tracker can be
|
||||||
|
seen here:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
def _update(self, context, values):
|
||||||
|
"""Update partial stats locally and populate them to Scheduler."""
|
||||||
|
self._write_ext_resources(values)
|
||||||
|
# NOTE(pmurray): the stats field is stored as a json string. The
|
||||||
|
# json conversion will be done automatically by the ComputeNode object
|
||||||
|
# so this can be removed when using ComputeNode.
|
||||||
|
values['stats'] = jsonutils.dumps(values['stats'])
|
||||||
|
|
||||||
|
if not self._resource_change(values):
|
||||||
|
return
|
||||||
|
if "service" in self.compute_node:
|
||||||
|
del self.compute_node['service']
|
||||||
|
# NOTE(sbauza): Now the DB update is asynchronous, we need to locally
|
||||||
|
# update the values
|
||||||
|
self.compute_node.update(values)
|
||||||
|
# Persist the stats to the Scheduler
|
||||||
|
self._update_resource_stats(context, values)
|
||||||
|
if self.pci_tracker:
|
||||||
|
self.pci_tracker.save(context)
|
||||||
|
|
||||||
|
If resources were actually modeled consistently, the above code would look like
|
||||||
|
this instead:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
def _update(self, context, resources):
|
||||||
|
if not self._resource_change(resources):
|
||||||
|
return
|
||||||
|
# Notify the scheduler about changed resources
|
||||||
|
scheduler_client.update_usage_for_compute_node(
|
||||||
|
context, self.compute_node.id, resources)
|
||||||
|
|
||||||
|
Similarly, the following code in the existing resource tracker:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
def _update_usage(self, context, resources, usage, sign=1):
|
||||||
|
mem_usage = usage['memory_mb']
|
||||||
|
|
||||||
|
overhead = self.driver.estimate_instance_overhead(usage)
|
||||||
|
mem_usage += overhead['memory_mb']
|
||||||
|
|
||||||
|
resources['memory_mb_used'] += sign * mem_usage
|
||||||
|
resources['local_gb_used'] += sign * usage.get('root_gb', 0)
|
||||||
|
resources['local_gb_used'] += sign * usage.get('ephemeral_gb', 0)
|
||||||
|
|
||||||
|
# free ram and disk may be negative, depending on policy:
|
||||||
|
resources['free_ram_mb'] = (resources['memory_mb'] -
|
||||||
|
resources['memory_mb_used'])
|
||||||
|
resources['free_disk_gb'] = (resources['local_gb'] -
|
||||||
|
resources['local_gb_used'])
|
||||||
|
|
||||||
|
resources['running_vms'] = self.stats.num_instances
|
||||||
|
self.ext_resources_handler.update_from_instance(usage, sign)
|
||||||
|
|
||||||
|
# Calculate the numa usage
|
||||||
|
free = sign == -1
|
||||||
|
updated_numa_topology = hardware.get_host_numa_usage_from_instance(
|
||||||
|
resources, usage, free)
|
||||||
|
resources['numa_topology'] = updated_numa_topology
|
||||||
|
|
||||||
|
would instead look like this:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
def _update_usage(self, context, resources, usage, sign=1):
|
||||||
|
for resource_type, amount in usage.items():
|
||||||
|
resources[resource_type].consume(amount*sign)
|
||||||
|
|
||||||
|
Use Cases
|
||||||
|
----------
|
||||||
|
|
||||||
|
Nova contributors wish to extend the functionality of the scheduler and intend
|
||||||
|
to break the scheduler out into the Gantt project. In order to do this
|
||||||
|
effectively, the internal interfaces around the resource tracker and the
|
||||||
|
scheduler must be cleaned up to use structured objects.
|
||||||
|
|
||||||
|
Project Priority
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
This blueprint is part of the `scheduler` refactoring effort defined as a
|
||||||
|
priority for the Kilo release.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
Modeling requested and used resource amounts is the foundational building block
|
||||||
|
that must be done first before any further refactoring or cleanup of the
|
||||||
|
scheduler or resource tracker interfaces.
|
||||||
|
|
||||||
|
This blueprint encompasses the addition of a set of model objects:
|
||||||
|
|
||||||
|
- A set of classes that represent amounts of different types of
|
||||||
|
resources
|
||||||
|
- A set of classes that represent usage of different types of
|
||||||
|
resources
|
||||||
|
- A set of classes that represent different types of resources
|
||||||
|
|
||||||
|
These amount, usage, and resource type classes will be `nova.objects` object
|
||||||
|
classes, and will enable Nova evolve, in a versioned manner, the way that it
|
||||||
|
tracks resources and exposes usage data about consumed resources.
|
||||||
|
|
||||||
|
The goals of the extensible resource tracker was to put in place a framework
|
||||||
|
that allowed adding new resource types and allowed accounting for those
|
||||||
|
resources in different ways. While this blueprint does indeed remove the
|
||||||
|
extensible resource tracker, because these amount, usage, and resource type
|
||||||
|
classes are being added as `nova.object` objects, we will gain the ability
|
||||||
|
to evolve and enhance our resource tracking with object versioning and
|
||||||
|
adding new resource type classes. This approach will provide the flexibility
|
||||||
|
that the ERT intended, but with the stability that the nova objects system.
|
||||||
|
|
||||||
|
The resource tracker code will then be converted to use the above classes when
|
||||||
|
representing usage of all resources on a compute node.
|
||||||
|
|
||||||
|
A new conductor RPC API method `update_compute_node_resource_usages` method
|
||||||
|
will be added that gets a `nova.objects.ComputeNode` object and a
|
||||||
|
`nova.objects.ResourceUsageCollection` object and updates the compute node
|
||||||
|
database record with the resource usage amounts/data coming from the resource
|
||||||
|
tracker.
|
||||||
|
|
||||||
|
Note that we do NOT propose any changes at this time to the database schema
|
||||||
|
of the `compute_nodes` table or the fields in `nova.objects.ComputeNode`,
|
||||||
|
however we do add translation methods to `nova.objects.ComputeNode` that will
|
||||||
|
be able to construct a set of `UsageSpec` objects for each resource type on
|
||||||
|
the compute node as well as set the existing fields on the
|
||||||
|
`nova.objects.ComputeNode` object from a supplied set of `UsageSpec` objects.
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
None. The objects added in this blueprint are not stored in a database. These
|
||||||
|
objects are a replacement for an unstructured nested dictionary that is
|
||||||
|
currently used to represent resource amounts.
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Notifications impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
The extensible resource tracker will be removed when this blueprint is
|
||||||
|
completed.
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
Once this blueprint is completed, code handling the construction of the
|
||||||
|
request_spec will be much more structured, and much of the spaghetti code
|
||||||
|
in the resource tracker around the ERT, PCI tracker, and NUMA topology
|
||||||
|
quirks will go away.
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
There will be a generalized usage spec class interface that looks like this:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
class UsageSpec(object):
|
||||||
|
|
||||||
|
"""Represents the used amount of a particular type of resource."""
|
||||||
|
|
||||||
|
def update(self, amount_spec):
|
||||||
|
"""Update the used amount of resources by the supplied amount.
|
||||||
|
|
||||||
|
:param amount_spec: `AmountSpec` to modify the usage with.
|
||||||
|
"""
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
def has_room_for(self, amount_spec):
|
||||||
|
"""Determine if there is room to fit the supplied amount of resources.
|
||||||
|
|
||||||
|
:param amount_spec: `AmountSpec` to determine if there is room for.
|
||||||
|
:returns True if the supplied requested amount of resources is able
|
||||||
|
to be consumed on the node, False otherwise.
|
||||||
|
|
||||||
|
If the supplied `amount_spec` is negative, returns False .
|
||||||
|
"""
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
Note that each concrete specialization of the UsageSpec class must be able
|
||||||
|
to handle overcommit ratios for the type of resource that it handles.
|
||||||
|
|
||||||
|
With the idea that *all* requested resources for an instance should be able
|
||||||
|
to be compared to *all* resource usage records for a compute node in the
|
||||||
|
same way, using code that looks like this:
|
||||||
|
|
||||||
|
.. code:: python
|
||||||
|
|
||||||
|
for resource_type, amount in request_spec.resources.items():
|
||||||
|
if compute_node.usages[resource_type].has_room_for(amount):
|
||||||
|
# do something... perhaps claim resources on the compute
|
||||||
|
# node, which might eventually call:
|
||||||
|
compute_node.usages[resource_type].update(amount)
|
||||||
|
|
||||||
|
The conductor RPC API should have a new `update_compute_node_resource_usages()`
|
||||||
|
method added to it that passes the new `nova.objects.ResourceUsageCollection`
|
||||||
|
object to the conductor, that the conductor decomposes and updates the
|
||||||
|
compute node records in the database.
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
jaypipes
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
- Add classes for requested and usage amount representation
|
||||||
|
|
||||||
|
- Add classes for resource type representation
|
||||||
|
|
||||||
|
- Add a class representing a collection of requested resource amounts
|
||||||
|
|
||||||
|
- Add a class representing a collection of resource usages
|
||||||
|
|
||||||
|
- Add translation methods to `nova.objects.ComputeNode` to construct and
|
||||||
|
decompose a set of `UsageSpec` objects w/ unit tests
|
||||||
|
|
||||||
|
- Convert resource tracker to use usage amount objects instead of triples of
|
||||||
|
free/total/used amounts in key/value pairs in a dictionary for the non-PCI,
|
||||||
|
non-ERT, non-NUMA resources.
|
||||||
|
|
||||||
|
- Remove the extensible resource tracker code.
|
||||||
|
|
||||||
|
- Convert resource tracker to use usage amount objects instead of
|
||||||
|
'numa_topology' key and `nova.virt.hardware.VirtNUMATopology` object
|
||||||
|
in the `old_resources` dictionary.
|
||||||
|
|
||||||
|
- Convert resource tracker to use usage amount objects instead of 'pci_devices'
|
||||||
|
and 'pci_passthrough_devices' keys and a `nova.pci.pci_stats.PciDeviceStats`
|
||||||
|
object in the `pci_tracker` attribute of the resource tracker.
|
||||||
|
|
||||||
|
- Convert the virt driver's `get_available_resources` method to return a
|
||||||
|
dictionary of resource objects.
|
||||||
|
|
||||||
|
- Add a new conductor RPC API (and scheduler client) method called
|
||||||
|
`update_compute_node_resource_usages()` that supplies the new
|
||||||
|
`nova.objects.ResourceUsageCollection` object representation of resource
|
||||||
|
usages.
|
||||||
|
|
||||||
|
- Deprecate the old update_resource_stats() conductor RPC API method.
|
||||||
|
|
||||||
|
- Convert the scheduler's `HostStateManager` to utilize the new
|
||||||
|
`ComputeNode.get_resource_usages()` method and
|
||||||
|
`ComputeNode.update_from_resource_usages` method.
|
||||||
|
|
||||||
|
- Add developer reference documentation for how resources are modeled.
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
New unit tests for the objects will be added. The existing unit tests of
|
||||||
|
resource tracker will be overhauled in the patch set that converts the resource
|
||||||
|
tracker to use the new resource object models instead of its current free-form
|
||||||
|
dictionary of things.
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
There are currently no developer reference docs that explain how the different
|
||||||
|
resources are tracked within Nova. Developer reference material that explains
|
||||||
|
the new resource type and amount classes will be delivered as a part of this
|
||||||
|
blueprint.
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
This blueprint is part of an overall effort to clean up, version, and stabilize
|
||||||
|
the interfaces between the nova-api, nova-scheduler, nova-conductor and
|
||||||
|
nova-compute daemons that involve scheduling and resource decisions.
|
||||||
|
|
||||||
|
- `detach-service-from-computenode`
|
||||||
|
- `resource-objects` <-- this blueprint
|
||||||
|
- `request-spec-object`
|
||||||
|
- `sched-select-destinations-use-request-spec-object`
|
||||||
|
- `placement-spec-object`
|
||||||
|
- `condition-objects`
|
||||||
|
- `sched-placement-spec-use-resource-objects`
|
||||||
|
- `sched-placement-spec-use-condition-objects`
|
||||||
|
- `sched-get-placement-claims`
|
||||||
Reference in New Issue
Block a user