From eb4e4b2dc4d7b9b0458c31cae3cd5042e9953b69 Mon Sep 17 00:00:00 2001 From: Tony Su Date: Wed, 6 May 2020 08:04:40 +0000 Subject: [PATCH] Re-propose provider-config-file spec for Victoria This is a proposal to configure resource provider visibility, ownership, inventory, trait, and aggregate settings using a standardized YAML file format. This was previously approved in Train [1] and Ussuri[2] but did not make it into the release. [1] http://specs.openstack.org/openstack/nova-specs/specs/train/approved/provider-config-file.html [2] https://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/provider-config-file.html Previously-approved: Train, Ussuri Blueprint: provider-config-file Change-Id: I24e67e3776d83ec1ac29f336b8392b1883d8934d --- .../approved/provider-config-file.rst | 379 ++++++++++++++++++ 1 file changed, 379 insertions(+) create mode 100644 specs/victoria/approved/provider-config-file.rst diff --git a/specs/victoria/approved/provider-config-file.rst b/specs/victoria/approved/provider-config-file.rst new file mode 100644 index 000000000..db55edeba --- /dev/null +++ b/specs/victoria/approved/provider-config-file.rst @@ -0,0 +1,379 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +=========================== +Provider Configuration File +=========================== + +https://blueprints.launchpad.net/nova/+spec/provider-config-file + +This is a proposal to configure resource provider inventory and traits using a +standardized YAML file format. + +.. note:: This work is derived from `Jay's Rocky provider-config-file + proposal`_ and `Konstantinos's device-placement-model spec`_ (which + is derived from `Eric's device-passthrough spec`_), but differs in + several substantive ways. + +.. note:: This work is influenced by requirements to Nova to support non + native compute resources that are managed by Resource Management + Daemon for finer grain control. PTG discussion notes available at + `Resource Management Daemon_PTG Summary`_ + +.. note:: We currently limit the ownership and consumption of the provider + config YAML as described by the file format to Nova only. + +.. note:: The provider config will currently only accept placement overrides + to create and manage inventories and traits for resources not + natively managed by the Nova virt driver. + +.. note:: This is intended to define a) a file format for currently active use + cases, and b) Nova's consumption of such files. Subsequent features + can define the semantics by which the framework can be used by other + consumers or enhanced to satisfy particular use cases. + +Problem description +=================== +In order to facilitate the proper management of resource provider information +in the placement API by agents within Nova (such as virt drivers and the +PCI passthrough subsystem), we require a way of expressing various +overrides for resource provider information. While we could continue to use +many existing and new configuration options for expressing this information, +having a standardized, versioned provider descriptor file format allows us to +decouple the management of provider information from the configuration of the +service or daemon that manages those resource providers. + +Use Cases +--------- +Note that the file format/schema defined here is designed to accommodate the +following use cases. The file format/schema currently addresses a few use cases +that require changes to resource provider information as consumed by virt +drivers in Nova but it should allow options for extensions to be consumed +by Nova or other services as described in the problem statement in the future. + +Inventory Customization +~~~~~~~~~~~~~~~~~~~~~~~ + +**An operator would like to describe inventories for new platform features** + +These features could be experimental or not yet completely supported by Nova. +The expectation is that Nova can manage these inventories and help schedule +workloads requesting support for new platform features against their +capacities. For instance, to report ``CUSTOM_LLC`` (last-level cache) +inventories. + +The file defined by this spec must allow its author to: + +* Identify a provider unambiguously. +* Create and manage inventories for resource classes not natively managed by + Nova virt driver (``CUSTOM_LLC``, ``CUSTOM_MEMORY_BANDWIDTH`` etc.) + +Trait Customization +~~~~~~~~~~~~~~~~~~~ + +**An operator wishes to associate new custom traits with a provider.** + +These features could be experimental or not yet completely supported by Nova. +The expectation is that Nova can manage these traits and help schedule +workloads with support to new platform features against their traits. + +The file defined by this spec must allow its author to: + +* Identify a provider unambiguously. +* Specify arbitrary custom traits which are to be associated with the provider. + +Proposed change +=============== + +Provider Config File Schema +--------------------------- +A versioned YAML file format with a formal schema is proposed. The scope of +this spec is the schema, code to parse a file into a Python dict, code to +validate the dict against the schema, and code to merge the resulting dict with +the provider tree as processed by the resource tracker. + +The code shall be introduced into the ``openstack/nova`` project initially and +consumed by the resource tracker. Parts of it (such as the schema definition, +file loading, and validation) may be moved to a separate oslo-ish library in +the future if it can be standardized for consumption outside of Nova. + +The following is a simplified pseudo-schema for the file format. + +.. code-block:: yaml + + meta: + # Version ($Major, $minor) of the schema must successfully parse documents + # conforming to ($Major, *). I.e. additionalProperties must be allowed at + # all levels; but code at a lower $minor will ignore fields it does not + # recognize. Schema changes representing optional additions should bump + # $minor. Any breaking schema change (e.g. removing fields, adding new + # required fields, imposing a stricter pattern on a value, etc.) must bump + # $Major. The question of whether/how old versions will be deprecated or + # become unsupported is left for future consideration. + schema_version: $Major.$minor + + providers: + # List of dicts + # Identify a single provider to configure. + # Exactly one of uuid or name is mandatory. Specifying both is an error. + # The consuming nova-compute service will error and fail to start if the + # same value is used more than once across all provider configs for name + # or uuid. + # NOTE: Caution should be exercised when identifying ironic nodes, + # especially via the `$COMPUTE_NODE` special value. If an ironic node + # moves to a different compute host with a different provider config, its + # attributes will change accordingly. + - identification: + # Name or UUID of the provider. + # The uuid can be set to the specialized string `$COMPUTE_NODE` which + # will cause the consuming compute service to apply the configuration + # in this section to each node it manages unless that node is also + # identified by name or uuid. + uuid: ($uuid_pattern|"$COMPUTE_NODE") + # Name of the provider. + name: $string + # Customize provider inventories + inventories: + # This section allows the admin to specify various adjectives to + # create and manage providers' inventories. This list of adjectives + # can be extended in the future as the schema evolves to meet new + # use cases. For now, only one adjective, `additional`, is supported. + additional: + # The following inventories should be created on the identified + # provider. Only CUSTOM_* resource classes are permitted. + # Specifying inventory of a resource class natively managed by + # nova-compute will cause the compute service to fail. + $resource_class: + # `total` is required. Other optional fields not specified + # get defaults from the Placement service. + total: $int + reserved: $int + min_unit: $int + max_unit: $int + step_size: $int + allocation_ratio: $float + # Next inventory dict, keyed by resource class... + ... + # Customize provider traits. + traits: + # This section allows the admin to specify various adjectives to + # create and manage providers' traits. This list of adjectives + # can be extended in the future as the schema evolves to meet new + # use cases. For now, only one adjective, `additional`, is supported. + additional: + # The following traits are added on the identified provider. Only + # CUSTOM_* traits are permitted. The consuming code is + # responsible for ensuring the existence of these traits in + # Placement. + - $trait_pattern + - ... + # Next provider... + - identification: + ... + +Example +~~~~~~~ +.. note:: This section is intended to describe at a very high level how this + file format could be consumed to provide ``CUSTOM_LLC`` inventory + information. + +.. note:: This section is intended to describe at a very high level how this + file format could be consumed to provide P-state compute trait + information. + +.. code-block:: yaml + + meta: + schema_version: 1.0 + + providers: + # List of dicts + - identification: + uuid: $COMPUTE_NODE + inventories: + additional: + CUSTOM_LLC: + # Describing LLC on this compute node + # max_unit indicates maximum size of single LLC + # total indicates sum of sizes of all LLC + total: 22 + reserved: 2 + min_unit: 1 + max_unit: 11 + step_size: 1 + allocation_ratio: 1 + traits: + additional: + # Describing that this compute node enables support for + # P-state control + - CUSTOM_P_STATE_ENABLED + +Provider config consumption from Nova +------------------------------------- +Provider config processing will be performed by the nova-compute process as +described below. There are no changes to virt drivers. In particular, virt +drivers have no control over the loading, parsing, validation, or integration +of provider configs. Such control may be added in the future if warranted. + +Configuration + A new config option is introduced:: + + [compute] + # Directory of yaml files containing resource provider configuration. + # Default: /etc/nova/provider_config/ + # Files in this directory will be processed in lexicographic order. + provider_config_location = $directory + +Loading, Parsing, Validation + On nova-compute startup, files in ``CONF.compute.provider_config_location`` + are loaded and parsed by standard libraries (e.g. ``yaml``), and + schema-validated (e.g. via ``jsonschema``). Schema validation failure or + multiple identifications of a node will cause nova-compute startup to fail. + Upon successful loading and validation, the resulting data structure is + stored in an instance attribute on the ResourceTracker. + +Provider Tree Merging + A generic (non-hypervisor/virt-specific) method will be written that merges + the provider config data into an existing ``ProviderTree`` data structure. + The method must detect conflicts whereby provider config data references + inventory of a resource class managed by the virt driver. Conflicts should + log a warning and cause the conflicting config inventory to be ignored. + The exact location and signature of this method, as well as how it detects + conflicts, is left to the implementation. In the event that a resource + provider is identified by both explicit UUID/NAME and $COMPUTE_NODE, only the + UUID/NAME record will be used. + +``_update_to_placement`` + In the ResourceTracker's ``_update_to_placement`` flow, the merging method is + invoked after ``update_provider_tree`` and automatic trait processing, *only* + in the ``update_provider_tree`` flow (not in the legacy ``get_inventory`` or + ``compute_node_to_inventory_dict`` flows). On startup (``startup == True``), + if the merge detects a conflict, the nova-compute service will fail. + +Alternatives +------------ +Ad hoc provider configuration is being performed today through an amalgam of +oslo.config options, more of which are being proposed or considered to deal +with VGPUs, NUMA, bandwidth resources, etc. The awkwardness of expressing +hierarchical data structures has led to such travesties as +``[pci]passthrough_whitelist`` and "dynamic config" mechanisms where config +groups and their options are created on the fly. YAML is natively suited for +this purpose as it is designed to express arbitrarily nested data structures +clearly, with minimal noisy punctuation. In addition, the schema is +self-documenting. + +Data model impact +----------------- +None + +REST API impact +--------------- +None + +Security impact +--------------- +Admins should ensure that provider config files have appropriate permissions +and ownership. Consuming services may wish to check this and generate an error +if a file is writable by anyone other than the process owner. + +Notifications impact +-------------------- +None + +Other end user impact +--------------------- +None + +Performance Impact +------------------ +None + +Other deployer impact +--------------------- +An understanding of this file and its implications is only required when the +operator desires provider customization. The deployer should be aware of the +precedence of records with UUID/NAME identification over $COMPUTE_NODE. + +Developer impact +---------------- +Subsequent specs will be needed for services consuming this file format. + +Upgrade impact +-------------- +None. (Consumers of this file format will need to address this - e.g. decide +how to deprecate existing config options which are being replaced). + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + tony su + +Other contributors: + dustinc + efried dakshinai + +Feature Liaison +--------------- + +Feature liaison: + gibi + +Work Items +---------- + +* Construct a formal schema +* Implement parsing and schema validation +* Implement merging of config to provider tree +* Incorporate above into ResourceTracker +* Compose a self-documenting sample file + +Dependencies +============ +None + + +Testing +======= +* Schema validation will be unit tested. +* Functional and integration testing to move updates from provider config file + to Placement via Nova virt driver. + +Documentation Impact +==================== +* The formal schema file and a self-documenting sample file for provider + config file. +* Admin-facing documentation on guide to update the file and how Nova + processes the updates. +* User-facing documentation (including release notes). + +References +========== +.. _Jay's Rocky provider-config-file proposal: https://review.openstack.org/#/c/550244/2/specs/rocky/approved/provider-config-file.rst +.. _Konstantinos's device-placement-model spec: https://review.openstack.org/#/c/591037/8/specs/stein/approved/device-placement-model.rst +.. _Eric's device-passthrough spec: https://review.openstack.org/#/c/579359/10/doc/source/specs/rocky/device-passthrough.rst +.. _Resource Management Daemon_PTG Summary: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005809.html +.. _Handling UUID/NAME and $COMPUTE_NODE conflicts: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-11-19.log.html#t2019-11-19T21:25:26 + +History +======= + +.. list-table:: Revisions + :header-rows: 1 + + * - Release Name + - Description + * - Stein + - Introduced + * - Train + - Re-proposed, simplified + * - Ussuri + - Re-proposed + * - Victoria + - Re-proposed