9.8 KiB
image metadata prefiltering
https://blueprints.launchpad.net/nova/+spec/image-metadata-prefiltering
Nova supports specifying hypervisor-specific device model via image
properties. Today when such properties are set on an image they are not
considered when scheduling unless the operator manually configures the
ImagePropertiesFilter. If the operator does not configure the
ImagePropertiesFilter and an instance is scheduled to a host that cannot
support the requested device model, a late check in the virt driver will
fail the spawn and trigger a reschedule. If only a subset of hosts can
support the requested device model this frequently results in a
NoValidHost
error.
This proposal suggests using standardised traits and placement to address device model compatibility by extending existing virt drivers to declare the device models they support as traits.
Problem description
Use Cases
As an operator I want the flexibility to deploy a heterogeneous cloud that has compute nodes with different device model capabilities, either temporarily during upgrades or permanently in a multi-architecture multi-hypervisor cloud, without requiring explicit scheduler configuration.
As an operator, I would like to be able to communicate with my customers what capabilities my cloud provides opaquely via standard traits instead of revealing the specific versions of the software that is deployed so that they can provide their own image that depends on those capabilities.
As an end user, I want to be able to succinctly specify device models or other hypervisor-dependent capabilities requirements for my instance without needing to be overly verbose.
Proposed change
Step 1: Standard Traits
Well-defined image metadata properties that have a finite set of values which represent virtualisation capabilities will be converted to standard traits.
The current proposed set is
{
"hw_vif_model": {
"title": "Virtual Network Interface",
"description": "Specifies the model of virtual network interface
device to use. The valid options depend on the hypervisor
configuration. libvirt driver options: KVM and QEMU:
e1000, ne2k_pci, pcnet, rtl8139, spapr-vlan, and virtio.
Xen: e1000, netfront, ne2k_pci, pcnet, and rtl8139.",
"type": "string",
"enum": [
"e1000",
"e1000e",
"ne2k_pci",
"netfront",
"pcnet",
"rtl8139",
"spapr-vlan",
"virtio"
]
},
"hw_video_model": {
"title": "Video Model",
"description": "The video image driver used.",
"type": "string",
"enum": [
"vga",
"cirrus",
"vmvga",
"xen",
"qxl"
]
},
"hw_disk_bus": {
"title": "Disk Bus",
"description": "Specifies the type of disk controller to
attach disk devices to.",
"type": "string",
"enum": [
"scsi",
"virtio",
"uml",
"xen",
"ide",
"usb",
"fdc",
"sata"
]
},
}
Note hw_cdrom_bus
supports the same values as
hw_disk_bus
but is not documented.
hw_cdrom_bus
will also be supported by this spec. Other
image properties that may also be worth considering are:
{
"hypervisor_type": {
"title": "Hypervisor Type",
"description": "Hypervisor type required by the image."
"type": "string",
"enum": [
"baremetal",
"hyperv",
"kvm",
"lxc",
"qemu",
"uml",
"vmware",
"vz",
"xen"
]
},
"vm_mode": {
"title": "VM Mode",
"description": "The virtual machine mode.
This represents the host/guest ABI (application binary interface)
used for the virtual machine."
"type": "string",
"enum": [
"hvm",
"xen",
"uml",
"exe"
]
}
}
While this spec primarily targets the device model specific image
metadata properties the same pattern could be applied to
hypervisor_type
and vm_mode
.
Creation of the standard traits will be tracked in a placement/os-traits StoryBoard story. Discussion of how these traits will be named/namespaced will happen there rather than in this spec.
Step 2: Reporting Of Capabilities By Virt Drivers
This spec primarily targets extending the libvirt driver; however as these properties are also used by the vmware and fake drivers they will also be extended.
To enable this feature, the virt drivers will be extended to report
traits for each device model they are capable of emulating on the
compute node resource provider. This will be done by introspection of
the libvirt version, qemu version, and Nova config such as
CONF.virt_type
.
To support upgrades without modifying existing images or flavors, the late checks for device model support in the virt driver will not be removed.
Step 3: Requesting A Device Model And Scheduling
A new scheduler prefilter will be added to automatically add the new traits to requests. As adding new options to the device models requires a change to Nova anyway, and these get updated infrequently, we can just have a mapping table in a prefilter that adds additional traits to the request spec by looking up the appropriate image metadata key and appending the traits to the unnumbered request group. This will not require changes to images to use the feature.
Alternatives
Operators can continue to use image property filters.
If the virt drivers are modified to report traits but a prefilter is
not added, the existing ability to specify required traits in an image
would be sufficient to consume the new traits. However, that would
require the image created to first specify the device model request and
then also the required traits. E.g.
hw_vif_model=e1000, traits:COMPUTE_NET_MODEL_E1000=required
.
This will work but it's verbose.
As with other recent features, we could use the traits request as a
replacement for an image metadata property. If we chose this option we
could deprecate the image metadata data values (e.g.
hw_vif_model
) in train and remove them in a later release.
To use the feature and request a device model going forward a trait
would be used e.g. traits:COMPUTE_NET_MODEL_E1000=required
.
While this may seem nicer in some respects it is more typing than the
selected option and has the disadvantage of requiring all images to be
updated to include the traits request. As such this is discarded due to
the upgrade impact.
Operators can also achieve the goals of this spec by manually
creating nova host aggregates or placement aggregates, then mapping the
images to aggregates using IsolatedHostsFilter or an aggregate
member_of
placement request.
Data model impact
A new set of standard traits will be added to os-traits. No other data model change should be required.
REST API impact
None
Security impact
None
Notifications impact
None
Other end user impact
None
Performance Impact
This is expected to improve boot performance in a heterogeneous cloud by reducing reschedules. By passing a more constrained request to placement this feautre should also reduce the resulting set of allocation_candidates that are returned.
Other deployer impact
A new config option will be added to enable the image metadata prefilter. Initially this will default to disabled but that can be changed in a future release based on feedback on the performance impact.
Developer impact
None
Upgrade impact
No action is required on upgrade. However just as with new deployments if the operator wishes to enable this feature they will need to update the nova config to enable it after upgrading.
Implementation
Assignee(s)
sean-k-mooney
Work Items
- Add new traits
- Modify libvirt virt driver to report traits
- Write prefilter
- Add config option
- Tests
Dependencies
Testing
This can be tested entirely in the gate. Functional and unit tests will be added.
While tempest tests could be added, since we do not have a multinode gate job with different hypervisors tempest will not be extended.
Documentation Impact
A release note will be added and documentation of the new config option for the prefilter will be provided. As there is no end user impact no user facing documentation will be required.
References
- Glance Metadata Definitions Catalog namespaces <image metadata properties_>
- Additions in os-traits
History
Release Name | Description |
---|---|
Train | Introduced |