Matt Riedemann 579e0cfd57 Document differences and similaries between extra specs and hints

Scheduler hints are not really documented very well at all except
for being mentioned per scheduler filter in the admin configuration
guide, nor are they documented within relation to flavor extra
specs which are both used for impacting scheduling decisions and
are choices that a deployer has to make based on how they configure
their cloud.

This change adds a document about scheduler hints and how they are
similar to and different from flavor extra specs, including end
user discoverability and interoperability, and thoughts on which
should be used if writing a custom scheduler filter.

The TODO in the API guide is also resolved by linking to this
document.

Change-Id: Ib1f35baacf59efafb9e4bccfcc4f0025d99ad5b2

2018-08-29 19:09:11 -04:00

7.7 KiB

Raw Blame History

Scheduler hints versus flavor extra specs

People deploying and working on Nova often have questions about flavor extra specs and scheduler hints and what role they play in scheduling decisions, and which is a better choice for exposing capability to an end user of the cloud. There are several things to consider and it can get complicated. This document attempts to explain at a high level some of the major differences and drawbacks with both flavor extra specs and scheduler hints.

Extra Specs

In general flavor extra specs are specific to the cloud and how it is organized for capabilities, and should be abstracted from the end user. Extra specs are tied to host aggregates </user/aggregates> and a lot of them also define how a guest is created in the hypervisor, for example what the watchdog action is for a VM. Extra specs are also generally interchangeable with image properties when it comes to VM behavior, like the watchdog example. How that is presented to the user is via the name of the flavor, or documentation specifically for that deployment, e.g. instructions telling a user how to setup a baremetal instance.

Scheduler Hints

Scheduler hints, also known simply as "hints", can be specified during server creation to influence the placement of the server by the scheduler depending on which scheduler filters are enabled. Hints are mapped to specific filters. For example, the ServerGroupAntiAffinityFilter scheduler filter is used with the group scheduler hint to indicate that the server being created should be a member of the specified anti-affinity group and the filter should place that server on a compute host which is different from all other current members of the group.

Hints are not more "dynamic" than flavor extra specs. The end user specifies a flavor and optionally a hint when creating a server, but ultimately what they can specify is static and defined by the deployment.

Similarities

Both scheduler hints and flavor extra specs can be used by scheduler filters </admin/configuration/schedulers>.
Both are totally customizable, meaning there is no whitelist within Nova of acceptable hints or extra specs, unlike image properties¹.
An end user cannot achieve a new behavior without deployer consent, i.e. even if the end user specifies the group hint, if the deployer did not configure the ServerGroupAntiAffinityFilter the end user cannot have the anti-affinity behavior.

Differences

A server's host location and/or behavior can change when resized with a flavor that has different extra specs from those used to create the server. Scheduler hints can only be specified during server creation, not during resize or any other "move" operation, but the original hints are still applied during the move operation.
The flavor extra specs used to create (or resize) a server can be retrieved from the compute API using the 2.47 microversion. As of the 19.0.0 Stein release, there is currently no way from the compute API to retrieve the scheduler hints used to create a server.

Note

Exposing the hints used to create a server has been proposed². Without this, it is possible to workaround the limitation by doing things such as including the scheduler hint in the server metadata so it can be retrieved via server metadata later.
In the case of hints the end user can decide not to include a hint. On the other hand the end user cannot create a new flavor (by default policy) to avoid passing a flavor with an extra spec - the deployer controls the flavors.

Discoverability

When it comes to discoverability, by the default os_compute_api:os-flavor-extra-specs:index policy rule, flavor extra specs are more "discoverable" by the end user since they can list them for a flavor. However, one should not expect an average end user to understand what different extra specs mean as they are just a key/value pair. There is some documentation for some "standard" extra specs though³. However, that is not an exhaustive list and it does not include anything that different deployments would define for things like linking a flavor to a set of host aggregates, for example, when creating flavors for baremetal instances, or what the chosen hypervisor driver </admin/configuration/hypervisors> might support for flavor extra specs.

Scheduler hints are less discoverable from an end user perspective than extra specs. There are some standard hints defined in the API request schema⁴. However:

Those hints are tied to scheduler filters and the scheduler filters are configurable per deployment, so for example the JsonFilter might not be enabled (it is not enabled by default), so the query hint would not do anything.
Scheduler hints are not restricted to just what is in that schema in the upstream nova code because of the additionalProperties: True entry in the schema. This allows deployments to define their own hints outside of that API request schema for their own custom scheduler filters <custom-scheduler-filters> which are not part of the upstream nova code.

Interoperability

The only way an end user can really use scheduler hints is based on documentation (or GUIs/SDKs) that a specific cloud deployment provides for their setup. So if CloudA defines a custom scheduler filter X and a hint for that filter in their documentation, an end user application can only run with that hint on that cloud and expect it to work as documented. If the user moves their application to CloudB which does not have that scheduler filter or hint, they will get different behavior.

So obviously both flavor extra specs and scheduler hints are not interoperable.

Which to use?

When it comes to defining a custom scheduler filter, you could use a hint or an extra spec. If you need a flavor extra spec anyway for some behavior in the hypervisor when creating the guest, or to be able to retrieve the original flavor extra specs used to create a guest later, then you might as well just use the extra spec. If you do not need that, then a scheduler hint may be an obvious choice, from an end user perspective, for exposing a certain scheduling behavior but it must be well documented and the end user should realize that hint might not be available in other clouds, and they do not have a good way of finding that out either. Long-term, flavor extra specs are likely to be more standardized than hints so ultimately extra specs are the recommended choice.

7.7 KiB Raw Blame History