When performing a resize, we'll want to (by default) select
target hosts from the source cell to do a traditional resize
if possible before considering target hosts in another cell
which will be slower and more complicated. If the source cell
is disabled or target flavor is not available in the source cell,
then we'll have no choice but to select a host from another cell.
But all things being equal between hosts, we want to stay within
the source cell (by default). Therefore this change adds a new
CrossCellWeigher and related configuration option to prefer hosts
within the source cell when moving a server. The weigher is
completely noop unless a cross-cell move is permitted by
configuration, which will be provided in a future change.
Part of blueprint cross-cell-resize
This spec proposes to add ability to allow users to use
``Aggregate``'s ``metadata`` to override the global config options
for weights to achieve more fine-grained control over resource
This resolves the TODO from Ocata change:
By adding a min=0.0 value to the soft affinity
weight multiplier configuration options.
It also removes the deprecated [DEFAULT] group
alias from Ocata change:
There is concern over the ability for compute nodes to reasonably
determine which events should count against its consecutive build
failures. Since the compute may erronenously disable itself in
response to mundane or otherwise intentional user-triggered events,
this patch adds a scheduler weigher that considers the build failure
counter and can negatively weigh hosts with recent failures. This
avoids taking computes fully out of rotation, rather treating them as
less likely to be picked for a subsequent scheduling
This introduces a new conf option to control this weight. The default
is set high to maintain the existing behavior of picking nodes that
are not experiencing high failure rates, and resetting the counter as
soon as a single successful build occurs. This is minimal visible
change from the existing behavior with default configuration.
The rationale behind the default value for this weigher comes from the
values likely to be generated by its peer weighers. The RAM and Disk
weighers will increase the score by number of available megabytes of
memory (range in thousands) and disk (range in millions). The default
value of 1000000 for the build failure weigher will cause competing
nodes with similar amounts of available disk and a small (less than ten)
number of failures to become less desirable than those without, even
with many terabytes of available disk.
nova provides many different types of filter, providing both resource
stacking/spreading and affinity/anti-affinity patterns. Curiously
enough, however, there is no way to configure a stack or spread policy
of CPUs. Add one.
Implements: bp vcpu-weighter
Take three hosts, one with a single PCI device, one with many PCI
devices, and one with no PCI devices. Nova should prioritise these
differently based on the demands of the instance. If the instance
requests a single PCI device, then the first of the hosts should be
preferred. Similarly, if the instance requests multiple PCI devices,
then the second of these hosts would be preferred. Finally, if the
instance does not request a PCI device, then the last of these hosts
should be preferred.
While the first and second of these cases is already somewhat handled by
the NUMA topology filter (which will ensure the correct number and type
of PCI devices are available on the hosts), there is nothing to stop a
instance that *does not* require PCI devices from occupying space on one
of the few hosts that *does* have free PCI devices.
The PCIWeigher is designed to keep hosts with PCI device(s) as free as
possible, to best ensure they can satisfy requests from instances with
PCI requirements when called upon. This change ensures (a) an instance
with PCI requirements has the best possible chance of scheduling
successfully and (b) an instance without these requirements won't clog
up hosts that have these valuable resources.
Partially Implements: blueprint reserve-numa-with-pci
There are five options marked with TODOs:
* periodic_task_interval - negative values are valid, making the TODO
* host_subset_size - code was in-place to prevent negative values, so
replace this code with a min parameter
* isolated_images - removing this option requires a little bit of
thought and would likely require a blueprint. Retain this for now.
* max_instances_per_host, attestation_auth_timeout,
soft_anti_affinity_weight_multiplier - support funky value ranges
which, semantically speaking, shouldn't be allowed. Start logging
any "invalid" values.
Move all scheduler options into their one of two groups. Many of the
options are simply renamed to remove their 'scheduler_' prefix, with
* scheduler_default_filters -> enabled_filters
* scheduler_baremetal_default_filters -> baremetal_enabled_filters
* scheduler_driver_task_period -> periodic_task_interval
* scheduler_tracks_instance_changes -> track_instance_changes
Co-Authored-By: Stephen Finucane <firstname.lastname@example.org>
Implements: blueprint centralize-config-options-ocata
ServerGroupSoftAffinityWeigher implements the soft-affinity
policy for server groups as it orders the hosts by the number
of instances running on them from the same group in decreasing
ServerGroupSoftAntiAffinityWeigher implements the
soft-anti-affinity policy for server groups as it orders the
hosts by the number of instances running on them from the same
group in increasing order.
Both weigher assumes the the RequestSpec object refers to a valid
InstanceGroup object which has a populated members field. This
will be provided by the subsequent patch.
New configuration options added:
Add docs about the new weigher classes and change the function
name 'weigh_object' to '_weigh_object' in the Weights section
to match the code.
Co-Authored-By: Balazs Gibizer <email@example.com>
Implements: blueprint soft-affinity-for-server-group
This change moves all of the configuration options previously defined in
nova/scheduler to the new centralized nova/conf/scheduler directory.
A subsequent patch will then improve the help texts.
Given that dict comprehensions don't allow to have None as a list, we had
a problem with the Ironic gate-ironic-inspector-dsvm job because
host_state.metrics was set to None by default.
Adding a unittest to make sure we don't regress.
The commit I4dfbea27ce6c3eecc1a8658b1f9dc0feb2298705 changes
'HostState.metrics' to 'objects.MonitorMetricList'. But MetricWeigher
still use the old dictionary of 'host_manager.MetricItem'. This patch
corrects the behavior of MetricWeigher with related tests, and delete
Co-Authored-By: Sylvain Bauza <firstname.lastname@example.org>
The oslo team is recommending everyone to switch to the
non-namespaced versions of libraries. Updating the hacking
rule to include a check to prevent oslo.* import from
creeping back in.
This commit includes:
- using oslo_utils instead of oslo.utils
- using oslo_serialization instead of oslo.serialization
- using oslo_db instead of oslo.db
- using oslo_i18n instead of oslo.i18n
- using oslo_middleware instead of oslo.middleware
- using oslo_config instead of oslo.config
- using oslo_messaging instead of "from oslo import messaging"
- using oslo_vmware instead of oslo.vmware
Add a new nova scheduler weighter, sort the filter
hosts according to host io ops number, aims to
booting instances on light workload hosts.
DocImpact: Adds io_ops_weight_multiplier to [DEFAULT]
group of nova.conf and an new default schedule weigher.
Implements: blueprint io-ops-weight
WeighedHost's __repr__ was poking under the HostState hood to use
.host in its own __repr__ but this hides some important details.
Specifically it hides the hypervisor_hostname, which is important for
Ironic as one 'node' can have thousands of hostnames. The other
details such as available RAM and so on are also useful for ops trying
to debug scheduling issues, so rather than add hypervisor_hostname, I
am just delegating to the underlying __repr__.
This patch added the support of unavailable metrics in the metric
weigher. Based on the configuration, if an unavailable metric is met,
the metric weigher might either throw an exception, or return an
configurable weight value.
This is part of the blueprint utilization-aware-scheduling.
DocImpact: Added support of unavailable metrics. 2 new config options
are added, 1) 'required' is a boolean option indicating whether all
the metrics set by 'weight_setting' are needed to be available when
calculating the weight. 2) 'weight_of_unavailable' is a float option
specifying the value to be returned if the metrics is unavailable. See
etc/nova/nova.conf.sample for details.
The scheduler filter MetricsFilter filters out those hosts which don't
have the metrics data available as required by metric settings.
This is part of the blueprint utilization-aware-scheduling.
DocImpact: Added a new metrics filter.
The weight system is being used by the scheduler and the cells code.
Currently this system is using the raw values instead of normalizing them.
This makes difficult to properly use multipliers for establishing the
relative importance between two wheighers (one big magnitude could
shade a smaller one). This change introduces weight normalization so
- From an operator point of view we can prioritize the weighers that
we are applying. The only way to do this is being sure that all the
weighers will give a value in a known range, so that it is
not needed to artificially use a huge multiplier to prioritize a
- From a weigher developer point of view, somebody willing to implement
one has to care about 1) returning a list of values, 2) setting the
minimum and maximum values where the weights can range, if they are
needed and they are significant for the weighing. For a weigher
developer there are two use cases:
Case 1: Use of a percentage instead of absolute values (for example, %
of free RAM). If we compare two nodes focusing on the percentage of free
ram, the maximum value for the weigher is 100. If we have two nodes one
with 2048 total/1024 free, and the second one 1024 total/512 free they
will get both the same weight, since they have the same % of free RAM
(that is, the 50%).
Case 2: Use of absolute values. In this case, the maximum of the weigher
will be the maximum of the values in the list (in the case above, 1024)
or the maximum value that the magnitude could take (in the case above,
2048). How this maximum is set, is a decision of the developer. He may
let the operator choose the behaviour of the weigher though.
- From the point of view of the scheduler we ensure that it is using
normalized values, and not leveraging the normalization mechanism to the
Changes introduced this commit:
1) it introduces weight normalization so that we can apply multipliers
easily. All the weights for an object will be normalized between 0.0 and
1.0 before being sumed up, so that the final weight for a host will be:
weight = w1_multiplier * norm(w1) + w2_multiplier * norm(w2) + ...
2) weights.BaseWeigher has been changed into an ABC so that we enforce
that all weighers have the expected methods.
3) weights.BaseWeigher.weigh_objects() does no longer sum up the
computer weighs to the object, but it rather returns a list that will be
then normalized and added to the existing weight by BaseWeightHandler
4) Adapt the existing weighers to the above changes. Namely
- New 'offset_weight_multiplier' for the cell weigher
- Changed the name of the existing multiplier methods.
5) unittests for all of the introduced changes.
Implements blueprint normalize-scheduler-weights
DocImpact: Now weights for an object are normalized before suming them
up. This means that each weigher will take a maximum value of 1. This
may have an impact for operators that are using more than one weigher
(currently there is only one weigher: RAMWeiger) and for operators using
cells (where we have several weighers). It is needed to review then the
multipliers used and adjust them properly in case they have been
Docimpact: There is a new configuration option 'offset_weight_multiplier'
The new metrics weigher can compute the weight based on the compute
node host's metrics data. The to-be weighed metrics and their
weighing ratio are specified in the configuration file as the
metrics_weight_setting = name1=1.0,name2=-1.0
The final weight would be name1.value * 1.0 + name2.value * (-1.0).
This is part of the blueprint utilization-aware-scheduling.
The cfg API is now available via the oslo-config library, so switch to
it and remove the copied-and-pasted version.
Add the 2013.1b4 tarball to tools/pip-requires - this will be changed
to 'oslo-config>=2013.1' when oslo-config is published to pypi. This
will happen in time for grizzly final.
Add dependency_links to setup.py so that oslo-config can be installed
from the tarball URL specified in pip-requires.
Remove the 'deps = pep8==1.3.3' from tox.ini as it means all the other
deps get installed with easy_install which can't install oslo-config
from the URL.
Make tools/hacking.py include oslo in IMPORT_EXCEPTIONS like it already
does for paste. It turns out imp.find_module() doesn't correct handle
Retain dummy cfg.py file until keystoneclient middleware has been
Modules import nova.config for two reasons right now - firstly, to
reference nova.config.CONF and, secondly, if they use one of the
options defined in nova.config.
Often modules import nova.openstack.common.cfg and nova.config
which is a bit pointless since they could just use cfg.CONF if
they just want to nova.config in order to reference CONF.
Let's just use cfg.CONF everywhere and we can explicitly state
where we actually require options defined in nova.config.
This makes scheduling weights more plugin friendly and creates shared
code that can be used by the host scheduler as well as the future cells
scheduler. Weighing classes can now be specified much like you can
specify scheduling host filters.
The new weights code reverses the old behavior where lower weights win.
Higher weights are now the winners.
The least_cost module and configs have been deprecated, but are still
supported for backwards compatibility. The code has moved to
nova.scheduler.weights.least_cost and been modified to work with the new
loadable-class code. If any of the least_cost related config options are
specified, this least_cost weigher will be used.
For those not overriding the default least_cost config values, the new
RamWeigher class will be used. The default behavior of the RamWeigher
class is the same default behavior as the old least_cost module.
The new weights code introduces a new config option
'scheduler_weight_classes' which is used to specify which weigher classes
to use. The default is 'all classes', but modified if least_cost
deprecated config options are used, as mentioned above.
The RamWeigher class introduces a new config option
'ram_weight_multiplier'. The default of 1.0 causes weights equal to the
free memory in MB to be returned, thus hosts with more free memory are
preferred (causes equal spreading). Changing this value to a negative
number such as -1.0 will cause reverse behavior (fill first).