Normalize the weights instead of using raw values

The weight system is being used by the scheduler and the cells code.
Currently this system is using the raw values instead of normalizing them.
This makes difficult to properly use multipliers for establishing the
relative importance between two wheighers (one big magnitude could
shade a smaller one). This change introduces weight normalization so

- From an operator point of view we can prioritize the weighers that
  we are applying. The only way to do this is being sure that all the
  weighers will give a value in a known range, so that it is
  not needed to artificially use a huge multiplier to prioritize a

- From a weigher developer point of view, somebody willing to implement
  one has to care about 1) returning a list of values, 2) setting the
  minimum and maximum values where the weights can range, if they are
  needed and they are significant for the weighing. For a weigher
  developer there are two use cases:

    Case 1: Use of a percentage instead of absolute values (for example, %
    of free RAM). If we compare two nodes focusing on the percentage of free
    ram, the maximum value for the weigher is 100. If we have two nodes one
    with 2048 total/1024 free, and the second one 1024 total/512 free they
    will get both the same weight, since they have the same % of free RAM
    (that is, the 50%).

    Case 2: Use of absolute values. In this case, the maximum of the weigher
    will be the maximum of the values in the list (in the case above, 1024)
    or the maximum value that the magnitude could take (in the case above,
    2048). How this maximum is set, is a decision of the developer. He may
    let the operator choose the behaviour of the weigher though.

- From the point of view of the scheduler we ensure that it is using
  normalized values, and not leveraging the normalization mechanism to the

Changes introduced this commit:

1) it introduces weight normalization so that we can apply multipliers
   easily. All the weights for an object will be normalized between 0.0 and
   1.0 before being sumed up, so that the final weight for a host will be:

    weight = w1_multiplier * norm(w1) + w2_multiplier * norm(w2) + ...

2) weights.BaseWeigher has been changed into an ABC so that we enforce
   that all weighers have the expected methods.

3) weights.BaseWeigher.weigh_objects() does no longer sum up the
   computer weighs to the object, but it rather returns a list that will be
   then normalized and added to the existing weight by BaseWeightHandler

4) Adapt the existing weighers to the above changes. Namely
    - New 'offset_weight_multiplier' for the cell weigher
    - Changed the name of the existing multiplier methods.

5) unittests for all of the introduced changes.

Implements blueprint normalize-scheduler-weights

DocImpact: Now weights for an object are normalized before suming them
up. This means that each weigher will take a maximum value of 1. This
may have an impact for operators that are using more than one weigher
(currently there is only one weigher: RAMWeiger) and for operators using
cells (where we have several weighers). It is needed to review then the
multipliers used and adjust them properly in case they have been

Docimpact: There is a new configuration option 'offset_weight_multiplier'
in nova.cells.weights.weight_offset.WeightOffsetWeigher

Change-Id: I81bf90898d3cb81541f4390596823cc00106eb20
Alvaro Lopez Garcia 10 years ago
parent 65c7027b5c
commit e5ba849437

@ -263,27 +263,35 @@ default when no filters are specified in the request.
Filter Scheduler uses so-called **weights** during its work.
Filter Scheduler uses the so called **weights** during its work. A weigher is a
way to select the best suitable host from a group of valid hosts by giving
weights to all the hosts in the list.
The Filter Scheduler weights hosts based on the config option
`scheduler_weight_classes`, this defaults to
`nova.scheduler.weights.all_weighers`, which selects all the available weighers
in the package nova.scheduler.weights. Hosts are then weighted and sorted with
the largest weight winning. For each host, the final weight is calculated by
summing up all weigher's weight value multiplying its own weight_mutiplier:
In order to prioritize one weigher against another, all the weighers have to
define a multiplier that will be applied before computing the weight for a node.
All the weights are normalized beforehand so that the multiplier can be applied
easily. Therefore the final weight for the object will be::
weight = w1_multiplier * norm(w1) + w2_multiplier * norm(w2) + ...
final_weight = 0
for each weigher:
final_weight += weigher's weight_mutiplier * weigher's calculated weight value
A weigher should be a subclass of ``weights.BaseHostWeigher`` and they must
implement the ``weight_multiplier`` and ``weight_object`` methods. If the
``weight_objects`` method is overriden it just return a list of weights, and not
modify the weight of the object directly, since final weights are normalized and
computed by ``weight.BaseWeightHandler``.
The weigher's weight_mutiplier can be set in the configuration file, e.g.
The Filter Scheduler weighs hosts based on the config option
`scheduler_weight_classes`, this defaults to
`nova.scheduler.weights.all_weighers`, which selects the following weighers:
* |RamWeigher| Hosts are then weighted and sorted with the largest weight winning.
If the multiplier is negative, the host with less RAM available will win (useful
for stacking hosts, instead of spreading).
* |MetricsWeigher| This weigher can compute the weight based on the compute node
host's various metrics. The to-be weighed metrics and their weighing ratio
are specified in the configuration file as the followings::
metrics_weight_setting = name1=1.0, name2=-1.0
Filter Scheduler finds local list of acceptable hosts by repeated filtering and
weighing. Each time it chooses a host, it virtually consumes resources on it,
@ -322,3 +330,5 @@ in :mod:``nova.tests.scheduler``.
.. |AggregateTypeAffinityFilter| replace:: :class:`AggregateTypeAffinityFilter <nova.scheduler.filters.type_filter.AggregateTypeAffinityFilter>`
.. |AggregateInstanceExtraSpecsFilter| replace:: :class:`AggregateInstanceExtraSpecsFilter <nova.scheduler.filters.aggregate_instance_extra_specs.AggregateInstanceExtraSpecsFilter>`
.. |AggregateMultiTenancyIsolation| replace:: :class:`AggregateMultiTenancyIsolation <nova.scheduler.filters.aggregate_multitenancy_isolation.AggregateMultiTenancyIsolation>`
.. |RamWeigher| replace:: :class:`RamWeigher <nova.scheduler.weights.all_weighers.RamWeigher>`

@ -3058,6 +3058,15 @@
# Options defined in nova.cells.weights.weight_offset
# Multiplier used to weigh offset weigher. (floating point
# value)

@ -48,7 +48,7 @@ class MuteChildWeigher(weights.BaseCellWeigher):
def _weight_multiplier(self):
def weight_multiplier(self):
# negative multiplier => lower weight
return CONF.cells.mute_weight_multiplier

@ -34,7 +34,7 @@ CONF.register_opts(ram_weigher_opts, group='cells')
class RamByInstanceTypeWeigher(weights.BaseCellWeigher):
"""Weigh cells by instance_type requested."""
def _weight_multiplier(self):
def weight_multiplier(self):
return CONF.cells.ram_weight_multiplier
def _weigh_object(self, cell, weight_properties):

@ -18,8 +18,19 @@ Weigh cells by their weight_offset in the DB. Cells with higher
weight_offsets in the DB will be preferred.
from oslo.config import cfg
from nova.cells import weights
weigher_opts = [
help='Multiplier used to weigh offset weigher.'),
CONF.register_opts(weigher_opts, group='cells')
class WeightOffsetWeigher(weights.BaseCellWeigher):
@ -28,6 +39,9 @@ class WeightOffsetWeigher(weights.BaseCellWeigher):
its weight_offset to 999999999999999 (highest weight wins)
def weight_multiplier(self):
return CONF.cells.offset_weight_multiplier
def _weigh_object(self, cell, weight_properties):
"""Returns whatever was in the DB for weight_offset."""
return cell.db_info.get('weight_offset', 0)

@ -76,7 +76,7 @@ class MetricsWeigher(weights.BaseHostWeigher):
" metrics_weight_setting: %s"),
def _weight_multiplier(self):
def weight_multiplier(self):
"""Override the weight multiplier."""
return CONF.metrics.weight_multiplier

@ -36,7 +36,9 @@ CONF.register_opts(ram_weight_opts)
class RAMWeigher(weights.BaseHostWeigher):
def _weight_multiplier(self):
minval = 0
def weight_multiplier(self):
"""Override the weight multiplier."""
return CONF.ram_weight_multiplier

@ -213,5 +213,5 @@ class MuteWeigherTestClass(_WeigherTestClass):
for i in range(2):
weighed_cell = weighed_cells.pop(0)
self.assertEqual(1000 * -10.0, weighed_cell.weight)
self.assertEqual(-10.0, weighed_cell.weight)
self.assertIn(, ['cell1', 'cell2'])

@ -72,37 +72,62 @@ class RamWeigherTestCase(test.NoDBTestCase):
# so, host4 should win:
weighed_host = self._get_weighed_host(hostinfo_list)
self.assertEqual(weighed_host.weight, 8192)
self.assertEqual(weighed_host.weight, 1.0)
self.assertEqual(, 'host4')
def test_ram_filter_multiplier1(self):
hostinfo_list = self._get_all_hosts()
# host1: free_ram_mb=-512
# host2: free_ram_mb=-1024
# host3: free_ram_mb=-3072
# host4: free_ram_mb=-8192
# host1: free_ram_mb=512
# host2: free_ram_mb=1024
# host3: free_ram_mb=3072
# host4: free_ram_mb=8192
# so, host1 should win:
# We do not know the host, all have same weight.
weighed_host = self._get_weighed_host(hostinfo_list)
self.assertEqual(weighed_host.weight, -512)
self.assertEqual(, 'host1')
self.assertEqual(weighed_host.weight, 0.0)
def test_ram_filter_multiplier2(self):
hostinfo_list = self._get_all_hosts()
# host1: free_ram_mb=512 * 2
# host2: free_ram_mb=1024 * 2
# host3: free_ram_mb=3072 * 2
# host4: free_ram_mb=8192 * 2
# host1: free_ram_mb=512
# host2: free_ram_mb=1024
# host3: free_ram_mb=3072
# host4: free_ram_mb=8192
# so, host4 should win:
weighed_host = self._get_weighed_host(hostinfo_list)
self.assertEqual(weighed_host.weight, 8192 * 2)
self.assertEqual(weighed_host.weight, 1.0 * 2)
self.assertEqual(, 'host4')
def test_ram_filter_negative(self):
hostinfo_list = self._get_all_hosts()
host_attr = {'id': 100, 'memory_mb': 8192, 'free_ram_mb': -512}
host_state = fakes.FakeHostState('negative', 'negative', host_attr)
hostinfo_list = list(hostinfo_list) + [host_state]
# host1: free_ram_mb=512
# host2: free_ram_mb=1024
# host3: free_ram_mb=3072
# host4: free_ram_mb=8192
# negativehost: free_ram_mb=-512
# so, host4 should win
weights = self.weight_handler.get_weighed_objects(self.weight_classes,
hostinfo_list, {})
weighed_host = weights[0]
self.assertEqual(weighed_host.weight, 1)
self.assertEqual(, "host4")
# and negativehost should lose
weighed_host = weights[-1]
self.assertEqual(weighed_host.weight, 0)
self.assertEqual(, "negative")
class MetricsWeigherTestCase(test.NoDBTestCase):
def setUp(self):
@ -139,7 +164,7 @@ class MetricsWeigherTestCase(test.NoDBTestCase):
# host4: foo=8192
# so, host4 should win:
setting = ['foo=1']
self._do_test(setting, 8192, 'host4')
self._do_test(setting, 1.0, 'host4')
def test_multiple_resource(self):
# host1: foo=512, bar=1
@ -148,7 +173,7 @@ class MetricsWeigherTestCase(test.NoDBTestCase):
# host4: foo=8192, bar=0
# so, host2 should win:
setting = ['foo=0.0001', 'bar=1']
self._do_test(setting, 2.1024, 'host2')
self._do_test(setting, 1.0, 'host2')
def test_single_resourcenegtive_ratio(self):
# host1: foo=512
@ -157,7 +182,7 @@ class MetricsWeigherTestCase(test.NoDBTestCase):
# host4: foo=8192
# so, host1 should win:
setting = ['foo=-1']
self._do_test(setting, -512, 'host1')
self._do_test(setting, 1.0, 'host1')
def test_multiple_resource_missing_ratio(self):
# host1: foo=512, bar=1
@ -166,7 +191,7 @@ class MetricsWeigherTestCase(test.NoDBTestCase):
# host4: foo=8192, bar=0
# so, host4 should win:
setting = ['foo=0.0001', 'bar']
self._do_test(setting, 0.8192, 'host4')
self._do_test(setting, 1.0, 'host4')
def test_multiple_resource_wrong_ratio(self):
# host1: foo=512, bar=1
@ -175,7 +200,7 @@ class MetricsWeigherTestCase(test.NoDBTestCase):
# host4: foo=8192, bar=0
# so, host4 should win:
setting = ['foo=0.0001', 'bar = 2.0t']
self._do_test(setting, 0.8192, 'host4')
self._do_test(setting, 1.0, 'host4')
def _check_parsing_result(self, weigher, setting, results):
self.flags(weight_setting=setting, group='metrics')

@ -0,0 +1,54 @@
# Copyright 2011-2012 OpenStack Foundation
# All Rights Reserved.
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
Tests For weights.
from nova import test
from nova import weights
class TestWeigher(test.NoDBTestCase):
def test_no_multiplier(self):
class FakeWeigher(weights.BaseWeigher):
def _weigh_object(self, *args, **kwargs):
def test_no_weight_object(self):
class FakeWeigher(weights.BaseWeigher):
def weight_multiplier(self, *args, **kwargs):
def test_normalization(self):
# weight_list, expected_result, minval, maxval
map_ = (
((), (), None, None),
((0.0, 0.0), (0.0, 0.0), None, None),
((1.0, 1.0), (0.0, 0.0), None, None),
((20.0, 50.0), (0.0, 1.0), None, None),
((20.0, 50.0), (0.0, 0.375), None, 100.0),
((20.0, 50.0), (0.4, 1.0), 0.0, None),
((20.0, 50.0), (0.2, 0.5), 0.0, 100.0),
normalize_to = (1.0, 10.0)
for seq, result, minval, maxval in map_:
ret = weights.normalize(seq, minval=minval, maxval=maxval)
self.assertEqual(tuple(ret), result)

@ -17,9 +17,40 @@
Pluggable Weighing support
import abc
from nova import loadables
def normalize(weight_list, minval=None, maxval=None):
"""Normalize the values in a list between 0 and 1.0.
The normalization is made regarding the lower and upper values present in
weight_list. If the minval and/or maxval parameters are set, these values
will be used instead of the minimum and maximum from the list.
If all the values are equal, they are normalized to 0.
if not weight_list:
return ()
if maxval is None:
maxval = max(weight_list)
if minval is None:
minval = min(weight_list)
maxval = float(maxval)
minval = float(minval)
if minval == maxval:
return [0] * len(weight_list)
range_ = maxval - minval
return ((i - minval) / range_ for i in weight_list)
class WeighedObject(object):
"""Object with weight information."""
def __init__(self, obj, weight):
@ -31,26 +62,59 @@ class WeighedObject(object):
class BaseWeigher(object):
"""Base class for pluggable weighers."""
def _weight_multiplier(self):
"""How weighted this weigher should be. Normally this would
be overridden in a subclass based on a config value.
"""Base class for pluggable weighers.
The attributes maxval and minval can be specified to set up the maximum
and minimum values for the weighed objects. These values will then be
taken into account in the normalization step, instead of taking the values
from the calculated weights.
__metaclass__ = abc.ABCMeta
minval = None
maxval = None
def weight_multiplier(self):
"""How weighted this weigher should be.
Override this method in a subclass, so that the returned value is
read from a configuration option to permit operators specify a
multiplier for the weigher.
return 1.0
def _weigh_object(self, obj, weight_properties):
"""Override in a subclass to specify a weight for a specific
return 0.0
"""Weigh an specific object."""
def weigh_objects(self, weighed_obj_list, weight_properties):
"""Weigh multiple objects. Override in a subclass if you need
need access to all objects in order to manipulate weights.
"""Weigh multiple objects.
Override in a subclass if you need access to all objects in order
to calculate weights. Do not modify the weight of an object here,
just return a list of weights.
# Calculate the weights
weights = []
for obj in weighed_obj_list:
obj.weight += (self._weight_multiplier() *
self._weigh_object(obj.obj, weight_properties))
weight = self._weigh_object(obj.obj, weight_properties)
# Record the min and max values if they are None. If they anything
# but none we assume that the weigher has set them
if self.minval is None:
self.minval = weight
if self.maxval is None:
self.maxval = weight
if weight < self.minval:
self.minval = weight
elif weight > self.maxval:
self.maxval = weight
return weights
class BaseWeightHandler(loadables.BaseLoader):
@ -58,7 +122,7 @@ class BaseWeightHandler(loadables.BaseLoader):
def get_weighed_objects(self, weigher_classes, obj_list,
"""Return a sorted (highest score first) list of WeighedObjects."""
"""Return a sorted (descending), normalized list of WeighedObjects."""
if not obj_list:
return []
@ -66,6 +130,15 @@ class BaseWeightHandler(loadables.BaseLoader):
weighed_objs = [self.object_class(obj, 0.0) for obj in obj_list]
for weigher_cls in weigher_classes:
weigher = weigher_cls()
weigher.weigh_objects(weighed_objs, weighing_properties)
weights = weigher.weigh_objects(weighed_objs, weighing_properties)
# Normalize the weights
weights = normalize(weights,
for i, weight in enumerate(weights):
obj = weighed_objs[i]
obj.weight += weigher.weight_multiplier() * weight
return sorted(weighed_objs, key=lambda x: x.weight, reverse=True)