Add CrossCellWeigher
When performing a resize, we'll want to (by default) select target hosts from the source cell to do a traditional resize if possible before considering target hosts in another cell which will be slower and more complicated. If the source cell is disabled or target flavor is not available in the source cell, then we'll have no choice but to select a host from another cell. But all things being equal between hosts, we want to stay within the source cell (by default). Therefore this change adds a new CrossCellWeigher and related configuration option to prefer hosts within the source cell when moving a server. The weigher is completely noop unless a cross-cell move is permitted by configuration, which will be provided in a future change. Part of blueprint cross-cell-resize Change-Id: Ib18752efa56cfeb860487fe6b26102bb4b1db038
This commit is contained in:
parent
83edfb7ec8
commit
18179aee4e
@ -906,6 +906,13 @@ Hosts are weighted based on the following options in the
|
|||||||
If the per aggregate ``build_failure_weight_multiplier``
|
If the per aggregate ``build_failure_weight_multiplier``
|
||||||
metadata is set, this multiplier will override the configuration option
|
metadata is set, this multiplier will override the configuration option
|
||||||
value.
|
value.
|
||||||
|
* - [filter_scheduler]
|
||||||
|
- ``cross_cell_move_weight_multiplier``
|
||||||
|
- Multiplier used for weighing hosts during a cross-cell move. By default,
|
||||||
|
prefers hosts within the same source cell when migrating a server.
|
||||||
|
If the per aggregate ``cross_cell_move_weight_multiplier``
|
||||||
|
metadata is set, this multiplier will override the configuration option
|
||||||
|
value.
|
||||||
* - [metrics]
|
* - [metrics]
|
||||||
- ``weight_multiplier``
|
- ``weight_multiplier``
|
||||||
- Multiplier for weighting meters. Use a floating-point value.
|
- Multiplier for weighting meters. Use a floating-point value.
|
||||||
|
@ -529,6 +529,16 @@ The Filter Scheduler weighs hosts based on the config option
|
|||||||
If more than one value is found for a host in aggregate metadata, the
|
If more than one value is found for a host in aggregate metadata, the
|
||||||
minimum value will be used.
|
minimum value will be used.
|
||||||
|
|
||||||
|
* |CrossCellWeigher| Weighs hosts based on which cell they are in. "Local"
|
||||||
|
cells are preferred when moving an instance. Use configuration option
|
||||||
|
:oslo.config:option:`filter_scheduler.cross_cell_move_weight_multiplier` to
|
||||||
|
control the weight. If per-aggregate value with the key
|
||||||
|
`cross_cell_move_weight_multiplier` is found, this value would be chosen
|
||||||
|
as the cross-cell move weight multiplier. Otherwise, it will fall back to the
|
||||||
|
:oslo.config:option:`filter_scheduler.cross_cell_move_weight_multiplier`.
|
||||||
|
If more than one value is found for a host in aggregate metadata, the
|
||||||
|
minimum value will be used.
|
||||||
|
|
||||||
Filter Scheduler makes a local list of acceptable hosts by repeated filtering and
|
Filter Scheduler makes a local list of acceptable hosts by repeated filtering and
|
||||||
weighing. Each time it chooses a host, it virtually consumes resources on it,
|
weighing. Each time it chooses a host, it virtually consumes resources on it,
|
||||||
so subsequent selections can adjust accordingly. It is useful if the customer
|
so subsequent selections can adjust accordingly. It is useful if the customer
|
||||||
@ -580,3 +590,4 @@ in :mod:`nova.tests.scheduler`.
|
|||||||
.. |ServerGroupSoftAntiAffinityWeigher| replace:: :class:`ServerGroupSoftAntiAffinityWeigher <nova.scheduler.weights.affinity.ServerGroupSoftAntiAffinityWeigher>`
|
.. |ServerGroupSoftAntiAffinityWeigher| replace:: :class:`ServerGroupSoftAntiAffinityWeigher <nova.scheduler.weights.affinity.ServerGroupSoftAntiAffinityWeigher>`
|
||||||
.. |DiskWeigher| replace:: :class:`DiskWeigher <nova.scheduler.weights.disk.DiskWeigher>`
|
.. |DiskWeigher| replace:: :class:`DiskWeigher <nova.scheduler.weights.disk.DiskWeigher>`
|
||||||
.. |BuildFailureWeigher| replace:: :class:`BuildFailureWeigher <nova.scheduler.weights.compute.BuildFailureWeigher>`
|
.. |BuildFailureWeigher| replace:: :class:`BuildFailureWeigher <nova.scheduler.weights.compute.BuildFailureWeigher>`
|
||||||
|
.. |CrossCellWeigher| replace:: :class:`CrossCellWeigher <nova.scheduler.weights.cross_cell.CrossCellWeigher>`
|
||||||
|
@ -533,6 +533,35 @@ Related options:
|
|||||||
|
|
||||||
* [compute]/consecutive_build_service_disable_threshold - Must be nonzero
|
* [compute]/consecutive_build_service_disable_threshold - Must be nonzero
|
||||||
for a compute to report data considered by this weigher.
|
for a compute to report data considered by this weigher.
|
||||||
|
"""),
|
||||||
|
cfg.FloatOpt(
|
||||||
|
"cross_cell_move_weight_multiplier",
|
||||||
|
default=1000000.0,
|
||||||
|
help="""
|
||||||
|
Multiplier used for weighing hosts during a cross-cell move.
|
||||||
|
|
||||||
|
This option determines how much weight is placed on a host which is within the
|
||||||
|
same source cell when moving a server, for example during cross-cell resize.
|
||||||
|
By default, when moving an instance, the scheduler will prefer hosts within
|
||||||
|
the same cell since cross-cell move operations can be slower and riskier due to
|
||||||
|
the complicated nature of cross-cell migrations.
|
||||||
|
|
||||||
|
This option is only used by the FilterScheduler and its subclasses; if you use
|
||||||
|
a different scheduler, this option has no effect. Similarly, if your cloud is
|
||||||
|
not configured to support cross-cell migrations, then this option has no
|
||||||
|
effect.
|
||||||
|
|
||||||
|
The value of this configuration option can be overridden per host aggregate
|
||||||
|
by setting the aggregate metadata key with the same name
|
||||||
|
(cross_cell_move_weight_multiplier).
|
||||||
|
|
||||||
|
Possible values:
|
||||||
|
|
||||||
|
* An integer or float value, where the value corresponds to the multiplier
|
||||||
|
ratio for this weigher. Positive values mean the weigher will prefer
|
||||||
|
hosts within the same cell in which the instance is currently running.
|
||||||
|
Negative values mean the weigher will prefer hosts in *other* cells from
|
||||||
|
which the instance is currently running.
|
||||||
"""),
|
"""),
|
||||||
cfg.BoolOpt(
|
cfg.BoolOpt(
|
||||||
"shuffle_best_same_weighed_hosts",
|
"shuffle_best_same_weighed_hosts",
|
||||||
|
63
nova/scheduler/weights/cross_cell.py
Normal file
63
nova/scheduler/weights/cross_cell.py
Normal file
@ -0,0 +1,63 @@
|
|||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||||
|
# not use this file except in compliance with the License. You may obtain
|
||||||
|
# a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||||
|
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||||
|
# License for the specific language governing permissions and limitations
|
||||||
|
# under the License.
|
||||||
|
"""
|
||||||
|
Cross-cell move weigher. Weighs hosts based on which cell they are in. "Local"
|
||||||
|
cells are preferred when moving an instance. In other words, select a host
|
||||||
|
from the source cell all other things being equal.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from nova import conf
|
||||||
|
from nova.scheduler import utils
|
||||||
|
from nova.scheduler import weights
|
||||||
|
|
||||||
|
CONF = conf.CONF
|
||||||
|
|
||||||
|
|
||||||
|
class CrossCellWeigher(weights.BaseHostWeigher):
|
||||||
|
|
||||||
|
def weight_multiplier(self, host_state):
|
||||||
|
"""How weighted this weigher should be."""
|
||||||
|
return utils.get_weight_multiplier(
|
||||||
|
host_state, 'cross_cell_move_weight_multiplier',
|
||||||
|
CONF.filter_scheduler.cross_cell_move_weight_multiplier)
|
||||||
|
|
||||||
|
def _weigh_object(self, host_state, weight_properties):
|
||||||
|
"""Higher weights win. Hosts within the "preferred" cell are weighed
|
||||||
|
higher than hosts in other cells.
|
||||||
|
|
||||||
|
:param host_state: nova.scheduler.host_manager.HostState object
|
||||||
|
representing a ComputeNode in a cell
|
||||||
|
:param weight_properties: nova.objects.RequestSpec - this is inspected
|
||||||
|
to see if there is a preferred cell via the requested_destination
|
||||||
|
field and if so, is the request spec allowing cross-cell move
|
||||||
|
:returns: 1 if cross-cell move and host_state is within the preferred
|
||||||
|
cell, -1 if cross-cell move and host_state is *not* within the
|
||||||
|
preferred cell, 0 for all other cases
|
||||||
|
"""
|
||||||
|
# RequestSpec.requested_destination.cell should only be set for
|
||||||
|
# move operations. The allow_cross_cell_move value will only be True if
|
||||||
|
# policy allows.
|
||||||
|
if ('requested_destination' in weight_properties and
|
||||||
|
weight_properties.requested_destination and
|
||||||
|
'cell' in weight_properties.requested_destination and
|
||||||
|
weight_properties.requested_destination.cell and
|
||||||
|
weight_properties.requested_destination.allow_cross_cell_move):
|
||||||
|
# Determine if the given host is in the "preferred" cell from
|
||||||
|
# the request spec. If it is, weigh it higher.
|
||||||
|
if (host_state.cell_uuid ==
|
||||||
|
weight_properties.requested_destination.cell.uuid):
|
||||||
|
return 1
|
||||||
|
# The host is in another cell, so weigh it lower.
|
||||||
|
return -1
|
||||||
|
# We don't know or don't care what cell we're going to be in, so noop.
|
||||||
|
return 0
|
@ -49,7 +49,11 @@ class TestMultiCellMigrate(integrated_helpers.ProviderUsageBaseTestCase):
|
|||||||
def setUp(self):
|
def setUp(self):
|
||||||
# Use our custom weigher defined above to make sure that we have
|
# Use our custom weigher defined above to make sure that we have
|
||||||
# a predictable scheduling sort order during server create.
|
# a predictable scheduling sort order during server create.
|
||||||
self.flags(weight_classes=[__name__ + '.HostNameWeigher'],
|
weight_classes = [
|
||||||
|
__name__ + '.HostNameWeigher',
|
||||||
|
'nova.scheduler.weights.cross_cell.CrossCellWeigher'
|
||||||
|
]
|
||||||
|
self.flags(weight_classes=weight_classes,
|
||||||
group='filter_scheduler')
|
group='filter_scheduler')
|
||||||
super(TestMultiCellMigrate, self).setUp()
|
super(TestMultiCellMigrate, self).setUp()
|
||||||
self.cinder = self.useFixture(nova_fixtures.CinderFixture(self))
|
self.cinder = self.useFixture(nova_fixtures.CinderFixture(self))
|
||||||
@ -790,11 +794,28 @@ class TestMultiCellMigrate(integrated_helpers.ProviderUsageBaseTestCase):
|
|||||||
'host3', cell_name=self.host_to_cell_mappings['host1'])
|
'host3', cell_name=self.host_to_cell_mappings['host1'])
|
||||||
self._resize_and_validate(target_host='host2')
|
self._resize_and_validate(target_host='host2')
|
||||||
|
|
||||||
# TODO(mriedem): Test cross-cell list where the source cell has two
|
def test_cold_migrate_cross_cell_weigher_stays_in_source_cell(self):
|
||||||
# hosts so the CrossCellWeigher picks the other host in the source cell
|
"""Tests cross-cell cold migrate where the source cell has two hosts
|
||||||
# and we do a traditional resize. Add a variant on this where the flavor
|
so the CrossCellWeigher picks the other host in the source cell and we
|
||||||
# being resized to is only available, via aggregate, on the host in the
|
do a traditional resize. Note that in this case, HostNameWeigher will
|
||||||
# other cell so the CrossCellWeigher is overruled by the filters.
|
actually weigh host2 (in cell2) higher than host3 (in cell1) but the
|
||||||
|
CrossCellWeigher will weigh host2 much lower than host3 since host3 is
|
||||||
|
in the same cell as the source host (host1).
|
||||||
|
"""
|
||||||
|
# Create the server first (should go in host1).
|
||||||
|
server = self._create_server(self.api.get_flavors()[0])
|
||||||
|
# Start another compute host service in cell1.
|
||||||
|
self._start_compute(
|
||||||
|
'host3', cell_name=self.host_to_cell_mappings['host1'])
|
||||||
|
# Cold migrate the server which should move the server to host3.
|
||||||
|
self.admin_api.post_server_action(server['id'], {'migrate': None})
|
||||||
|
server = self._wait_for_state_change(server, 'VERIFY_RESIZE')
|
||||||
|
self.assertEqual('host3', server['OS-EXT-SRV-ATTR:host'])
|
||||||
|
|
||||||
|
# TODO(mriedem): Add a variant of
|
||||||
|
# test_cold_migrate_cross_cell_weigher_stays_in_source_cell where the
|
||||||
|
# flavor being resized to is only available, via aggregate, on the host in
|
||||||
|
# the other cell so the CrossCellWeigher is overruled by the filters.
|
||||||
|
|
||||||
# TODO(mriedem): Test a bunch of rollback scenarios.
|
# TODO(mriedem): Test a bunch of rollback scenarios.
|
||||||
|
|
||||||
|
142
nova/tests/unit/scheduler/weights/test_cross_cell.py
Normal file
142
nova/tests/unit/scheduler/weights/test_cross_cell.py
Normal file
@ -0,0 +1,142 @@
|
|||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||||
|
# not use this file except in compliance with the License. You may obtain
|
||||||
|
# a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||||
|
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||||
|
# License for the specific language governing permissions and limitations
|
||||||
|
# under the License.
|
||||||
|
|
||||||
|
from oslo_utils.fixture import uuidsentinel as uuids
|
||||||
|
|
||||||
|
from nova import conf
|
||||||
|
from nova import objects
|
||||||
|
from nova.scheduler import weights
|
||||||
|
from nova.scheduler.weights import cross_cell
|
||||||
|
from nova import test
|
||||||
|
from nova.tests.unit.scheduler import fakes
|
||||||
|
|
||||||
|
CONF = conf.CONF
|
||||||
|
|
||||||
|
|
||||||
|
class CrossCellWeigherTestCase(test.NoDBTestCase):
|
||||||
|
"""Tests for the FilterScheduler CrossCellWeigher."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
super(CrossCellWeigherTestCase, self).setUp()
|
||||||
|
self.weight_handler = weights.HostWeightHandler()
|
||||||
|
self.weighers = [cross_cell.CrossCellWeigher()]
|
||||||
|
|
||||||
|
def _get_weighed_hosts(self, request_spec):
|
||||||
|
hosts = self._get_all_hosts()
|
||||||
|
return self.weight_handler.get_weighed_objects(
|
||||||
|
self.weighers, hosts, request_spec)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _get_all_hosts():
|
||||||
|
"""Provides two hosts, one in cell1 and one in cell2."""
|
||||||
|
host_values = [
|
||||||
|
('host1', 'node1', {'cell_uuid': uuids.cell1}),
|
||||||
|
('host2', 'node2', {'cell_uuid': uuids.cell2}),
|
||||||
|
]
|
||||||
|
return [fakes.FakeHostState(host, node, values)
|
||||||
|
for host, node, values in host_values]
|
||||||
|
|
||||||
|
def test_get_weighed_hosts_no_requested_destination_or_cell(self):
|
||||||
|
"""Weights should all be 0.0 given there is no requested_destination
|
||||||
|
or source cell in the RequestSpec, e.g. initial server create scenario.
|
||||||
|
"""
|
||||||
|
# Test the requested_destination field not being set.
|
||||||
|
request_spec = objects.RequestSpec()
|
||||||
|
weighed_hosts = self._get_weighed_hosts(request_spec)
|
||||||
|
self.assertTrue(all([wh.weight == 0.0
|
||||||
|
for wh in weighed_hosts]))
|
||||||
|
# Test the requested_destination field being set to None.
|
||||||
|
request_spec.requested_destination = None
|
||||||
|
weighed_hosts = self._get_weighed_hosts(request_spec)
|
||||||
|
self.assertTrue(all([wh.weight == 0.0
|
||||||
|
for wh in weighed_hosts]))
|
||||||
|
# Test the requested_destination field being set but without the
|
||||||
|
# cell field set.
|
||||||
|
request_spec.requested_destination = objects.Destination()
|
||||||
|
weighed_hosts = self._get_weighed_hosts(request_spec)
|
||||||
|
self.assertTrue(all([wh.weight == 0.0
|
||||||
|
for wh in weighed_hosts]))
|
||||||
|
# Test the requested_destination field being set with the cell field
|
||||||
|
# set but to None.
|
||||||
|
request_spec.requested_destination = objects.Destination(cell=None)
|
||||||
|
weighed_hosts = self._get_weighed_hosts(request_spec)
|
||||||
|
self.assertTrue(all([wh.weight == 0.0
|
||||||
|
for wh in weighed_hosts]))
|
||||||
|
|
||||||
|
def test_get_weighed_hosts_allow_cross_cell_move_false(self):
|
||||||
|
"""Tests the scenario that the source cell is set in the requested
|
||||||
|
destination but it's not a cross cell move so the weights should all
|
||||||
|
be 0.0.
|
||||||
|
"""
|
||||||
|
request_spec = objects.RequestSpec(
|
||||||
|
requested_destination=objects.Destination(
|
||||||
|
cell=objects.CellMapping(uuid=uuids.cell1)))
|
||||||
|
weighed_hosts = self._get_weighed_hosts(request_spec)
|
||||||
|
self.assertTrue(all([wh.weight == 0.0
|
||||||
|
for wh in weighed_hosts]))
|
||||||
|
|
||||||
|
def test_get_weighed_hosts_allow_cross_cell_move_true_positive(self):
|
||||||
|
"""Tests a cross-cell move where the host in the source (preferred)
|
||||||
|
cell should be weighed higher than the host in the other cell based
|
||||||
|
on the default configuration.
|
||||||
|
"""
|
||||||
|
request_spec = objects.RequestSpec(
|
||||||
|
requested_destination=objects.Destination(
|
||||||
|
cell=objects.CellMapping(uuid=uuids.cell1),
|
||||||
|
allow_cross_cell_move=True))
|
||||||
|
weighed_hosts = self._get_weighed_hosts(request_spec)
|
||||||
|
multiplier = CONF.filter_scheduler.cross_cell_move_weight_multiplier
|
||||||
|
self.assertEqual([multiplier, 0.0],
|
||||||
|
[wh.weight for wh in weighed_hosts])
|
||||||
|
# host1 should be preferred since it's in cell1
|
||||||
|
preferred_host = weighed_hosts[0]
|
||||||
|
self.assertEqual('host1', preferred_host.obj.host)
|
||||||
|
|
||||||
|
def test_get_weighed_hosts_allow_cross_cell_move_true_negative(self):
|
||||||
|
"""Tests a cross-cell move where the host in another cell should be
|
||||||
|
weighed higher than the host in the source cell because the weight
|
||||||
|
value is negative.
|
||||||
|
"""
|
||||||
|
self.flags(cross_cell_move_weight_multiplier=-1000,
|
||||||
|
group='filter_scheduler')
|
||||||
|
request_spec = objects.RequestSpec(
|
||||||
|
requested_destination=objects.Destination(
|
||||||
|
# cell1 is the source cell
|
||||||
|
cell=objects.CellMapping(uuid=uuids.cell1),
|
||||||
|
allow_cross_cell_move=True))
|
||||||
|
weighed_hosts = self._get_weighed_hosts(request_spec)
|
||||||
|
multiplier = CONF.filter_scheduler.cross_cell_move_weight_multiplier
|
||||||
|
self.assertEqual([0.0, multiplier],
|
||||||
|
[wh.weight for wh in weighed_hosts])
|
||||||
|
# host2 should be preferred since it's *not* in cell1
|
||||||
|
preferred_host = weighed_hosts[0]
|
||||||
|
self.assertEqual('host2', preferred_host.obj.host)
|
||||||
|
|
||||||
|
def test_multiplier(self):
|
||||||
|
weigher = self.weighers[0]
|
||||||
|
host1 = fakes.FakeHostState('fake-host', 'node', {})
|
||||||
|
# By default, return the cross_cell_move_weight_multiplier
|
||||||
|
# configuration directly since the host is not in an aggregate.
|
||||||
|
self.assertEqual(
|
||||||
|
CONF.filter_scheduler.cross_cell_move_weight_multiplier,
|
||||||
|
weigher.weight_multiplier(host1))
|
||||||
|
host1.aggregates = [
|
||||||
|
objects.Aggregate(
|
||||||
|
id=1,
|
||||||
|
name='foo',
|
||||||
|
hosts=['fake-host'],
|
||||||
|
metadata={'cross_cell_move_weight_multiplier': '-1.0'},
|
||||||
|
)]
|
||||||
|
# Read the weight multiplier from aggregate metadata to override the
|
||||||
|
# config.
|
||||||
|
self.assertEqual(-1.0, weigher.weight_multiplier(host1))
|
Loading…
Reference in New Issue
Block a user