Add CrossCellWeigher

When performing a resize, we'll want to (by default) select
target hosts from the source cell to do a traditional resize
if possible before considering target hosts in another cell
which will be slower and more complicated. If the source cell
is disabled or target flavor is not available in the source cell,
then we'll have no choice but to select a host from another cell.
But all things being equal between hosts, we want to stay within
the source cell (by default). Therefore this change adds a new
CrossCellWeigher and related configuration option to prefer hosts
within the source cell when moving a server. The weigher is
completely noop unless a cross-cell move is permitted by
configuration, which will be provided in a future change.

Part of blueprint cross-cell-resize

Change-Id: Ib18752efa56cfeb860487fe6b26102bb4b1db038

doc/source/admin/configuration/schedulers.rst

@@ -906,6 +906,13 @@ Hosts are weighted based on the following options in the
        If the per aggregate ``build_failure_weight_multiplier``
        metadata is set, this multiplier will override the configuration option
        value.
+   * - [filter_scheduler]
+     - ``cross_cell_move_weight_multiplier``
+     - Multiplier used for weighing hosts during a cross-cell move. By default,
+       prefers hosts within the same source cell when migrating a server.
+       If the per aggregate ``cross_cell_move_weight_multiplier``
+       metadata is set, this multiplier will override the configuration option
+       value.
    * - [metrics]
      - ``weight_multiplier``
      - Multiplier for weighting meters. Use a floating-point value.

doc/source/user/filter-scheduler.rst

@@ -529,6 +529,16 @@ The Filter Scheduler weighs hosts based on the config option
   If more than one value is found for a host in aggregate metadata, the
   minimum value will be used.
 
+* |CrossCellWeigher| Weighs hosts based on which cell they are in. "Local"
+  cells are preferred when moving an instance. Use configuration option
+  :oslo.config:option:`filter_scheduler.cross_cell_move_weight_multiplier` to
+  control the weight. If per-aggregate value with the key
+  `cross_cell_move_weight_multiplier` is found, this value would be chosen
+  as the cross-cell move weight multiplier. Otherwise, it will fall back to the
+  :oslo.config:option:`filter_scheduler.cross_cell_move_weight_multiplier`.
+  If more than one value is found for a host in aggregate metadata, the
+  minimum value will be used.
+
 Filter Scheduler makes a local list of acceptable hosts by repeated filtering and
 weighing. Each time it chooses a host, it virtually consumes resources on it,
 so subsequent selections can adjust accordingly. It is useful if the customer
@@ -580,3 +590,4 @@ in :mod:`nova.tests.scheduler`.
 .. |ServerGroupSoftAntiAffinityWeigher| replace:: :class:`ServerGroupSoftAntiAffinityWeigher <nova.scheduler.weights.affinity.ServerGroupSoftAntiAffinityWeigher>`
 .. |DiskWeigher| replace:: :class:`DiskWeigher <nova.scheduler.weights.disk.DiskWeigher>`
 .. |BuildFailureWeigher| replace:: :class:`BuildFailureWeigher <nova.scheduler.weights.compute.BuildFailureWeigher>`
+.. |CrossCellWeigher| replace:: :class:`CrossCellWeigher <nova.scheduler.weights.cross_cell.CrossCellWeigher>`

nova/conf/scheduler.py

@@ -533,6 +533,35 @@ Related options:
 
 * [compute]/consecutive_build_service_disable_threshold - Must be nonzero
   for a compute to report data considered by this weigher.
+"""),
+    cfg.FloatOpt(
+        "cross_cell_move_weight_multiplier",
+        default=1000000.0,
+        help="""
+Multiplier used for weighing hosts during a cross-cell move.
+
+This option determines how much weight is placed on a host which is within the
+same source cell when moving a server, for example during cross-cell resize.
+By default, when moving an instance, the scheduler will prefer hosts within
+the same cell since cross-cell move operations can be slower and riskier due to
+the complicated nature of cross-cell migrations.
+
+This option is only used by the FilterScheduler and its subclasses; if you use
+a different scheduler, this option has no effect. Similarly, if your cloud is
+not configured to support cross-cell migrations, then this option has no
+effect.
+
+The value of this configuration option can be overridden per host aggregate
+by setting the aggregate metadata key with the same name
+(cross_cell_move_weight_multiplier).
+
+Possible values:
+
+* An integer or float value, where the value corresponds to the multiplier
+  ratio for this weigher. Positive values mean the weigher will prefer
+  hosts within the same cell in which the instance is currently running.
+  Negative values mean the weigher will prefer hosts in *other* cells from
+  which the instance is currently running.
 """),
     cfg.BoolOpt(
         "shuffle_best_same_weighed_hosts",

nova/scheduler/weights/cross_cell.py

@@ -0,0 +1,63 @@
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+"""
+Cross-cell move weigher. Weighs hosts based on which cell they are in. "Local"
+cells are preferred when moving an instance. In other words, select a host
+from the source cell all other things being equal.
+"""
+
+from nova import conf
+from nova.scheduler import utils
+from nova.scheduler import weights
+
+CONF = conf.CONF
+
+
+class CrossCellWeigher(weights.BaseHostWeigher):
+
+    def weight_multiplier(self, host_state):
+        """How weighted this weigher should be."""
+        return utils.get_weight_multiplier(
+            host_state, 'cross_cell_move_weight_multiplier',
+            CONF.filter_scheduler.cross_cell_move_weight_multiplier)
+
+    def _weigh_object(self, host_state, weight_properties):
+        """Higher weights win. Hosts within the "preferred" cell are weighed
+        higher than hosts in other cells.
+
+        :param host_state: nova.scheduler.host_manager.HostState object
+            representing a ComputeNode in a cell
+        :param weight_properties: nova.objects.RequestSpec - this is inspected
+            to see if there is a preferred cell via the requested_destination
+            field and if so, is the request spec allowing cross-cell move
+        :returns: 1 if cross-cell move and host_state is within the preferred
+            cell, -1 if cross-cell move and host_state is *not* within the
+            preferred cell, 0 for all other cases
+        """
+        # RequestSpec.requested_destination.cell should only be set for
+        # move operations. The allow_cross_cell_move value will only be True if
+        # policy allows.
+        if ('requested_destination' in weight_properties and
+                weight_properties.requested_destination and
+                'cell' in weight_properties.requested_destination and
+                weight_properties.requested_destination.cell and
+                weight_properties.requested_destination.allow_cross_cell_move):
+            # Determine if the given host is in the "preferred" cell from
+            # the request spec. If it is, weigh it higher.
+            if (host_state.cell_uuid ==
+                    weight_properties.requested_destination.cell.uuid):
+                return 1
+            # The host is in another cell, so weigh it lower.
+            return -1
+        # We don't know or don't care what cell we're going to be in, so noop.
+        return 0

nova/tests/functional/test_cross_cell_migrate.py

@@ -49,7 +49,11 @@ class TestMultiCellMigrate(integrated_helpers.ProviderUsageBaseTestCase):
     def setUp(self):
         # Use our custom weigher defined above to make sure that we have
         # a predictable scheduling sort order during server create.
-        self.flags(weight_classes=[__name__ + '.HostNameWeigher'],
+        weight_classes = [
+            __name__ + '.HostNameWeigher',
+            'nova.scheduler.weights.cross_cell.CrossCellWeigher'
+        ]
+        self.flags(weight_classes=weight_classes,
                    group='filter_scheduler')
         super(TestMultiCellMigrate, self).setUp()
         self.cinder = self.useFixture(nova_fixtures.CinderFixture(self))
@@ -790,11 +794,28 @@ class TestMultiCellMigrate(integrated_helpers.ProviderUsageBaseTestCase):
             'host3', cell_name=self.host_to_cell_mappings['host1'])
         self._resize_and_validate(target_host='host2')
 
-    # TODO(mriedem): Test cross-cell list where the source cell has two
-    # hosts so the CrossCellWeigher picks the other host in the source cell
-    # and we do a traditional resize. Add a variant on this where the flavor
-    # being resized to is only available, via aggregate, on the host in the
-    # other cell so the CrossCellWeigher is overruled by the filters.
+    def test_cold_migrate_cross_cell_weigher_stays_in_source_cell(self):
+        """Tests cross-cell cold migrate where the source cell has two hosts
+        so the CrossCellWeigher picks the other host in the source cell and we
+        do a traditional resize. Note that in this case, HostNameWeigher will
+        actually weigh host2 (in cell2) higher than host3 (in cell1) but the
+        CrossCellWeigher will weigh host2 much lower than host3 since host3 is
+        in the same cell as the source host (host1).
+        """
+        # Create the server first (should go in host1).
+        server = self._create_server(self.api.get_flavors()[0])
+        # Start another compute host service in cell1.
+        self._start_compute(
+            'host3', cell_name=self.host_to_cell_mappings['host1'])
+        # Cold migrate the server which should move the server to host3.
+        self.admin_api.post_server_action(server['id'], {'migrate': None})
+        server = self._wait_for_state_change(server, 'VERIFY_RESIZE')
+        self.assertEqual('host3', server['OS-EXT-SRV-ATTR:host'])
+
+    # TODO(mriedem): Add a variant of
+    # test_cold_migrate_cross_cell_weigher_stays_in_source_cell where the
+    # flavor being resized to is only available, via aggregate, on the host in
+    # the other cell so the CrossCellWeigher is overruled by the filters.
 
     # TODO(mriedem): Test a bunch of rollback scenarios.
 

nova/tests/unit/scheduler/weights/test_cross_cell.py

@@ -0,0 +1,142 @@
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from oslo_utils.fixture import uuidsentinel as uuids
+
+from nova import conf
+from nova import objects
+from nova.scheduler import weights
+from nova.scheduler.weights import cross_cell
+from nova import test
+from nova.tests.unit.scheduler import fakes
+
+CONF = conf.CONF
+
+
+class CrossCellWeigherTestCase(test.NoDBTestCase):
+    """Tests for the FilterScheduler CrossCellWeigher."""
+
+    def setUp(self):
+        super(CrossCellWeigherTestCase, self).setUp()
+        self.weight_handler = weights.HostWeightHandler()
+        self.weighers = [cross_cell.CrossCellWeigher()]
+
+    def _get_weighed_hosts(self, request_spec):
+        hosts = self._get_all_hosts()
+        return self.weight_handler.get_weighed_objects(
+            self.weighers, hosts, request_spec)
+
+    @staticmethod
+    def _get_all_hosts():
+        """Provides two hosts, one in cell1 and one in cell2."""
+        host_values = [
+            ('host1', 'node1', {'cell_uuid': uuids.cell1}),
+            ('host2', 'node2', {'cell_uuid': uuids.cell2}),
+        ]
+        return [fakes.FakeHostState(host, node, values)
+                for host, node, values in host_values]
+
+    def test_get_weighed_hosts_no_requested_destination_or_cell(self):
+        """Weights should all be 0.0 given there is no requested_destination
+        or source cell in the RequestSpec, e.g. initial server create scenario.
+        """
+        # Test the requested_destination field not being set.
+        request_spec = objects.RequestSpec()
+        weighed_hosts = self._get_weighed_hosts(request_spec)
+        self.assertTrue(all([wh.weight == 0.0
+                             for wh in weighed_hosts]))
+        # Test the requested_destination field being set to None.
+        request_spec.requested_destination = None
+        weighed_hosts = self._get_weighed_hosts(request_spec)
+        self.assertTrue(all([wh.weight == 0.0
+                             for wh in weighed_hosts]))
+        # Test the requested_destination field being set but without the
+        # cell field set.
+        request_spec.requested_destination = objects.Destination()
+        weighed_hosts = self._get_weighed_hosts(request_spec)
+        self.assertTrue(all([wh.weight == 0.0
+                             for wh in weighed_hosts]))
+        # Test the requested_destination field being set with the cell field
+        # set but to None.
+        request_spec.requested_destination = objects.Destination(cell=None)
+        weighed_hosts = self._get_weighed_hosts(request_spec)
+        self.assertTrue(all([wh.weight == 0.0
+                             for wh in weighed_hosts]))
+
+    def test_get_weighed_hosts_allow_cross_cell_move_false(self):
+        """Tests the scenario that the source cell is set in the requested
+        destination but it's not a cross cell move so the weights should all
+        be 0.0.
+        """
+        request_spec = objects.RequestSpec(
+            requested_destination=objects.Destination(
+                cell=objects.CellMapping(uuid=uuids.cell1)))
+        weighed_hosts = self._get_weighed_hosts(request_spec)
+        self.assertTrue(all([wh.weight == 0.0
+                             for wh in weighed_hosts]))
+
+    def test_get_weighed_hosts_allow_cross_cell_move_true_positive(self):
+        """Tests a cross-cell move where the host in the source (preferred)
+        cell should be weighed higher than the host in the other cell based
+        on the default configuration.
+        """
+        request_spec = objects.RequestSpec(
+            requested_destination=objects.Destination(
+                cell=objects.CellMapping(uuid=uuids.cell1),
+                allow_cross_cell_move=True))
+        weighed_hosts = self._get_weighed_hosts(request_spec)
+        multiplier = CONF.filter_scheduler.cross_cell_move_weight_multiplier
+        self.assertEqual([multiplier, 0.0],
+                         [wh.weight for wh in weighed_hosts])
+        # host1 should be preferred since it's in cell1
+        preferred_host = weighed_hosts[0]
+        self.assertEqual('host1', preferred_host.obj.host)
+
+    def test_get_weighed_hosts_allow_cross_cell_move_true_negative(self):
+        """Tests a cross-cell move where the host in another cell should be
+        weighed higher than the host in the source cell because the weight
+        value is negative.
+        """
+        self.flags(cross_cell_move_weight_multiplier=-1000,
+                   group='filter_scheduler')
+        request_spec = objects.RequestSpec(
+            requested_destination=objects.Destination(
+                # cell1 is the source cell
+                cell=objects.CellMapping(uuid=uuids.cell1),
+                allow_cross_cell_move=True))
+        weighed_hosts = self._get_weighed_hosts(request_spec)
+        multiplier = CONF.filter_scheduler.cross_cell_move_weight_multiplier
+        self.assertEqual([0.0, multiplier],
+                         [wh.weight for wh in weighed_hosts])
+        # host2 should be preferred since it's *not* in cell1
+        preferred_host = weighed_hosts[0]
+        self.assertEqual('host2', preferred_host.obj.host)
+
+    def test_multiplier(self):
+        weigher = self.weighers[0]
+        host1 = fakes.FakeHostState('fake-host', 'node', {})
+        # By default, return the cross_cell_move_weight_multiplier
+        # configuration directly since the host is not in an aggregate.
+        self.assertEqual(
+            CONF.filter_scheduler.cross_cell_move_weight_multiplier,
+            weigher.weight_multiplier(host1))
+        host1.aggregates = [
+            objects.Aggregate(
+                id=1,
+                name='foo',
+                hosts=['fake-host'],
+                metadata={'cross_cell_move_weight_multiplier': '-1.0'},
+            )]
+        # Read the weight multiplier from aggregate metadata to override the
+        # config.
+        self.assertEqual(-1.0, weigher.weight_multiplier(host1))