Add PCIWeigher

Take three hosts, one with a single PCI device, one with many PCI
devices, and one with no PCI devices. Nova should prioritise these
differently based on the demands of the instance. If the instance
requests a single PCI device, then the first of the hosts should be
preferred. Similarly, if the instance requests multiple PCI devices,
then the second of these hosts would be preferred. Finally, if the
instance does not request a PCI device, then the last of these hosts
should be preferred.

While the first and second of these cases is already somewhat handled by
the NUMA topology filter (which will ensure the correct number and type
of PCI devices are available on the hosts), there is nothing to stop a
instance that *does not* require PCI devices from occupying space on one
of the few hosts that *does* have free PCI devices.

The PCIWeigher is designed to keep hosts with PCI device(s) as free as
possible, to best ensure they can satisfy requests from instances with
PCI requirements when called upon.  This change ensures (a) an instance
with PCI requirements has the best possible chance of scheduling
successfully and (b) an instance without these requirements won't clog
up hosts that have these valuable resources.

Change-Id: I04bf7e6b8324dcac6c93b0cb69c38c30fb05be56
Partially Implements: blueprint reserve-numa-with-pci

doc/source/filter_scheduler.rst

@@ -433,6 +433,24 @@ The Filter Scheduler weighs hosts based on the config option
   hosts. If the multiplier is positive, the weigher prefer choosing heavy
   workload compute hosts, the weighing has the opposite effect of the default.
 
+* |PCIWeigher| Compute a weighting based on the number of PCI devices on the
+  host and the number of PCI devices requested by the instance. For example,
+  given three hosts - one with a single PCI device, one with many PCI devices,
+  and one with no PCI devices - nova should prioritise these differently based
+  on the demands of the instance. If the instance requests a single PCI device,
+  then the first of the hosts should be preferred. Similarly, if the instance
+  requests multiple PCI devices, then the second of these hosts would be
+  preferred. Finally, if the instance does not request a PCI device, then the
+  last of these hosts should be preferred.
+
+  For this to be of any value, at least one of the |PciPassthroughFilter| or
+  |NUMATopologyFilter| filters must be enabled.
+
+  :Configuration Option: ``[filter_scheduler] pci_weight_multiplier``. Only
+    positive values are allowed for the multiplier as a negative value would
+    force non-PCI instances away from non-PCI hosts, thus, causing future
+    scheduling issues.
+
 * |ServerGroupSoftAffinityWeigher| The weigher can compute the weight based
   on the number of instances that run on the same server group. The largest
   weight defines the preferred host for the new instance. For the multiplier
@@ -496,6 +514,7 @@ in :mod:`nova.tests.scheduler`.
 .. |MetricsFilter| replace:: :class:`MetricsFilter <nova.scheduler.filters.metrics_filter.MetricsFilter>`
 .. |MetricsWeigher| replace:: :class:`MetricsWeigher <nova.scheduler.weights.metrics.MetricsWeigher>`
 .. |IoOpsWeigher| replace:: :class:`IoOpsWeigher <nova.scheduler.weights.io_ops.IoOpsWeigher>`
+.. |PCIWeigher| replace:: :class:`PCIWeigher <nova.scheduler.weights.pci.PCIWeigher>`
 .. |ServerGroupSoftAffinityWeigher| replace:: :class:`ServerGroupSoftAffinityWeigher <nova.scheduler.weights.affinity.ServerGroupSoftAffinityWeigher>`
 .. |ServerGroupSoftAntiAffinityWeigher| replace:: :class:`ServerGroupSoftAntiAffinityWeigher <nova.scheduler.weights.affinity.ServerGroupSoftAntiAffinityWeigher>`
 .. |DiskWeigher| replace:: :class:`DiskWeigher <nova.scheduler.weights.disk.DiskWeigher>`

nova/conf/scheduler.py

@@ -438,6 +438,25 @@ Possible values:
 * An integer or float value, where the value corresponds to the multipler
   ratio for this weigher.
 """),
+    cfg.FloatOpt("pci_weight_multiplier",
+        default=1.0,
+        min=0.0,
+        help="""
+PCI device affinity weight multiplier.
+
+The PCI device affinity weighter computes a weighting based on the number of
+PCI devices on the host and the number of PCI devices requested by the
+instance. The ``NUMATopologyFilter`` filter must be enabled for this to have
+any significance. For more information, refer to the filter documentation:
+
+    https://docs.openstack.org/developer/nova/filter_scheduler.html
+
+Possible values:
+
+* A positive integer or float value, where the value corresponds to the
+  multiplier ratio for this weigher.
+"""),
+    # TODO(sfinucan): Add 'min' parameter and remove warning in 'affinity.py'
     cfg.FloatOpt("soft_affinity_weight_multiplier",
         default=1.0,
         deprecated_group="DEFAULT",

nova/scheduler/weights/pci.py

@@ -0,0 +1,65 @@
+# Copyright (c) 2016, Red Hat Inc.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""
+PCI Affinity Weigher.  Weigh hosts by their PCI availability.
+
+Prefer hosts with PCI devices for instances with PCI requirements and vice
+versa. Configure the importance of this affinitization using the
+'pci_weight_multiplier' option.
+"""
+
+import nova.conf
+from nova.scheduler import weights
+
+CONF = nova.conf.CONF
+
+# An arbitrary value used to ensure PCI-requesting instances are stacked rather
+# than spread on hosts with PCI devices. The actual value of this filter is in
+# the scarcity case, where there are very few PCI devices left in the cloud and
+# we want to preserve the ones that do exist. To this end, we don't really mind
+# if a host with 2000 PCI devices is weighted the same as one with 500 devices,
+# as there's clearly no shortage there.
+MAX_DEVS = 100
+
+
+class PCIWeigher(weights.BaseHostWeigher):
+
+    def weight_multiplier(self):
+        """Override the weight multiplier."""
+        return CONF.filter_scheduler.pci_weight_multiplier
+
+    def _weigh_object(self, host_state, request_spec):
+        """Higher weights win. We want to keep PCI hosts free unless needed.
+
+        Prefer hosts with the least number of PCI devices. If the instance
+        requests PCI devices, this will ensure a stacking behavior and reserve
+        as many totally free PCI hosts as possible. If PCI devices are not
+        requested, this will ensure hosts with PCI devices are avoided
+        completely, if possible.
+        """
+        pools = host_state.pci_stats.pools if host_state.pci_stats else []
+        free = sum(pool['count'] for pool in pools) or 0
+
+        # reverse the "has PCI" values. For instances *without* PCI device
+        # requests, this ensures we avoid the hosts with the most free PCI
+        # devices. For the instances *with* PCI devices requests, this helps to
+        # prevent fragmentation. If we didn't do this, hosts with the most PCI
+        # devices would be weighted highest and would be used first which would
+        # prevent instances requesting a larger number of PCI devices from
+        # launching successfully.
+        weight = MAX_DEVS - min(free, MAX_DEVS - 1)
+
+        return weight

nova/tests/unit/scheduler/weights/test_weights_pci.py

@@ -0,0 +1,171 @@
+# Copyright (c) 2016, Red Hat Inc.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""Tests for Scheduler PCI weights."""
+
+import copy
+
+from nova import objects
+from nova.pci import stats
+from nova.scheduler import weights
+from nova.scheduler.weights import pci
+from nova import test
+from nova.tests.unit import fake_pci_device_pools as fake_pci
+from nova.tests.unit.scheduler import fakes
+
+
+class PCIWeigherTestCase(test.NoDBTestCase):
+    def setUp(self):
+        super(PCIWeigherTestCase, self).setUp()
+        self.weight_handler = weights.HostWeightHandler()
+        self.weighers = [pci.PCIWeigher()]
+
+    def _get_weighed_hosts(self, hosts, request_spec):
+        return self.weight_handler.get_weighed_objects(self.weighers,
+                hosts, request_spec)
+
+    def _get_all_hosts(self, host_values):
+
+        def _create_pci_pool(count):
+            test_dict = copy.copy(fake_pci.fake_pool_dict)
+            test_dict['count'] = count
+            return objects.PciDevicePool.from_dict(test_dict)
+
+        def _create_pci_stats(counts):
+            if counts is None:  # the pci_stats column is nullable
+                return None
+
+            pools = [_create_pci_pool(count) for count in counts]
+            return stats.PciDeviceStats(pools)
+
+        return [fakes.FakeHostState(
+            host, node, {'pci_stats': _create_pci_stats(values)})
+            for host, node, values in host_values]
+
+    def test_multiplier_no_pci_empty_hosts(self):
+        """Test weigher with a no PCI device instance on no PCI device hosts.
+
+        Ensure that the host with no PCI devices receives the highest
+        weighting.
+        """
+        hosts = [
+            ('host1', 'node1', [3, 1]),  # 4 devs
+            ('host2', 'node2', []),  # 0 devs
+        ]
+        hostinfo_list = self._get_all_hosts(hosts)
+
+        # we don't request PCI devices
+        spec_obj = objects.RequestSpec(pci_requests=None)
+
+        # host2, which has the least PCI devices, should win
+        weighed_host = self._get_weighed_hosts(hostinfo_list, spec_obj)[0]
+        self.assertEqual(1.0, weighed_host.weight)
+        self.assertEqual('host2', weighed_host.obj.host)
+
+    def test_multiplier_no_pci_non_empty_hosts(self):
+        """Test weigher with a no PCI device instance on PCI device hosts.
+
+        Ensure that the host with the least PCI devices receives the highest
+        weighting.
+        """
+        hosts = [
+            ('host1', 'node1', [2, 2, 2]),  # 6 devs
+            ('host2', 'node2', [3, 1]),  # 4 devs
+        ]
+        hostinfo_list = self._get_all_hosts(hosts)
+
+        # we don't request PCI devices
+        spec_obj = objects.RequestSpec(pci_requests=None)
+
+        # host2, which has the least free PCI devices, should win
+        weighed_host = self._get_weighed_hosts(hostinfo_list, spec_obj)[0]
+        self.assertEqual(1.0, weighed_host.weight)
+        self.assertEqual('host2', weighed_host.obj.host)
+
+    def test_multiplier_with_pci(self):
+        """Test weigher with a PCI device instance and a multiplier.
+
+        Ensure that the host with the smallest number of free PCI devices
+        capable of meeting the requirements of the instance is chosen,
+        enforcing a stacking (rather than spreading) behavior.
+        """
+        # none of the hosts will have less than the number of devices required
+        # by the instance: the NUMATopologyFilter takes care of this for us
+        hosts = [
+            ('host1', 'node1', [4, 1]),  # 5 devs
+            ('host2', 'node2', [10]),  # 10 devs
+            ('host3', 'node3', [1, 1, 1, 1]),  # 4 devs
+        ]
+        hostinfo_list = self._get_all_hosts(hosts)
+
+        # we request PCI devices
+        request = objects.InstancePCIRequest(count=4,
+            spec=[{'vendor_id': '8086'}])
+        requests = objects.InstancePCIRequests(requests=[request])
+        spec_obj = objects.RequestSpec(pci_requests=requests)
+
+        # host3, which has the least free PCI devices, should win
+        weighed_host = self._get_weighed_hosts(hostinfo_list, spec_obj)[0]
+        self.assertEqual(1.0, weighed_host.weight)
+        self.assertEqual('host3', weighed_host.obj.host)
+
+    def test_multiplier_with_many_pci(self):
+        """Test weigher with a PCI device instance and huge hosts.
+
+        Ensure that the weigher gracefully degrades when the number of PCI
+        devices on the host exceeeds MAX_DEVS.
+        """
+        hosts = [
+            ('host1', 'node1', [500]),  # 500 devs
+            ('host2', 'node2', [2000]),  # 2000 devs
+        ]
+        hostinfo_list = self._get_all_hosts(hosts)
+
+        # we request PCI devices
+        request = objects.InstancePCIRequest(count=4,
+            spec=[{'vendor_id': '8086'}])
+        requests = objects.InstancePCIRequests(requests=[request])
+        spec_obj = objects.RequestSpec(pci_requests=requests)
+
+        # we do not know the host as all have same weight
+        weighed_hosts = self._get_weighed_hosts(hostinfo_list, spec_obj)
+        for weighed_host in weighed_hosts:
+            # the weigher normalizes all weights to 0 if they're all equal
+            self.assertEqual(0.0, weighed_host.weight)
+
+    def test_multiplier_none(self):
+        """Test weigher with a PCI device instance and a 0.0 multiplier.
+
+        Ensure that the 0.0 multiplier disables the weigher entirely.
+        """
+        self.flags(pci_weight_multiplier=0.0, group='filter_scheduler')
+
+        hosts = [
+            ('host1', 'node1', [4, 1]),  # 5 devs
+            ('host2', 'node2', [10]),  # 10 devs
+            ('host3', 'node3', [1, 1, 1, 1]),  # 4 devs
+        ]
+        hostinfo_list = self._get_all_hosts(hosts)
+
+        request = objects.InstancePCIRequest(count=1,
+            spec=[{'vendor_id': '8086'}])
+        requests = objects.InstancePCIRequests(requests=[request])
+        spec_obj = objects.RequestSpec(pci_requests=requests)
+
+        # we do not know the host as all have same weight
+        weighed_hosts = self._get_weighed_hosts(hostinfo_list, spec_obj)
+        for weighed_host in weighed_hosts:
+            # the weigher normalizes all weights to 0 if they're all equal
+            self.assertEqual(0.0, weighed_host.weight)

releasenotes/notes/add-pci-weigher-4a7e0a7b8e908975.yaml

@@ -0,0 +1,7 @@
+---
+features:
+  - |
+    Add ``PCIWeigher`` weigher. This can be used to ensure non-PCI instances
+    don't occupy resources on hosts with PCI devices. This can be configured
+    using the ``[filter_scheduler] pci_weight_multiplier`` configuration
+    option.