Normalize the weights instead of using raw values
The weight system is being used by the scheduler and the cells code.
Currently this system is using the raw values instead of normalizing them.
This makes difficult to properly use multipliers for establishing the
relative importance between two wheighers (one big magnitude could
shade a smaller one). This change introduces weight normalization so
that:
- From an operator point of view we can prioritize the weighers that
we are applying. The only way to do this is being sure that all the
weighers will give a value in a known range, so that it is
not needed to artificially use a huge multiplier to prioritize a
weigher.
- From a weigher developer point of view, somebody willing to implement
one has to care about 1) returning a list of values, 2) setting the
minimum and maximum values where the weights can range, if they are
needed and they are significant for the weighing. For a weigher
developer there are two use cases:
Case 1: Use of a percentage instead of absolute values (for example, %
of free RAM). If we compare two nodes focusing on the percentage of free
ram, the maximum value for the weigher is 100. If we have two nodes one
with 2048 total/1024 free, and the second one 1024 total/512 free they
will get both the same weight, since they have the same % of free RAM
(that is, the 50%).
Case 2: Use of absolute values. In this case, the maximum of the weigher
will be the maximum of the values in the list (in the case above, 1024)
or the maximum value that the magnitude could take (in the case above,
2048). How this maximum is set, is a decision of the developer. He may
let the operator choose the behaviour of the weigher though.
- From the point of view of the scheduler we ensure that it is using
normalized values, and not leveraging the normalization mechanism to the
weighers.
Changes introduced this commit:
1) it introduces weight normalization so that we can apply multipliers
easily. All the weights for an object will be normalized between 0.0 and
1.0 before being sumed up, so that the final weight for a host will be:
weight = w1_multiplier * norm(w1) + w2_multiplier * norm(w2) + ...
2) weights.BaseWeigher has been changed into an ABC so that we enforce
that all weighers have the expected methods.
3) weights.BaseWeigher.weigh_objects() does no longer sum up the
computer weighs to the object, but it rather returns a list that will be
then normalized and added to the existing weight by BaseWeightHandler
4) Adapt the existing weighers to the above changes. Namely
- New 'offset_weight_multiplier' for the cell weigher
nova.cells.weights.weight_offset.WeightOffsetWeigher
- Changed the name of the existing multiplier methods.
5) unittests for all of the introduced changes.
Implements blueprint normalize-scheduler-weights
DocImpact: Now weights for an object are normalized before suming them
up. This means that each weigher will take a maximum value of 1. This
may have an impact for operators that are using more than one weigher
(currently there is only one weigher: RAMWeiger) and for operators using
cells (where we have several weighers). It is needed to review then the
multipliers used and adjust them properly in case they have been
modified.
Docimpact: There is a new configuration option 'offset_weight_multiplier'
in nova.cells.weights.weight_offset.WeightOffsetWeigher
Change-Id: I81bf90898d3cb81541f4390596823cc00106eb20
This commit is contained in:
Alvaro Lopez Garcia
parent
65c7027b5c
commit
e5ba849437
|
|
@ -263,27 +263,35 @@ default when no filters are specified in the request.
|
|||
Weights
|
||||
-------
|
||||
|
||||
Filter Scheduler uses so-called **weights** during its work.
|
||||
Filter Scheduler uses the so called **weights** during its work. A weigher is a
|
||||
way to select the best suitable host from a group of valid hosts by giving
|
||||
weights to all the hosts in the list.
|
||||
|
||||
The Filter Scheduler weights hosts based on the config option
|
||||
In order to prioritize one weigher against another, all the weighers have to
|
||||
define a multiplier that will be applied before computing the weight for a node.
|
||||
All the weights are normalized beforehand so that the multiplier can be applied
|
||||
easily. Therefore the final weight for the object will be::
|
||||
|
||||
weight = w1_multiplier * norm(w1) + w2_multiplier * norm(w2) + ...
|
||||
|
||||
A weigher should be a subclass of ``weights.BaseHostWeigher`` and they must
|
||||
implement the ``weight_multiplier`` and ``weight_object`` methods. If the
|
||||
``weight_objects`` method is overriden it just return a list of weights, and not
|
||||
modify the weight of the object directly, since final weights are normalized and
|
||||
computed by ``weight.BaseWeightHandler``.
|
||||
|
||||
The Filter Scheduler weighs hosts based on the config option
|
||||
`scheduler_weight_classes`, this defaults to
|
||||
`nova.scheduler.weights.all_weighers`, which selects all the available weighers
|
||||
in the package nova.scheduler.weights. Hosts are then weighted and sorted with
|
||||
the largest weight winning. For each host, the final weight is calculated by
|
||||
summing up all weigher's weight value multiplying its own weight_mutiplier:
|
||||
`nova.scheduler.weights.all_weighers`, which selects the following weighers:
|
||||
|
||||
::
|
||||
* |RamWeigher| Hosts are then weighted and sorted with the largest weight winning.
|
||||
If the multiplier is negative, the host with less RAM available will win (useful
|
||||
for stacking hosts, instead of spreading).
|
||||
* |MetricsWeigher| This weigher can compute the weight based on the compute node
|
||||
host's various metrics. The to-be weighed metrics and their weighing ratio
|
||||
are specified in the configuration file as the followings::
|
||||
|
||||
final_weight = 0
|
||||
for each weigher:
|
||||
final_weight += weigher's weight_mutiplier * weigher's calculated weight value
|
||||
|
||||
The weigher's weight_mutiplier can be set in the configuration file, e.g.
|
||||
|
||||
::
|
||||
|
||||
[metrics]
|
||||
weight_multiplier=1.0
|
||||
metrics_weight_setting = name1=1.0, name2=-1.0
|
||||
|
||||
Filter Scheduler finds local list of acceptable hosts by repeated filtering and
|
||||
weighing. Each time it chooses a host, it virtually consumes resources on it,
|
||||
|
|
@ -322,3 +330,5 @@ in :mod:``nova.tests.scheduler``.
|
|||
.. |AggregateTypeAffinityFilter| replace:: :class:`AggregateTypeAffinityFilter <nova.scheduler.filters.type_filter.AggregateTypeAffinityFilter>`
|
||||
.. |AggregateInstanceExtraSpecsFilter| replace:: :class:`AggregateInstanceExtraSpecsFilter <nova.scheduler.filters.aggregate_instance_extra_specs.AggregateInstanceExtraSpecsFilter>`
|
||||
.. |AggregateMultiTenancyIsolation| replace:: :class:`AggregateMultiTenancyIsolation <nova.scheduler.filters.aggregate_multitenancy_isolation.AggregateMultiTenancyIsolation>`
|
||||
|
||||
.. |RamWeigher| replace:: :class:`RamWeigher <nova.scheduler.weights.all_weighers.RamWeigher>`
|
||||
|
|
|
|||
|
|
@ -3058,6 +3058,15 @@
|
|||
#ram_weight_multiplier=10.0
|
||||
|
||||
|
||||
#
|
||||
# Options defined in nova.cells.weights.weight_offset
|
||||
#
|
||||
|
||||
# Multiplier used to weigh offset weigher. (floating point
|
||||
# value)
|
||||
#offset_weight_multiplier=1.0
|
||||
|
||||
|
||||
[baremetal]
|
||||
|
||||
#
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@ class MuteChildWeigher(weights.BaseCellWeigher):
|
|||
weight.
|
||||
"""
|
||||
|
||||
def _weight_multiplier(self):
|
||||
def weight_multiplier(self):
|
||||
# negative multiplier => lower weight
|
||||
return CONF.cells.mute_weight_multiplier
|
||||
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ CONF.register_opts(ram_weigher_opts, group='cells')
|
|||
class RamByInstanceTypeWeigher(weights.BaseCellWeigher):
|
||||
"""Weigh cells by instance_type requested."""
|
||||
|
||||
def _weight_multiplier(self):
|
||||
def weight_multiplier(self):
|
||||
return CONF.cells.ram_weight_multiplier
|
||||
|
||||
def _weigh_object(self, cell, weight_properties):
|
||||
|
|
|
|||
|
|
@ -18,8 +18,19 @@ Weigh cells by their weight_offset in the DB. Cells with higher
|
|||
weight_offsets in the DB will be preferred.
|
||||
"""
|
||||
|
||||
from oslo.config import cfg
|
||||
|
||||
from nova.cells import weights
|
||||
|
||||
weigher_opts = [
|
||||
cfg.FloatOpt('offset_weight_multiplier',
|
||||
default=1.0,
|
||||
help='Multiplier used to weigh offset weigher.'),
|
||||
]
|
||||
|
||||
CONF = cfg.CONF
|
||||
CONF.register_opts(weigher_opts, group='cells')
|
||||
|
||||
|
||||
class WeightOffsetWeigher(weights.BaseCellWeigher):
|
||||
"""
|
||||
|
|
@ -28,6 +39,9 @@ class WeightOffsetWeigher(weights.BaseCellWeigher):
|
|||
its weight_offset to 999999999999999 (highest weight wins)
|
||||
"""
|
||||
|
||||
def weight_multiplier(self):
|
||||
return CONF.cells.offset_weight_multiplier
|
||||
|
||||
def _weigh_object(self, cell, weight_properties):
|
||||
"""Returns whatever was in the DB for weight_offset."""
|
||||
return cell.db_info.get('weight_offset', 0)
|
||||
|
|
|
|||
|
|
@ -76,7 +76,7 @@ class MetricsWeigher(weights.BaseHostWeigher):
|
|||
" metrics_weight_setting: %s"),
|
||||
",".join(bad))
|
||||
|
||||
def _weight_multiplier(self):
|
||||
def weight_multiplier(self):
|
||||
"""Override the weight multiplier."""
|
||||
return CONF.metrics.weight_multiplier
|
||||
|
||||
|
|
|
|||
|
|
@ -36,7 +36,9 @@ CONF.register_opts(ram_weight_opts)
|
|||
|
||||
|
||||
class RAMWeigher(weights.BaseHostWeigher):
|
||||
def _weight_multiplier(self):
|
||||
minval = 0
|
||||
|
||||
def weight_multiplier(self):
|
||||
"""Override the weight multiplier."""
|
||||
return CONF.ram_weight_multiplier
|
||||
|
||||
|
|
|
|||
|
|
@ -213,5 +213,5 @@ class MuteWeigherTestClass(_WeigherTestClass):
|
|||
|
||||
for i in range(2):
|
||||
weighed_cell = weighed_cells.pop(0)
|
||||
self.assertEqual(1000 * -10.0, weighed_cell.weight)
|
||||
self.assertEqual(-10.0, weighed_cell.weight)
|
||||
self.assertIn(weighed_cell.obj.name, ['cell1', 'cell2'])
|
||||
|
|
|
|||
|
|
@ -72,37 +72,62 @@ class RamWeigherTestCase(test.NoDBTestCase):
|
|||
|
||||
# so, host4 should win:
|
||||
weighed_host = self._get_weighed_host(hostinfo_list)
|
||||
self.assertEqual(weighed_host.weight, 8192)
|
||||
self.assertEqual(weighed_host.weight, 1.0)
|
||||
self.assertEqual(weighed_host.obj.host, 'host4')
|
||||
|
||||
def test_ram_filter_multiplier1(self):
|
||||
self.flags(ram_weight_multiplier=-1.0)
|
||||
self.flags(ram_weight_multiplier=0.0)
|
||||
hostinfo_list = self._get_all_hosts()
|
||||
|
||||
# host1: free_ram_mb=-512
|
||||
# host2: free_ram_mb=-1024
|
||||
# host3: free_ram_mb=-3072
|
||||
# host4: free_ram_mb=-8192
|
||||
# host1: free_ram_mb=512
|
||||
# host2: free_ram_mb=1024
|
||||
# host3: free_ram_mb=3072
|
||||
# host4: free_ram_mb=8192
|
||||
|
||||
# so, host1 should win:
|
||||
# We do not know the host, all have same weight.
|
||||
weighed_host = self._get_weighed_host(hostinfo_list)
|
||||
self.assertEqual(weighed_host.weight, -512)
|
||||
self.assertEqual(weighed_host.obj.host, 'host1')
|
||||
self.assertEqual(weighed_host.weight, 0.0)
|
||||
|
||||
def test_ram_filter_multiplier2(self):
|
||||
self.flags(ram_weight_multiplier=2.0)
|
||||
hostinfo_list = self._get_all_hosts()
|
||||
|
||||
# host1: free_ram_mb=512 * 2
|
||||
# host2: free_ram_mb=1024 * 2
|
||||
# host3: free_ram_mb=3072 * 2
|
||||
# host4: free_ram_mb=8192 * 2
|
||||
# host1: free_ram_mb=512
|
||||
# host2: free_ram_mb=1024
|
||||
# host3: free_ram_mb=3072
|
||||
# host4: free_ram_mb=8192
|
||||
|
||||
# so, host4 should win:
|
||||
weighed_host = self._get_weighed_host(hostinfo_list)
|
||||
self.assertEqual(weighed_host.weight, 8192 * 2)
|
||||
self.assertEqual(weighed_host.weight, 1.0 * 2)
|
||||
self.assertEqual(weighed_host.obj.host, 'host4')
|
||||
|
||||
def test_ram_filter_negative(self):
|
||||
self.flags(ram_weight_multiplier=1.0)
|
||||
hostinfo_list = self._get_all_hosts()
|
||||
host_attr = {'id': 100, 'memory_mb': 8192, 'free_ram_mb': -512}
|
||||
host_state = fakes.FakeHostState('negative', 'negative', host_attr)
|
||||
hostinfo_list = list(hostinfo_list) + [host_state]
|
||||
|
||||
# host1: free_ram_mb=512
|
||||
# host2: free_ram_mb=1024
|
||||
# host3: free_ram_mb=3072
|
||||
# host4: free_ram_mb=8192
|
||||
# negativehost: free_ram_mb=-512
|
||||
|
||||
# so, host4 should win
|
||||
weights = self.weight_handler.get_weighed_objects(self.weight_classes,
|
||||
hostinfo_list, {})
|
||||
|
||||
weighed_host = weights[0]
|
||||
self.assertEqual(weighed_host.weight, 1)
|
||||
self.assertEqual(weighed_host.obj.host, "host4")
|
||||
|
||||
# and negativehost should lose
|
||||
weighed_host = weights[-1]
|
||||
self.assertEqual(weighed_host.weight, 0)
|
||||
self.assertEqual(weighed_host.obj.host, "negative")
|
||||
|
||||
|
||||
class MetricsWeigherTestCase(test.NoDBTestCase):
|
||||
def setUp(self):
|
||||
|
|
@ -139,7 +164,7 @@ class MetricsWeigherTestCase(test.NoDBTestCase):
|
|||
# host4: foo=8192
|
||||
# so, host4 should win:
|
||||
setting = ['foo=1']
|
||||
self._do_test(setting, 8192, 'host4')
|
||||
self._do_test(setting, 1.0, 'host4')
|
||||
|
||||
def test_multiple_resource(self):
|
||||
# host1: foo=512, bar=1
|
||||
|
|
@ -148,7 +173,7 @@ class MetricsWeigherTestCase(test.NoDBTestCase):
|
|||
# host4: foo=8192, bar=0
|
||||
# so, host2 should win:
|
||||
setting = ['foo=0.0001', 'bar=1']
|
||||
self._do_test(setting, 2.1024, 'host2')
|
||||
self._do_test(setting, 1.0, 'host2')
|
||||
|
||||
def test_single_resourcenegtive_ratio(self):
|
||||
# host1: foo=512
|
||||
|
|
@ -157,7 +182,7 @@ class MetricsWeigherTestCase(test.NoDBTestCase):
|
|||
# host4: foo=8192
|
||||
# so, host1 should win:
|
||||
setting = ['foo=-1']
|
||||
self._do_test(setting, -512, 'host1')
|
||||
self._do_test(setting, 1.0, 'host1')
|
||||
|
||||
def test_multiple_resource_missing_ratio(self):
|
||||
# host1: foo=512, bar=1
|
||||
|
|
@ -166,7 +191,7 @@ class MetricsWeigherTestCase(test.NoDBTestCase):
|
|||
# host4: foo=8192, bar=0
|
||||
# so, host4 should win:
|
||||
setting = ['foo=0.0001', 'bar']
|
||||
self._do_test(setting, 0.8192, 'host4')
|
||||
self._do_test(setting, 1.0, 'host4')
|
||||
|
||||
def test_multiple_resource_wrong_ratio(self):
|
||||
# host1: foo=512, bar=1
|
||||
|
|
@ -175,7 +200,7 @@ class MetricsWeigherTestCase(test.NoDBTestCase):
|
|||
# host4: foo=8192, bar=0
|
||||
# so, host4 should win:
|
||||
setting = ['foo=0.0001', 'bar = 2.0t']
|
||||
self._do_test(setting, 0.8192, 'host4')
|
||||
self._do_test(setting, 1.0, 'host4')
|
||||
|
||||
def _check_parsing_result(self, weigher, setting, results):
|
||||
self.flags(weight_setting=setting, group='metrics')
|
||||
|
|
|
|||
|
|
@ -0,0 +1,54 @@
|
|||
# Copyright 2011-2012 OpenStack Foundation
|
||||
# All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
"""
|
||||
Tests For weights.
|
||||
"""
|
||||
|
||||
from nova import test
|
||||
from nova import weights
|
||||
|
||||
|
||||
class TestWeigher(test.NoDBTestCase):
|
||||
def test_no_multiplier(self):
|
||||
class FakeWeigher(weights.BaseWeigher):
|
||||
def _weigh_object(self, *args, **kwargs):
|
||||
pass
|
||||
|
||||
self.assertEqual(1.0,
|
||||
FakeWeigher().weight_multiplier())
|
||||
|
||||
def test_no_weight_object(self):
|
||||
class FakeWeigher(weights.BaseWeigher):
|
||||
def weight_multiplier(self, *args, **kwargs):
|
||||
pass
|
||||
self.assertRaises(TypeError,
|
||||
FakeWeigher)
|
||||
|
||||
def test_normalization(self):
|
||||
# weight_list, expected_result, minval, maxval
|
||||
map_ = (
|
||||
((), (), None, None),
|
||||
((0.0, 0.0), (0.0, 0.0), None, None),
|
||||
((1.0, 1.0), (0.0, 0.0), None, None),
|
||||
|
||||
((20.0, 50.0), (0.0, 1.0), None, None),
|
||||
((20.0, 50.0), (0.0, 0.375), None, 100.0),
|
||||
((20.0, 50.0), (0.4, 1.0), 0.0, None),
|
||||
((20.0, 50.0), (0.2, 0.5), 0.0, 100.0),
|
||||
)
|
||||
normalize_to = (1.0, 10.0)
|
||||
for seq, result, minval, maxval in map_:
|
||||
ret = weights.normalize(seq, minval=minval, maxval=maxval)
|
||||
self.assertEqual(tuple(ret), result)
|
||||
101
nova/weights.py
101
nova/weights.py
|
|
@ -17,9 +17,40 @@
|
|||
Pluggable Weighing support
|
||||
"""
|
||||
|
||||
import abc
|
||||
|
||||
from nova import loadables
|
||||
|
||||
|
||||
def normalize(weight_list, minval=None, maxval=None):
|
||||
"""Normalize the values in a list between 0 and 1.0.
|
||||
|
||||
The normalization is made regarding the lower and upper values present in
|
||||
weight_list. If the minval and/or maxval parameters are set, these values
|
||||
will be used instead of the minimum and maximum from the list.
|
||||
|
||||
If all the values are equal, they are normalized to 0.
|
||||
"""
|
||||
|
||||
if not weight_list:
|
||||
return ()
|
||||
|
||||
if maxval is None:
|
||||
maxval = max(weight_list)
|
||||
|
||||
if minval is None:
|
||||
minval = min(weight_list)
|
||||
|
||||
maxval = float(maxval)
|
||||
minval = float(minval)
|
||||
|
||||
if minval == maxval:
|
||||
return [0] * len(weight_list)
|
||||
|
||||
range_ = maxval - minval
|
||||
return ((i - minval) / range_ for i in weight_list)
|
||||
|
||||
|
||||
class WeighedObject(object):
|
||||
"""Object with weight information."""
|
||||
def __init__(self, obj, weight):
|
||||
|
|
@ -31,26 +62,59 @@ class WeighedObject(object):
|
|||
|
||||
|
||||
class BaseWeigher(object):
|
||||
"""Base class for pluggable weighers."""
|
||||
def _weight_multiplier(self):
|
||||
"""How weighted this weigher should be. Normally this would
|
||||
be overridden in a subclass based on a config value.
|
||||
"""Base class for pluggable weighers.
|
||||
|
||||
The attributes maxval and minval can be specified to set up the maximum
|
||||
and minimum values for the weighed objects. These values will then be
|
||||
taken into account in the normalization step, instead of taking the values
|
||||
from the calculated weights.
|
||||
"""
|
||||
|
||||
__metaclass__ = abc.ABCMeta
|
||||
|
||||
minval = None
|
||||
maxval = None
|
||||
|
||||
def weight_multiplier(self):
|
||||
"""How weighted this weigher should be.
|
||||
|
||||
Override this method in a subclass, so that the returned value is
|
||||
read from a configuration option to permit operators specify a
|
||||
multiplier for the weigher.
|
||||
"""
|
||||
return 1.0
|
||||
|
||||
@abc.abstractmethod
|
||||
def _weigh_object(self, obj, weight_properties):
|
||||
"""Override in a subclass to specify a weight for a specific
|
||||
object.
|
||||
"""
|
||||
return 0.0
|
||||
"""Weigh an specific object."""
|
||||
|
||||
def weigh_objects(self, weighed_obj_list, weight_properties):
|
||||
"""Weigh multiple objects. Override in a subclass if you need
|
||||
need access to all objects in order to manipulate weights.
|
||||
"""Weigh multiple objects.
|
||||
|
||||
Override in a subclass if you need access to all objects in order
|
||||
to calculate weights. Do not modify the weight of an object here,
|
||||
just return a list of weights.
|
||||
"""
|
||||
# Calculate the weights
|
||||
weights = []
|
||||
for obj in weighed_obj_list:
|
||||
obj.weight += (self._weight_multiplier() *
|
||||
self._weigh_object(obj.obj, weight_properties))
|
||||
weight = self._weigh_object(obj.obj, weight_properties)
|
||||
|
||||
# Record the min and max values if they are None. If they anything
|
||||
# but none we assume that the weigher has set them
|
||||
if self.minval is None:
|
||||
self.minval = weight
|
||||
if self.maxval is None:
|
||||
self.maxval = weight
|
||||
|
||||
if weight < self.minval:
|
||||
self.minval = weight
|
||||
elif weight > self.maxval:
|
||||
self.maxval = weight
|
||||
|
||||
weights.append(weight)
|
||||
|
||||
return weights
|
||||
|
||||
|
||||
class BaseWeightHandler(loadables.BaseLoader):
|
||||
|
|
@ -58,7 +122,7 @@ class BaseWeightHandler(loadables.BaseLoader):
|
|||
|
||||
def get_weighed_objects(self, weigher_classes, obj_list,
|
||||
weighing_properties):
|
||||
"""Return a sorted (highest score first) list of WeighedObjects."""
|
||||
"""Return a sorted (descending), normalized list of WeighedObjects."""
|
||||
|
||||
if not obj_list:
|
||||
return []
|
||||
|
|
@ -66,6 +130,15 @@ class BaseWeightHandler(loadables.BaseLoader):
|
|||
weighed_objs = [self.object_class(obj, 0.0) for obj in obj_list]
|
||||
for weigher_cls in weigher_classes:
|
||||
weigher = weigher_cls()
|
||||
weigher.weigh_objects(weighed_objs, weighing_properties)
|
||||
weights = weigher.weigh_objects(weighed_objs, weighing_properties)
|
||||
|
||||
# Normalize the weights
|
||||
weights = normalize(weights,
|
||||
minval=weigher.minval,
|
||||
maxval=weigher.maxval)
|
||||
|
||||
for i, weight in enumerate(weights):
|
||||
obj = weighed_objs[i]
|
||||
obj.weight += weigher.weight_multiplier() * weight
|
||||
|
||||
return sorted(weighed_objs, key=lambda x: x.weight, reverse=True)
|
||||
|
|
|
|||
Loading…
Reference in New Issue