Support filtering of allocation_candidates by forbidden aggregates
This blueprint proposes to specify forbidden aggregates in the `member_of` query parameter of the `GET /allocation_candidates` placement API. Depends-On: https://review.openstack.org/#/c/603352 Change-Id: I1a94f50d42e57ff9d746e2f160a89dfab1978894 Implements: blueprint placement-req-filter-forbidden-aggregates
This commit is contained in:
parent
a2a3d72034
commit
2e5a8e0f3a
|
@ -0,0 +1,269 @@
|
||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
===================================================================
|
||||||
|
Support filtering of allocation_candidates by forbidden aggregates
|
||||||
|
===================================================================
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/nova/+spec/placement-req-filter-forbidden-aggregates
|
||||||
|
|
||||||
|
This blueprint proposes to specify forbidden aggregates in the `member_of`
|
||||||
|
query parameter of the `GET /allocation_candidates` placement API during
|
||||||
|
scheduling.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
If flavor or image doesn't contain any ``required`` or ``forbidden``
|
||||||
|
traits, then all resource providers will be eligible to be returned in the
|
||||||
|
``GET /allocation_candidates`` API call depending on the availability of the
|
||||||
|
requested resources. Some of the resource providers (compute host) could be
|
||||||
|
special ones like `Licensed Windows Compute Host`, meaning any VM booted on
|
||||||
|
this compute host will be considered as licensed Windows image and depending
|
||||||
|
on the usage of VM operators will charge it to their end-users. As an operator,
|
||||||
|
I want to avoid booting non-windows OS images/volumes on aggregates which
|
||||||
|
contains `Licensed Windows Compute Hosts`. The existing
|
||||||
|
``AggregateImagePropertiesIsolationFilter`` scheduler filter does restricts
|
||||||
|
windows license images to windows license host aggregates but the problem is it
|
||||||
|
doesn't exclude other images without matching metadata.
|
||||||
|
|
||||||
|
Consider following example to depict the licensing use case.
|
||||||
|
Operator adds image metadata to classify images as below:
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
image 1 properties: {"os_distro": "windows"} (added by an operator)
|
||||||
|
image 2 properties: {} (added by a normal end user)
|
||||||
|
|
||||||
|
Host aggregate 1 metadata: {"os_distro": "windows"}
|
||||||
|
|
||||||
|
Now when user boots an instance using image 2, then this scheduler filter
|
||||||
|
allows to boot instance in host aggregate 1 which is a problem.
|
||||||
|
|
||||||
|
Use Cases
|
||||||
|
---------
|
||||||
|
|
||||||
|
Some of the compute hosts are `Licensed Windows Compute Host`, meaning any VMs
|
||||||
|
booted on such compute host will be considered as licensed Windows image and
|
||||||
|
depending on the usage of VM, operator will charge it to the end-users.
|
||||||
|
As an operator, I want to avoid booting non Windows OS images/volumes on
|
||||||
|
the `Licensed Windows Compute Hosts` thereby enabling operators to
|
||||||
|
|
||||||
|
* Avoid wasting licensing resources.
|
||||||
|
|
||||||
|
* Charge users correctly for their VM usage.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
Add a new placement request filter ``forbidden_aggregates`` and a new config
|
||||||
|
option of type boolean ``enable_forbidden_aggregates_filter``. Operator will
|
||||||
|
set `True` to enable the request filter. By default the value will be set to
|
||||||
|
`False`. Operator will need to set aggregate metadata key/value pairs
|
||||||
|
`trait:<trait_name>=required` with traits which they expect to match with the
|
||||||
|
`trait:<trait_name>=required` set in the flavor and images of the create
|
||||||
|
server request from request_spec object. In the new request filter, it will
|
||||||
|
get the required traits set in both flavor and images from request_spec object
|
||||||
|
and compare it with the required traits set in the aggregate metadata.
|
||||||
|
If any of the traits are not matching with the aggregate metadata, it will
|
||||||
|
include that aggregate as forbidden aggregate in the ``member_of`` query
|
||||||
|
parameter of ``GET /allocation_candidates`` API. If there are multiple
|
||||||
|
forbidden aggregates, then the query parameter should be like:
|
||||||
|
|
||||||
|
``&member_of=!in:<agg1>,<agg2>,<agg3>``
|
||||||
|
|
||||||
|
Example, how to set multiple traits to the metadata of an aggregate,
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
openstack aggregate set --property trait:CUSTOM_WINDOWS_LICENSED=required 123
|
||||||
|
openstack aggregate set --property trait:CUSTOM_XYZ=required 123
|
||||||
|
|
||||||
|
Operator will need to set ``trait:<trait_name>=required`` to images for
|
||||||
|
windows OS images.
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
openstack image set --property trait:CUSTOM_WINDOWS_LICENSED=required <image_uuid>
|
||||||
|
|
||||||
|
Example, how to enable ``forbidden_aggregates`` placement request filter:
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
[scheduler]
|
||||||
|
enable_forbidden_aggregates_filter = True
|
||||||
|
|
||||||
|
This ``forbidden_aggregates`` placement request filter supersedes
|
||||||
|
existing ``IsolatedHostsFilter`` except it:-
|
||||||
|
|
||||||
|
* Relies on aggregates rather than individual hosts (which won't scale in
|
||||||
|
large environments like a public cloud).
|
||||||
|
|
||||||
|
* Relies on image properties rather than specific image IDs, which again
|
||||||
|
won't scale.
|
||||||
|
|
||||||
|
With this placement request filter in place, there is a possibility we can
|
||||||
|
deprecate ``IsolatedHostsFilter`` scheduler filter for reasons as stated above.
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
Option 1: `Strict-isolation-group-hosts-images`_ spec
|
||||||
|
|
||||||
|
The main issues with this spec are:
|
||||||
|
|
||||||
|
* Adding a new scheduler filter which yet again depends on metadata key for
|
||||||
|
host aggregates.
|
||||||
|
|
||||||
|
* A compute node associated with multiple host aggregates. This is a
|
||||||
|
fundamental problem with nova host aggregates that doesn't exist in placement
|
||||||
|
aggregates.
|
||||||
|
|
||||||
|
Option 2: `Bi-directional-enforcement-of-traits`_ spec
|
||||||
|
|
||||||
|
The main issue with this spec is:
|
||||||
|
|
||||||
|
It's not placement's job to make operators have an easy life. Operators
|
||||||
|
should be required to set up their deployment's providers with an appropriate
|
||||||
|
set of traits, put providers into appropriate aggregates, put appropriate
|
||||||
|
metadata on their own images and flavors, and configure *Nova* with the set
|
||||||
|
of configuration options that would allow these things to be used properly.
|
||||||
|
|
||||||
|
Option 3: Use `IsolatedHostsFilter`_ scheduler filter
|
||||||
|
|
||||||
|
It doesn't really scale in a large public cloud with thousands of hosts and
|
||||||
|
images. Also, if you add new hosts in the system, you will need to modify the
|
||||||
|
config option ``isolated_hosts`` from ``filter_scheduler`` section and restart
|
||||||
|
nova scheduler services.
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Notifications impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
DB call to fetch aggregates with value `required` in the
|
||||||
|
``forbidden_aggregates`` placement request filter will marginally impact the
|
||||||
|
overall processing time of each `select_destination` request.
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
A new config boolean option ``enable_forbidden_aggregates_filter`` will be
|
||||||
|
added in nova.conf which will be used by nova-scheduler service.
|
||||||
|
The default value of this config option will be set to false.
|
||||||
|
|
||||||
|
.. code::
|
||||||
|
|
||||||
|
enable_forbidden_aggregates_filter=False
|
||||||
|
|
||||||
|
To enable `forbidden_aggregates` request filter, operator should set this
|
||||||
|
config option to true.
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
Upgrade impact
|
||||||
|
--------------
|
||||||
|
|
||||||
|
Starting from Rocky release, nova host aggregates are mirrored in placement
|
||||||
|
service (Implemented in `mirror_nova_host_aggregates`_). But if there is any
|
||||||
|
problem in mirroring, operator can sync it manually with ``nova-manage``
|
||||||
|
command:
|
||||||
|
|
||||||
|
``nova-manage placement sync_aggregates``
|
||||||
|
|
||||||
|
We will modify this command to synchronize traits to placement and add these
|
||||||
|
traits to the compute node resource providers that belongs to the aggregates
|
||||||
|
if aggregate has metadata key=value pair with syntax
|
||||||
|
`trait:<trait_name>=required`.
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
shilpa.devharakar <shilpa.devharakar@nttdata.com>
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
* Add a placement request filter ``forbidden_aggregates``.
|
||||||
|
* Modify ``resources_from_request_spec`` method to add forbidden aggregates to
|
||||||
|
the Destination object.
|
||||||
|
* Modify ``RequestGroup`` class `to_querystring` method to generate a
|
||||||
|
`member_of` query parameter to pass forbidden aggregates in format
|
||||||
|
``&member_of=!in:<agg1_uuid>,<agg2_uuid>,<agg3_uuid>``.
|
||||||
|
* Add unit and functional tests for the changes.
|
||||||
|
* Add releasenotes.
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
This spec is dependent on `negative-aggregate-membership`_ which supports
|
||||||
|
passing forbidden aggregates in the `member_of` query parameter.
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
Add normal functional and unit testing.
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
Add documentation to explain how to use ``forbidden_aggregates`` placement
|
||||||
|
request filter.
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
.. _negative-aggregate-membership: https://review.openstack.org/#/c/603352/4/specs/stein/approved/negative-aggregate-membership.rst
|
||||||
|
.. Launchpad Bug: https://bugs.launchpad.net/nova/+bug/1677217
|
||||||
|
.. _Bi-directional-enforcement-of-traits: https://review.openstack.org/#/c/593475/2/specs/stein/approved/bi-directional-traits.rst
|
||||||
|
.. _Strict-isolation-group-hosts-images: https://review.openstack.org/#/c/381912/17/specs/rocky/approved/strict_isolation_of_group_of_hosts_for_image.rst
|
||||||
|
.. _mirror_nova_host_aggregates: https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/placement-mirror-host-aggregates.html
|
||||||
|
.. _IsolatedHostsFilter: https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#isolatedhostsfilter
|
||||||
|
.. Rocky PTG: https://etherpad.openstack.org/p/nova-ptg-rocky
|
||||||
|
.. Stein PTG: https://etherpad.openstack.org/p/nova-ptg-stein
|
||||||
|
|
||||||
|
History
|
||||||
|
=======
|
||||||
|
|
||||||
|
.. list-table:: Revisions
|
||||||
|
:header-rows: 1
|
||||||
|
|
||||||
|
* - Release Name
|
||||||
|
- Description
|
||||||
|
* - Stein
|
||||||
|
- Introduced
|
||||||
|
* - Train
|
||||||
|
- Re-proposed
|
Loading…
Reference in New Issue