openstack/placement
Code Issues Proposed changes

Allow [a-zA-Z0-9_-]{1,64} for request group suffix

Browse Source 
Add a 1.33 microversion to move from numeric suffixes to string
suffixes that can be 64 chars longs made from '-', '_', and
mixed-case alphanumeric. The format is shared between schema
and RequestGroup parsing.

Docs, api-ref, api history and microversion upper limit are updated
to indicate the new form in the new microversion.

A release note is added.

Story: 2005575
Task: 30781
Change-Id: Ia44b0922d151695d406883262e891bd932536f38
This commit is contained in:
Chris Dent 2019-05-06 10:24:53 -07:00
parent 1281806c99
commit fb0f6f2608
11 changed files with 215 additions and 47 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

68
api-ref/source/parameters.yaml
View File

@ -50,11 +50,11 @@ allocation_candidates_group_policy:
description: >
When more than one ``resourcesN`` query parameter is supplied,
``group_policy`` is required to indicate how the groups should interact.
With ``group_policy=none``, separate groupings - numbered or unnumbered -
With ``group_policy=none``, separate groupings - with or without a suffix -
may or may not be satisfied by the same provider. With
``group_policy=isolate``, numbered groups are guaranteed to be satisfied by
``group_policy=isolate``, suffixed groups are guaranteed to be satisfied by
*different* providers - though there may still be overlap with the
unnumbered group.
suffixless group.
allocation_candidates_in_tree: &allocation_candidates_in_tree
type: string
in: query
@ -68,11 +68,15 @@ allocation_candidates_in_tree_granular:
<<: *allocation_candidates_in_tree
description: >
A string representing a resource provider uuid. The parameter key is
``in_treeN``, where ``N`` represents a positive integer suffix
corresponding with a ``resourcesN`` parameter. When supplied, it will
filter the returned allocation candidates for that numbered group to only
those resource providers that are in the same tree with the given resource
provider.
``in_treeN``, where ``N`` represents a suffix corresponding with a
``resourcesN`` parameter. When supplied, it will filter the returned
allocation candidates for that suffixed group to only those resource
providers that are in the same tree with the given resource provider.
**In microversions 1.25 - 1.32** the suffix is a number.
**Starting from microversion 1.33** the suffix is a string that may be 1-64
characters long and consist of numbers, ``a-z``, ``A-Z``, ``-``, and ``_``.
allocation_candidates_limit:
type: integer
in: query
@ -151,12 +155,18 @@ allocation_candidates_member_of_granular:
aggregate uuids. The returned resource providers must not directly be
associated with any of the aggregates identified by uuid.
The parameter key is ``member_ofN``, where ``N`` represents a positive
integer suffix corresponding with a ``resourcesN`` parameter. The value
format is the same as for the (unnumbered) ``member_of`` parameter; but
all of the resources and traits specified in a numbered grouping will
always be satisfied by the same resource provider.
Separate groupings - numbered or unnumbered - may or may not be satisfied
The parameter key is ``member_ofN``, where ``N`` represents a suffix
corresponding with a ``resourcesN`` parameter. The value format is the
same as for the (not granular) ``member_of`` parameter; but all of the
resources and traits specified in a granular grouping will always be
satisfied by the same resource provider.
**In microversions 1.25 - 1.32** the suffix is a number.
**Starting from microversion 1.33** the suffix is a string that may be 1-64
characters long and consist of numbers, ``a-z``, ``A-Z``, ``-``, and ``_``.
Separate groupings - with or without a suffix - may or may not be satisfied
by the same provider, depending on the value of the ``group_policy``
parameter.
@ -179,14 +189,20 @@ required_traits_granular:
required42=HW_CPU_X86_AVX,HW_CPU_X86_SSE,!HW_CPU_X86_AVX2
The parameter key is ``requiredN``, where ``N`` represents a
positive integer suffix corresponding with a ``resourcesN`` parameter.
The value format is the same as for the (unnumbered) ``required``
parameter; but all of the resources and traits specified in a numbered
The parameter key is ``requiredN``, where ``N`` represents a suffix
corresponding with a ``resourcesN`` parameter.
The value format is the same as for the (not granular) ``required``
parameter; but all of the resources and traits specified in a suffixed
grouping will always be satisfied by the same resource provider. Separate
groupings - numbered or unnumbered - may or may not be satisfied by the
groupings - with or without a suffix - may or may not be satisfied by the
same provider, depending on the value of the ``group_policy`` parameter.
**In microversions 1.25 - 1.32** the suffix is a number.
**Starting from microversion 1.33** the suffix is a string that may be 1-64
characters long and consist of numbers, ``a-z``, ``A-Z``, ``-``, and ``_``.
It is an error to specify a ``requiredN`` parameter without a corresponding
``resourcesN`` parameter with the same suffix.
min_version: 1.25
@ -322,10 +338,16 @@ resources_query_granular:
resources42=VCPU:4,DISK_GB:64,MEMORY_MB:2048
The parameter key is ``resourcesN``, where ``N`` represents a unique
positive integer suffix. The value format is the same as for the
(unnumbered) ``resources`` parameter, but the resources specified in a
``resourcesN`` parameter will always be satisfied by a single provider.
Separate groupings - numbered or unnumbered - may or may not be satisfied
suffix. The value format is the same as for the (not granular)
``resources`` parameter, but the resources specified in a ``resourcesN``
parameter will always be satisfied by a single provider.
**In microversions 1.25 - 1.32** the suffix is a number.
**Starting from microversion 1.33** the suffix is a string that may be 1-64
characters long and consist of numbers, ``a-z``, ``A-Z``, ``-``, and ``_``.
Separate groupings - with or without a suffix - may or may not be satisfied
by the same provider depending on the value of the ``group_policy``
parameter.
min_version: 1.25

11
doc/source/usage/provider-tree.rst
View File

@ -358,9 +358,10 @@ with ``NUMA1_1`` resource provider.
proposed separately and in progress. See the `Support subtree filter`_
specification for details.
The numbered syntax ``in_tree<N>`` is also supported according to
`Granular Resource Requests`_. This restricts providers satisfying the Nth
granular request group to the tree of the specified provider.
The suffixed syntax ``in_tree<$S>`` (where ``$S`` is a number in microversions
``1.25-1.32`` and ``[a-zA-Z0-9_-]{1,64}`` from ``1.33``) is also supported
according to `Granular Resource Requests`_. This restricts providers satisfying
the suffixed granular request group to the tree of the specified provider.
For example, in the environment above, when you want to have ``VCPU`` from
``CN1`` and ``DISK_GB`` from wherever, the request may look like::
@ -377,8 +378,8 @@ which will return the sharing providers as well as the local disk.
5. ``NUMA1_1`` (``VCPU``) + ``SS2`` (``DISK_GB``)
6. ``NUMA1_2`` (``VCPU``) + ``SS2`` (``DISK_GB``)
This is because the unnumbered ``in_tree`` is applied to only the unnumbered
resource of ``VCPU``, and not applied to the numbered resource, ``DISK_GB``.
This is because the unsuffixed ``in_tree`` is applied to only the unsuffixed
resource of ``VCPU``, and not applied to the suffixed resource, ``DISK_GB``.
When you want to have ``VCPU`` from wherever and ``DISK_GB`` from ``SS1``,
the request may look like::

4
placement/handlers/allocation_candidate.py
View File

@ -243,7 +243,9 @@ def list_allocation_candidates(req):
context.can(policies.LIST)
want_version = req.environ[microversion.MICROVERSION_ENVIRON]
get_schema = schema.GET_SCHEMA_1_10
if want_version.matches((1, 31)):
if want_version.matches((1, 33)):
get_schema = schema.GET_SCHEMA_1_33
elif want_version.matches((1, 31)):
get_schema = schema.GET_SCHEMA_1_31
elif want_version.matches((1, 25)):
get_schema = schema.GET_SCHEMA_1_25

35
placement/lib.py
View File

@ -19,6 +19,7 @@ import re
import webob
from placement import microversion
from placement.schemas import allocation_candidate
from placement import util
@ -28,8 +29,13 @@ _QS_REQUIRED = 'required'
_QS_MEMBER_OF = 'member_of'
_QS_IN_TREE = 'in_tree'
_QS_KEY_PATTERN = re.compile(
r"^(%s)([1-9][0-9]*)?$" % '|'.join(
(_QS_RESOURCES, _QS_REQUIRED, _QS_MEMBER_OF, _QS_IN_TREE)))
r"^(%s)(%s)?$" % ('|'.join(
(_QS_RESOURCES, _QS_REQUIRED, _QS_MEMBER_OF, _QS_IN_TREE)),
allocation_candidate.GROUP_PAT))
_QS_KEY_PATTERN_1_33 = re.compile(
r"^(%s)(%s)?$" % ('|'.join(
(_QS_RESOURCES, _QS_REQUIRED, _QS_MEMBER_OF, _QS_IN_TREE)),
allocation_candidate.GROUP_PAT_1_33))
class RequestGroup(object):
@ -74,14 +80,16 @@ class RequestGroup(object):
return ret
@staticmethod
def _parse_request_items(req, allow_forbidden):
def _parse_request_items(req, allow_forbidden, verbose_suffix):
ret = {}
pattern = _QS_KEY_PATTERN_1_33 if verbose_suffix else _QS_KEY_PATTERN
for key, val in req.GET.items():
match = _QS_KEY_PATTERN.match(key)
match = pattern.match(key)
if not match:
continue
# `prefix` is 'resources', 'required', 'member_of', or 'in_tree'
# `suffix` is an integer string, or None
# `suffix` is a number in microversion < 1.33, a string 1-64
# characters long of [a-zA-Z0-9_-] in microversion >= 1.33, or None
prefix, suffix = match.groups()
suffix = suffix or ''
if suffix not in ret:
@ -135,7 +143,7 @@ class RequestGroup(object):
# The above would still pass if there were no request groups
if not by_suffix:
msg = (
"At least one request group (`resources` or `resources{N}`) "
"At least one request group (`resources` or `resources{$S}`) "
"is required.")
raise webob.exc.HTTPBadRequest(msg)
@ -161,7 +169,7 @@ class RequestGroup(object):
@classmethod
def dict_from_request(cls, req):
"""Parse numbered resources, traits, and member_of groupings out of a
"""Parse suffixed resources, traits, and member_of groupings out of a
querystring dict found in a webob Request.
The input req contains a query string of the form:
@ -174,11 +182,11 @@ class RequestGroup(object):
&resources2=$RESOURCE_CLASS_NAME:$AMOUNT,RESOURCE_CLASS_NAME:$AMOUNT
&required2=$TRAIT_NAME,$TRAIT_NAME&member_of2=$AGG_UUID
These are parsed in groups according to the numeric suffix of the key.
These are parsed in groups according to the arbitrary suffix of the key.
For each group, a RequestGroup instance is created containing that
group's resources, required traits, and member_of. For the (single)
group with no suffix, the RequestGroup.use_same_provider attribute is
False; for the numbered groups it is True.
False; for the granular groups it is True.
If a trait in the required parameter is prefixed with ``!`` this
indicates that that trait must not be present on the resource
@ -186,8 +194,8 @@ class RequestGroup(object):
traits are only processed if ``allow_forbidden`` is True. This allows
the caller to control processing based on microversion handling.
The return is a dict, keyed by the numeric suffix of these RequestGroup
instances (or the empty string for the unnumbered group).
The return is a dict, keyed by the suffix of these RequestGroup
instances (or the empty string for the unidentified group).
As an example, if qsdict represents the query string:
@ -250,8 +258,11 @@ class RequestGroup(object):
want_version = req.environ[microversion.MICROVERSION_ENVIRON]
# Control whether we handle forbidden traits.
allow_forbidden = want_version.matches((1, 22))
# Control whether we want verbose suffixes
verbose_suffix = want_version.matches((1, 33))
# dict of the form: { suffix: RequestGroup } to be returned
by_suffix = cls._parse_request_items(req, allow_forbidden)
by_suffix = cls._parse_request_items(
req, allow_forbidden, verbose_suffix)
cls._check_for_orphans(by_suffix)

2
placement/microversion.py
View File

@ -81,6 +81,8 @@ VERSIONS = [
# `GET /allocation_candidates` API
'1.32', # Support negative member_of queryparams on
# `GET /resource_providers` and `GET /allocation_candidates`
'1.33', # Support granular resource requests with suffixes that match
# [A-Za-z0-9_-]{1,64}.
]

22
placement/rest_api_version_history.rst
View File

@ -603,3 +603,25 @@ which is equivalent to::
?member_of=!<agg1>&member_of=!<agg2>&member_of=!<agg3>``
where candidate resource providers must not be in agg1, agg2, or agg3.
1.33 - Support string request group suffixes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. versionadded:: Train
The syntax for granular groupings of resource, required/forbidden trait, and
aggregate association requests introduced in ``1.25`` has been extended to
allow, in addition to numbers, strings from 1 to 64 characters in length
consisting of a-z, A-Z, 0-9, ``_``, and ``-``. This is done to allow naming
conventions (e.g., ``resources_COMPUTE`` and ``resources_NETWORK``) to emerge
in situations where multiple services are collaborating to make requests.
For example, in addition to the already supported::
resources42=XXX&required42=YYY&member_of42=ZZZ
it is now possible to use more complex strings, including UUIDs::
resources_PORT_fccc7adb-095e-4bfd-8c9b-942f41990664=XXX
&required_PORT_fccc7adb-095e-4bfd-8c9b-942f41990664=YYY
&member_of_PORT_fccc7adb-095e-4bfd-8c9b-942f41990664=ZZZ

15
placement/schemas/allocation_candidate.py
View File

@ -14,6 +14,12 @@
import copy
# The suffix used with request groups. Prior to 1.33, the group were numbered.
# With 1.33 they become alphanumeric, '_', and '-' with a length limit of 64.
GROUP_PAT = r'[1-9][0-9]*'
GROUP_PAT_1_33 = r'[a-zA-Z0-9_-]{1,64}'
# Represents the allowed query string parameters to the GET
# /allocation_candidates API call
GET_SCHEMA_1_10 = {
@ -60,7 +66,7 @@ del GET_SCHEMA_1_25["required"]
del GET_SCHEMA_1_25["properties"]["required"]
del GET_SCHEMA_1_25["properties"]["member_of"]
# Pattern property key format for a numbered or un-numbered grouping
_GROUP_PAT_FMT = "^%s([1-9][0-9]*)?$"
_GROUP_PAT_FMT = "^%s(" + GROUP_PAT + ")?$"
GET_SCHEMA_1_25["patternProperties"] = {
_GROUP_PAT_FMT % "resources": {
"type": "string",
@ -81,3 +87,10 @@ GET_SCHEMA_1_25["properties"]["group_policy"] = {
GET_SCHEMA_1_31 = copy.deepcopy(GET_SCHEMA_1_25)
GET_SCHEMA_1_31["patternProperties"][_GROUP_PAT_FMT % "in_tree"] = {
"type": "string"}
# Microversion 1.33 allows more complex resource group suffixes.
GET_SCHEMA_1_33 = copy.deepcopy(GET_SCHEMA_1_31)
_GROUP_PAT_FMT_1_33 = "^%s(" + GROUP_PAT_1_33 + ")?$"
GET_SCHEMA_1_33["patternProperties"] = {
_GROUP_PAT_FMT_1_33 % group_type: {"type": "string"}
for group_type in ('resources', 'required', 'member_of', 'in_tree')}

45
placement/tests/functional/gabbits/granular.yaml
View File

@ -287,6 +287,30 @@ tests:
$.provider_summaries["$ENVIRON['CN_RIGHT']"].resources[DISK_GB][capacity]: 500
$.provider_summaries["$ENVIRON['SHR_DISK_1']"].resources[DISK_GB][capacity]: 1000
- name: required, forbidden, member_of in long suffix
desc: same as above, but using complex suffixes
GET: /allocation_candidates
query_parameters:
resources_compute: VCPU:1
required_compute: "!HW_CPU_X86_SSE"
resources_disk: DISK_GB:100
required_disk: CUSTOM_DISK_SSD
member_of_disk: in:$ENVIRON['AGGA'],$ENVIRON['AGGC']
group_policy: none
request_headers:
openstack-api-version: placement 1.33
status: 200
response_json_paths:
$.allocation_requests.`len`: 2
$.allocation_requests..allocations["$ENVIRON['CN_LEFT']"].resources[VCPU]: 1
$.allocation_requests..allocations["$ENVIRON['CN_RIGHT']"].resources[VCPU]: 1
$.allocation_requests..allocations["$ENVIRON['SHR_DISK_1']"].resources[DISK_GB]: 100
$.provider_summaries.`len`: 3
$.provider_summaries["$ENVIRON['CN_LEFT']"].resources[VCPU][capacity]: 8
$.provider_summaries["$ENVIRON['CN_RIGHT']"].resources[VCPU][capacity]: 8
$.provider_summaries["$ENVIRON['CN_RIGHT']"].resources[DISK_GB][capacity]: 500
$.provider_summaries["$ENVIRON['SHR_DISK_1']"].resources[DISK_GB][capacity]: 1000
- name: multiple member_of
GET: /allocation_candidates
query_parameters:
@ -467,11 +491,26 @@ tests:
- "'resources01' does not match any of the regexes"
- name: bogus suffix
GET: /allocation_candidates?resources1a=VCPU:1
desc: this is bogus because of unsupported character
GET: /allocation_candidates?resources1@=VCPU:1
request_headers:
openstack-api-version: placement 1.33
status: 400
response_strings:
- Invalid query string parameters
- "'resources1a' does not match any of the regexes"
- "'resources1@' does not match any of the regexes"
- "^member_of([a-zA-Z0-9_-]{1,64})?$"
- name: bogus length
desc: 65 character suffix is too long
GET: /allocation_candidates?resources_0123456701234567012345670123456701234567012345670123456701234567=VCPU:1
request_headers:
openstack-api-version: placement 1.33
status: 400
response_strings:
- Invalid query string parameters
- "'resources_0123456701234567012345670123456701234567012345670123456701234567' does not match any of the regexes"
- "^member_of([a-zA-Z0-9_-]{1,64})?$"
- name: invalid group_policy value
GET: /allocation_candidates?resources=VCPU:1&group_policy=bogus
@ -502,4 +541,4 @@ tests:
GET: /allocation_candidates?group_policy=isolate
status: 400
response_strings:
- At least one request group (`resources` or `resources{N}`) is required.
- At least one request group (`resources` or `resources{$S}`) is required.

4
placement/tests/functional/gabbits/microversion.yaml
View File

@ -41,13 +41,13 @@ tests:
response_json_paths:
$.errors[0].title: Not Acceptable
- name: latest microversion is 1.32
- name: latest microversion is 1.33
GET: /
request_headers:
openstack-api-version: placement latest
response_headers:
vary: /openstack-api-version/
openstack-api-version: placement 1.32
openstack-api-version: placement 1.33
- name: other accept header bad version
GET: /

44
placement/tests/unit/test_util.py
View File

@ -900,6 +900,50 @@ class TestParseQsRequestGroups(testtools.TestCase):
self.assertRequestGroupsEqual(
expected, self.do_parse(qs, version=(1, 22)))
def test_group_suffix_length_1_33(self):
longstring = '01234567' * 8
qs = 'resources_%s=CUSTOM_MAGIC:1' % longstring
exc = self.assertRaises(
webob.exc.HTTPBadRequest, self.do_parse, qs, version=(1, 33))
# NOTE(cdent): This error message is not what an API user would see.
# They would get an error during JSON schema processing.
self.assertIn('least one request group', str(exc))
def test_group_suffix_character_limits_1_33(self):
qs = 'resources!#%=CUSTOM_MAGIC:1'
exc = self.assertRaises(
webob.exc.HTTPBadRequest, self.do_parse, qs, version=(1, 33))
# NOTE(cdent): This error message is not what an API user would see.
# They would get an error during JSON schema processing.
self.assertIn('least one request group', str(exc))
def test_group_suffix_character_limits_1_22(self):
qs = 'resources!#%=CUSTOM_MAGIC:1'
exc = self.assertRaises(
webob.exc.HTTPBadRequest, self.do_parse, qs, version=(1, 22))
# NOTE(cdent): This error message is not what an API user would see.
# They would get an error during JSON schema processing.
self.assertIn('least one request group', str(exc))
def test_good_suffix_1_33(self):
qs = ('resources_car_HOUSE_10=CUSTOM_MAGIC:1'
'&required_car_HOUSE_10=CUSTOM_PHYSNET1')
expected = [
pl.RequestGroup(
use_same_provider=True,
resources={
'CUSTOM_MAGIC': 1,
},
required_traits={
'CUSTOM_PHYSNET1',
}
),
]
self.assertRequestGroupsEqual(
expected, self.do_parse(qs, version=(1, 33)))
self.assertRaises(
webob.exc.HTTPBadRequest, self.do_parse, qs, version=(1, 22))
class TestPickLastModified(testtools.TestCase):

12
releasenotes/notes/granular-request-suffix-a7fd857eadc16b56.yaml Normal file
View File

@ -0,0 +1,12 @@
---
features:
- |
In microversion 1.33, the syntax for granular groupings of resource,
required/forbidden trait, and aggregate association requests introduced in
`1.25`_ has been extended to allow, in addition to numbers, strings from 1
to 64 characters in length consisting of a-z, A-Z, 0-9, ``_``, and ``-``.
This is done to allow naming conventions (e.g., ``resources_COMPUTE`` and
``resources_NETWORK``) to emerge in situations where multiple services are
collaborating to make requests.
.. _1.25: https://docs.openstack.org/placement/latest/placement-api-microversion-history.html#granular-resource-requests-to-get-allocation-candidates