From e8095800887e05857056707698a35452fbba3df8 Mon Sep 17 00:00:00 2001 From: Monty Taylor Date: Mon, 24 Apr 2017 11:03:48 -0500 Subject: [PATCH] Add document describing consuming version discovery The next patch actually describes desired state of version discovery. But in an epic amount of cart-before-the-horse, we have the process for consuming the discovery already because the process must take in to account the present as well as the past. This process has kept in mind what consuming the recommended discovery process _wants_ to look like in the future and in calls that out in a few places. The intent would be that the algorithm here would work for all clouds, but that as clouds and services adopt API-SIG recommendations, the interactions with the clouds would become more efficient. (so for clients using the complete algorithm they should be upwards compatible with forthcoming API-SIG guidelines and will just naturally do less work over time). I believe this is consistent in defaults, fallbacks and error conditions with what is currently implemented in keystoneauth, although there is additional logic presented here which is not yet in keystoneauth. The intent is for the process presented here to not change the behavior experienced by current keystoneauth users, with the exception that when the complete algorithm is implemented it's possible that an additional API call may be made on older clouds. That is to say, keystoneauth should not need to make any incompatible changes, but may need to add some features to be a fully compliant implementation. Apologies for the size and complexity. It turns out there are many historical oddities still lurking out there and advice to client authors that does not take them in to account would be incomplete. On the other hand, as we drive guidelines forward into being implemented, the need for this much crazy logic should go away. Co-Authored-By: Dmitry Tantsur Change-Id: I241f76bca8ac27fc3d27028ae284b9012a2da7e9 --- guidelines/consuming-catalog.rst | 669 +++-------- guidelines/consuming-catalog/authority.rst | 42 + guidelines/consuming-catalog/endpoint.rst | 471 ++++++++ .../consuming-catalog/version-discovery.rst | 1068 +++++++++++++++++ guidelines/discoverability.rst | 2 + guidelines/microversion_specification.rst | 2 + 6 files changed, 1783 insertions(+), 471 deletions(-) create mode 100644 guidelines/consuming-catalog/authority.rst create mode 100644 guidelines/consuming-catalog/endpoint.rst create mode 100644 guidelines/consuming-catalog/version-discovery.rst diff --git a/guidelines/consuming-catalog.rst b/guidelines/consuming-catalog.rst index 5a9031e..474f892 100644 --- a/guidelines/consuming-catalog.rst +++ b/guidelines/consuming-catalog.rst @@ -22,9 +22,18 @@ from the Service Catalog. .. note:: The use of the word "object" in this document refers to a JSON object, not an Object from any particular programming language. +.. _catalog-user-request: + User Request ============ +.. note:: It is worth noting that 'user' is a maleable concept. For instance, + the shade library performs service discovery on behalf of its users + so does not expect its users to provide a 'service-type'. In that + case, shade is the 'user' of the keystoneauth library which is the + discovery implementation. It is definitely not required that all + consumers of OpenStack clouds know all of these things. + The ultimate goal of this process is for a user to find the information about an endpoint for a service given some inputs. The user will start the process knowing some number of these parameters. Each additional input expected from @@ -35,69 +44,192 @@ in helping the user ask the right question. .. note:: Be liberal with what you accept and strict with what you emit. -There is one piece of information that is absolutely required that the -user know: +The following is a list of such pieces of information that can be provided +as user input. When an implementation exposes the ability for a user to +express these parameters it is **STRONGLY** recommended that these names +be used, as they show up across the OpenStack ecosystem and make discussion +easier. -service-type +It is assumed that the user has an ``{auth-url}`` and authentication +information. The authentication process itself is out of the scope of this +document. + +Required Inputs +--------------- + +There is one piece of information that is absolutely required that the +user know. + +``service-type`` The official name of the service, such as ``compute``, ``image`` or - ``block-storage`` as listed in the `OpenStack Service Types Authority`_. + ``block-storage`` as listed in the :doc:`OpenStack Service Types Authority + `. Required. It is impossible for a user to consume service discovery without knowing what service they want to discover. -The user may also wish to express an alteration to the general algorithm: - -be-strict - Forgo lenient backwards compatibility concessions and be more strict in - input and output validation. +Optional Filters +---------------- There are several optional pieces of information that the user might know, -or additional constraints the user might wish to express. +or additional constraints the user might wish to express to control how the +endpoints for a service are selected. -region-name +``region-name`` The region of the service the user desires to work with. May be optional, depending on whether the cloud has more than one region. Services all exist within regions, but some clouds only have one region. - If ``{be-strict}`` has been given, ``{region-name}`` is required. + If ``{be-strict}`` (see below) has been given, ``{region-name}`` is required. -.. note:: It is highly recommended that ``{region-name}`` always be required - to protect against single-region clouds adding a region in the - future. However, keystoneauth today allows region-name to be omitted - and there are a large number of clouds in existence with a single - region named ``RegionOne``. For completely new libraries or major - versions where breaking behavior is acceptable, requiring region-name - by default would be preferred. + .. note:: It is highly recommended that ``{region-name}`` always be required + to protect against single-region clouds adding a region in the + future. However, the canonical OpenStack implementation + *keystoneauth* today allows region name to be omitted and there are + a large number of clouds in existence with a single region named + ``RegionOne``. For completely new libraries or major versions + where breaking behavior is acceptable, requiring region name + by default would be preferred, but breaking users just to introduce + the restriction is discouraged. -interface - Which API interface, such as ``public``, ``internal``, or ``admin`` - the user wants to use. A user can also request a list of interfaces they find - acceptable in the order of their preference, such as - ``['internal', 'public']`` (Optional, defaults to ``public``.) +``interface`` + Which API interface, such as ``public``, ``internal`` or ``admin``, that + the user wants to use. A user should be able to request a list of interfaces + they find acceptable in the order of their preference, such as + ``['internal', 'public']`` (Optional, defaults to ``public``) -service-name +``endpoint-version`` OR ``min-endpoint-version``, ``max-endpoint-version`` + The **major** version of the service the user desires to work with. Optional. + + An endpoint version is inherently a range with a minimum and a maximum value. + Whether it is presented to the user as a single parameter or a pair of + parameters is an implementation detail. + + Each endpoint version is a string with one (``3``) or two (``3.1``) numbers, + separated by a dot. + + .. warning:: Care has to be taken to not confuse major versions consisting + of two numbers with microversions. Microversions usually exist + within a certain major version, and also have a form of ``X.Y``. + No services currently use both major versions and microversions + in the form of ``X.Y``. + + .. TODO(dtantsur): so, what if a service has both major versions in the form + of ``X.Y`` and microversions? + + Version strings are not decimals, the are a tuple of 2 numbers combined with + a dot. Therefore, ``3.10`` is higher than ``3.9``. + + A user can omit the endpoint-version indicating that they want to use + whatever endpoint is in the ``{service-catalog}``. + + A user can desire to work with the latest available version, in which + case the ``{endpoint-version}`` should be ``latest``. If s + ``{min-endpoint-version}`` is ``latest``, ``{max-endpoint-version}`` must be + omitted or also ``latest``. + + A version can be specified with a minor value of ``latest`` to indicate + the highest minor version of a given major version. For instance, + ``3.latest`` would match the highest of ``3.3`` and ``3.4`` but not ``4.0``. + + If the parameter is presented as a single string, a single value should be + interpreted as if ``{min-endpoint-version}`` is the value given and + ``{max-endpoint-version}`` is ``MAJOR.latest``. For instance, if ``3.4`` is + given as a single value, ``{min-endpoint-version}`` is ``3.4`` and + ``{max-endpoint-version}`` is ``3.latest``. + + It may seem strange from an individual user perspective to want a range or + ``latest`` - but from a library and framework perspective, things like shade + or terraform may have internal logic that can handle more than one version of + a service and want to use the best version available. + + .. note:: Guidance around 'latest' is different from that found in + :ref:`the microversion specification + `. It is acceptable for a client + library or framework to be interested in the latest version + available but such a specification is internal and not sent to + the server. In the client case with major versions, ``latest`` acts + as an input to the version discovery process. + +``service-name`` Arbitrary name given to the service by the deployer. Optional. -.. note:: In all except the most extreme cases this should never be needed and - its use as a meaningful identifier by Deployers is strongly - discouraged. However, the Consumer has no way to otherwise mitigate - the situation if their Deployer has provided them with a catalog - where a ``service-name`` must be used, so ``service-name`` must be - accepted as input. If ``{be-strict}`` has been requested, - supplying ``{service-name}`` should be an error. + .. note:: In all except the most extreme cases this should never be needed + and its use as a meaningful identifier by Deployers is strongly + discouraged. However, the Consumer has no way to otherwise mitigate + the situation if their Deployer has provided them with a catalog + where a ``service-name`` must be used, so ``service-name`` must be + accepted as input. + If ``{be-strict}`` (see below) has been requested, a user supplying + ``{service-name}`` should be an error. -service-id +``service-id`` Unique identifier for an endpoint in the catalog. Optional. -.. note:: On clouds with well-formed catalogs ``service-id`` should never be - needed. If ``{be-strict}`` has been requested, supplying - ``{service-id}`` should be an error. + .. note:: On clouds with well-formed catalogs ``service-id`` should never be + needed. If ``{be-strict}`` has been requested, supplying + ``{service-id}`` should be an error. -endpoint-override +``endpoint-override`` An endpoint for the service that the user has procured from some other source. (Optional, defaults to omitted.) -At the end of the discovery process, the user should know the -``{service-endpoint}``, which is the endpoint to use as the root of the -service, and the ``{interface}`` of the endpoint that was found. +Discovery Behavior Modifiers +---------------------------- + +The user may also wish to express alterations to the general algorithm. +Implementations may present these flags under any name that makes sense, +or may choose to not present them as behavioral modification options at all. + +``be-strict`` + Forgo leniant backwards compatibility concessions and be more strict in + input and output validation. Defaults to False. + +``skip-discovery`` + If the user wants to completely skip the version discovery process even if + logic would otherwise do it. This is useful if the user has specified an + ``{endpoint-override}`` or they know they just want to use whatever is in + the catalog and do not need additional metadata about the endpoint. Defaults + to False + +``fetch-version-information`` + If the user has specified an ``{endpoint-version}`` which can be known to + match just from looking at the URL, the version discovery process will not + fetch version information documents. However, the user may need the + information, such as microversion ranges. Using + ``{fetch-version-information}`` allows them to request that the version + document be fetched even when an optimization in the process would otherwise + allow fetching the document to be skipped. Defaults to False. + + +Discovery Results +================= + +At the end of the discovery process, the user should know the following: + +If the process was successful: + +* The actual values found for all of the input values above. + + Found values will be referred to in these documents as ``found-{value}`` to + differentiate. So if a user requested an ``{endpoint-version}`` of + ``latest``, ``{found-endpoint-version}`` might be ``3.5``. + +* ``service-endpoint`` + The endpoint to use as the root of the service. + +* ``max-version`` + If the service supports microversions, what is the maximum microversion the + service supports. Optional, defaults to omitted, which implies that + microversions are not supported. + +* ``min-version`` + If the service supports microversions, what is the minimum microversion the + service supports. Optional, defaults to omitted, which implies that + microversions are not supported. + +If the process was unsuccessful, an error should be returned explaining which +part failed. For instance, was a matching service not found at all or was +a matching version not found. If a matching version was not found, the error +should contain a list of version that were found. In the description that follows, each of the above inputs and outputs will be referred to like ``{endpoint-override}`` so that it is clear whether a user @@ -107,459 +239,54 @@ referred to at a later point are similarly referred to like ``{service-catalog}``. Names will not be reused within the process to hold different content at different times. -It is also assumed that the user has an ``{auth-url}`` and authentication -information. The authentication process itself is out of the scope of this -document. - Discovery Algorithm =================== Services should be registered in the ``{service-catalog}`` using their -``{service-type}`` from the `OpenStack Service Types Authority`_. However, -for historical reasons there are some services that have old service types -found in the wild. To facilitate moving forward with the correct -``{service-type}`` names, but also support existing users and installations, -the `OpenStack Service Types Authority`_ contains a list of historical -aliases for such services. See `Consuming Service Types Authority`_ for -information on the data itself. +``{service-type}`` from the :doc:`OpenStack Service Types Authority +`. However, for historical reasons there are some +services that have old service types found in the wild. To facilitate moving +forward with the correct ``{service-type}`` names, but also support existing +users and installations, the OpenStack Service Types Authority contains a list +of historical aliases for such services. Clients will need a copy of the data published in the -`OpenStack Service Types Authority`_ to be able to complete the full Discovery +OpenStack Service Types Authority to be able to complete the full Discovery Algorithm. A client library could either keep a local copy or fetch the data from https://service-types.openstack.org/service-types.json and potentially cache it. It is recommended that client libraries handle consumption of the historical data for their users but also allow some mechanism for the user to -provide a more up to date version of the data if necessary. See -`Consuming Service Types Authority`_ for information on how to fetch the data. +provide a more up to date verison of the data if necessary. The basic process is: -#. If the user has provided ``{endpoint-override}``, STOP. This is the - ``{service-endpoint}``. - #. Authenticate to keystone at the ``{auth-url}``, retreiving a ``token`` which contains the ``{service-catalog}``. -#. Retrieve ``{catalog-endpoint}`` from the ``{service-catalog}`` given - some combination of ``{service-type}``, ``{interface}``, ``{service-name}``, - ``{region-name}`` and ``{service-id}``. (See :ref:`endpoint-from-catalog`.) + .. note:: This step is obviously skipped for clouds without authentication. -.. _endpoint-from-catalog: +#. If the user has provided ``{endpoint-override}``, it is used as + ``{catalog-endpoint}``. -Endpoint from Catalog -===================== +#. If the user has not provided ``{endpoint-override}``, retrieve matching + ``{catalog-endpoint}`` from the ``{service-catalog}`` using the procedure + explained in :doc:`consuming-catalog/endpoint`. -The ``{service-catalog}`` can be found in the ``token`` returned from -keystone authentication. - -If v3 auth is used, the catalog will be in the ``catalog`` property of the -top-level ``token`` object. Such as: - -.. code-block:: json - - { - "token": { - "catalog": {} - } - } - -If v2 auth is used it will be in the ``serviceCatalog`` property of the -top-level ``access`` object. Such as: - -.. code-block:: json - - { - "access": { - "serviceCatalog": {} - } - } - -In both cases, the catalog content itself is a list of objects. Each object has -two main keys that concern discovery: - -type - Matches ``{service-type}`` - -endpoints - List of endpoint objects for that service - -Additionally, for backwards compatibility reasons, the following keys may -need to be checked. - -name - Matches ``{service-name}`` - -id - Matches ``{service-id}`` - -The list of endpoints has a different format depending on whether v2 or v3 auth -was used. For both versions each endpoint object has a ``region`` key, -which should match ``{region-name}`` if one was given. - -In v2 auth the endpoint object has three keys ``publicURL``, -``internalURL``, ``adminURL``. The endpoint for the ``{interface}`` requested -by the user is found in the key with the name matching ``{interface}`` plus -the string ``URL``. - -In v3 auth the endpoint object has a ``url`` that is the endpoint that is -being requested if the value of ``interface`` matches ``{interface}``. - -Concrete examples of tokens with catalogs: - -V3 Catalog Objects: - -.. code-block:: json - - { - "token": { - "catalog": [ - { - "endpoints": [ - { - "id": "39dc322ce86c4111b4f06c2eeae0841b", - "interface": "public", - "region": "RegionOne", - "url": "https://identity.example.com" - }, - { - "id": "ec642f27474842e78bf059f6c48f4e99", - "interface": "internal", - "region": "RegionOne", - "url": "https://identity.example.com" - }, - { - "id": "c609fc430175452290b62a4242e8a7e8", - "interface": "admin", - "region": "RegionOne", - "url": "https://identity.example.com" - } - ], - "id": "4363ae44bdf34a3981fde3b823cb9aa2", - "type": "identity", - "name": "keystone" - } - ], - } - -V2 Catalog Objects: - -.. code-block:: json - - { - "access": { - "serviceCatalog": [ - { - "endpoints_links": [], - "endpoints": [ - { - "adminURL": "https://identity.example.com/v2.0", - "region": "RegionOne", - "publicURL": "https://identity.example.com/v2.0", - "internalURL": "https://identity.example.com/v2.0", - "id": "4deb4d0504a044a395d4480741ba628c" - } - ], - "type": "identity", - "name": "keystone" - }, - ] - } - } - -The algorithm is: - -#. Find the objects in the ``{service-catalog}`` that match the requested - ``{service-type}``. (See `Match Candidate Entries`_.) - -#. If ``{service-name}`` was given and the objects remaining have a ``name`` - field, keep only the ones where ``name`` matches ``{service-name}``. - -.. note:: Catalogs from Keystone v3 before v3.3 do not have a name field. If - ``{be-strict}`` was not requested and the catalog does not have a - ``name`` field, ``{service-name}`` should be ignored. - -#. If ``{service-id}`` was given and the objects remaining have a ``id`` - field, keep only the ones where ``id`` matches ``{service-id}``. - -.. note:: Catalogs from Keystone v2 do not have an id field. If - ``{be-strict}`` was not requested and the catalog does not have a - ``id`` field, ``{service-id}`` should be ignored. - -The list of remaining objects are the ``{candidate-catalog-objects}``. If there -are no endpoints, return an error that there are no endpoints matching -``{service-type}`` and ``{service-name}``. - -Use ``{candidate-catalog-objects}`` to produce the list of -``{candidate-endpoints}``. - -For each endpoint object in each of the ``{candidate-catalog-objects}``: - -#. If v2, if there is no key of the form ``{interface}URL`` for any of the - the ``{interface}`` values given, discard the endpoint. - -#. If v3, if ``interface`` does not match any of the ``{interface}`` values - given, discard the endpoint. - -If there are no endpoints left, return an error that there are no endpoints -matching any of the ``{interface}`` values, preferrably including the list of -interfaces that were found. - -For each remaining endpoint in ``{candidate-endpoints}``: - -#. If ``{region_name}`` was given and does not match either of ``region`` - or ``region_id``, discard the endpoint. - -If there are no remaining endpoints, return an error that there are no -endpoints matching ``{region_name}``, preferrably including the list of -regions that were found. - -#. From the set of remaining candidate endpoints, find the ones that best - matches the requested ``{service-type}``. - (See `Find Endpoint Matching Best Service Type`_.) - -The remaining ``{candidate-endpoints}`` match the request. If there is more -than one of them, use the first, but emit a warning to the user that more -than one endpoint was left. If ``{be-strict}`` has been requested, return an -error instead with information about each of the endpoints left in the list. - -.. note:: It would be more correct to raise an error if there is more than one - endpoint left, but the keystoneauth library returns the first and - changing that would break a large number of existing users. If one - is writing a completely new library from scratch, or a new major - version where behavior change is acceptable, it is preferable to - raise an error here if there is more than one endpoint left. - -#. If v2, the ``{catalog-endpoint}`` is the value of ``{interface}URL``. - -#. If v3, the ``{catalog-endpoint}`` is the value of ``url``. - -Match Candidate Entries ------------------------ - -For every entry in the catalog: - -#. If the entry's type matches the requested ``{service-type}``, it is a - candidate. - -#. If the requested type is an official type from the - `OpenStack Service Types Authority`_ that has aliases and one of the aliases - matches the entry's type, it is a candidate. - -#. If the requested type is an alias of an official type from the - `OpenStack Service Types Authority`_ and the entry's type matches the - official type, it is a candidate. - -.. note:: Requesting one alias and finding a different alias is not supported - at this point because most aliases carry implied information about - major versions as well. A subsequent spec adds the process for - version discovery at which point it can be safe to attempt to return - an endpoint listed under an alias different than what was requested. - -Find Endpoint Matching Best Service Type ----------------------------------------- - -Given a list of candidate endpoints that have matched the other criteria: - -#. Check the list of candidate endpoints to see if one of them matches the - requested ``{service-type}``. If any are an exact match, - `Find Endpoint Matching Best Interface`_. - -#. If the requested ``{service-type}`` is an official type in the - `OpenStack Service Types Authority`_ that has aliases, check each alias - in order of preference as listed in the Authority to see if it has a - matching endpoint from the candidate endpoints. For all endpoints that - match the first alias with matching endpoints, - `Find Endpoint Matching Best Interface`_. +#. If ``{skip-discovery}`` is true, STOP and use ``{catalog-endpoint}`` as + ``{service-endpoint}``. Otherwise, discover the available API versions + and find the suitable ``{service-endpoint}`` using the version discovery + procedure from :doc:`consuming-catalog/version-discovery`. #. If the requested ``{service-type}`` is an alias of an official type in the - `OpenStack Service Types Authority`_ and any endpoints match the official - type, `Find Endpoint Matching Best Interface`_. + OpenStack Service Types Authority and any endpoints match the official + type, :ref:`find-endpoint-matching-best-service-type`. -Find Endpoint Matching Best Interface -------------------------------------- -Given a list of candidate endpoints that have matched the other criteria: +Table of Contents +================= -#. In order of preference of ``{interface}`` list, return all endpoints that - match the first ``{interface}`` with matching endpoints. +.. toctree:: -For example, given the following catalog: - -.. code-block:: json - - { - "token": { - "catalog": [ - { - "endpoints": [ - { - "interface": "public", - "region": "RegionOne", - "url": "https://block-storage.example.com/v3" - } - ], - "id": "4363ae44bdf34a3981fde3b823cb9aa3", - "type": "volumev3", - "name": "cinder" - }, - { - "endpoints": [ - { - "interface": "public", - "region": "RegionOne", - "url": "https://block-storage.example.com/v2" - } - ], - "id": "4363ae44bdf34a3981fde3b823cb9aa2", - "type": "volumev2", - "name": "cinder" - } - ], - } - -Then the following: - -:: - - service_type = 'block-storage' - # block-storage is not found, get list of aliases - # volumev3 is found, return it - - service_type = 'volumev2' - # volumev2 not an official type in authority, but is in catalog - # return volumev2 entry - - service_type = 'volume' - # volume not in authority or catalog - # volume is an alias of block-storage - # block-storage is not found. Return error. - -Given the following catalog: - -.. code-block:: json - - { - "token": { - "catalog": [ - { - "endpoints": [ - { - "interface": "public", - "region": "RegionOne", - "url": "https://block-storage.example.com" - } - ], - "id": "4363ae44bdf34a3981fde3b823cb9aa3", - "type": "block-storage", - "name": "cinder" - } - ], - } - -Then the following: - -:: - - service_type = 'block-storage' - # block-storage is found, return it - - service_type = 'volumev2' - # volumev2 not in authority, is an alias for block-storage - # block-storage is in the catalog, return it - -Given the following catalog: - -.. code-block:: json - - { - "token": { - "catalog": [ - { - "endpoints": [ - { - "interface": "public", - "region": "RegionOne", - "url": "https://block-storage.example.com" - } - ], - "id": "4363ae44bdf34a3981fde3b823cb9aa3", - "type": "block-storage", - "name": "cinder" - }, - { - "endpoints": [ - { - "interface": "public", - "region": "RegionOne", - "url": "https://block-storage.example.com/v2" - }, - { - "interface": "internal", - "region": "RegionOne", - "url": "https://block-storage.example.int/v2" - } - ], - "id": "4363ae44bdf34a3981fde3b823cb9aa2", - "type": "volumev2", - "name": "cinder" - } - ], - } - -Then the following: - -:: - - service_type = 'block-storage' - interface = ['internal', 'public'] - # block-storage is found - # block-storage does not have internal, but has public - # return block-storage public - - service_type = 'volumev2' - interface = ['internal', 'public'] - # volumev2 not an official type in authority, but is in catalog - # volumev2 has an internal interface - # return volumev2 internal entry - -Consuming Service Types Authority -================================= - -The `OpenStack Service Types Authority`_ is data about official service type -names and historical service type names commonly in use from before there was -an official list. It is made available to allow libraries and other client -API consumers to be able to provide a consistent interface based on the -official list but still support existing names. Providing this support is -highly recommended, but is ultimately optional. The first step in the matching -process is always to return direct matches between the catalog and the user -request, so the existing consumption models from before the existence of the -authority should always work. - -In order to consume the information in the `OpenStack Service Types Authority`_ -it is important to know a few things: - -#. The data is maintained in YAML format in git. This is the ultimately - authoritative source code for the list. - -#. The data is published in JSON format at - https://service-types.openstack.org/service-types.json and has a JSONSchema - at https://service-types.openstack.org/published-schema.json. - -#. The published data contains a version which is date based in - `ISO Date Time Format`_, a sha which contains the git sha of the - commit the published data was built from, and pre-built forward and reverse - mappings between official types and aliases. - -#. The JSON file is served with ETag support and should be considered highly - cacheable. - -#. The current version of the JSON file should always be the preferred file to - use. - -#. The JSON file is similar to timezone data. It should not be considered - versioned such that stable releases of distros should provide a - frozen version of it. Distro packages should instead update for all - active releases when a new version of the file is published. - -.. _OpenStack Service Types Authority: https://opendev.org/openstack/service-types-authority/ -.. _ISO Date Time Format: https://tools.ietf.org/html/rfc3339#section-5.6 + consuming-catalog/endpoint + consuming-catalog/version-discovery + consuming-catalog/authority diff --git a/guidelines/consuming-catalog/authority.rst b/guidelines/consuming-catalog/authority.rst new file mode 100644 index 0000000..05f834e --- /dev/null +++ b/guidelines/consuming-catalog/authority.rst @@ -0,0 +1,42 @@ +Consuming Service Types Authority +================================= + +The `OpenStack Service Types Authority`_ is data about official service type +names and historical service type names commonly in use from before there was +an official list. It is made available to allow libraries and other client +API consumers to be able to provide a consistent interface based on the +official list but still support existing names. Providing this support is +highly recommended, but is ultimately optional. The first step in the matching +process is always to return direct matches between the catalog and the user +request, so the existing consumption models from before the existence of the +authority should always work. + +In order to consume the information in the `OpenStack Service Types Authority`_ +it is important to know a few things: + +#. The data is maintained in YAML format in git. This is the ultimately + authoritative source code for the list. + +#. The data is published in JSON format at + https://service-types.openstack.org/service-types.json and has a JSONSchema + at https://service-types.openstack.org/published-schema.json. + +#. The published data contains a version which is date based in + `ISO Date Time Format`_, a sha which contains the git sha of the + commit the published data was built from, and pre-built forward and reverse + mappings between official types and aliases. + +#. The JSON file is served with ETag support and should be considered highly + cacheable. + +#. The current version of the JSON file should always be the preferred file to + use. + +#. The JSON file is similar to timezone data. It should not be considered + versioned such that stable releases of distros should provide a + frozen version of it. Distro packages should instead update for all + active releases when a new version of the file is published. + + +.. _OpenStack Service Types Authority: https://opendev.org/openstack/service-types-authority/ +.. _ISO Date Time Format: https://tools.ietf.org/html/rfc3339#section-5.6 diff --git a/guidelines/consuming-catalog/endpoint.rst b/guidelines/consuming-catalog/endpoint.rst new file mode 100644 index 0000000..06bc893 --- /dev/null +++ b/guidelines/consuming-catalog/endpoint.rst @@ -0,0 +1,471 @@ +Endpoint Discovery +================== + +Endpoint from Catalog +--------------------- + +The ``{service-catalog}`` can be found in the ``token`` returned from +keystone authentication. + +If v3 auth is used, the catalog will be in the ``catalog`` property of the +top-level ``token`` object. Such as: + +.. code-block:: json + + { + "token": { + "catalog": {} + } + } + +If v2 auth is used it will be in the ``serviceCatalog`` property of the +top-level ``access`` object. Such as: + +.. code-block:: json + + { + "access": { + "serviceCatalog": {} + } + } + +In both cases, the catalog content itself is a list of objects. Each object has +two main keys that concern discovery: + +``type`` + Matches ``{service-type}`` + +``endpoints`` + List of endpoint objects for that service + +Additionally, for backwards compatibility reasons, the following keys may +need to be checked. + +``name`` + Matches ``{service-name}`` + +``id`` + Matches ``{service-id}`` + +The list of endpoints has a different format depending on whether v2 or v3 auth +was used. For both versions each endpoint object has a ``region`` key, +which should match ``{region-name}`` if one was given. + +In v2 auth the endpoint object has three keys ``publicURL``, +``internalURL``, ``adminURL``. The endpoint for the ``{interface}`` requested +by the user is found in the key with the name matching ``{interface}`` plus +the string ``URL``. + +In v3 auth the endpoint object has a ``url`` that is the endpoint that is +being requested if the value of ``interface`` matches ``{interface}``. + +Examples of Tokens with Catalogs +-------------------------------- + +V3 Catalog Objects: + +.. code-block:: json + + { + "token": { + "catalog": [ + { + "endpoints": [ + { + "id": "39dc322ce86c4111b4f06c2eeae0841b", + "interface": "public", + "region": "RegionOne", + "url": "https://identity.example.com" + }, + { + "id": "ec642f27474842e78bf059f6c48f4e99", + "interface": "internal", + "region": "RegionOne", + "url": "https://identity.example.com" + }, + { + "id": "c609fc430175452290b62a4242e8a7e8", + "interface": "admin", + "region": "RegionOne", + "url": "https://identity.example.com" + } + ], + "id": "4363ae44bdf34a3981fde3b823cb9aa2", + "type": "identity", + "name": "keystone" + } + ], + } + +V2 Catalog Objects: + +.. code-block:: json + + { + "access": { + "serviceCatalog": [ + { + "endpoints_links": [], + "endpoints": [ + { + "adminURL": "https://identity.example.com/v2.0", + "region": "RegionOne", + "publicURL": "https://identity.example.com/v2.0", + "internalURL": "https://identity.example.com/v2.0", + "id": "4deb4d0504a044a395d4480741ba628c" + } + ], + "type": "identity", + "name": "keystone" + }, + ] + } + } + +Endpoint Discovery Algorithm +---------------------------- + +#. If ``{endpoint-version}`` was given and ``{service-type}`` ends with a + suffix of ``v[0-9]+$`` and ``{endpoint-version}`` does not match that suffix + (see `Comparing Major Versions`_), STOP. Return an error that the user + has requested a versioned ``{service-type}`` alias and an incompatible + ``{endpoint-version}``. + +#. Find the objects in the ``{service-catalog}`` that match the requested + ``{service-type}`` (see `Match Candidate Entries`_). + +#. If ``{service-name}`` was given and the objects remaining have a ``name`` + field, keep only the ones where ``name`` matches ``{service-name}``. + + .. note:: Catalogs from Keystone v3 before v3.3 do not have a name field. If + ``{be-strict}`` was not requested and the catalog does not have a + ``name`` field, ``{service-name}`` should be ignored. + +#. If ``{service-id}`` was given and the objects remaining have a ``id`` + field, keep only the ones where ``id`` matches ``{service-id}``. + + .. note:: Catalogs from Keystone v2 do not have an id field. If + ``{be-strict}`` was not requested and the catalog does not have a + ``id`` field, ``{service-id}`` should be ignored. + +The list of remaining objects are the ``{candidate-catalog-objects}``. If this +list is empty, return an error that there are no endpoints matching +``{service-type}`` and ``{service-name}``. + +#. Use ``{candidate-catalog-objects}`` to produce the list of + ``{candidate-endpoints}``. For each endpoint object in each of the + ``{candidate-catalog-objects}``: + + #. If v2, if there is no key of the form ``{interface}URL`` for any of the + the ``{interface}`` values given, discard the endpoint. + + #. If v3, if ``interface`` does not match any of the ``{interface}`` values + given, discard the endpoint. + +#. If there are no endpoints left, return an error that there are no endpoints + matching any of the ``{interface}`` values, preferrably including the list + of interfaces that were found. + +#. For each remaining endpoint in ``{candidate-endpoints}``, if + ``{region_name}`` was given and does not match either of ``region`` or + ``region_id``, discard the endpoint. + + If there are no remaining endpoints, return an error that there are no + endpoints matching ``{region_name}``, preferrably including the list of + regions that were found. + +#. From the set of remaining candidate endpoints, find the ones that best + matches the requested ``{service-type}`` (see `Find Endpoint Matching Best + Service Type`_). + +#. From the set of remaining candidate endpoints, find the ones that best + matches the best available requested ``{interface}``: in order of + preference of the ``{interface}`` list, return all endpoints that match + the first ``{interface}`` that has any matching endpoints. + +The remaining ``{candidate-endpoints}`` match the request. If there is more +than one of them, use the first, but emit a warning to the user that more +than one endpoint was left. If ``{be-strict}`` has been requested, return an +error instead with information about each of the endpoints left in the list. + +.. note:: It would be more correct to raise an error if there is more than one + endpoint left, but the keystoneauth library returns the first and + changing that would break a large number of existing users. If one + is writing a completely new library from scratch, or a new major + version where behavior change is acceptable, it is preferable to + raise an error here if there is more than one endpoint left. + +#. If v2, the ``{catalog-endpoint}`` is the value of ``{interface}URL``. + +#. If v3, the ``{catalog-endpoint}`` is the value of ``url``. + +Match Candidate Entries +~~~~~~~~~~~~~~~~~~~~~~~ + +For every entry in the catalog: + +#. If the entry's type matches the requested ``{service-type}``, it is a + candidate. + +#. If the requested type is an official type from the + :doc:`OpenStack Service Types Authority ` that has aliases and + one of the aliases matches the entry's type, it is a candidate. + +#. If the requested type is an alias of an official type from the + :doc:`OpenStack Service Types Authority ` and the entry's type + matches the official type, it is a candidate. + +#. If the requested type is an alias of an official type from the + :doc:`OpenStack Service Types Authority ` that has aliases and + the entry's type matches one of the aliases and ``{endpoint-version}`` was + given and the found alias ends with a suffix of ``v[0-9]+$`` and + ``{endpoint-version}`` matches the version in the suffix (see `Comparing + Major Versions`_) it is a candidate. + +.. _find-endpoint-matching-best-service-type: + +Find Endpoint Matching Best Service Type +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Given a list of candidate endpoints that have matched the other criteria: + +#. Check the list of candidate endpoints to see if one of them matches the + requested ``{service-type}``. If any are an exact match, return them. + +#. If the requested ``{service-type}`` + + * is an official type from the :doc:`OpenStack Service Types Authority + ` that has aliases + * ``{endpoint-version}`` was given + + Look for aliases that end with a version suffix of the form ``v[0-9]+$``. + If there are any aliases with a version suffix that matches the + ``{endpoint-version}`` (see `Comparing Major Versions`_), look for those + aliases in the list of candidate endpoints. If any are a match, return them. + +#. If the requested ``{service-type}`` + + * is an official type in the :doc:`OpenStack Service Types Authority + ` that has aliases + * ``{endpoint-version}`` was not given + + check each alias in the order listed to see if it has a matching endpoint + from the candidate endpoints. Return the endpoints that match the first + alias that has matching endpoints. + +#. If the requested ``{service-type}`` + + * is an alias of an official type in the + :doc:`OpenStack Service Types Authority ` + * ``{endpoint-version}`` was given + + look for aliases that end with a version suffix of the form ``v[0-9]+$``. If + there are any aliases with a version suffix that matches the + ``{endpoint-version}`` (see `Comparing Major Versions`_), look for those + aliases in the list of candidate endpoints. + + Return the endpoints that match the alias with the highest matching version. + +#. If there are no matching endpoints, return an error. + +.. note:: The case where + + * an alias was requested + * no ``{endpoint-version}`` was given + * there is a different alias in the catalog + + is not safe and so is treated as a lack of matching endpoint on + purpose. Many of the aliases carry an implied version, so absent + a requested ``{endpoint-version}`` from the user, returning + an endpoint different than the one explicitly requested has a high + chance of not being the endpoint the user expected. + +.. _comparing-major-versions: + +Comparing Major Versions +~~~~~~~~~~~~~~~~~~~~~~~~ + +When comparing Major Versions, there is a ``required`` and a ``candidate``: + +* The ``required`` is what the user has requested. +* The ``candidate`` is the possible version being tested. + +To be suitable a ``candidate`` must be of the same major version as +``required`` and be at least a match in minor level: ``candidate`` ``3.3`` +is a match for ``required`` ``3.1`` but ``4.1`` is not. + +Leading 'v' strings should be discarded in all cases. + +#. Versions with only a single number normalize to ``.0``. That is, + a version of ``2`` should be treated as if it was ``2.0``. + +#. If ``required`` is the string ``latest`` or contains no value, ``candidate`` + matches. + +#. If ``required`` is a range, any ``candidate`` that is greater than or equal + to the first value and less than or equal to the second value is a match. + Equality is judged by the above rules. Greater than and less than are judged + as expected: first by comparing the first number, and if those match then by + comparing the second number. Thus, a ``{required}`` of ``2,4`` matches + ``2``, ``2.3``, ``3``, ``4`` and ``4.7``. A ``{required}`` of ``2.1,4.0`` + matches ``2.3``, ``3``, ``4`` and ``4.7`` but not ``2``. + +#. If ``required`` is a range without a maximum value, maximum should be + treated as if it is ``latest``. + +Examples of discovery +--------------------- + +For example, given the following catalog: + +.. code-block:: json + + { + "token": { + "catalog": [ + { + "endpoints": [ + { + "interface": "public", + "region": "RegionOne", + "url": "https://block-storage.example.com/v3" + } + ], + "id": "4363ae44bdf34a3981fde3b823cb9aa3", + "type": "volumev3", + "name": "cinder" + }, + { + "endpoints": [ + { + "interface": "public", + "region": "RegionOne", + "url": "https://block-storage.example.com/v2" + } + ], + "id": "4363ae44bdf34a3981fde3b823cb9aa2", + "type": "volumev2", + "name": "cinder" + } + ], + } + +Then the following: + +:: + + service_type = 'block-storage' + # block-storage is not found, get list of aliases + # volumev3 is found, return it + + service_type = 'volumev2' + # volumev2 not an official type in authority, but is in catalog + # return volumev2 entry + + service_type = 'volume' + # volume not in authority or catalog + # volume is an alias of block-storage + # block-storage is not found. Return error. + + service_type = 'volume' + api_version = 2 + # volume not in authority or catalog + # volume is an alias of block-storage + # block-storage is not found. + # volumev2 is an alias of block-storage and ends with v2 which matches + # api_version of 2 + # return volumev2 + +Given the following catalog: + +.. code-block:: json + + { + "token": { + "catalog": [ + { + "endpoints": [ + { + "interface": "public", + "region": "RegionOne", + "url": "https://block-storage.example.com" + } + ], + "id": "4363ae44bdf34a3981fde3b823cb9aa3", + "type": "block-storage", + "name": "cinder" + } + ], + } + +Then the following: + +:: + + service_type = 'block-storage' + # block-storage is found, return it + + service_type = 'volumev2' + # volumev2 not in authority, is an alias for block-storage + # block-storage is in the catalog, return it + + service_type = 'volumev2' + api_version = '3' + # volumev2 ends with a version suffix of v2 which does not match 3 + # return an error before even fetching the catalog + +Given the following catalog: + +.. code-block:: json + + { + "token": { + "catalog": [ + { + "endpoints": [ + { + "interface": "public", + "region": "RegionOne", + "url": "https://block-storage.example.com" + } + ], + "id": "4363ae44bdf34a3981fde3b823cb9aa3", + "type": "block-storage", + "name": "cinder" + }, + { + "endpoints": [ + { + "interface": "public", + "region": "RegionOne", + "url": "https://block-storage.example.com/v2" + }, + { + "interface": "internal", + "region": "RegionOne", + "url": "https://block-storage.example.int/v2" + } + ], + "id": "4363ae44bdf34a3981fde3b823cb9aa2", + "type": "volumev2", + "name": "cinder" + } + ], + } + +Then the following: + +:: + + service_type = 'block-storage' + interface = ['internal', 'public'] + # block-storage is found + # block-storage does not have internal, but has public + # return block-storage public + + service_type = 'volumev2' + interface = ['internal', 'public'] + # volumev2 not an official type in authority, but is in catalog + # volumev2 has an internal interface + # return volumev2 internal entry + diff --git a/guidelines/consuming-catalog/version-discovery.rst b/guidelines/consuming-catalog/version-discovery.rst new file mode 100644 index 0000000..ef2befc --- /dev/null +++ b/guidelines/consuming-catalog/version-discovery.rst @@ -0,0 +1,1068 @@ +================= +Version Discovery +================= + +The topic document on :ref:`discoverability` describes how REST services can +expose version discovery information. However, due to seven years of existence +that pre-date the existence of that document, there are a few non-optimal +setups in the wild. This document describes the complete algorithm to correctly +consume OpenStack version discovery. The intent with this algorithm is that for +all clouds that fully implement :ref:`discoverability` guidelines the path +through the system should be the most efficient, but that process degrades +gracefully for systems that do not yet... ultimately degrading all the way back +to being behaviorally the same as the "just use what's in the catalog" method. + +.. note:: This document contains references to dealing with all known forms + of things encountered in the wild. Where it doesn't distract from the + rest of the description, care is taken to indicate which form is the + preferred form and which are supported for legacy reasons. + Mention of a form in this document should not be construed as + endorsement. Definitions of preferred forms of data will be found + in other documents. + +.. _version-discovery-algorithm: + +Version Discovery Algorithm +=========================== + +The Version Discovery Algorithm is a part of :doc:`../consuming-catalog`. Its +input parameters and return values are a subset of the input parameters and +return values described in :ref:`catalog-user-request`. It is expeced at this +point that the ``{catalog-endpoint}`` is already known, either from the Service +Catalog or directly from ``{endpoint-override}``. + +The algorithm is as follows: + +#. If the user has omitted ``{endpoint-version}``, follow + `User Omitted API Version`_. + +#. Infer the ``{found-endpoint-version}`` from the ``{catalog-endpoint}`` using + the `Inferring Version`_ process. + +#. If ``{found-endpoint-version}`` exists and ``{fetch-version-information}`` + is false, STOP. Return ``{catalog-endpoint}`` as ``{service-endpoint}``. + +#. If the `Inferring Version`_ process returned an error, the + ``{catalog-endpoint}`` does not match the ``{endpoint-version}``. Attempt to + `Find a Document`_. + + .. note:: If the :ref:`discoverability` guidelines have been implemented, + there will always be a ``{discovery-document}``. + +#. If it is not possible to find a ``{discovery-document}`` and ``{be-strict}`` + is true, STOP. Return an error that version discovery has failed. + +#. Determine ``{single-or-multiple}`` for the ``{discovery-document}`` + (see `Single or Multiple Version Documents`_). + + .. note:: If the :ref:`discoverability` guidelines have been implemented, + ``{single-or-multiple}`` will always be ``multiple``. + +At this point, there is a matrix of four possibilities: + +#. If ``{endpoint-version}`` is ``latest`` and ``{single-or-multiple}`` is + ``single``, follow `Latest Single Version`_. + +#. If ``{endpoint-version}`` is ``latest`` and ``{single-or-multiple}`` is + ``multiple``, follow `Latest Multiple Versions`_. + +#. If ``{endpoint-version}`` is a version and ``{single-or-multiple}`` is + ``single``, follow `Requested Single Version`_. + +#. If ``{endpoint-version}`` is a version and ``{single-or-multiple}`` is + ``multiple``, follow `Requested Multiple Versions`_. + +User Omitted API Version +------------------------ + +If the user has omitted the API Version, then the user is indicating that they +want to use the ``{catalog-endpoint}`` as their ``{service-endpoint}``. +Discovery is only run to find out version information about that endpoint. + +#. ``{service-endpoint}`` is ``{catalog-endpoint}``. + +#. If ``{fetch-version-information}`` is false, STOP. Infer the + ``{found-endpoint-version}`` from ``{service-endpoint}``. + (see `Inferring Version`_) + +#. Retrieve ``{discovery-document}`` at ``{service-endpoint}``. + +#. If a ``{discovery-document}`` is found, STOP. Return the + ``{endpoint-information}`` in it (see `Return Information`_). + +#. If there is no ``{discovery-document}``, attempt to `Find a Document`_. + +#. If there is no ``{discovery-document}``, STOP Infer the + ``{found-endpoint-version}`` from ``{service-endpoint}``. + (see `Inferring Version`_) + +#. Determine if the ``{single-or-multiple}`` of the ``{discovery-document}`` is + ``single`` or ``multiple`` (see `Single or Multiple Version Documents`_). + +#. If ``{single-or-multiple}`` is ``single``, STOP. Return the + ``{endpoint-information}`` in it (see `Return Information`_). + +#. If ``{single-or-multiple}`` is ``multiple``, find the + ``{endpoint-information}`` in the ``{discovery-document}`` that matches + ``{service-endpoint}`` (see `Matching Endpoints`_). + +#. If there is no ``{endpoint-information}``, STOP. Infer the + ``{found-endpoint-version}`` from ``{catalog-endpoint}``. + (see `Inferring Version`_) + +#. STOP. Return the information in ``{endpoint-information}`` (see + `Return Information`_). + +Find a Document +--------------- + +In some cases, the ``{discovery-endpoint}`` will either not return a document, +or will not return the document we want, so we need to look for a new one. + +The Unversioned Document is always preferred over the Versioned Document, +because the Unversioned Document supplies the list of possible versions, +allowing Discovery to process the list and make decisions in one step. The +Versioned Document only contains one Version, so additional calls must be +made if the version in it does not match the user's request. + +The algorithm for finding a new document is as follows: + +#. If there is an existing ``{discovery-document}`` and + ``{single-or-multiple}`` is ``multiple``, STOP. There is no better document. + +#. If + + * there is an existing ``{discovery-document}`` + * ``{single-or-multiple}`` is ``single`` + * the ``collection`` link in the links section is different than the + current ``{discovery-endpoint}`` + + make the endpoint at the ``collection`` link the new + ``{discovery-endpoint}`` and fetch a new ``{discovery-document}``. STOP. + Return the new ``{discovery-document}``. + +#. Get the curently scoped ``project_id`` from the ``token``, if one exists. + +#. If the ``{discovery-endpoint}`` ends with a path element that ends with + the ``project_id``, remove that path element and make the resulting URL + the new ``{discovery-endpoint}``. + +#. If the current ``{discovery-endpoint}`` ends with a path element that ends + with a version string of the form "v[0-9]+(\.[0-9]+)?$", remove that path + element but save it as ``{removed-version-path-element}``. Make the + resulting URL the new ``{discovery-endpoint}``. + +#. If the ``{discovery-endpoint}`` matches the ``{catalog-endpoint}``, STOP. + Return an error reporting no working ``{discovery-document}``. + +#. Attempt to fetch a ``{discovery-document}`` from the + ``{discovery-endpoint}``. If one exists, STOP. Normalize it (see + `Normalizing Documents`_) and return it as the ``{dicovery-document}``. + +#. If no new ``{discovery-document}`` can be found at the new endpoint but + there is a saved value in ``{removed-version-path-element}``, append + the ``{removed-version-path-element}`` to the ``{discovery-endpoint}`` and + make the resulting URL the new ``{discovery-endpoint}``. + +#. Attempt to fetch a ``{discovery-document}`` from the + ``{discovery-endpoint}``. If one exists, STOP. Normalize it (see + `Normalizing Documents`_) and return it as the ``{dicovery-document}``. + +#. If no document can be found, return an error reporting no working + ``{discovery-document}``. + +For example: + +.. code-block:: python + + # Given a discovery document from the cloud + original_document = { + "version": { + "status": "SUPPORTED", + "id": "v2.0", + "links": [ + { + "href": "http://compute.example.com/v2/", + "rel": "self" + }, + { + "href": "http://compute.example.com/", + "rel": "collection" + } + ] + } + } + + # It is a single version document + single_or_multiple = 'single' + + # We apply the normalization process + normalized_document = { + "versions": [ + { + "status": "SUPPORTED", + "id": "v2.0", + "min_version": "", + "max_version": "", + "links": [ + { + "href": "http://compute.example.com/v2/", + "rel": "self" + }, + { + "href": "http://compute.example.com/", + "rel": "collection" + } + ] + } + ] + } + + # We see that a collection link exists, so we'll use it as the new discovery + # endpoint. + discovery_endpoint = "http://compute.example.com/" + + # We fetch the document from that endpoint and normalize it. + normalized_better_discovery_document = { + "versions": [ + { + "status": "SUPPORTED", + "links": [ + { + "href": "http://compute.example.com/v2/", + "rel": "self" + } + ], + "min_version": "", + "max_version": "", + "id": "v2.0" + }, { + "status": "CURRENT", + "links": [ + { + "href": "http://compute.example.com/v2.1/", + "rel": "self" + } + ], + "min_version": "2.1", + "max_version": "2.38", + "id": "v2.1" + } + ] + } + + # single-or-multiple is multiple, so it's better + return normalized_better_discovery_document + +Example with project_id: + +.. code-block:: python + + # The user has requested service-type=file-storage + + # The user's token reports the project_id + project_id = '45f0034e8c5a4ef4895b5a87b6b57def' + # The service-catalog contains an entry for filestorage + catalog_endpoint = 'https://file-storage.example.com/v2/45f0034e8c5a4ef4895b5a87b6b57def' + + # The catalog_endpoint ends with the user's project_id, so we pop it. + new_endpoint = 'https://file-storage.example.com/v2' + + # Fetch the document, normalize it and return it + return { + "versions": [ + { + "status": "CURRENT", + "id": "v2.0", + "links": [ + { + "href": "http://file-storage.example.com/v2/", + "rel": "self" + }, + { + "href": "http://file-storage.example.com/", + "rel": "collection" + } + ] + } + ] + } + +More pathological example: + +.. code-block:: python + + # The user has requested service-type=file-storage + + # The user's token reports the project_id + project_id = '45f0034e8c5a4ef4895b5a87b6b57def' + catalog_endpoint = 'https://file-storage.example.com/v2/45f0034e8c5a4ef4895b5a87b6b57def' + + # The catalog_endpoint ends with the user's project_id, so we pop it. + discovery_endpoint = 'https://file-storage.example.com/v2' + + # We try to fetch https://file-storage.example.com/v2 but it returns an error + + # Pop version string from the endpoint + new_discovery_endpoint = 'https://file-storage.example.com/' + + # Fetch the document, normalize and return it + return { + "versions": [ + { + "status": "SUPPORTED", + "links": [ + { + "href": "http://file-storage.example.com/v1/", + "rel": "self" + } + ], + "min_version": "", + "max_version": "", + "id": "v1.0" + }, + { + "status": "CURRENT", + "links": [ + { + "href": "http://file-storage.example.com/v2/", + "rel": "self" + } + ], + "min_version": "2.0", + "max_version": "2.22", + "id": "v2.0" + } + ] + } + +Inferring Version +----------------- + +In most cases the version of the ``{service-endpoint}`` should be retrievable +from the ``{discovery-document}``, and in those cases it should be considered +the version of the service at the ``{service-endpoint}``. In some cases no +discovery document can be found corresponding with the ``{service-endpoint}`` +in question. Alternately, in some cases the ``{catalog-endpoint}`` contains +version information and the user is not looking for microversion information. + +Microversion information will always be empty when this procedure is used. + +The algorithm for inferring the version is as follows: + +#. Get the curently scoped ``project_id`` from the token, if one exists. + +#. If the endpoint ends with a path element that ends with ``project_id``, + remove it. + +#. If the endpoint ends with a path element that is of the form, + ``^v[0-9]+(\.[0-9]+)?$``, strip the ``v`` and use the rest of that element + as ``{found-endpoint-version}``. + +#. If the endpoint contains no version elements, a version cannot be inferred. + Return a null value for ``{found-endpoint-version}``. + +#. If ``{endpoint-version}`` was given and does not match + ``{found-endpoint-version}``, STOP. Return an error that says that user + requested a version and that the version inferred from the URL did not + match. + +#. Return ``{found-endpoint-version}``. + +For example: + +.. code-block:: python + + catalog_endpoint = 'https://file-storage.example.com/v2/45f0034e8c5a4ef4895b5a87b6b57def' + # Match path elements - /v2/ matches ... + found_api_version = '2' + + catalog_endpoint = 'https://identity-storage.example.com/' + # Match path elements - no matches + found_api_version = None + + catalog_endpoint = 'https://object-store.example.com/v1/AUTH_622b11a1-5dfa-43b4-9f58-4ad3c6dbc4a0' + # Match path elements - /v1/ matches ... + found_api_version = '1' + + catalog_endpoint = 'https://compute.example.com/v2.1' + # Match path elements - /v2.1/ matches ... + found_api_version = '2.1' + +Matching Endpoints +------------------ + +If ``{single-or-multiple}`` is ``multiple`` and the discovery algorithm has +chosen to fall back to the endpoint provided by the catalog, a URL matching the +catalog URL should be found so that the version can be extracted. + +#. Sort the endpoints in the ``{discovery-document}`` by ``id`` in descending + order using version comparision. + +#. For each endpoint in the list, expand it (see `Expanding Endpoints`_) + and compare it to the catalog endpoint. The first endpoint that matches is + the winner. + +For example: + +.. code-block:: python + + catalog_endpoint = 'https://file-storage.example.com/v2/45f0034e8c5a4ef4895b5a87b6b57def' + + discovery_document = { + "versions": [ + { + "status": "CURRENT", + "id": "v2.0", + "links": [ + { + "href": "http://file-storage.example.com/v2/", + "rel": "self" + } + ], + } + ] + } + + # Expand endpoint http://file-storage.example.com/v2/ + expanded_endpoint = "https://file-storage.example.com/v2/45f0034e8c5a4ef4895b5a87b6b57def" + + # expanded_endpoint matches catalog_endpoint - id v2.0 is the match + +Expanding Endpoints +------------------- + +Endpoints in discovery documents can be relative and can also be erroneous in +known ways. Before using endpoints from discovery documents, they must be +expanded. The algorithm is as follows: + +#. Join the endpoint from the discovery document with the endpoint the + discovery document was fetched from. If the endpoint in the document is + an absolute url, this should result in the endpoint from the document being + unchanged. If the endpoint from the document is relative, it should be + be appended to the endpoint the document was fetched from following normal + relative URL rules. The python module ``six.moves.urllib.parse.urljoin`` is + an example of an implementation of url joining that behaves as expected. + +#. Replace the ``scheme`` and ``host`` of the endpoint from the discovery + document with the ``scheme`` and ``host`` from the endpoint it was fetched + from. This is to work around older buggy discovery documents seen in the + wild. + + For example: + +.. code-block:: python + + def replace_scheme(endpoint, discovery_url): + parsed_endpoint = urllib.parse.urlparse(endpoint) + parsed_discovery_url = urllib.parse.urlparse(discovery_url) + + return urllib.parse.ParseResult( + parsed_discovery_url.scheme, + parsed_discovery_url.netloc, + parsed_endpoint.path, + parsed_endpoint.params, + parsed_endpoint.query, + parsed_endpoint.fragment).geturl() + +#. Get the curently scoped ``project_id`` from the token, if one exists. + +#. If the ``{catalog-endpoint}`` ends with a path element that ends with + ``project_id`` but the endpoint does not, append the final element of the + path of the ``{catalog-endpoint}`` to the end of the endpoint. + +.. note:: Some services prepend a string to the project_id in their endpoint, + so just appending the project_id to the catalog-endpoint is not + sufficient. + +For example: + +.. code-block:: python + + project_id = '45f0034e8c5a4ef4895b5a87b6b57def' + catalog_endpoint = 'https://file-storage.example.com/v2/45f0034e8c5a4ef4895b5a87b6b57def' + + discovery_document = { + "versions": [ + { + "status": "CURRENT", + "id": "v2.0", + "links": [ + { + "href": "/v2.0", + "rel": "self" + } + ] + } + ] + } + + # Pop project_id from catalog_endpoint + shortened_catalog_endpoint = 'https://file-storage.example.com/v2' + + # Apply URL join to https://file-storage.example.com/v2 and /v2.0 + joined_endpoint = 'https://file-storage.example.com/v2.0' + + # catalog_endpoint ends with project_id, append project_id + service_endpoint = 'http://file-storage.example.com/v2.0/45f0034e8c5a4ef4895b5a87b6b57def' + +With broken service endpoint in discovery: + +.. code-block:: python + + project_id = '45f0034e8c5a4ef4895b5a87b6b57def' + catalog_endpoint = 'https://file-storage.example.com/v2/45f0034e8c5a4ef4895b5a87b6b57def' + + # This discovery_document is the result of a service with a broken + # configuration. Obviously the service is not on "localhost". Similarly, + # since the discovery endpoint is an https endpoint, it can be assumed + # that the actual service endpoint is https. + discovery_document = { + "versions": [ + { + "status": "CURRENT", + "id": "v2.0", + "links": [ + { + "href": "http://localhost/v2.0", + "rel": "self" + } + ], + } + ] + } + + # Pop project_id from catalog_endpoint + shortened_catalog_endpoint = 'https://file-storage.example.com/v2' + + # Apply URL join to https://file-storage.example.com/v2 and + # http://localhost/v2.0 - endpoint from discovery is absolute + joined_endpoint = 'http://localhost/v2.0' + + # Replace scheme and host from https://file-storage.example.com/v2 + joined_endpoint = 'https://file-storage.example.com/v2.0' + + # catalog_endpoint ends with project_id, append project_id + service_endpoint = 'http://file-storage.example.com/v2.0/45f0034e8c5a4ef4895b5a87b6b57def' + +Single or Multiple Version Documents +------------------------------------ + +Even with the version documents normalized as per `Normalizing Documents`_ +into the form described by :ref:`discoverability`, it is still +important to know if the document lists all available versions or only a +single out of a larger set. As it's also possible that there is only one +version, merely looking at the length of the list is not sufficient. + +.. note:: Once all services implement the full recommendations in + :ref:`discoverability` there will never be a document + with a single version out of a larger set, so this logic will not + be needed. However, the logic is upwards compatible with that + desired future state. + +In order to apply the discovery algorithm, the type of document must be +detected. + +* If the document has a link description in the ``links`` list with a ``rel`` + of ``collection`` and the ``href`` of that link is different than the + ``href`` of the link with a ``rel`` of ``self``, then it is a Single + Version Document. + +* Otherwise it is a Multiple Version Document and can be relied on to contain + the complete set of available versions. + +.. note:: TODO(mordred) add examples + +Normalizing Documents +--------------------- + +.. note:: If the API-SIG recommendations in :ref:`discoverability` + are implemented, all of the logic in this section can be skipped. + +There are three forms of existing version discovery documents in addition to +the one that is preferred and described in :ref:`discoverability`. +In order to apply the algorithm sanely, the fetched documents should be +normalized to align with the :ref:`discoverability`. + +.. note:: It is not actually required that normalization take place in a + client or library. It is described here for the purposes of + simplifying other parts of this document and being able to describe + the process in terms of the correct document formats. + +* If the document has a key named ``versions`` which contains a dict with a + key named ``values``, move the list contained in ``values`` to be directly + under ``versions``. That list is then the list of Version objects. + +For example: + +.. code-block:: json + + { + "versions": { + "values": [ + { + "status": "stable", + "updated": "2016-10-06T00:00:00Z", + "id": "v3.7", + "links": [ + { + "href": "https://auth.example.com/v3/", + "rel": "self" + } + ] + }, + { + "status": "deprecated", + "updated": "2016-08-04T00:00:00Z", + "id": "v2.0", + "links": [ + { + "href": "https://auth.example.com/v2.0/", + "rel": "self" + } + ] + } + ] + } + } + +becomes: + +.. code-block:: json + + { + "versions": [ + { + "status": "stable", + "updated": "2016-10-06T00:00:00Z", + "id": "v3.7", + "links": [ + { + "href": "https://auth.example.com/v3/", + "rel": "self" + } + ] + }, + { + "status": "deprecated", + "updated": "2016-08-04T00:00:00Z", + "id": "v2.0", + "links": [ + { + "href": "https://auth.example.com/v2.0/", + "rel": "self" + } + ] + } + ] + } + +* If the document has a key named ``id`` make a key named ``version`` and + place all of the values under it. + +For example: + +.. code-block:: json + + { + "status": "CURRENT", + "id": "v2.0", + "links": [ + { + "href": "http://network.example.com/v2.0", + "rel": "self" + } + ] + } + +becomes: + +.. code-block:: json + + { + "version": { + "status": "CURRENT", + "id": "v2.0", + "links": [ + { + "href": "http://network.example.com/v2.0", + "rel": "self" + } + ] + } + } + +* If the document has a key named ``version``, (even if you just created it) + grab the ``href`` for the link where ``rel`` is ``self`` link. If the + ``href`` ends with with a version string of the form "v[0-9]+(\.[0-9]*)?$", + pop that element from the end of the endpoint and add an entry to the + ``links`` list with a ``rel`` of ``collection`` and the resulting endpoint. + +For example: + +.. code-block:: json + + { + "version": { + "status": "CURRENT", + "id": "v2.0", + "links": [ + { + "href": "http://network.example.com/v2.0", + "rel": "self" + } + ] + } + } + +becomes: + +.. code-block:: json + + { + "version": { + "status": "CURRENT", + "id": "v2.0", + "links": [ + { + "href": "http://network.example.com/v2.0", + "rel": "self" + }, + { + "href": "http://network.example.com/", + "rel": "collection" + } + ] + } + } + +* If the document has a key named ``version``, create a top level key called + ``versions`` that contains a list. Move the contents of ``version`` into + a dict in the ``versions`` list and remove the top level key ``version``. + +For example: + +.. code-block:: json + + { + "version": { + "status": "CURRENT", + "id": "v2.0", + "links": [ + { + "href": "http://network.example.com/v2.0", + "rel": "self" + }, + { + "href": "http://network.example.com/", + "rel": "collection" + } + ] + } + } + +becomes: + +.. code-block:: json + + { + "versions": [ + { + "status": "CURRENT", + "id": "v2.0", + "links": [ + { + "href": "http://network.example.com/v2.0", + "rel": "self" + }, + { + "href": "http://network.example.com/", + "rel": "collection" + } + ] + } + ] + } + +For each Version object in the ``versions`` list: + +#. Keys other than ``id``, ``version``, ``min_version``, ``max_version``, + ``status`` and ``links`` can be ignored or removed. + +#. Convert the value in the ``status`` field to upper case. + +#. If ``status`` is ``STABLE``, change it to ``CURRENT``. (handles keystone) + +#. If there is a ``version`` field and not a ``max_version`` field, make a + ``max_version`` field with the value from the ``version`` field. (handles + nova, cinder, manila and ironic microversions) + +#. The ``links`` key should contain a list, and that list should contain one + dict with ``rel`` equal to ``self`` and additionally may contain a second + dict with ``rel`` equal to ``collection``. Any other entries can be + discarded. + +Some examples of the total normalization follow. + +Original document: + +.. code-block:: json + + { + "versions": [ + { + "status": "stable", + "updated": "2016-10-06T00:00:00Z", + "id": "v3.7", + "links": [ + { + "href": "https://auth.example.com/v3/", + "rel": "self" + } + ] + }, + { + "status": "deprecated", + "updated": "2016-08-04T00:00:00Z", + "id": "v2.0", + "links": [ + { + "href": "https://auth.example.com/v2.0/", + "rel": "self" + } + ] + } + ] + } + +becomes: + +.. code-block:: json + + { + "versions": [ + { + "status": "CURRENT", + "id": "v3.7", + "links": [ + { + "href": "https://auth.example.com/v3/", + "rel": "self" + } + ] + }, + { + "status": "DEPRECATED", + "id": "v2.0", + "links": [ + { + "href": "https://auth.example.com/v2.0/", + "rel": "self" + } + ] + } + ] + } + +Original document: + +.. code-block:: json + + { + "versions": [ + { + "status": "SUPPORTED", + "updated": "2011-01-21T11:33:21Z", + "links": [ + { + "href": "http://compute.example.com/v2/", + "rel": "self" + } + ], + "min_version": "", + "version": "", + "id": "v2.0" + }, + { + "status": "CURRENT", + "updated": "2013-07-23T11:33:21Z", + "links": [ + { + "href": "http://compute.example.com/v2.1/", + "rel": "self" + } + ], + "min_version": "2.1", + "version": "2.38", + "id": "v2.1" + } + ] + } + +becomes: + +.. code-block:: json + + { + "versions": [ + { + "status": "SUPPORTED", + "links": [ + { + "href": "http://compute.example.com/v2/", + "rel": "self" + } + ], + "min_version": "", + "max_version": "", + "id": "v2.0" + }, + { + "status": "CURRENT", + "links": [ + { + "href": "http://compute.example.com/v2.1/", + "rel": "self" + } + ], + "min_version": "2.1", + "max_version": "2.38", + "id": "v2.1" + } + ] + } + +Find Matching Version +===================== + +Finding a version out of a list of endpoint descriptions is done by comparing +``{endpoint-version}`` with the ``id`` field of the description to find a list +of ``{candidate-endpoints}`` (see :ref:`comparing-major-versions`). + +If there is more than one ``{id}`` that matches the requested +``{endpoint-version}`` and one of them has ``status`` of ``CURRENT``, it should +be returned. + +If there is more than one ``{id}`` that matches the requested +``{endpoint-version}`` and none has the ``status`` of ``CURRENT``, the highest +should be returned. + +If there is more than one ``{id}`` that matches the requested +``{endpoint-version}`` and more than one has the ``status`` of ``CURRENT``, the +highest should be returned. + +Latest Single Version +--------------------- + +``{endpoint-version}`` is ``latest`` and ``{single-or-multiple}`` is +``single``. + +#. If ``status`` in the ``{discovery-document}`` is ``CURRENT``, STOP. + Return the ``{endpoint-information}`` in the ``{discovery-document}`` + (see `Return Information`_). + +#. Attempt to `Find a Document`_ + +#. If there is a new ``{discovery-document}`` determine if the + ``{single-or-multiple}`` is ``single`` or ``multiple`` + (see `Single or Multiple Version Documents`_). + +#. If new ``{single-or-multiple}`` is ``multiple``, follow + `Latest Multiple Versions`_. + +#. If new ``{single-or-multiple}`` is ``single``, or there is no new + ``{discovery-document}``, STOP. Return the ``{endpoint-information}`` in + the ``{discovery-document}`` (see `Return Information`_). + +Latest Multiple Versions +------------------------ + +``{endpoint-version}`` is ``latest`` and ``{single-or-multiple}`` is +``multiple``. + +#. Find the ``{endpoint-information}`` in the ``{discovery-document}`` + with the latest version, see `Find Latest Version`_. + +#. When ``{endpoint-information}`` is found, STOP. Return the information in + the ``{endpoint-information}`` (see `Return Information`_). + +Requested Single Version +------------------------ + +``{endpoint-version}`` is a version or range and ``{single-or-multiple}`` is +``single``. + +#. Check to see if the version in the ``{discovery-document}`` matches the + ``{endpoint-version}`` by following `Find Matching Version`_. + +#. Find a matching ``{endpoint-information}`` in the ``{discovery-document}`` + that matches the ``{endpoint-version}``. (see `Find Matching Version`_) + +#. If ``{endpoint-information}`` is found, STOP. Return the information in the + ``{endpoint-information}`` (see `Return Information`_). + +#. If the version does not match, attempt to `Find a Document`_. + +#. If there is a new ``{discovery-document}`` determine if the + ``{single-or-multiple}`` is ``single`` or ``multiple`` + (see `Single or Multiple Version Documents`_). + +#. If the ``{single-or-multiple}`` is ``multiple``, follow + `Requested Multiple Versions`_. + +#. If there is no new ``{discovery-document}``, STOP. Return an error telling + the user their requested version could not be found. Include the version + that was found in the error. + +Requested Multiple Versions +--------------------------- + +``{endpoint-version}`` is a version or range and ``{single-or-multiple}`` is +``multiple``. + +#. Find a matching ``{endpoint-information}`` in the ``{discovery-document}`` + (see `Find Matching Version`_) + +#. If ``{endpoint-information}`` is found, STOP. Return the information in the + ``{endpoint-information}`` (see `Return Information`_). + +#. If no matching ``{endpoint-information}`` is found and + ``{be-strict}`` is ``True``, STOP. Return an error telling the + user their requested version could not be found. Include the list of + versions that were found in the error. + +#. If no matching ``{endpoint-information}`` is found and + ``{be-strict}`` is False, use the ``{catalog-endpoint}`` as the + ``{service-endpoint}``. Find the ``{endpoint-information}`` + in the document that matches the ``{catalog-endpoint}`` and use it. + (see `Matching Endpoints`_). + +#. If there is no ``{endpoint-information}``, STOP. Infer the + ``{found-endpoint-version}`` from the ``{service-endpoint}`` + (see `Inferring Version`_). + +#. STOP. Return the information in ``{endpoint-information}`` (see + `Return Information`_). + +Find Latest Version +------------------- + +If one of the versions in the list has ``status`` of ``CURRENT``, use it. + +Otherwise, select the version with the highest ``id``, excluding any with +``status`` of ``EXPERIMENTAL`` or ``DEPRECATED`` sorted using version +comparison not lexical sorting. + +Return Information +================== + +When endpoint information has been selected, return the information in the +following manner: + +#. Strip the leading "v" from ``{id}`` and return it as + ``{found-endpoint-version}``. + +#. Expand the ``href`` of the entry in ``links`` where ``rel`` is ``self`` + and return it as the ``{service-endpoint}`` (see `Expanding Endpoints`_). + +#. Return ``{min-version}`` and ``{max-version}`` if they exist. diff --git a/guidelines/discoverability.rst b/guidelines/discoverability.rst index 6ea1b7b..647eb39 100644 --- a/guidelines/discoverability.rst +++ b/guidelines/discoverability.rst @@ -8,6 +8,8 @@ API expose the URIs and resources to end users in a machine-readable way. See also the topic document on :ref:`consuming-catalog`. +See also the topic document on :doc:`consuming-catalog/version-discovery`. + Guidance -------- diff --git a/guidelines/microversion_specification.rst b/guidelines/microversion_specification.rst index f5ada22..6c52bbe 100644 --- a/guidelines/microversion_specification.rst +++ b/guidelines/microversion_specification.rst @@ -50,6 +50,8 @@ For example, you cannot request the feature which was introduced at microversion 2.100 without backwards incompatible changes which were introduced in microversion 2.99 and earlier. +.. _microversion-client-interaction: + Client Interaction ------------------