Merge "Microversions documentation"

barbican/api/controllers/versions.py

@@ -34,6 +34,9 @@ _LAST_UPDATED = '2021-02-10T00:00:00Z'
 # across all of the v1 REST API.
 # When introducing a new microversion, the _MAX_MICROVERSION
 # needs to be incremented by 1 and the _LAST_UPDATED string updated.
+# Additionally, the new microversion has to be documented in
+# doc/source/api/microversion_history.rst
+#
 # The following is the complete (ordered) list of supported versions
 # used by the microversion middleware to parse what is allowed and
 # supported.

bindep.txt

@@ -5,6 +5,7 @@ nss-devel [platform:rpm]
 libnss3-dev [platform:dpkg]
 
 gettext [test]
+graphviz [doc test]
 
 # Required for the Dogtag plugin
 # Comment out for now -- these are not installing due to need to

doc/source/api/index.rst

@@ -5,6 +5,9 @@ Barbican API Documentation
 User Guide
 ##########
 
+The OpenStack Key Manager API version 1.0 supports microversions.
+See `doc/source/api/microversions.rst` for details.
+
 API guide docs are built to:
 https://docs.openstack.org/api-guide/key-manager/
 
@@ -23,3 +26,6 @@ API Reference
     ./reference/quotas.rst
     ./reference/consumers.rst
     ./reference/orders.rst
+    ./microversions.rst
+    ./microversion_history.rst
+

doc/source/api/microversion_history.rst

@@ -0,0 +1,30 @@
+REST API Version History
+========================
+
+This documents the changes made to the REST API with every
+microversion change. The description for each version should be a
+verbose one which has enough information to be suitable for use in
+user documentation.
+
+1.0
+---
+
+This is the initial version of the v1.0 API which supports
+microversions.
+
+A user can specify a header in the API request::
+
+  OpenStack-API-Version: key-manager <version>
+
+where ``<version>`` is any valid api version for this API.
+
+If no version is specified then the API will behave as if a version
+request of v1.0 was requested.
+
+1.1 (Maximum in Wallaby)
+---
+
+Added Secret Consumers to Secrets.
+
+When requesting Secrets (individual Secret or a list), the results contain an
+additional ``consumers`` key, which contains references to Secret Consumers.

doc/source/api/microversions.rst

@@ -0,0 +1,102 @@
+=============
+Microversions
+=============
+
+API v1.0 supports microversions: small, documented changes to the API. A user
+can use microversions to discover the latest API microversion supported in
+their cloud. A cloud that is upgraded to support newer microversions will
+still support all older microversions to maintain the backward compatibility
+for those users, who depend on older microversions. Users can also discover
+new features easily with microversions, so that they can benefit from all the
+advantages and improvements of the current cloud.
+
+There are multiple cases which you can resolve with microversions:
+
+- **Older clients with new cloud**
+
+Before using an old client to talk to a newer cloud, the old client can check
+the minimum version of microversions to verify whether the cloud is compatible
+with the old API. This prevents the old client from breaking with backwards
+incompatible API changes.
+
+Currently the minimum version of microversions is `1.0`, which is a
+microversion compatible with the legacy v1 API. That means the legacy v1 API
+user doesn't need to worry that their older client software will be broken
+when their cloud is upgraded with new versions. The cloud operator doesn't
+need to worry that upgrading their cloud to newer versions will break any
+user with older clients that don't expect these changes.
+
+- **User discovery of available features between clouds**
+
+The new features can be discovered by microversions. The user client should
+first check the microversions supported by the server. New features are only
+enabled when clouds support it. In this way, the user client can work with
+clouds that have deployed different microversions simultaneously.
+
+Version Discovery
+=================
+
+The Version API will return the minimum and maximum microversions. These
+values are used by the client to discover the API's supported microversion(s).
+
+Requests to '/' will get version info for all endpoints. A response would look
+as follows::
+
+  {
+    "versions": [
+        {
+            "id": "v1.0",
+            "links": [
+                {
+                    "href": "http://openstack.example.com/v1/",
+                    "rel": "self"
+                }
+            ],
+            "max_version": "1.1",
+            "min_version": "1.0",
+            "updated": "2021-02-10T00:00:00Z"
+        }
+    ]
+  }
+
+"max_version" is the maximum microversion, "min_version" is the minimum
+microversion. The client should specify a microversion between
+(and including) the minimum and maximum microversion to access the endpoint.
+
+Client Interaction
+==================
+
+A client specifies the microversion of the API they want by using the following HTTP header::
+
+  OpenStack-API-Version: key-manager 1.1
+
+.. note:: For more detail on the syntax see the `Microversion Specification
+   <http://specs.openstack.org/openstack/api-wg/guidelines/microversion_specification.html>`_.
+
+This acts conceptually like the "Accept" header. Semantically this means:
+
+* If `OpenStack-API-Version` (specifying `key-manager`) is provided, act as
+  if the minimum supported microversion was specified.
+
+* If `OpenStack-API-Version` is provided, respond with the API at
+  that microversion. If that's outside of the range
+  of microversions supported, return 406 Not Acceptable.
+
+* `OpenStack-API-Version` has a value of ``latest`` (special keyword),
+  act as if maximum was specified.
+
+.. warning:: The ``latest`` value is mostly meant for integration testing and
+  would be dangerous to rely on in client code since microversions are not
+  following semver and therefore backward compatibility is not guaranteed.
+  Clients should always require a specific microversion but limit what is
+  acceptable to the microversion range that it understands at the time.
+
+This means that out of the box, an old client without any knowledge of
+microversions can work with an OpenStack installation with microversions
+support.
+
+From microversion `1.1` two additional headers are added to the
+response::
+
+    OpenStack-API-Version: key-manager microversion_number
+    Vary: OpenStack-API-Version

doc/source/conf.py

@@ -21,6 +21,7 @@ sys.path.insert(0, os.path.abspath('../..'))
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = [
     'sphinx.ext.autodoc',
+    'sphinx.ext.graphviz',
     # 'sphinx.ext.intersphinx',
     'openstackdocstheme',
     'oslo_config.sphinxext',

doc/source/contributor/index.rst

@@ -28,5 +28,6 @@ When you're ready to dive deeper in to barbican take a look at:
    dataflow.rst
    dependencies.rst
    database_migrations.rst
+   microversions.rst
    plugin/index.rst
    testing.rst

doc/source/contributor/microversions.rst

@@ -0,0 +1,275 @@
+API Microversions
+=================
+
+Background
+----------
+
+Barbican uses a framework we call 'API Microversions' for allowing changes
+to the API while preserving backward compatibility. The basic idea is
+that a user has to explicitly ask for their request to be treated with
+a particular version of the API. So breaking changes can be added to
+the API without breaking users who don't specifically ask for it. This
+is done with an HTTP header ``OpenStack-API-Version`` which has as its
+value a string containing the name of the service, ``key-manager``, and a
+monotonically increasing semantic version number starting from ``1.0``.
+The full form of the header takes the form::
+
+    OpenStack-API-Version: key-manager 1.1
+
+If a user makes a request without specifying a version, they will get
+the ``MIN_API_VERSION`` as calculated from the defined _MIN_MICROVERSION in
+``barbican/api/controllers/versions.py``.  This value is currently ``1.0`` and
+is expected to remain so for quite a long time.
+
+There is a special value ``latest`` which can be specified, which will
+allow a client to always receive the most recent version of API
+responses from the server.
+
+.. warning:: The ``latest`` value is mostly meant for integration testing and
+  would be dangerous to rely on in client code since microversions are not
+  following semver and therefore backward compatibility is not guaranteed.
+  Clients, like python-barbicanclient, should always require a specific
+  microversion but limit what is acceptable to the version range that it
+  understands at the time.
+
+For full details please read the `Microversion Specification
+<http://specs.openstack.org/openstack/api-wg/guidelines/microversion_specification.html>`_.
+
+When do I need a new Microversion?
+----------------------------------
+
+A microversion is needed when the contract to the user is
+changed. The user contract covers many kinds of information such as:
+
+- the Request
+
+  - the list of resource urls which exist on the server
+
+    Example: adding a new servers/{ID}/foo which didn't exist in a
+    previous version of the code
+
+  - the list of query parameters that are valid on urls
+
+    Example: adding a new parameter ``is_yellow`` servers/{ID}?is_yellow=True
+
+  - the list of query parameter values for non free form fields
+
+    Example: parameter filter_by takes a small set of constants/enums "A",
+    "B", "C". Adding support for new enum "D".
+
+  - new headers accepted on a request
+
+  - the list of attributes and data structures accepted.
+
+    Example: adding a new attribute 'consumer': '...' to the request body
+
+- the Response
+
+  - the list of attributes and data structures returned
+
+    Example: adding a new attribute 'consumers': [] to the output
+    of secrets/{ID}
+
+  - the allowed values of non free form fields
+
+    Example: adding a new allowed ``secret_type`` to secrets/{ID}
+
+  - the list of status codes allowed for a particular request
+
+    Example: an API previously could return 200, 400, 403, 404 and the
+    change would make the API now also be allowed to return 409.
+
+    See [#f2]_ for the 400, 403, 404 and 415 cases.
+
+  - changing a status code on a particular response
+
+    Example: changing the return code of an API from 501 to 400.
+
+    .. note:: Fixing a bug so that a 400+ code is returned rather than a 500 or
+      503 does not require a microversion change. It's assumed that clients are
+      not expected to handle a 500 or 503 response and therefore should not
+      need to opt-in to microversion changes that fixes a 500 or 503 response
+      from happening.
+      According to the OpenStack API Working Group, a
+      **500 Internal Server Error** should **not** be returned to the user for
+      failures due to user error that can be fixed by changing the request on
+      the client side. See [#f1]_.
+
+  - new headers returned on a response
+
+The following flow chart attempts to walk through the process of "do
+we need a microversion".
+
+
+.. graphviz::
+
+   digraph states {
+
+    label="Do I need a microversion?"
+
+    silent_fail[shape="diamond", style="", group=g1, label="Did we silently
+   fail to do what is asked?"];
+    ret_500[shape="diamond", style="", group=g1, label="Did we return a 500
+   before?"];
+    new_error[shape="diamond", style="", group=g1, label="Are we changing what
+    status code is returned?"];
+    new_attr[shape="diamond", style="", group=g1, label="Did we add or remove an
+    attribute to a payload?"];
+    new_param[shape="diamond", style="", group=g1, label="Did we add or remove
+    an accepted query string parameter or value?"];
+    new_resource[shape="diamond", style="", group=g1, label="Did we add or remove a
+   resource url?"];
+
+
+   no[shape="box", style=rounded, label="No microversion needed"];
+   yes[shape="box", style=rounded, label="Yes, you need a microversion"];
+   no2[shape="box", style=rounded, label="No microversion needed, it's
+   a bug"];
+
+   silent_fail -> ret_500[label=" no"];
+   silent_fail -> no2[label="yes"];
+
+    ret_500 -> no2[label="yes [1]"];
+    ret_500 -> new_error[label=" no"];
+
+    new_error -> new_attr[label=" no"];
+    new_error -> yes[label="yes"];
+
+    new_attr -> new_param[label=" no"];
+    new_attr -> yes[label="yes"];
+
+    new_param -> new_resource[label=" no"];
+    new_param -> yes[label="yes"];
+
+    new_resource -> no[label=" no"];
+    new_resource -> yes[label="yes"];
+
+   {rank=same; yes new_attr}
+   {rank=same; no2 ret_500}
+   {rank=min; silent_fail}
+   }
+
+
+**Footnotes**
+
+.. [#f1] When fixing 500 errors that previously caused stack traces, try
+  to map the new error into the existing set of errors that API call
+  could previously return (400 if nothing else is appropriate). Changing
+  the set of allowed status codes from a request is changing the
+  contract, and should be part of a microversion (except in [#f2]_).
+
+  The reason why we are so strict on contract is that we'd like
+  application writers to be able to know, for sure, what the contract is
+  at every microversion in Barbican. If they do not, they will need to write
+  conditional code in their application to handle ambiguities.
+
+  When in doubt, consider application authors. If it would work with no
+  client side changes on both Barbican versions, you probably don't need a
+  microversion. If, on the other hand, there is any ambiguity, a
+  microversion is probably needed.
+
+.. [#f2] The exception to not needing a microversion when returning a
+  previously unspecified error code is the 400, 403, 404 and 415 cases. This is
+  considered OK to return even if previously unspecified in the code since
+  it's implied given keystone authentication can fail with a 403 and API
+  validation can fail with a 400 for invalid json request body. Request to
+  url/resource that does not exist always fails with 404. Invalid content types
+  are handled before API methods are called which results in a 415.
+
+
+When a microversion is not needed
+---------------------------------
+
+A microversion is not needed in the following situation:
+
+- the response
+
+  - Changing the error message without changing the response code
+    does not require a new microversion.
+
+  - Removing an inapplicable HTTP header, for example, suppose the Retry-After
+    HTTP header is being returned with a 4xx code. This header should only be
+    returned with a 503 or 3xx response, so it may be removed without bumping
+    the microversion.
+
+  - An obvious regression bug in an admin-only API where the bug can still
+    be fixed upstream on active stable branches. Admin-only APIs are less of
+    a concern for interoperability and generally a regression in behavior can
+    be dealt with as a bug fix when the documentation clearly shows the API
+    behavior was unexpectedly regressed. See [#f3]_ for an example from Nova.
+    Intentional behavior changes to an admin-only API *do* require a
+    microversion.
+
+**Footnotes**
+
+.. [#f3] https://review.opendev.org/#/c/523194/
+
+In Code
+-------
+
+In ``barbican/api/controllers/versions.py`` we define the ``is_supported``
+function which is intended to be used in Controller methods to check if API
+request version satisfies version restrictions. The function accepts
+``min_version`` and ``max_version`` arguments, and returns ``True`` when the
+requested version meets those constrainst.
+
+.. note:: Originally Nova also implemented a decorator API, but it frequently
+  lead to code duplication. In Barbican it was decided to limit the
+  microversion API to just the ``is_supported`` function.
+
+
+If you are adding a patch which adds a new microversion, it is
+necessary to add changes to other places which describe your change:
+
+* Update ``_MAX_MICROVERSION`` and bump ``_LAST_UPDATED`` in
+  ``barbican/api/controllers/versions.py``
+
+* Add a verbose description to
+  ``doc/source/api/microversion_history.rst``.
+
+* Add a release note with a ``features`` section announcing the new or
+  changed feature and the microversion.
+
+* Update the expected versions in affected tests, add new tests to test
+  both the old and new behavior to avoid regressions.
+
+* Make a new commit to python-barbicanclient and update corresponding
+  files to enable the newly added microversion API.
+
+* If the microversion changes the response schema, a new schema and test for
+  the microversion must be added to Tempest.
+
+* Update the `API Reference`_ documentation as appropriate.  The source is
+  located under `doc/source/api/reference/`.
+
+.. _API Reference: https://docs.openstack.org/api-ref/key-manager/
+
+Allocating a microversion
+-------------------------
+
+If you are adding a patch which adds a new microversion, it is
+necessary to allocate the next microversion number. Except under
+extremely unusual circumstances and this would have been mentioned in
+the barbican spec for the change, the ``_MAX_MICROVERSION`` will be
+incremented. This will also be the new minor version number for the API
+change.
+
+It is possible that multiple microversion patches would be proposed in
+parallel and the microversions would conflict between patches.  This
+will cause a merge conflict. We don't reserve a microversion for each
+patch in advance as we don't know the final merge order. Developers
+may need over time to rebase their patch calculating a new version
+number as above based on the updated value of ``_MAX_MICROVERSION``.
+
+Testing Microversioned API Methods
+----------------------------------
+
+Testing a microversioned API method is very similar to a normal controller
+method test, you just need to add the ``OpenStack-API-Version`` header
+For unit tests, 'barbican.test.utils.set_version' function can be used,
+for example::
+
+    def test_should_get_secret_as_json_v1(self):
+        utils.set_version(self.app, '1.1')
+        secret = self._test_should_get_secret_as_json()
+        self.assertIn('consumers', secret)