openstack/tempest
Code Issues Proposed changes

Document the policy for Tempest & its plugins release

Browse Source 
Tempest and its plugin are very tightly coupled for testing
Upstream or Production Cloud. Due to its branchless model,
one of the challenges we faced/facing is to provide their
compatible version set.

Since Queens release and with operator request, we started
doing a coordinated release of plugins along with Tempest
which will be marked as the First compatible version for
that particular release.

For example https://releases.openstack.org/train/index.html#tempest-plugins

We test our stable branch with master version of Tempest &
plugins until it is moved to Extended Maintainance state.
One stable release is in EM state, Tempest officially declares
the end of support tag so that we can use that as the last compatible
version for that EM stable testing. But we do not have the correct
last compatible version from plugins which can be used for testing
the EM branch. It should be with hash from the master at the time
stable release is moved to EM not the hash when the stable release
was done otherwise it will be incompatible with Tempest.

We have faced and fixed many issue due to that during py2-drop work
where we have to pick and try all different versions from plugin to
make it work against EM testing.

Change-Id: I99a6df0a2c02e19efaf484fcd97b875887f08d8e
This commit is contained in:
Ghanshyam Mann 2020-04-16 16:55:56 -05:00
parent 8f3e503413
commit 235c264457
2 changed files with 63 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
doc/source/index.rst
View File

@ -80,6 +80,14 @@ Plugins
plugin
plugin-registry
Tempest & Plugins Compatible Version Policy
-------------------------------------------
.. toctree::
:maxdepth: 2
tempest_and_plugins_compatible_version_policy
Library
-------
@ -104,4 +112,4 @@ Search
* :ref:`Tempest document search <search>`: Search the contents of this document.
* `OpenStack wide search <https://docs.openstack.org>`_: Search the wider
set of OpenStack documentation, including forums.
set of OpenStack documentation, including forums.

54
doc/source/tempest_and_plugins_compatible_version_policy.rst Normal file
View File

@ -0,0 +1,54 @@
Tempest and Plugins compatible version policy
=============================================
Tempest and its plugins are responsible for the integrated
testing of OpenStack. These tools have two use cases:
#. Testing upstream code at gate
#. Testing Production Cloud
Upstream code is tested by the master version of branchless Tempest & plugins
for all supported stable branches in `Maintained phase`_.
Production Cloud can be tested by using the compatible version or using
master version. It depends on the testing strategy of cloud. To provide
the compatible version of Tempest and its Plugins per OpenStack release,
we started the coordinated release of all plugins and Tempest per OpenStack
release.
These versions are the first set of versions from Tempest and its Plugins to
officially start the support of a particular OpenStack release. For example:
OpenStack Train release first compatible versions `Tempest plugins version`_.
Because of branchless nature of Tempest and its plugins, first version
released during OpenStack release is not the last version to support that
OpenStack release. This means the next (or master) versions can also be used
for upstream testing as well as in production testing.
Since the `Extended Maintenance policy`_ for stable branch, Tempest
started releasing the ``end of support`` version once stable release
is moved to EM state, which used to happen on EOL of stable release. This is
the last compatible version of Tempest for the OpenStack release moved to EM.
Because of branchless nature as explained above, we have a range of versions
which can be considered a compatible version for particular OpenStack release.
How we should release those versions is mentioned in the below table.
+-----------------------------+-----------------+------------------------------------+
| First compatible version -> | OpenStack 'XYZ' | <- Last compatible version |
+=============================+=================+====================================+
|This is the latest version | |This is the version released |
|released when OpenStack | |when OpenStack 'XYZ' is moved to |
|'XYZ' is released. | |EM state. Hash used for this should |
|Example: | |be the hash from master at the time |
|`Tempest plugins version`_ | |of branch is EM not the one used for|
| | |First compatible version |
+-----------------------------+-----------------+------------------------------------+
Tempest & the Plugins should follow the above mentioned policy for the
``First compatible version`` and the ``Last compatible version.``
so that we provide the right set of compatible versions to Upstream as well as to
Production Cloud testing.
.. _Maintained phase: https://docs.openstack.org/project-team-guide/stable-branches.html#maintained
.. _Extended Maintenance policy: https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.html
.. _Tempest plugins version: https://releases.openstack.org/train/#tempest-plugins