235c264457
Tempest and its plugin are very tightly coupled for testing Upstream or Production Cloud. Due to its branchless model, one of the challenges we faced/facing is to provide their compatible version set. Since Queens release and with operator request, we started doing a coordinated release of plugins along with Tempest which will be marked as the First compatible version for that particular release. For example https://releases.openstack.org/train/index.html#tempest-plugins We test our stable branch with master version of Tempest & plugins until it is moved to Extended Maintainance state. One stable release is in EM state, Tempest officially declares the end of support tag so that we can use that as the last compatible version for that EM stable testing. But we do not have the correct last compatible version from plugins which can be used for testing the EM branch. It should be with hash from the master at the time stable release is moved to EM not the hash when the stable release was done otherwise it will be incompatible with Tempest. We have faced and fixed many issue due to that during py2-drop work where we have to pick and try all different versions from plugin to make it work against EM testing. Change-Id: I99a6df0a2c02e19efaf484fcd97b875887f08d8e
3.1 KiB
Tempest and Plugins compatible version policy
Tempest and its plugins are responsible for the integrated testing of OpenStack. These tools have two use cases:
- Testing upstream code at gate
- Testing Production Cloud
Upstream code is tested by the master version of branchless Tempest & plugins for all supported stable branches in Maintained phase.
Production Cloud can be tested by using the compatible version or using master version. It depends on the testing strategy of cloud. To provide the compatible version of Tempest and its Plugins per OpenStack release, we started the coordinated release of all plugins and Tempest per OpenStack release. These versions are the first set of versions from Tempest and its Plugins to officially start the support of a particular OpenStack release. For example: OpenStack Train release first compatible versions Tempest plugins version.
Because of branchless nature of Tempest and its plugins, first version released during OpenStack release is not the last version to support that OpenStack release. This means the next (or master) versions can also be used for upstream testing as well as in production testing.
Since the Extended
Maintenance policy for stable branch, Tempest started releasing the
end of support version once stable release is moved to EM
state, which used to happen on EOL of stable release. This is the last
compatible version of Tempest for the OpenStack release moved to EM.
Because of branchless nature as explained above, we have a range of versions which can be considered a compatible version for particular OpenStack release. How we should release those versions is mentioned in the below table.
First compatible version -> OpenStack 'XYZ' <- Last compatible version This is the latest version released when OpenStack 'XYZ' is released. Example: Tempest plugins version This is the version released when OpenStack 'XYZ' is moved to EM state. Hash used for this should be the hash from master at the time of branch is EM not the one used for First compatible version
Tempest & the Plugins should follow the above mentioned policy
for the First compatible version and the
Last compatible version. so that we provide the right set
of compatible versions to Upstream as well as to Production Cloud
testing.