470 lines
16 KiB
ReStructuredText
470 lines
16 KiB
ReStructuredText
=================================
|
|
Microversion Testing With Tempest
|
|
=================================
|
|
|
|
Many OpenStack Services provide their APIs with `microversion`_
|
|
support and want to test them in Tempest.
|
|
|
|
.. _microversion: http://specs.openstack.org/openstack/api-wg/guidelines/microversion_specification.html
|
|
|
|
This document covers how to test microversions for each project and
|
|
whether tests should live in Tempest or on project side.
|
|
|
|
Tempest Scope For Microversion Testing
|
|
""""""""""""""""""""""""""""""""""""""
|
|
APIs microversions for any OpenStack service grow rapidly and
|
|
testing each and every microversion in Tempest is not feasible and
|
|
efficient way.
|
|
Also not every API microversion changes the complete system behavior
|
|
and many of them only change the API or DB layer to accept and return more
|
|
data on API.
|
|
|
|
Tempest is an integration test suite, but not all API microversion testing fall under this category.
|
|
As a result, Tempest mainly covers integration test cases for microversions, Other testing coverage
|
|
for microversion should be hosted on project side as functional tests or via Tempest plugin as per
|
|
project guidelines.
|
|
|
|
.. note:: Integration tests are those tests which involve more than one service to
|
|
verify the expected behavior by single or combination of API requests.
|
|
If a test is just to verify the API behavior as success and failure cases
|
|
or verify its expected response object, then it does not fall under integration
|
|
tests.
|
|
|
|
Tempest will cover only integration testing of applicable microversions with
|
|
below exceptions:
|
|
|
|
#. Test covers a feature which is important for interoperability. This covers tests requirement
|
|
from Defcore.
|
|
#. Test needed to fill Schema gaps.
|
|
Tempest validates API responses with defined JSON schema. API responses can be different on
|
|
each microversion and the JSON schemas need to be defined separately for the microversion.
|
|
While implementing new integration tests for a specific microversion, there
|
|
may be a gap in the JSON schemas (caused by previous microversions) implemented
|
|
in Tempest.
|
|
Filling that gap while implementing the new integration test cases is not efficient due to
|
|
many reasons:
|
|
|
|
* Hard to review
|
|
* Sync between multiple integration tests patches which try to fill the same schema gap at same
|
|
time
|
|
* Might delay the microversion change on project side where project team wants Tempest
|
|
tests to verify the results.
|
|
|
|
Tempest will allow to fill the schema gaps at the end of each cycle, or more
|
|
often if required.
|
|
Schema gap can be filled with testing those with a minimal set of tests. Those
|
|
tests might not be integration tests and might be already covered on project
|
|
side also.
|
|
This exception is needed because:
|
|
|
|
* Allow to create microversion response schema in Tempest at the same time that projects are
|
|
implementing their API microversions. This will make implementation easier for adding
|
|
required tests before a new microversion change can be merged in the corresponding project
|
|
and hence accelerate the development of microversions.
|
|
* New schema must be verified by at least one test case which exercises such schema.
|
|
|
|
For example:
|
|
If any projects implemented 4 API microversion say- v2.3, v2.4, v2.5, v2.6
|
|
Assume microversion v2.3, v2.4, v2.6 change the API Response which means Tempest
|
|
needs to add JSON schema for v2.3, v2.4, v2.6.
|
|
In that case if only 1 or 2 tests can verify all new schemas then we do not need
|
|
separate tests for each new schemas. In worst case, we have to add 3 separate tests.
|
|
#. Test covers service behavior at large scale with involvement of more deep layer like hypervisor
|
|
etc not just API/DB layer. This type of tests will be added case by case basis and
|
|
with project team consultation about why it cannot be covered on project side and worth to test
|
|
in Tempest.
|
|
|
|
Project Scope For Microversion Testing
|
|
""""""""""""""""""""""""""""""""""""""
|
|
All microversions testing which are not covered under Tempest as per above section, should be
|
|
tested on project side as functional tests or as Tempest plugin as per project decision.
|
|
|
|
|
|
Configuration options for Microversion
|
|
""""""""""""""""""""""""""""""""""""""
|
|
|
|
* Add configuration options for specifying test target Microversions.
|
|
We need to specify test target Microversions because the supported
|
|
Microversions may be different between OpenStack clouds. For operating
|
|
multiple Microversion tests in a single Tempest operation, configuration
|
|
options should represent the range of test target Microversions.
|
|
New configuration options are:
|
|
|
|
* min_microversion
|
|
* max_microversion
|
|
|
|
Those should be defined under respective section of each service.
|
|
For example:
|
|
|
|
.. code-block:: ini
|
|
|
|
[compute]
|
|
min_microversion = None
|
|
max_microversion = latest
|
|
|
|
|
|
How To Implement Microversion Tests
|
|
"""""""""""""""""""""""""""""""""""
|
|
|
|
Tempest provides stable interfaces to test API Microversion.
|
|
For Details, see: `API Microversion testing Framework`_
|
|
This document explains how to implement Microversion tests using those
|
|
interfaces.
|
|
|
|
.. _API Microversion testing Framework: https://docs.openstack.org/tempest/latest/library/api_microversion_testing.html
|
|
|
|
|
|
Step1: Add skip logic based on configured Microversion range
|
|
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
|
|
|
|
Add logic to skip the tests based on Tests class and configured Microversion
|
|
range.
|
|
api_version_utils.check_skip_with_microversion function can be used
|
|
to automatically skip the tests which do not fall under configured
|
|
Microversion range.
|
|
For example:
|
|
|
|
.. code-block:: python
|
|
|
|
class BaseTestCase1(api_version_utils.BaseMicroversionTest):
|
|
|
|
[..]
|
|
@classmethod
|
|
def skip_checks(cls):
|
|
super(BaseTestCase1, cls).skip_checks()
|
|
api_version_utils.check_skip_with_microversion(cls.min_microversion,
|
|
cls.max_microversion,
|
|
CONF.compute.min_microversion,
|
|
CONF.compute.max_microversion)
|
|
|
|
Skip logic can be added in tests base class or any specific test class depends on
|
|
tests class structure.
|
|
|
|
Step2: Selected API request microversion
|
|
''''''''''''''''''''''''''''''''''''''''
|
|
|
|
Select appropriate Microversion which needs to be used
|
|
to send with API request.
|
|
api_version_utils.select_request_microversion function can be used
|
|
to select the appropriate Microversion which will be used for API request.
|
|
For example:
|
|
|
|
.. code-block:: python
|
|
|
|
@classmethod
|
|
def resource_setup(cls):
|
|
super(BaseTestCase1, cls).resource_setup()
|
|
cls.request_microversion = (
|
|
api_version_utils.select_request_microversion(
|
|
cls.min_microversion,
|
|
CONF.compute.min_microversion))
|
|
|
|
|
|
Step3: Set Microversion on Service Clients
|
|
''''''''''''''''''''''''''''''''''''''''''
|
|
|
|
Microversion selected by Test Class in previous step needs to be set on
|
|
service clients so that APIs can be requested with selected Microversion.
|
|
|
|
Microversion can be defined as global variable on service clients which
|
|
can be set using fixture.
|
|
Also Microversion header name needs to be defined on service clients which
|
|
should be constant because it is not supposed to be changed by project
|
|
as per API contract.
|
|
For example:
|
|
|
|
.. code-block:: python
|
|
|
|
COMPUTE_MICROVERSION = None
|
|
|
|
class BaseClient1(rest_client.RestClient):
|
|
api_microversion_header_name = 'X-OpenStack-Nova-API-Version'
|
|
|
|
Now test class can set the selected Microversion on required service clients
|
|
using fixture which can take care of resetting the same once tests is completed.
|
|
For example:
|
|
|
|
.. code-block:: python
|
|
|
|
def setUp(self):
|
|
super(BaseTestCase1, self).setUp()
|
|
self.useFixture(api_microversion_fixture.APIMicroversionFixture(
|
|
self.request_microversion))
|
|
|
|
Service clients needs to add set Microversion in API request header which
|
|
can be done by overriding the get_headers() method of rest_client.
|
|
For example:
|
|
|
|
.. code-block:: python
|
|
|
|
COMPUTE_MICROVERSION = None
|
|
|
|
class BaseClient1(rest_client.RestClient):
|
|
api_microversion_header_name = 'X-OpenStack-Nova-API-Version'
|
|
|
|
def get_headers(self):
|
|
headers = super(BaseClient1, self).get_headers()
|
|
if COMPUTE_MICROVERSION:
|
|
headers[self.api_microversion_header_name] = COMPUTE_MICROVERSION
|
|
return headers
|
|
|
|
|
|
Step4: Separate Test classes for each Microversion
|
|
''''''''''''''''''''''''''''''''''''''''''''''''''
|
|
|
|
This is last step to implement Microversion test class.
|
|
|
|
For any Microversion tests, basically we need to implement a
|
|
separate test class. In addition, each test class defines its
|
|
Microversion range with class variable like min_microversion
|
|
and max_microversion. Tests will be valid for that defined range.
|
|
If that range is out of configured Microversion range then, test
|
|
will be skipped.
|
|
|
|
.. note:: Microversion testing is supported at test class level not at
|
|
individual test case level.
|
|
|
|
For example:
|
|
|
|
Below test is applicable for Microversion from 2.2 till 2.9:
|
|
|
|
.. code-block:: python
|
|
|
|
class BaseTestCase1(api_version_utils.BaseMicroversionTest,
|
|
tempest.test.BaseTestCase):
|
|
|
|
[..]
|
|
|
|
|
|
class Test1(BaseTestCase1):
|
|
min_microversion = '2.2'
|
|
max_microversion = '2.9'
|
|
|
|
[..]
|
|
|
|
Below test is applicable for Microversion from 2.10 till latest:
|
|
|
|
.. code-block:: python
|
|
|
|
class Test2(BaseTestCase1):
|
|
min_microversion = '2.10'
|
|
max_microversion = 'latest'
|
|
|
|
[..]
|
|
|
|
|
|
Notes about Compute Microversion Tests
|
|
""""""""""""""""""""""""""""""""""""""
|
|
|
|
Some of the compute Microversion tests have been already implemented
|
|
with the Microversion testing framework. So for further tests only
|
|
step 4 is needed.
|
|
|
|
Along with that JSON response schema might need versioning if needed.
|
|
|
|
Compute service clients strictly validate the response against defined JSON
|
|
schema and does not allow additional elements in response.
|
|
So if that Microversion changed the API response then schema needs to be versioned.
|
|
New JSON schema file needs to be defined with new response attributes and service
|
|
client methods will select the schema based on requested microversion.
|
|
|
|
If Microversion tests are implemented randomly meaning not
|
|
in sequence order(v2.20 tests added and previous Microversion tests are not yet added)
|
|
then, still schema might need to be version for older Microversion if they changed
|
|
the response.
|
|
This is because Nova Microversion includes all the previous Microversions behavior.
|
|
|
|
For Example:
|
|
Implementing the v2.20 Microversion tests before v2.9 and 2.19-
|
|
v2.20 API request will respond as latest behavior of Nova till v2.20,
|
|
and in v2.9 and 2.19, server response has been changed so response schema needs
|
|
to be versioned accordingly.
|
|
|
|
That can be done by using the get_schema method in below module:
|
|
|
|
The base_compute_client module
|
|
''''''''''''''''''''''''''''''
|
|
|
|
.. automodule:: tempest.lib.services.compute.base_compute_client
|
|
:members:
|
|
|
|
|
|
Microversion tests implemented in Tempest
|
|
"""""""""""""""""""""""""""""""""""""""""
|
|
|
|
* Compute
|
|
|
|
* `2.1`_
|
|
|
|
.. _2.1: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id1
|
|
|
|
* `2.2`_
|
|
|
|
.. _2.2: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id2
|
|
|
|
* `2.3`_
|
|
|
|
.. _2.3: http://docs.openstack.org/developer/nova/api_microversion_history.html#maximum-in-kilo
|
|
|
|
* `2.6`_
|
|
|
|
.. _2.6: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id5
|
|
|
|
* `2.8`_
|
|
|
|
.. _2.8: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id7
|
|
|
|
* `2.9`_
|
|
|
|
.. _2.9: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id8
|
|
|
|
* `2.10`_
|
|
|
|
.. _2.10: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id9
|
|
|
|
* `2.19`_
|
|
|
|
.. _2.19: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id17
|
|
|
|
* `2.20`_
|
|
|
|
.. _2.20: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id18
|
|
|
|
* `2.21`_
|
|
|
|
.. _2.21: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id19
|
|
|
|
* `2.25`_
|
|
|
|
.. _2.25: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-mitaka
|
|
|
|
* `2.26`_
|
|
|
|
.. _2.26: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id23
|
|
|
|
* `2.28`_
|
|
|
|
.. _2.28: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id25
|
|
|
|
* `2.32`_
|
|
|
|
.. _2.32: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id29
|
|
|
|
* `2.36`_
|
|
|
|
.. _2.36: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#microversion
|
|
|
|
* `2.37`_
|
|
|
|
.. _2.37: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id35
|
|
|
|
* `2.39`_
|
|
|
|
.. _2.39: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id36
|
|
|
|
* `2.41`_
|
|
|
|
.. _2.41: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id38
|
|
|
|
* `2.42`_
|
|
|
|
.. _2.42: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-ocata
|
|
|
|
* `2.47`_
|
|
|
|
.. _2.47: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id43
|
|
|
|
* `2.48`_
|
|
|
|
.. _2.48: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id44
|
|
|
|
* `2.49`_
|
|
|
|
.. _2.49: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id45
|
|
|
|
* `2.53`_
|
|
|
|
.. _2.53: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-pike
|
|
|
|
* `2.54`_
|
|
|
|
.. _2.54: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id50
|
|
|
|
* `2.55`_
|
|
|
|
.. _2.55: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id51
|
|
|
|
* `2.57`_
|
|
|
|
.. _2.57: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id53
|
|
|
|
* `2.59`_
|
|
|
|
.. _2.59: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id55
|
|
|
|
* `2.60`_
|
|
|
|
.. _2.60: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-queens
|
|
|
|
* `2.61`_
|
|
|
|
.. _2.61: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id56
|
|
|
|
* `2.63`_
|
|
|
|
.. _2.63: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id58
|
|
|
|
* `2.70`_
|
|
|
|
.. _2.70: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id64
|
|
|
|
* `2.71`_
|
|
|
|
.. _2.71: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id65
|
|
|
|
* `2.73`_
|
|
|
|
.. _2.73: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id66
|
|
|
|
* `2.79`_
|
|
|
|
.. _2.79: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-train
|
|
|
|
* Volume
|
|
|
|
* `3.3`_
|
|
|
|
.. _3.3: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id3
|
|
|
|
* `3.9`_
|
|
|
|
.. _3.9: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id9
|
|
|
|
* `3.11`_
|
|
|
|
.. _3.11: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id11
|
|
|
|
* `3.12`_
|
|
|
|
.. _3.12: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id12
|
|
|
|
* `3.13`_
|
|
|
|
.. _3.13: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id13
|
|
|
|
* `3.14`_
|
|
|
|
.. _3.14: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id14
|
|
|
|
* `3.19`_
|
|
|
|
.. _3.19: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id18
|
|
|
|
* `3.20`_
|
|
|
|
.. _3.20: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#id19
|
|
|
|
* `3.55`_
|
|
|
|
.. _3.55: https://docs.openstack.org/cinder/latest/contributor/api_microversion_history.html#maximum-in-rocky
|