openstack/tempest
Code Issues Proposed changes

General doc updates

Browse Source 
Added missing doc, for example for serial tests, fixed 404 links,
spellings, formatting and etc

Change-Id: I07a8cf9508d3cba33bd04afdc301bb7145153d59
This commit is contained in:
Martin Kopec 2024-01-16 20:43:56 +01:00
parent 01c2e2ff7e
commit d76178e682
16 changed files with 144 additions and 55 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
HACKING.rst
View File

@ -184,7 +184,7 @@ This long runtime has performance implications on `Zuul`_ jobs, which is why
the ``slow`` attribute is leveraged to run slow tests on a selective basis,
to keep total `Zuul`_ job runtime down to a reasonable time frame.
.. _Zuul: https://docs.openstack.org/infra/zuul/
.. _Zuul: https://zuul-ci.org/docs/zuul/latest/
Smoke Attribute
^^^^^^^^^^^^^^^
@ -488,7 +488,7 @@ following procedure::
Otherwise the bug fix won't be able to land in the project.
Handily, `Zuul's cross-repository dependencies
<https://docs.openstack.org/infra/zuul/user/gating.html#cross-project-dependencies>`_.
<https://zuul-ci.org/docs/zuul/latest/gating.html#cross-project-dependencies>`_.
can be leveraged to do without step 2 and to have steps 3 and 4 happen
"atomically". To do that, make the patch written in step 1 to depend (refer to
Zuul's documentation above) on the patch written in step 4. The commit message

1
README.rst
View File

@ -3,7 +3,6 @@ Team and repository tags
========================
.. image:: https://governance.openstack.org/tc/badges/tempest.svg
:target: https://governance.openstack.org/tc/reference/tags/index.html
.. Change things from this point on

19
doc/source/contributor/contributing.rst
View File

@ -14,8 +14,9 @@ with Tempest.
Communication
~~~~~~~~~~~~~
* IRC channel ``#openstack-qa`` at OFTC
* Mailing list (prefix subjects with ``[qa]`` for faster responses)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss
* `Mailing list <https://lists.openstack.org/mailman3/lists/openstack-discuss.lists.openstack.org/>`_
(prefix subjects with ``[qa]`` for faster responses)
Contacting the Core Team
~~~~~~~~~~~~~~~~~~~~~~~~
@ -26,25 +27,29 @@ New Feature Planning
~~~~~~~~~~~~~~~~~~~~
If you want to propose a new feature please read `Feature Proposal Process`_
Tempest features are tracked on `Launchpad BP <https://blueprints.launchpad.net/tempest>`_.
It also helps to bring the feature up during the next PTG and contact
`the current PTL of QA project <https://governance.openstack.org/tc/reference/projects/>`_.
Information about PTG is always posted on `the Mailing list
<https://lists.openstack.org/mailman3/lists/openstack-discuss.lists.openstack.org/>`_.
Task Tracking
~~~~~~~~~~~~~
We track our tasks in `Launchpad <https://bugs.launchpad.net/tempest>`_.
If you're looking for some smaller, easier work item to pick up and get started
If you're looking for some smaller, easier work items to pick up and get started
on, search for the 'low-hanging-fruit' tag.
Reporting a Bug
~~~~~~~~~~~~~~~
You found an issue and want to make sure we are aware of it? You can do so on
`Launchpad <https://bugs.launchpad.net/tempest/+filebug>`__.
Have you found an issue and want to make sure we are aware of it? You can do so
on `Launchpad <https://bugs.launchpad.net/tempest/+filebug>`__.
More info about Launchpad usage can be found on `OpenStack docs page
<https://docs.openstack.org/contributors/common/task-tracking.html#launchpad>`_
Getting Your Patch Merged
~~~~~~~~~~~~~~~~~~~~~~~~~
All changes proposed to the Tempest require single ``Code-Review +2`` votes from
Tempest core reviewers by giving ``Workflow +1`` vote. More detailed guidelines
All changes proposed to the Tempest require a single ``Code-Review +2`` vote
from a Tempest core followed by a ``Workflow +1`` vote. More detailed guidelines
for reviewers are available at :doc:`../REVIEWING`.
Project Team Lead Duties

1
doc/source/index.rst
View File

@ -83,6 +83,7 @@ Development
:maxdepth: 2
HACKING
serial_tests
REVIEWING
microversion_testing
test_removal

4
doc/source/library/api_microversion_testing.rst
View File

@ -9,9 +9,9 @@ Framework to support API Microversion testing
Many of the OpenStack components have implemented API microversions.
It is important to test those microversions in Tempest or external plugins.
Tempest now provides stable interfaces to support to test the API microversions.
Tempest now provides stable interfaces to support testing the API microversions.
Based on the microversion range coming from the combination of both configuration
and each test case, APIs request will be made with selected microversion.
and each test case, APIs requests will be made with the selected microversion.
This document explains the interfaces needed for microversion testing.

4
doc/source/library/clients.rst
View File

@ -6,11 +6,11 @@ Service Clients Usage
Tests make requests against APIs using service clients. Service clients are
specializations of the ``RestClient`` class. The service clients that cover the
APIs exposed by a service should be grouped in a service clients module.
A service clients module is python module where all service clients are
A service clients module is a Python module where all service clients are
defined. If major API versions are available, submodules should be defined,
one for each version.
The ``ClientsFactory`` class helps initializing all clients of a specific
The ``ClientsFactory`` class helps to initialize all clients of a specific
service client module from a set of shared parameters.
The ``ServiceClients`` class provides a convenient way to get access to all

16
doc/source/library/credential_providers.rst
View File

@ -4,12 +4,12 @@ Credential Providers
====================
These library interfaces are used to deal with allocating credentials on demand
either dynamically by calling keystone to allocate new credentials, or from
either dynamically by calling Keystone to allocate new credentials, or from
a list of preprovisioned credentials. These 2 modules are implementations of
the same abstract credential providers class and can be used interchangeably.
However, each implementation has some additional parameters that are used to
influence the behavior of the modules. The API reference at the bottom of this
doc shows the interface definitions for both modules, however that may be a bit
doc shows the interface definitions for both modules, however, that may be a bit
opaque. You can see some examples of how to leverage this interface below.
Initialization Example
@ -30,7 +30,7 @@ on config::
# If a test requires a new account to work, it can have it via forcing
# dynamic credentials. A new account will be produced only for that test.
# In case admin credentials are not available for the account creation,
# the test should be skipped else it would fail.
# the test should be skipped else it will fail.
identity_version = identity_version or CONF.identity.auth_version
if CONF.auth.use_dynamic_credentials or force_tenant_isolation:
admin_creds = get_configured_admin_credentials(
@ -81,12 +81,12 @@ Dealing with Credentials
Once you have a credential provider object created the access patterns for
allocating and removing credentials are the same across both the dynamic
and preprovisioned credentials. These are defined in the abstract
CredentialProvider class. At a high level the credentials provider enables
you to get 3 basic types of credentials at once (per object): a primary, alt,
CredentialProvider class. At a high level, the credentials provider enables
you to get 3 basic types of credentials at once (per object): primary, alt,
and admin. You're also able to allocate a credential by role. These credentials
are tracked by the provider object and delete must manually be called otherwise
the created resources will not be deleted (or returned to the pool in the case
of preprovisioned creds)
are tracked by the provider object and delete must be called manually,
otherwise, the created resources will not be deleted (or returned to the pool
in the case of preprovisioned creds).
Examples
''''''''

38
doc/source/plugins/plugin.rst
View File

@ -80,30 +80,30 @@ Standalone Plugin vs In-repo Plugin
Since all that's required for a plugin to be detected by Tempest is a valid
setuptools entry point in the proper namespace there is no difference from the
Tempest perspective on either creating a separate python package to
house the plugin or adding the code to an existing python project. However,
Tempest perspective on either creating a separate Python package to
house the plugin or adding the code to an existing Python project. However,
there are tradeoffs to consider when deciding which approach to take when
creating a new plugin.
If you create a separate python project for your plugin this makes a lot of
If you create a separate Python project for your plugin this makes a lot of
things much easier. Firstly it makes packaging and versioning much simpler, you
can easily decouple the requirements for the plugin from the requirements for
the other project. It lets you version the plugin independently and maintain a
single version of the test code across project release boundaries (see the
`Branchless Tempest Spec`_ for more details on this). It also greatly
simplifies the install time story for external users. Instead of having to
install the right version of a project in the same python namespace as Tempest
install the right version of a project in the same Python namespace as Tempest
they simply need to pip install the plugin in that namespace. It also means
that users don't have to worry about inadvertently installing a Tempest plugin
when they install another package.
.. _Branchless Tempest Spec: https://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/branchless-tempest.html
The sole advantage to integrating a plugin into an existing python project is
The sole advantage of integrating a plugin into an existing Python project is
that it enables you to land code changes at the same time you land test changes
in the plugin. This reduces some of the burden on contributors by not having
to land 2 changes to add a new API feature and then test it and doing it as a
single combined commit.
to land 2 changes to add a new API feature and then test it, and do it as a
single combined commit instead.
Plugin Class
@ -122,7 +122,7 @@ you would do something like the following:
class MyPlugin(plugins.TempestPlugin):
Then you need to ensure you locally define all of the mandatory methods in the
abstract class, you can refer to the api doc below for a reference of what that
abstract class, you can refer to the API doc below for a reference of what that
entails.
Abstract Plugin Class
@ -214,7 +214,7 @@ Example implementation of ``get_service_clients``:
Parameters:
* **name**: Name of the attribute used to access the ``ClientsFactory`` from
the ``ServiceClients`` instance. See example below.
the ``ServiceClients`` instance. See the example below.
* **service_version**: Tempest enforces a single implementation for each
service client. Available service clients are held in a ``ClientsRegistry``
singleton, and registered with ``service_version``, which means that
@ -229,7 +229,7 @@ Example usage of the service clients in tests:
.. code-block:: python
# my_creds is instance of tempest.lib.auth.Credentials
# my_creds is an instance of tempest.lib.auth.Credentials
# identity_uri is v2 or v3 depending on the configuration
from tempest.lib.services import clients
@ -241,13 +241,13 @@ Automatic configuration and registration of service clients imposes some extra
constraints on the structure of the configuration options exposed by the
plugin.
First ``service_version`` should be in the format `service_config[.version]`.
Firstly, ``service_version`` should be in the format `service_config[.version]`.
The `.version` part is optional, and should only be used if there are multiple
versions of the same API available. The `service_config` must match the name of
a configuration options group defined by the plugin. Different versions of one
API must share the same configuration group.
Second the configuration options group `service_config` must contain the
Secondly, the configuration options group `service_config` must contain the
following options:
* `catalog_type`: corresponds to `service` in the catalog
@ -257,10 +257,10 @@ The following options will be honoured if defined, but they are not mandatory,
as they do not necessarily apply to all service clients.
* `region`: default to identity.region
* `build_timeout` : default to compute.build_timeout
* `build_timeout`: default to compute.build_timeout
* `build_interval`: default to compute.build_interval
Third the service client classes should inherit from ``RestClient``, should
Thirdly, the service client classes should inherit from ``RestClient``, should
accept generic keyword arguments, and should pass those arguments to the
``__init__`` method of ``RestClient``. Extra arguments can be added. For
instance:
@ -276,7 +276,7 @@ instance:
self.my_arg = my_arg
self.my_args2 = my_arg
Finally the service client should be structured in a python module, so that all
Finally, the service client should be structured in a Python module, so that all
service client classes are importable from it. Each major API version should
have its own module.
@ -299,7 +299,7 @@ The content of __init__.py module should be:
__all__ = ['API1Client', 'API2Client']
The following folder and module structure is recommended for multiple major
API version::
API versions::
plugin_dir/
services/
@ -325,14 +325,14 @@ Using Plugins
=============
Tempest will automatically discover any installed plugins when it is run. So by
just installing the python packages which contain your plugin you'll be using
just installing the Python packages, which contain your plugin, you'll be using
them with Tempest, nothing else is really required.
However, you should take care when installing plugins. By their very nature
However, you should take care when installing plugins. By their very nature,
there are no guarantees when running Tempest with plugins enabled about the
quality of the plugin. Additionally, while there is no limitation on running
with multiple plugins, it's worth noting that poorly written plugins might not
properly isolate their tests which could cause unexpected cross interactions
properly isolate their tests which could cause unexpected cross-interactions
between plugins.
Notes for using plugins with virtualenvs

1
doc/source/serial_tests.rst Symbolic link
View File

@ -0,0 +1 @@
../../tempest/serial_tests/README.rst

7
doc/source/tests/modules.rst
View File

@ -19,3 +19,10 @@ OpenStack Services API Tests
network/modules
object_storage/modules
volume/modules
Serial Tests
------------
.. toctree::
:maxdepth: 2
serial_tests/modules

12
tempest/README.rst
View File

@ -8,7 +8,7 @@ OpenStack core projects, being used to validate OpenStack cloud
implementations for both correctness, as well as a burn in tool for
OpenStack clouds.
As such Tempest tests come in many flavors, each with their own rules
As such Tempest tests come in many flavors, each with its own rules
and guidelines. Below is the overview of the Tempest repository structure
to make this clear.
@ -17,6 +17,7 @@ to make this clear.
tempest/
api/ - API tests
scenario/ - complex scenario tests
serial_tests/ - tests that run always in the serial mode
tests/ - unit tests for Tempest internals
Each of these directories contains different types of tests. What
@ -41,13 +42,20 @@ frameworks.
---------------------------
Scenario tests are complex "through path" tests for OpenStack
functionality. They are typically a series of steps where complicated
functionality. They are typically a series of steps where a complicated
state requiring multiple services is set up exercised, and torn down.
Scenario tests should not use the existing Python clients for OpenStack,
but should instead use the Tempest implementations of clients.
:ref:`serial_tests_guide`
--------------------------------
Tests within this category will always be executed serially from the rest of
the test cases.
:ref:`unit_tests_field_guide`
-----------------------------

12
tempest/api/README.rst
View File

@ -7,20 +7,20 @@ Tempest Field Guide to API tests
What are these tests?
---------------------
One of Tempest's prime function is to ensure that your OpenStack cloud
One of Tempest's prime functions is to ensure that your OpenStack cloud
works with the OpenStack API as documented. The current largest
portion of Tempest code is devoted to test cases that do exactly this.
It's also important to test not only the expected positive path on
APIs, but also to provide them with invalid data to ensure they fail
in expected and documented ways. The latter type of tests is called
``negative tests`` in Tempest source code. Over the course of the OpenStack
project Tempest has discovered many fundamental bugs by doing just
``negative tests`` in Tempest source code. Throughout the OpenStack
project, Tempest has discovered many fundamental bugs by doing just
this.
In order for some APIs to return meaningful results, there must be
enough data in the system. This means these tests might start by
spinning up a server, image, etc, then operating on it.
spinning up a server, image, etc., and then operating on it.
Why are these tests in Tempest?
@ -32,7 +32,7 @@ their clouds haven't broken the OpenStack API.
It could be argued that some of the negative testing could be done
back in the projects themselves, and we might evolve there over time,
but currently in the OpenStack gate this is a fundamentally important
but currently, in the OpenStack gate, this is a fundamentally important
place to keep things.
@ -43,7 +43,7 @@ API tests should always use the Tempest implementation of the
OpenStack API, as we want to ensure that bugs aren't hidden by the
official clients.
They should test specific API calls, and can build up complex state if
They should test specific API calls and can build up complex states if
it's needed for the API call to be meaningful.
They should send not only good data, but bad data at the API and look

9
tempest/scenario/README.rst
View File

@ -7,14 +7,14 @@ Tempest Field Guide to Scenario tests
What are these tests?
---------------------
Scenario tests are "through path" tests of OpenStack
function. Complicated setups where one part might depend on completion
Scenario tests are "through path" tests of OpenStack function.
Complicated setups where one part might depend on the completion
of a previous part. They ideally involve the integration between
multiple OpenStack services to exercise the touch points between them.
Any scenario test should have a real-life use case. An example would be:
- "As operator I want to start with a blank environment":
- "As an operator, I want to start with a blank environment":
1. upload a glance image
2. deploy a vm from it
@ -24,12 +24,14 @@ Any scenario test should have a real-life use case. An example would be:
Why are these tests in Tempest?
-------------------------------
This is one of Tempest's core purposes, testing the integration between
projects.
Scope of these tests
--------------------
Scenario tests should always use the Tempest implementation of the
OpenStack API, as we want to ensure that bugs aren't hidden by the
official clients.
@ -40,6 +42,7 @@ determined by which client libraries are used directly by the test.
Example of a good test
----------------------
While we are looking for interaction of 2 or more services, be
specific in your interactions. A giant "this is my data center" smoke
test is hard to debug when it goes wrong.

61
tempest/serial_tests/README.rst Normal file
View File

@ -0,0 +1,61 @@
.. _serial_tests_guide:
Tempest Field Guide to Serial tests
===================================
What are these tests?
---------------------
Tempest can run tests serially as well as in parallel, depending on the
configuration that is fully up to the user. However, sometimes you need to
make sure that tests are not interfering with each other via OpenStack
resources with the other tests running in parallel. Tempest creates separate
projects for each test class to separate project based resources between test
cases.
If your tests use resources outside of projects, e.g. host aggregates then
you might need to explicitly separate interfering test cases. If you only need
to separate a small set of test cases from each other then you can use the
``LockFixture``.
However, in some cases, a small set of tests needs to be run serially. For
example, some of the host aggregate and availability zone testing needs
compute nodes without any running nova server to be able to move compute hosts
between availability zones. But many tempest tests start one or more nova
servers.
Why are these tests in Tempest?
-------------------------------
This is one of Tempest's core purposes, testing the integration between
projects.
Scope of these tests
--------------------
The tests should always use the Tempest implementation of the OpenStack API,
as we want to ensure that bugs aren't hidden by the official clients.
Tests should be tagged with which services they exercise, as
determined by which client libraries are used directly by the test.
Example of a good test
----------------------
While we are looking for interaction of 2 or more services, be specific in
your interactions. A giant "this is my data center" smoke test is hard to
debug when it goes wrong.
The tests that need to be run serially need to be marked with the
``@serial`` class decorator. This will make sure that even if tempest is
configured to run the tests in parallel, these tests will always be executed
separately from the rest of the test cases.
Please note that due to test ordering optimization reasons test cases marked
for ``@serial`` execution need to be put under ``tempest/serial_tests``
directory. This will ensure that the serial tests will block the parallel tests
in the least amount of time.

2
tempest/tests/README.rst
View File

@ -14,6 +14,7 @@ and should be able to run solely from the Tempest tree.
Why are these tests in Tempest?
-------------------------------
These tests exist to make sure that the mechanisms that we use inside of
Tempest are valid and remain functional. They are only here for self
validation of Tempest.
@ -21,6 +22,7 @@ validation of Tempest.
Scope of these tests
--------------------
Unit tests should not require an external service to be running or any extra
configuration to run. Any state that is required for a test should either be
mocked out or created in a temporary test directory. (see test_wrappers.py for

6
tox.ini
View File

@ -359,8 +359,9 @@ commands =
sphinx-apidoc -f -o doc/source/tests/image tempest/api/image
sphinx-apidoc -f -o doc/source/tests/network tempest/api/network
sphinx-apidoc -f -o doc/source/tests/object_storage tempest/api/object_storage
sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario
sphinx-apidoc -f -o doc/source/tests/volume tempest/api/volume
sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario
sphinx-apidoc -f -o doc/source/tests/serial_tests tempest/serial_tests
rm -rf doc/build
sphinx-build -W -b html doc/source doc/build/html
allowlist_externals =
@ -377,8 +378,9 @@ commands =
sphinx-apidoc -f -o doc/source/tests/image tempest/api/image
sphinx-apidoc -f -o doc/source/tests/network tempest/api/network
sphinx-apidoc -f -o doc/source/tests/object_storage tempest/api/object_storage
sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario
sphinx-apidoc -f -o doc/source/tests/volume tempest/api/volume
sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario
sphinx-apidoc -f -o doc/source/tests/serial_tests tempest/serial_tests
sphinx-build -W -b latex doc/source doc/build/pdf
make -C doc/build/pdf