diff --git a/HACKING.rst b/HACKING.rst index caf954b4fa..23bc61ba6f 100644 --- a/HACKING.rst +++ b/HACKING.rst @@ -184,7 +184,7 @@ This long runtime has performance implications on `Zuul`_ jobs, which is why the ``slow`` attribute is leveraged to run slow tests on a selective basis, to keep total `Zuul`_ job runtime down to a reasonable time frame. -.. _Zuul: https://docs.openstack.org/infra/zuul/ +.. _Zuul: https://zuul-ci.org/docs/zuul/latest/ Smoke Attribute ^^^^^^^^^^^^^^^ @@ -488,7 +488,7 @@ following procedure:: Otherwise the bug fix won't be able to land in the project. Handily, `Zuul's cross-repository dependencies -`_. +`_. can be leveraged to do without step 2 and to have steps 3 and 4 happen "atomically". To do that, make the patch written in step 1 to depend (refer to Zuul's documentation above) on the patch written in step 4. The commit message diff --git a/README.rst b/README.rst index 3cde2bf376..788035797d 100644 --- a/README.rst +++ b/README.rst @@ -3,7 +3,6 @@ Team and repository tags ======================== .. image:: https://governance.openstack.org/tc/badges/tempest.svg - :target: https://governance.openstack.org/tc/reference/tags/index.html .. Change things from this point on diff --git a/doc/source/contributor/contributing.rst b/doc/source/contributor/contributing.rst index 139f0b758e..81a1874653 100644 --- a/doc/source/contributor/contributing.rst +++ b/doc/source/contributor/contributing.rst @@ -14,8 +14,9 @@ with Tempest. Communication ~~~~~~~~~~~~~ * IRC channel ``#openstack-qa`` at OFTC -* Mailing list (prefix subjects with ``[qa]`` for faster responses) - http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss +* `Mailing list `_ + (prefix subjects with ``[qa]`` for faster responses) + Contacting the Core Team ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -26,25 +27,29 @@ New Feature Planning ~~~~~~~~~~~~~~~~~~~~ If you want to propose a new feature please read `Feature Proposal Process`_ Tempest features are tracked on `Launchpad BP `_. +It also helps to bring the feature up during the next PTG and contact +`the current PTL of QA project `_. +Information about PTG is always posted on `the Mailing list +`_. Task Tracking ~~~~~~~~~~~~~ We track our tasks in `Launchpad `_. -If you're looking for some smaller, easier work item to pick up and get started +If you're looking for some smaller, easier work items to pick up and get started on, search for the 'low-hanging-fruit' tag. Reporting a Bug ~~~~~~~~~~~~~~~ -You found an issue and want to make sure we are aware of it? You can do so on -`Launchpad `__. +Have you found an issue and want to make sure we are aware of it? You can do so +on `Launchpad `__. More info about Launchpad usage can be found on `OpenStack docs page `_ Getting Your Patch Merged ~~~~~~~~~~~~~~~~~~~~~~~~~ -All changes proposed to the Tempest require single ``Code-Review +2`` votes from -Tempest core reviewers by giving ``Workflow +1`` vote. More detailed guidelines +All changes proposed to the Tempest require a single ``Code-Review +2`` vote +from a Tempest core followed by a ``Workflow +1`` vote. More detailed guidelines for reviewers are available at :doc:`../REVIEWING`. Project Team Lead Duties diff --git a/doc/source/index.rst b/doc/source/index.rst index 2f29cf2b42..0340f8da9e 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -83,6 +83,7 @@ Development :maxdepth: 2 HACKING + serial_tests REVIEWING microversion_testing test_removal diff --git a/doc/source/library/api_microversion_testing.rst b/doc/source/library/api_microversion_testing.rst index 8be924d0a3..e97968385e 100644 --- a/doc/source/library/api_microversion_testing.rst +++ b/doc/source/library/api_microversion_testing.rst @@ -9,9 +9,9 @@ Framework to support API Microversion testing Many of the OpenStack components have implemented API microversions. It is important to test those microversions in Tempest or external plugins. -Tempest now provides stable interfaces to support to test the API microversions. +Tempest now provides stable interfaces to support testing the API microversions. Based on the microversion range coming from the combination of both configuration -and each test case, APIs request will be made with selected microversion. +and each test case, APIs requests will be made with the selected microversion. This document explains the interfaces needed for microversion testing. diff --git a/doc/source/library/clients.rst b/doc/source/library/clients.rst index 0f4ba4c1a8..fe9f4ca816 100644 --- a/doc/source/library/clients.rst +++ b/doc/source/library/clients.rst @@ -6,11 +6,11 @@ Service Clients Usage Tests make requests against APIs using service clients. Service clients are specializations of the ``RestClient`` class. The service clients that cover the APIs exposed by a service should be grouped in a service clients module. -A service clients module is python module where all service clients are +A service clients module is a Python module where all service clients are defined. If major API versions are available, submodules should be defined, one for each version. -The ``ClientsFactory`` class helps initializing all clients of a specific +The ``ClientsFactory`` class helps to initialize all clients of a specific service client module from a set of shared parameters. The ``ServiceClients`` class provides a convenient way to get access to all diff --git a/doc/source/library/credential_providers.rst b/doc/source/library/credential_providers.rst index d25f85c674..8c9a16aa2f 100644 --- a/doc/source/library/credential_providers.rst +++ b/doc/source/library/credential_providers.rst @@ -4,12 +4,12 @@ Credential Providers ==================== These library interfaces are used to deal with allocating credentials on demand -either dynamically by calling keystone to allocate new credentials, or from +either dynamically by calling Keystone to allocate new credentials, or from a list of preprovisioned credentials. These 2 modules are implementations of the same abstract credential providers class and can be used interchangeably. However, each implementation has some additional parameters that are used to influence the behavior of the modules. The API reference at the bottom of this -doc shows the interface definitions for both modules, however that may be a bit +doc shows the interface definitions for both modules, however, that may be a bit opaque. You can see some examples of how to leverage this interface below. Initialization Example @@ -30,7 +30,7 @@ on config:: # If a test requires a new account to work, it can have it via forcing # dynamic credentials. A new account will be produced only for that test. # In case admin credentials are not available for the account creation, - # the test should be skipped else it would fail. + # the test should be skipped else it will fail. identity_version = identity_version or CONF.identity.auth_version if CONF.auth.use_dynamic_credentials or force_tenant_isolation: admin_creds = get_configured_admin_credentials( @@ -81,12 +81,12 @@ Dealing with Credentials Once you have a credential provider object created the access patterns for allocating and removing credentials are the same across both the dynamic and preprovisioned credentials. These are defined in the abstract -CredentialProvider class. At a high level the credentials provider enables -you to get 3 basic types of credentials at once (per object): a primary, alt, +CredentialProvider class. At a high level, the credentials provider enables +you to get 3 basic types of credentials at once (per object): primary, alt, and admin. You're also able to allocate a credential by role. These credentials -are tracked by the provider object and delete must manually be called otherwise -the created resources will not be deleted (or returned to the pool in the case -of preprovisioned creds) +are tracked by the provider object and delete must be called manually, +otherwise, the created resources will not be deleted (or returned to the pool +in the case of preprovisioned creds). Examples '''''''' diff --git a/doc/source/plugins/plugin.rst b/doc/source/plugins/plugin.rst index 07713180d9..31aa1349e7 100644 --- a/doc/source/plugins/plugin.rst +++ b/doc/source/plugins/plugin.rst @@ -80,30 +80,30 @@ Standalone Plugin vs In-repo Plugin Since all that's required for a plugin to be detected by Tempest is a valid setuptools entry point in the proper namespace there is no difference from the -Tempest perspective on either creating a separate python package to -house the plugin or adding the code to an existing python project. However, +Tempest perspective on either creating a separate Python package to +house the plugin or adding the code to an existing Python project. However, there are tradeoffs to consider when deciding which approach to take when creating a new plugin. -If you create a separate python project for your plugin this makes a lot of +If you create a separate Python project for your plugin this makes a lot of things much easier. Firstly it makes packaging and versioning much simpler, you can easily decouple the requirements for the plugin from the requirements for the other project. It lets you version the plugin independently and maintain a single version of the test code across project release boundaries (see the `Branchless Tempest Spec`_ for more details on this). It also greatly simplifies the install time story for external users. Instead of having to -install the right version of a project in the same python namespace as Tempest +install the right version of a project in the same Python namespace as Tempest they simply need to pip install the plugin in that namespace. It also means that users don't have to worry about inadvertently installing a Tempest plugin when they install another package. .. _Branchless Tempest Spec: https://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/branchless-tempest.html -The sole advantage to integrating a plugin into an existing python project is +The sole advantage of integrating a plugin into an existing Python project is that it enables you to land code changes at the same time you land test changes in the plugin. This reduces some of the burden on contributors by not having -to land 2 changes to add a new API feature and then test it and doing it as a -single combined commit. +to land 2 changes to add a new API feature and then test it, and do it as a +single combined commit instead. Plugin Class @@ -122,7 +122,7 @@ you would do something like the following: class MyPlugin(plugins.TempestPlugin): Then you need to ensure you locally define all of the mandatory methods in the -abstract class, you can refer to the api doc below for a reference of what that +abstract class, you can refer to the API doc below for a reference of what that entails. Abstract Plugin Class @@ -135,7 +135,7 @@ Plugin Structure ================ While there are no hard and fast rules for the structure of a plugin, there are basically no constraints on what the plugin looks like as long as the 2 steps -above are done. However, there are some recommended patterns to follow to make +above are done. However, there are some recommended patterns to follow to make it easy for people to contribute and work with your plugin. For example, if you create a directory structure with something like:: @@ -214,7 +214,7 @@ Example implementation of ``get_service_clients``: Parameters: * **name**: Name of the attribute used to access the ``ClientsFactory`` from - the ``ServiceClients`` instance. See example below. + the ``ServiceClients`` instance. See the example below. * **service_version**: Tempest enforces a single implementation for each service client. Available service clients are held in a ``ClientsRegistry`` singleton, and registered with ``service_version``, which means that @@ -229,7 +229,7 @@ Example usage of the service clients in tests: .. code-block:: python - # my_creds is instance of tempest.lib.auth.Credentials + # my_creds is an instance of tempest.lib.auth.Credentials # identity_uri is v2 or v3 depending on the configuration from tempest.lib.services import clients @@ -241,13 +241,13 @@ Automatic configuration and registration of service clients imposes some extra constraints on the structure of the configuration options exposed by the plugin. -First ``service_version`` should be in the format `service_config[.version]`. +Firstly, ``service_version`` should be in the format `service_config[.version]`. The `.version` part is optional, and should only be used if there are multiple versions of the same API available. The `service_config` must match the name of a configuration options group defined by the plugin. Different versions of one API must share the same configuration group. -Second the configuration options group `service_config` must contain the +Secondly, the configuration options group `service_config` must contain the following options: * `catalog_type`: corresponds to `service` in the catalog @@ -257,10 +257,10 @@ The following options will be honoured if defined, but they are not mandatory, as they do not necessarily apply to all service clients. * `region`: default to identity.region -* `build_timeout` : default to compute.build_timeout +* `build_timeout`: default to compute.build_timeout * `build_interval`: default to compute.build_interval -Third the service client classes should inherit from ``RestClient``, should +Thirdly, the service client classes should inherit from ``RestClient``, should accept generic keyword arguments, and should pass those arguments to the ``__init__`` method of ``RestClient``. Extra arguments can be added. For instance: @@ -276,7 +276,7 @@ instance: self.my_arg = my_arg self.my_args2 = my_arg -Finally the service client should be structured in a python module, so that all +Finally, the service client should be structured in a Python module, so that all service client classes are importable from it. Each major API version should have its own module. @@ -299,7 +299,7 @@ The content of __init__.py module should be: __all__ = ['API1Client', 'API2Client'] The following folder and module structure is recommended for multiple major -API version:: +API versions:: plugin_dir/ services/ @@ -325,14 +325,14 @@ Using Plugins ============= Tempest will automatically discover any installed plugins when it is run. So by -just installing the python packages which contain your plugin you'll be using +just installing the Python packages, which contain your plugin, you'll be using them with Tempest, nothing else is really required. -However, you should take care when installing plugins. By their very nature +However, you should take care when installing plugins. By their very nature, there are no guarantees when running Tempest with plugins enabled about the quality of the plugin. Additionally, while there is no limitation on running with multiple plugins, it's worth noting that poorly written plugins might not -properly isolate their tests which could cause unexpected cross interactions +properly isolate their tests which could cause unexpected cross-interactions between plugins. Notes for using plugins with virtualenvs diff --git a/doc/source/serial_tests.rst b/doc/source/serial_tests.rst new file mode 120000 index 0000000000..6709115f67 --- /dev/null +++ b/doc/source/serial_tests.rst @@ -0,0 +1 @@ +../../tempest/serial_tests/README.rst \ No newline at end of file diff --git a/doc/source/tests/modules.rst b/doc/source/tests/modules.rst index 026a7a5a82..697b0112e2 100644 --- a/doc/source/tests/modules.rst +++ b/doc/source/tests/modules.rst @@ -19,3 +19,10 @@ OpenStack Services API Tests network/modules object_storage/modules volume/modules + +Serial Tests +------------ +.. toctree:: + :maxdepth: 2 + + serial_tests/modules diff --git a/tempest/README.rst b/tempest/README.rst index b34503255b..b300dcb847 100644 --- a/tempest/README.rst +++ b/tempest/README.rst @@ -8,7 +8,7 @@ OpenStack core projects, being used to validate OpenStack cloud implementations for both correctness, as well as a burn in tool for OpenStack clouds. -As such Tempest tests come in many flavors, each with their own rules +As such Tempest tests come in many flavors, each with its own rules and guidelines. Below is the overview of the Tempest repository structure to make this clear. @@ -17,6 +17,7 @@ to make this clear. tempest/ api/ - API tests scenario/ - complex scenario tests + serial_tests/ - tests that run always in the serial mode tests/ - unit tests for Tempest internals Each of these directories contains different types of tests. What @@ -41,13 +42,20 @@ frameworks. --------------------------- Scenario tests are complex "through path" tests for OpenStack -functionality. They are typically a series of steps where complicated +functionality. They are typically a series of steps where a complicated state requiring multiple services is set up exercised, and torn down. Scenario tests should not use the existing Python clients for OpenStack, but should instead use the Tempest implementations of clients. +:ref:`serial_tests_guide` +-------------------------------- + +Tests within this category will always be executed serially from the rest of +the test cases. + + :ref:`unit_tests_field_guide` ----------------------------- diff --git a/tempest/api/README.rst b/tempest/api/README.rst index a796922193..705123037d 100644 --- a/tempest/api/README.rst +++ b/tempest/api/README.rst @@ -7,20 +7,20 @@ Tempest Field Guide to API tests What are these tests? --------------------- -One of Tempest's prime function is to ensure that your OpenStack cloud +One of Tempest's prime functions is to ensure that your OpenStack cloud works with the OpenStack API as documented. The current largest portion of Tempest code is devoted to test cases that do exactly this. It's also important to test not only the expected positive path on APIs, but also to provide them with invalid data to ensure they fail in expected and documented ways. The latter type of tests is called -``negative tests`` in Tempest source code. Over the course of the OpenStack -project Tempest has discovered many fundamental bugs by doing just +``negative tests`` in Tempest source code. Throughout the OpenStack +project, Tempest has discovered many fundamental bugs by doing just this. In order for some APIs to return meaningful results, there must be enough data in the system. This means these tests might start by -spinning up a server, image, etc, then operating on it. +spinning up a server, image, etc., and then operating on it. Why are these tests in Tempest? @@ -32,7 +32,7 @@ their clouds haven't broken the OpenStack API. It could be argued that some of the negative testing could be done back in the projects themselves, and we might evolve there over time, -but currently in the OpenStack gate this is a fundamentally important +but currently, in the OpenStack gate, this is a fundamentally important place to keep things. @@ -43,7 +43,7 @@ API tests should always use the Tempest implementation of the OpenStack API, as we want to ensure that bugs aren't hidden by the official clients. -They should test specific API calls, and can build up complex state if +They should test specific API calls and can build up complex states if it's needed for the API call to be meaningful. They should send not only good data, but bad data at the API and look diff --git a/tempest/scenario/README.rst b/tempest/scenario/README.rst index efcd1394eb..6c51f22749 100644 --- a/tempest/scenario/README.rst +++ b/tempest/scenario/README.rst @@ -7,14 +7,14 @@ Tempest Field Guide to Scenario tests What are these tests? --------------------- -Scenario tests are "through path" tests of OpenStack -function. Complicated setups where one part might depend on completion +Scenario tests are "through path" tests of OpenStack function. +Complicated setups where one part might depend on the completion of a previous part. They ideally involve the integration between multiple OpenStack services to exercise the touch points between them. Any scenario test should have a real-life use case. An example would be: -- "As operator I want to start with a blank environment": +- "As an operator, I want to start with a blank environment": 1. upload a glance image 2. deploy a vm from it @@ -24,12 +24,14 @@ Any scenario test should have a real-life use case. An example would be: Why are these tests in Tempest? ------------------------------- + This is one of Tempest's core purposes, testing the integration between projects. Scope of these tests -------------------- + Scenario tests should always use the Tempest implementation of the OpenStack API, as we want to ensure that bugs aren't hidden by the official clients. @@ -40,6 +42,7 @@ determined by which client libraries are used directly by the test. Example of a good test ---------------------- + While we are looking for interaction of 2 or more services, be specific in your interactions. A giant "this is my data center" smoke test is hard to debug when it goes wrong. diff --git a/tempest/serial_tests/README.rst b/tempest/serial_tests/README.rst new file mode 100644 index 0000000000..40a50f4edb --- /dev/null +++ b/tempest/serial_tests/README.rst @@ -0,0 +1,61 @@ +.. _serial_tests_guide: + +Tempest Field Guide to Serial tests +=================================== + + +What are these tests? +--------------------- + +Tempest can run tests serially as well as in parallel, depending on the +configuration that is fully up to the user. However, sometimes you need to +make sure that tests are not interfering with each other via OpenStack +resources with the other tests running in parallel. Tempest creates separate +projects for each test class to separate project based resources between test +cases. + +If your tests use resources outside of projects, e.g. host aggregates then +you might need to explicitly separate interfering test cases. If you only need +to separate a small set of test cases from each other then you can use the +``LockFixture``. + +However, in some cases, a small set of tests needs to be run serially. For +example, some of the host aggregate and availability zone testing needs +compute nodes without any running nova server to be able to move compute hosts +between availability zones. But many tempest tests start one or more nova +servers. + + +Why are these tests in Tempest? +------------------------------- + +This is one of Tempest's core purposes, testing the integration between +projects. + + +Scope of these tests +-------------------- + +The tests should always use the Tempest implementation of the OpenStack API, +as we want to ensure that bugs aren't hidden by the official clients. + +Tests should be tagged with which services they exercise, as +determined by which client libraries are used directly by the test. + + +Example of a good test +---------------------- + +While we are looking for interaction of 2 or more services, be specific in +your interactions. A giant "this is my data center" smoke test is hard to +debug when it goes wrong. + +The tests that need to be run serially need to be marked with the +``@serial`` class decorator. This will make sure that even if tempest is +configured to run the tests in parallel, these tests will always be executed +separately from the rest of the test cases. + +Please note that due to test ordering optimization reasons test cases marked +for ``@serial`` execution need to be put under ``tempest/serial_tests`` +directory. This will ensure that the serial tests will block the parallel tests +in the least amount of time. diff --git a/tempest/tests/README.rst b/tempest/tests/README.rst index 0587e7b887..081dd07e6a 100644 --- a/tempest/tests/README.rst +++ b/tempest/tests/README.rst @@ -14,6 +14,7 @@ and should be able to run solely from the Tempest tree. Why are these tests in Tempest? ------------------------------- + These tests exist to make sure that the mechanisms that we use inside of Tempest are valid and remain functional. They are only here for self validation of Tempest. @@ -21,6 +22,7 @@ validation of Tempest. Scope of these tests -------------------- + Unit tests should not require an external service to be running or any extra configuration to run. Any state that is required for a test should either be mocked out or created in a temporary test directory. (see test_wrappers.py for diff --git a/tox.ini b/tox.ini index de81707517..7c50a3bb5f 100644 --- a/tox.ini +++ b/tox.ini @@ -359,8 +359,9 @@ commands = sphinx-apidoc -f -o doc/source/tests/image tempest/api/image sphinx-apidoc -f -o doc/source/tests/network tempest/api/network sphinx-apidoc -f -o doc/source/tests/object_storage tempest/api/object_storage - sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario sphinx-apidoc -f -o doc/source/tests/volume tempest/api/volume + sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario + sphinx-apidoc -f -o doc/source/tests/serial_tests tempest/serial_tests rm -rf doc/build sphinx-build -W -b html doc/source doc/build/html allowlist_externals = @@ -377,8 +378,9 @@ commands = sphinx-apidoc -f -o doc/source/tests/image tempest/api/image sphinx-apidoc -f -o doc/source/tests/network tempest/api/network sphinx-apidoc -f -o doc/source/tests/object_storage tempest/api/object_storage - sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario sphinx-apidoc -f -o doc/source/tests/volume tempest/api/volume + sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario + sphinx-apidoc -f -o doc/source/tests/serial_tests tempest/serial_tests sphinx-build -W -b latex doc/source doc/build/pdf make -C doc/build/pdf