General doc updates

Added missing doc, for example for serial tests, fixed 404 links,
spellings, formatting and etc

Change-Id: I07a8cf9508d3cba33bd04afdc301bb7145153d59

HACKING.rst

@@ -184,7 +184,7 @@ This long runtime has performance implications on `Zuul`_ jobs, which is why
 the ``slow`` attribute is leveraged to run slow tests on a selective basis,
 to keep total `Zuul`_ job runtime down to a reasonable time frame.
 
-.. _Zuul: https://docs.openstack.org/infra/zuul/
+.. _Zuul: https://zuul-ci.org/docs/zuul/latest/
 
 Smoke Attribute
 ^^^^^^^^^^^^^^^
@@ -488,7 +488,7 @@ following procedure::
 Otherwise the bug fix won't be able to land in the project.
 
 Handily, `Zuul's cross-repository dependencies
-<https://docs.openstack.org/infra/zuul/user/gating.html#cross-project-dependencies>`_.
+<https://zuul-ci.org/docs/zuul/latest/gating.html#cross-project-dependencies>`_.
 can be leveraged to do without step 2 and to have steps 3 and 4 happen
 "atomically". To do that, make the patch written in step 1 to depend (refer to
 Zuul's documentation above) on the patch written in step 4. The commit message

README.rst

@@ -3,7 +3,6 @@ Team and repository tags
 ========================
 
 .. image:: https://governance.openstack.org/tc/badges/tempest.svg
-    :target: https://governance.openstack.org/tc/reference/tags/index.html
 
 .. Change things from this point on
 

doc/source/contributor/contributing.rst

@@ -14,8 +14,9 @@ with Tempest.
 Communication
 ~~~~~~~~~~~~~
 * IRC channel ``#openstack-qa`` at OFTC
-* Mailing list (prefix subjects with ``[qa]`` for faster responses)
+* `Mailing list <https://lists.openstack.org/mailman3/lists/openstack-discuss.lists.openstack.org/>`_
-  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss
+  (prefix subjects with ``[qa]`` for faster responses)
+
 
 Contacting the Core Team
 ~~~~~~~~~~~~~~~~~~~~~~~~
@@ -26,25 +27,29 @@ New Feature Planning
 ~~~~~~~~~~~~~~~~~~~~
 If you want to propose a new feature please read `Feature Proposal Process`_
 Tempest features are tracked on `Launchpad BP <https://blueprints.launchpad.net/tempest>`_.
+It also helps to bring the feature up during the next PTG and contact
+`the current PTL of QA project <https://governance.openstack.org/tc/reference/projects/>`_.
+Information about PTG is always posted on `the Mailing list
+<https://lists.openstack.org/mailman3/lists/openstack-discuss.lists.openstack.org/>`_.
 
 Task Tracking
 ~~~~~~~~~~~~~
 We track our tasks in `Launchpad <https://bugs.launchpad.net/tempest>`_.
 
-If you're looking for some smaller, easier work item to pick up and get started
+If you're looking for some smaller, easier work items to pick up and get started
 on, search for the 'low-hanging-fruit' tag.
 
 Reporting a Bug
 ~~~~~~~~~~~~~~~
-You found an issue and want to make sure we are aware of it? You can do so on
+Have you found an issue and want to make sure we are aware of it? You can do so
-`Launchpad <https://bugs.launchpad.net/tempest/+filebug>`__.
+on `Launchpad <https://bugs.launchpad.net/tempest/+filebug>`__.
 More info about Launchpad usage can be found on `OpenStack docs page
 <https://docs.openstack.org/contributors/common/task-tracking.html#launchpad>`_
 
 Getting Your Patch Merged
 ~~~~~~~~~~~~~~~~~~~~~~~~~
-All changes proposed to the Tempest require single ``Code-Review +2`` votes from
+All changes proposed to the Tempest require a single ``Code-Review +2`` vote
-Tempest core reviewers by giving ``Workflow +1`` vote. More detailed guidelines
+from a Tempest core followed by a ``Workflow +1`` vote. More detailed guidelines
 for reviewers are available at :doc:`../REVIEWING`.
 
 Project Team Lead Duties

doc/source/index.rst

@@ -83,6 +83,7 @@ Development
    :maxdepth: 2
 
    HACKING
+   serial_tests
    REVIEWING
    microversion_testing
    test_removal

doc/source/library/api_microversion_testing.rst

@@ -9,9 +9,9 @@ Framework to support API Microversion testing
 
 Many of the OpenStack components have implemented API microversions.
 It is important to test those microversions in Tempest or external plugins.
-Tempest now provides stable interfaces to support to test the API microversions.
+Tempest now provides stable interfaces to support testing the API microversions.
 Based on the microversion range coming from the combination of both configuration
-and each test case, APIs request will be made with selected microversion.
+and each test case, APIs requests will be made with the selected microversion.
 
 This document explains the interfaces needed for microversion testing.
 

doc/source/library/clients.rst

@@ -6,11 +6,11 @@ Service Clients Usage
 Tests make requests against APIs using service clients. Service clients are
 specializations of the ``RestClient`` class. The service clients that cover the
 APIs exposed by a service should be grouped in a service clients module.
-A service clients module is python module where all service clients are
+A service clients module is a Python module where all service clients are
 defined. If major API versions are available, submodules should be defined,
 one for each version.
 
-The ``ClientsFactory`` class helps initializing all clients of a specific
+The ``ClientsFactory`` class helps to initialize all clients of a specific
 service client module from a set of shared parameters.
 
 The ``ServiceClients`` class provides a convenient way to get access to all

doc/source/library/credential_providers.rst

@@ -4,12 +4,12 @@ Credential Providers
 ====================
 
 These library interfaces are used to deal with allocating credentials on demand
-either dynamically by calling keystone to allocate new credentials, or from
+either dynamically by calling Keystone to allocate new credentials, or from
 a list of preprovisioned credentials. These 2 modules are implementations of
 the same abstract credential providers class and can be used interchangeably.
 However, each implementation has some additional parameters that are used to
 influence the behavior of the modules. The API reference at the bottom of this
-doc shows the interface definitions for both modules, however that may be a bit
+doc shows the interface definitions for both modules, however, that may be a bit
 opaque. You can see some examples of how to leverage this interface below.
 
 Initialization Example
@@ -30,7 +30,7 @@ on config::
       # If a test requires a new account to work, it can have it via forcing
       # dynamic credentials. A new account will be produced only for that test.
       # In case admin credentials are not available for the account creation,
-      # the test should be skipped else it would fail.
+      # the test should be skipped else it will fail.
       identity_version = identity_version or CONF.identity.auth_version
       if CONF.auth.use_dynamic_credentials or force_tenant_isolation:
           admin_creds = get_configured_admin_credentials(
@@ -81,12 +81,12 @@ Dealing with Credentials
 Once you have a credential provider object created the access patterns for
 allocating and removing credentials are the same across both the dynamic
 and preprovisioned credentials. These are defined in the abstract
-CredentialProvider class. At a high level the credentials provider enables
+CredentialProvider class. At a high level, the credentials provider enables
-you to get 3 basic types of credentials at once (per object): a primary, alt,
+you to get 3 basic types of credentials at once (per object): primary, alt,
 and admin. You're also able to allocate a credential by role. These credentials
-are tracked by the provider object and delete must manually be called otherwise
+are tracked by the provider object and delete must be called manually,
-the created resources will not be deleted (or returned to the pool in the case
+otherwise, the created resources will not be deleted (or returned to the pool
-of preprovisioned creds)
+in the case of preprovisioned creds).
 
 Examples
 ''''''''

doc/source/plugins/plugin.rst

@@ -80,30 +80,30 @@ Standalone Plugin vs In-repo Plugin
 
 Since all that's required for a plugin to be detected by Tempest is a valid
 setuptools entry point in the proper namespace there is no difference from the
-Tempest perspective on either creating a separate python package to
+Tempest perspective on either creating a separate Python package to
-house the plugin or adding the code to an existing python project. However,
+house the plugin or adding the code to an existing Python project. However,
 there are tradeoffs to consider when deciding which approach to take when
 creating a new plugin.
 
-If you create a separate python project for your plugin this makes a lot of
+If you create a separate Python project for your plugin this makes a lot of
 things much easier. Firstly it makes packaging and versioning much simpler, you
 can easily decouple the requirements for the plugin from the requirements for
 the other project. It lets you version the plugin independently and maintain a
 single version of the test code across project release boundaries (see the
 `Branchless Tempest Spec`_ for more details on this). It also greatly
 simplifies the install time story for external users. Instead of having to
-install the right version of a project in the same python namespace as Tempest
+install the right version of a project in the same Python namespace as Tempest
 they simply need to pip install the plugin in that namespace. It also means
 that users don't have to worry about inadvertently installing a Tempest plugin
 when they install another package.
 
 .. _Branchless Tempest Spec: https://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/branchless-tempest.html
 
-The sole advantage to integrating a plugin into an existing python project is
+The sole advantage of integrating a plugin into an existing Python project is
 that it enables you to land code changes at the same time you land test changes
 in the plugin. This reduces some of the burden on contributors by not having
-to land 2 changes to add a new API feature and then test it and doing it as a
+to land 2 changes to add a new API feature and then test it, and do it as a
-single combined commit.
+single combined commit instead.
 
 
 Plugin Class
@@ -122,7 +122,7 @@ you would do something like the following:
   class MyPlugin(plugins.TempestPlugin):
 
 Then you need to ensure you locally define all of the mandatory methods in the
-abstract class, you can refer to the api doc below for a reference of what that
+abstract class, you can refer to the API doc below for a reference of what that
 entails.
 
 Abstract Plugin Class
@@ -135,7 +135,7 @@ Plugin Structure
 ================
 While there are no hard and fast rules for the structure of a plugin, there are
 basically no constraints on what the plugin looks like as long as the 2 steps
-above are done. However,  there are some recommended patterns to follow to make
+above are done. However, there are some recommended patterns to follow to make
 it easy for people to contribute and work with your plugin. For example, if you
 create a directory structure with something like::
 
@@ -214,7 +214,7 @@ Example implementation of ``get_service_clients``:
 Parameters:
 
 * **name**: Name of the attribute used to access the ``ClientsFactory`` from
-  the ``ServiceClients`` instance. See example below.
+  the ``ServiceClients`` instance. See the example below.
 * **service_version**: Tempest enforces a single implementation for each
   service client. Available service clients are held in a ``ClientsRegistry``
   singleton, and registered with ``service_version``, which means that
@@ -229,7 +229,7 @@ Example usage of the service clients in tests:
 
 .. code-block:: python
 
-   # my_creds is instance of tempest.lib.auth.Credentials
+   # my_creds is an instance of tempest.lib.auth.Credentials
    # identity_uri is v2 or v3 depending on the configuration
    from tempest.lib.services import clients
 
@@ -241,13 +241,13 @@ Automatic configuration and registration of service clients imposes some extra
 constraints on the structure of the configuration options exposed by the
 plugin.
 
-First ``service_version`` should be in the format `service_config[.version]`.
+Firstly, ``service_version`` should be in the format `service_config[.version]`.
 The `.version` part is optional, and should only be used if there are multiple
 versions of the same API available. The `service_config` must match the name of
 a configuration options group defined by the plugin. Different versions of one
 API must share the same configuration group.
 
-Second the configuration options group `service_config` must contain the
+Secondly, the configuration options group `service_config` must contain the
 following options:
 
 * `catalog_type`: corresponds to `service` in the catalog
@@ -257,10 +257,10 @@ The following options will be honoured if defined, but they are not mandatory,
 as they do not necessarily apply to all service clients.
 
 * `region`: default to identity.region
-* `build_timeout` : default to compute.build_timeout
+* `build_timeout`: default to compute.build_timeout
 * `build_interval`: default to compute.build_interval
 
-Third the service client classes should inherit from ``RestClient``, should
+Thirdly, the service client classes should inherit from ``RestClient``, should
 accept generic keyword arguments, and should pass those arguments to the
 ``__init__`` method of ``RestClient``. Extra arguments can be added. For
 instance:
@@ -276,7 +276,7 @@ instance:
            self.my_arg = my_arg
            self.my_args2 = my_arg
 
-Finally the service client should be structured in a python module, so that all
+Finally, the service client should be structured in a Python module, so that all
 service client classes are importable from it. Each major API version should
 have its own module.
 
@@ -299,7 +299,7 @@ The content of __init__.py module should be:
    __all__ = ['API1Client', 'API2Client']
 
 The following folder and module structure is recommended for multiple major
-API version::
+API versions::
 
     plugin_dir/
       services/
@@ -325,14 +325,14 @@ Using Plugins
 =============
 
 Tempest will automatically discover any installed plugins when it is run. So by
-just installing the python packages which contain your plugin you'll be using
+just installing the Python packages, which contain your plugin, you'll be using
 them with Tempest, nothing else is really required.
 
-However, you should take care when installing plugins. By their very nature
+However, you should take care when installing plugins. By their very nature,
 there are no guarantees when running Tempest with plugins enabled about the
 quality of the plugin. Additionally, while there is no limitation on running
 with multiple plugins, it's worth noting that poorly written plugins might not
-properly isolate their tests which could cause unexpected cross interactions
+properly isolate their tests which could cause unexpected cross-interactions
 between plugins.
 
 Notes for using plugins with virtualenvs

doc/source/serial_tests.rst

@@ -0,0 +1 @@
+../../tempest/serial_tests/README.rst

doc/source/tests/modules.rst

@@ -19,3 +19,10 @@ OpenStack Services API Tests
    network/modules
    object_storage/modules
    volume/modules
+
+Serial Tests
+------------
+.. toctree::
+   :maxdepth: 2
+
+   serial_tests/modules

tempest/README.rst

@@ -8,7 +8,7 @@ OpenStack core projects, being used to validate OpenStack cloud
 implementations for both correctness, as well as a burn in tool for
 OpenStack clouds.
 
-As such Tempest tests come in many flavors, each with their own rules
+As such Tempest tests come in many flavors, each with its own rules
 and guidelines. Below is the overview of the Tempest repository structure
 to make this clear.
 
@@ -17,6 +17,7 @@ to make this clear.
    tempest/
       api/ - API tests
       scenario/ - complex scenario tests
+      serial_tests/ - tests that run always in the serial mode
       tests/ - unit tests for Tempest internals
 
 Each of these directories contains different types of tests. What
@@ -41,13 +42,20 @@ frameworks.
 ---------------------------
 
 Scenario tests are complex "through path" tests for OpenStack
-functionality. They are typically a series of steps where complicated
+functionality. They are typically a series of steps where a complicated
 state requiring multiple services is set up exercised, and torn down.
 
 Scenario tests should not use the existing Python clients for OpenStack,
 but should instead use the Tempest implementations of clients.
 
 
+:ref:`serial_tests_guide`
+--------------------------------
+
+Tests within this category will always be executed serially from the rest of
+the test cases.
+
+
 :ref:`unit_tests_field_guide`
 -----------------------------
 

tempest/api/README.rst

@@ -7,20 +7,20 @@ Tempest Field Guide to API tests
 What are these tests?
 ---------------------
 
-One of Tempest's prime function is to ensure that your OpenStack cloud
+One of Tempest's prime functions is to ensure that your OpenStack cloud
 works with the OpenStack API as documented. The current largest
 portion of Tempest code is devoted to test cases that do exactly this.
 
 It's also important to test not only the expected positive path on
 APIs, but also to provide them with invalid data to ensure they fail
 in expected and documented ways. The latter type of tests is called
-``negative tests`` in Tempest source code. Over the course of the OpenStack
+``negative tests`` in Tempest source code. Throughout the OpenStack
-project Tempest has discovered many fundamental bugs by doing just
+project, Tempest has discovered many fundamental bugs by doing just
 this.
 
 In order for some APIs to return meaningful results, there must be
 enough data in the system. This means these tests might start by
-spinning up a server, image, etc, then operating on it.
+spinning up a server, image, etc., and then operating on it.
 
 
 Why are these tests in Tempest?
@@ -32,7 +32,7 @@ their clouds haven't broken the OpenStack API.
 
 It could be argued that some of the negative testing could be done
 back in the projects themselves, and we might evolve there over time,
-but currently in the OpenStack gate this is a fundamentally important
+but currently, in the OpenStack gate, this is a fundamentally important
 place to keep things.
 
 
@@ -43,7 +43,7 @@ API tests should always use the Tempest implementation of the
 OpenStack API, as we want to ensure that bugs aren't hidden by the
 official clients.
 
-They should test specific API calls, and can build up complex state if
+They should test specific API calls and can build up complex states if
 it's needed for the API call to be meaningful.
 
 They should send not only good data, but bad data at the API and look

tempest/scenario/README.rst

@@ -7,14 +7,14 @@ Tempest Field Guide to Scenario tests
 What are these tests?
 ---------------------
 
-Scenario tests are "through path" tests of OpenStack
+Scenario tests are "through path" tests of OpenStack function.
-function. Complicated setups where one part might depend on completion
+Complicated setups where one part might depend on the completion
 of a previous part. They ideally involve the integration between
 multiple OpenStack services to exercise the touch points between them.
 
 Any scenario test should have a real-life use case. An example would be:
 
-- "As operator I want to start with a blank environment":
+- "As an operator, I want to start with a blank environment":
 
   1. upload a glance image
   2. deploy a vm from it
@@ -24,12 +24,14 @@ Any scenario test should have a real-life use case. An example would be:
 
 Why are these tests in Tempest?
 -------------------------------
+
 This is one of Tempest's core purposes, testing the integration between
 projects.
 
 
 Scope of these tests
 --------------------
+
 Scenario tests should always use the Tempest implementation of the
 OpenStack API, as we want to ensure that bugs aren't hidden by the
 official clients.
@@ -40,6 +42,7 @@ determined by which client libraries are used directly by the test.
 
 Example of a good test
 ----------------------
+
 While we are looking for interaction of 2 or more services, be
 specific in your interactions. A giant "this is my data center" smoke
 test is hard to debug when it goes wrong.

tempest/serial_tests/README.rst

@@ -0,0 +1,61 @@
+.. _serial_tests_guide:
+
+Tempest Field Guide to Serial tests
+===================================
+
+
+What are these tests?
+---------------------
+
+Tempest can run tests serially as well as in parallel, depending on the
+configuration that is fully up to the user. However, sometimes you need to
+make sure that tests are not interfering with each other via OpenStack
+resources with the other tests running in parallel. Tempest creates separate
+projects for each test class to separate project based resources between test
+cases.
+
+If your tests use resources outside of projects, e.g. host aggregates then
+you might need to explicitly separate interfering test cases. If you only need
+to separate a small set of test cases from each other then you can use the
+``LockFixture``.
+
+However, in some cases, a small set of tests needs to be run serially. For
+example, some of the host aggregate and availability zone testing needs
+compute nodes without any running nova server to be able to move compute hosts
+between availability zones. But many tempest tests start one or more nova
+servers.
+
+
+Why are these tests in Tempest?
+-------------------------------
+
+This is one of Tempest's core purposes, testing the integration between
+projects.
+
+
+Scope of these tests
+--------------------
+
+The tests should always use the Tempest implementation of the OpenStack API,
+as we want to ensure that bugs aren't hidden by the official clients.
+
+Tests should be tagged with which services they exercise, as
+determined by which client libraries are used directly by the test.
+
+
+Example of a good test
+----------------------
+
+While we are looking for interaction of 2 or more services, be specific in
+your interactions. A giant "this is my data center" smoke test is hard to
+debug when it goes wrong.
+
+The tests that need to be run serially need to be marked with the
+``@serial`` class decorator. This will make sure that even if tempest is
+configured to run the tests in parallel, these tests will always be executed
+separately from the rest of the test cases.
+
+Please note that due to test ordering optimization reasons test cases marked
+for ``@serial`` execution need to be put under ``tempest/serial_tests``
+directory. This will ensure that the serial tests will block the parallel tests
+in the least amount of time.

tempest/tests/README.rst

@@ -14,6 +14,7 @@ and should be able to run solely from the Tempest tree.
 
 Why are these tests in Tempest?
 -------------------------------
+
 These tests exist to make sure that the mechanisms that we use inside of
 Tempest are valid and remain functional. They are only here for self
 validation of Tempest.
@@ -21,6 +22,7 @@ validation of Tempest.
 
 Scope of these tests
 --------------------
+
 Unit tests should not require an external service to be running or any extra
 configuration to run. Any state that is required for a test should either be
 mocked out or created in a temporary test directory. (see test_wrappers.py for

tox.ini

@@ -359,8 +359,9 @@ commands =
   sphinx-apidoc -f -o doc/source/tests/image tempest/api/image
   sphinx-apidoc -f -o doc/source/tests/network tempest/api/network
   sphinx-apidoc -f -o doc/source/tests/object_storage tempest/api/object_storage
-  sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario
   sphinx-apidoc -f -o doc/source/tests/volume tempest/api/volume
+  sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario
+  sphinx-apidoc -f -o doc/source/tests/serial_tests tempest/serial_tests
   rm -rf doc/build
   sphinx-build -W -b html doc/source doc/build/html
 allowlist_externals =
@@ -377,8 +378,9 @@ commands =
    sphinx-apidoc -f -o doc/source/tests/image tempest/api/image
    sphinx-apidoc -f -o doc/source/tests/network tempest/api/network
    sphinx-apidoc -f -o doc/source/tests/object_storage tempest/api/object_storage
-   sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario
    sphinx-apidoc -f -o doc/source/tests/volume tempest/api/volume
+   sphinx-apidoc -f -o doc/source/tests/scenario tempest/scenario
+   sphinx-apidoc -f -o doc/source/tests/serial_tests tempest/serial_tests
    sphinx-build -W -b latex doc/source doc/build/pdf
    make -C doc/build/pdf