Doc: fix markups, capitalization and add 2 REVIEWING advices

I've been reading our entire documentation to see where it could
be improved. It's guide good actually. While reading I've fixed
some typos, added some capitalization to project names, mostly
trivial stuff.

What's worth reviewing is the 2 paragraphs I added to the REVIEWING
guidelines.

Change-Id: I977de335119f4ff7b9aedcfbed31c264ed531ea8
changes/28/460128/6
Jordan Pittier 6 years ago
parent 21dd8a5ee2
commit 74a56aba03
  1. 48
      HACKING.rst
  2. 54
      README.rst
  3. 24
      REVIEWING.rst
  4. 62
      doc/source/configuration.rst
  5. 18
      doc/source/write_tests.rst
  6. 15
      tempest/README.rst
  7. 5
      tempest/api/README.rst
  8. 6
      tempest/scenario/README.rst
  9. 12
      tempest/tests/README.rst

@ -102,20 +102,20 @@ to work even if just one ``test_method`` is selected for execution.
Service Tagging
---------------
Service tagging is used to specify which services are exercised by a particular
test method. You specify the services with the tempest.test.services decorator.
For example:
test method. You specify the services with the ``tempest.test.services``
decorator. For example:
@services('compute', 'image')
Valid service tag names are the same as the list of directories in tempest.api
that have tests.
For scenario tests having a service tag is required. For the api tests service
tags are only needed if the test method makes an api call (either directly or
For scenario tests having a service tag is required. For the API tests service
tags are only needed if the test method makes an API call (either directly or
indirectly through another service) that differs from the parent directory
name. For example, any test that make an api call to a service other than nova
in tempest.api.compute would require a service tag for those services, however
they do not need to be tagged as compute.
name. For example, any test that make an API call to a service other than Nova
in ``tempest.api.compute`` would require a service tag for those services,
however they do not need to be tagged as ``compute``.
Test fixtures and resources
---------------------------
@ -198,7 +198,7 @@ will expand the guideline based on our discussion and experience.
Test skips because of Known Bugs
--------------------------------
If a test is broken because of a bug it is appropriate to skip the test until
bug has been fixed. You should use the skip_because decorator so that
bug has been fixed. You should use the ``skip_because`` decorator so that
Tempest's skip tracking tool can watch the bug status.
Example::
@ -229,7 +229,7 @@ parallel.
require admin privileges are outside of projects.
- Races between methods in the same class are not a problem because
parallelization in tempest is at the test class level, but if there is a json
parallelization in Tempest is at the test class level, but if there is a json
and xml version of the same test class there could still be a race between
methods.
@ -238,8 +238,8 @@ parallel.
avoided to prevent resource conflicts.
- If the execution of a set of tests is required to be serialized then locking
can be used to perform this. See AggregatesAdminTest in
tempest.api.compute.admin for an example of using locking.
can be used to perform this. See usage of ``LockFixture`` for examples of
using locking.
Sample Configuration File
-------------------------
@ -251,7 +251,7 @@ regenerated. This can be done running::
Unit Tests
----------
Unit tests are a separate class of tests in tempest. They verify tempest
Unit tests are a separate class of tests in Tempest. They verify Tempest
itself, and thus have a different set of guidelines around them:
1. They can not require anything running externally. All you should need to
@ -321,8 +321,8 @@ format of the metadata looks like::
Tempest.lib includes a ``check-uuid`` tool that will test for the existence
and uniqueness of idempotent_id metadata for every test. If you have
tempest installed you run the tool against Tempest by calling from the
tempest repo::
Tempest installed you run the tool against Tempest by calling from the
Tempest repo::
check-uuid
@ -337,7 +337,7 @@ one::
check-uuid --fix
The ``check-uuid`` tool is used as part of the tempest gate job
The ``check-uuid`` tool is used as part of the Tempest gate job
to ensure that all tests have an ``idempotent_id`` decorator.
Branchless Tempest Considerations
@ -350,7 +350,7 @@ branches are also gated by the Tempest master branch, which also means that
proposed commits to Tempest must work against both the master and all the
currently supported stable branches of the projects. As such there are a few
special considerations that have to be accounted for when pushing new changes
to tempest.
to Tempest.
1. New Tests for new features
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -370,14 +370,22 @@ be able to merge.
When trying to land a bug fix which changes a tested API you'll have to use the
following procedure::
- Propose change to the project, get a +2 on the change even with failing
- Propose skip on Tempest which will only be approved after the
1. Propose change to the project, get a +2 on the change even with failing
2. Propose skip on Tempest which will only be approved after the
corresponding change in the project has a +2 on change
- Land project change in master and all open stable branches (if required)
- Land changed test in Tempest
3. Land project change in master and all open stable branches (if required)
4. Land changed test in Tempest
Otherwise the bug fix won't be able to land in the project.
Handily, `Zuul’s cross-repository dependencies
<https://docs.openstack.org/infra/zuul/gating.html#cross-repository-dependencies>`_.
can be leveraged to do without step 2 and to have steps 3 and 4 happen
"atomically". To do that, make the patch written in step 1 to depend (refer to
Zuul's documentation above) on the patch written in step 4. The commit message
for the Tempest change should have a link to the Gerrit review that justifies
that change.
3. New Tests for existing features
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

@ -23,7 +23,7 @@ Design Principles
Tempest Design Principles that we strive to live by.
- Tempest should be able to run against any OpenStack cloud, be it a
one node devstack install, a 20 node lxc cloud, or a 1000 node kvm
one node DevStack install, a 20 node LXC cloud, or a 1000 node KVM
cloud.
- Tempest should be explicit in testing features. It is easy to auto
discover features of a cloud incorrectly, and give people an
@ -65,13 +65,13 @@ as it is simpler, and quicker to work with.
$ pip install tempest/
This can be done within a venv, but the assumption for this guide is that
the Tempest cli entry point will be in your shell's PATH.
the Tempest CLI entry point will be in your shell's PATH.
#. Installing Tempest may create a /etc/tempest dir, however if one isn't
created you can create one or use ~/.tempest/etc or ~/.config/tempest in
place of /etc/tempest. If none of these dirs are created tempest will create
~/.tempest/etc when it's needed. The contents of this dir will always
automatically be copied to all etc/ dirs in local workspaces as an initial
#. Installing Tempest may create a ``/etc/tempest dir``, however if one isn't
created you can create one or use ``~/.tempest/etc`` or ``~/.config/tempest`` in
place of ``/etc/tempest``. If none of these dirs are created Tempest will create
``~/.tempest/etc`` when it's needed. The contents of this dir will always
automatically be copied to all ``etc/`` dirs in local workspaces as an initial
setup step. So if there is any common configuration you'd like to be shared
between local Tempest workspaces it's recommended that you pre-populate it
before running ``tempest init``.
@ -90,12 +90,12 @@ as it is simpler, and quicker to work with.
is that you'll create a new working directory for each to maintain separate
configuration files and local artifact storage for each.
#. Then cd into the newly created working dir and also modify the local
config files located in the etc/ subdir created by the ``tempest init``
command. Tempest is expecting a tempest.conf file in etc/ so if only a
#. Then ``cd`` into the newly created working dir and also modify the local
config files located in the ``etc/`` subdir created by the ``tempest init``
command. Tempest is expecting a ``tempest.conf`` file in etc/ so if only a
sample exists you must rename or copy it to tempest.conf before making
any changes to it otherwise Tempest will not know how to load it. For
details on configuring tempest refer to the :ref:`tempest-configuration`.
details on configuring Tempest refer to the :ref:`tempest-configuration`.
#. Once the configuration is done you're now ready to run Tempest. This can
be done using the :ref:`tempest_run` command. This can be done by either
@ -124,8 +124,8 @@ Library
Tempest exposes a library interface. This interface is a stable interface and
should be backwards compatible (including backwards compatibility with the
old tempest-lib package, with the exception of the import). If you plan to
directly consume tempest in your project you should only import code from the
tempest library interface, other pieces of tempest do not have the same
directly consume Tempest in your project you should only import code from the
Tempest library interface, other pieces of Tempest do not have the same
stable interface and there are no guarantees on the Python API unless otherwise
stated.
@ -137,7 +137,7 @@ Release Versioning
shows what changes have been released on each version.
Tempest's released versions are broken into 2 sets of information. Depending on
how you intend to consume tempest you might need
how you intend to consume Tempest you might need
The version is a set of 3 numbers:
@ -146,12 +146,12 @@ X.Y.Z
While this is almost `semver`_ like, the way versioning is handled is slightly
different:
X is used to represent the supported OpenStack releases for tempest tests
in-tree, and to signify major feature changes to tempest. It's a monotonically
X is used to represent the supported OpenStack releases for Tempest tests
in-tree, and to signify major feature changes to Tempest. It's a monotonically
increasing integer where each version either indicates a new supported OpenStack
release, the drop of support for an OpenStack release (which will coincide with
the upstream stable branch going EOL), or a major feature lands (or is removed)
from tempest.
from Tempest.
Y.Z is used to represent library interface changes. This is treated the same
way as minor and patch versions from `semver`_ but only for the library
@ -166,16 +166,16 @@ Configuration
Detailed configuration of Tempest is beyond the scope of this
document see :ref:`tempest-configuration` for more details on configuring
Tempest. The etc/tempest.conf.sample attempts to be a self-documenting version
of the configuration.
Tempest. The ``etc/tempest.conf.sample`` attempts to be a self-documenting
version of the configuration.
You can generate a new sample tempest.conf file, run the following
command from the top level of the Tempest directory::
$ tox -e genconfig
The most important pieces that are needed are the user ids, openstack
endpoint, and basic flavors and images needed to run tests.
The most important pieces that are needed are the user ids, OpenStack
endpoints, and basic flavors and images needed to run tests.
Unit Tests
----------
@ -190,13 +190,13 @@ should only be run on the unit test directory. The default value of OS_TEST_PATH
is OS_TEST_PATH=./tempest/test_discover which will only run test discover on the
Tempest suite.
Alternatively, there are the py27 and py34 tox jobs which will run the unit
Alternatively, there are the py27 and py35 tox jobs which will run the unit
tests with the corresponding version of python.
Python 2.6
----------
Starting in the kilo release the OpenStack services dropped all support for
Starting in the Kilo release the OpenStack services dropped all support for
python 2.6. This change has been mirrored in Tempest, starting after the
tempest-2 tag. This means that proposed changes to Tempest which only fix
python 2.6 compatibility will be rejected, and moving forward more features not
@ -208,8 +208,8 @@ that has python 2.7)
Python 3.x
----------
Starting during the Pike cycle Tempest has a gating CI job that runs tempest
with Python 3. Any tempest release after 15.0.0 should fully support running
Starting during the Pike cycle Tempest has a gating CI job that runs Tempest
with Python 3. Any Tempest release after 15.0.0 should fully support running
under Python 3 as well as Python 2.7.
Legacy run method
@ -239,10 +239,10 @@ and reference data to be used in testing.
.. note::
If you have a running devstack environment, Tempest will be
If you have a running DevStack environment, Tempest will be
automatically configured and placed in ``/opt/stack/tempest``. It
will have a configuration file already set up to work with your
devstack installation.
DevStack installation.
Tempest is not tied to any single test runner, but `testr`_ is the most commonly
used tool. Also, the nosetests test runner is **not** recommended to run Tempest.

@ -21,6 +21,15 @@ manual/developers.html#cross-repository-dependencies>`_ or a simple link
to a Gerrit review.
Execution time
--------------
While checking in the job logs that a new test is actually executed, also
pay attention to the execution time of that test. Keep in mind that each test
is going to be executed hundreds of time each day, because Tempest tests
run in many OpenStack projects. It's worth considering how important/critical
the feature under test is with how costly the new test is.
Unit Tests
----------
@ -48,6 +57,17 @@ the maintenance burden. Such changes should not be approved if it is easy to
abstract the duplicated code into a function or method.
Tests overlap
-------------
When a new test is being proposed, question whether this feature is not already
tested with Tempest. Tempest has more than 1200 tests, spread amongst many
directories, so it's easy to introduce test duplication. For example, testing
volume attachment to a server could be a compute test or a volume test, depending
on how you see it. So one must look carefully in the entire code base for possible
overlap. As a rule of thumb, the older a feature is, the more likely it's
already tested.
Being explicit
--------------
When tests are being added that depend on a configurable feature or extension,
@ -60,8 +80,8 @@ whether to skip or not.
Configuration Options
---------------------
With the introduction of the tempest external test plugin interface we needed
to provide a stable contract for tempest's configuration options. This means
With the introduction of the Tempest external test plugin interface we needed
to provide a stable contract for Tempest's configuration options. This means
we can no longer simply remove a configuration option when it's no longer used.
Patches proposed that remove options without a deprecation cycle should not
be approved. Similarly when changing default values with configuration we need

@ -28,7 +28,7 @@ can be used to:
- Run tests for admin APIs
- Generate test credentials on the fly (see `Dynamic Credentials`_)
When keystone uses a policy that requires domain scoped tokens for admin
When Keystone uses a policy that requires domain scoped tokens for admin
actions, the flag ``admin_domain_scope`` must be set to ``True``.
The admin user configured, if any, must have a role assigned to the domain to
be usable.
@ -39,7 +39,7 @@ This can be done using the accounts.yaml file (see
number of users available to run tests with.
You can specify the location of the file in the ``auth`` section in the
tempest.conf file. To see the specific format used in the file please refer to
the accounts.yaml.sample file included in Tempest.
the ``accounts.yaml.sample`` file included in Tempest.
Keystone Connection Info
^^^^^^^^^^^^^^^^^^^^^^^^
@ -47,18 +47,17 @@ In order for Tempest to be able to talk to your OpenStack deployment you need
to provide it with information about how it communicates with keystone.
This involves configuring the following options in the ``identity`` section:
#. ``auth_version``
#. ``uri``
#. ``uri_v3``
- ``auth_version``
- ``uri``
- ``uri_v3``
The ``auth_version`` option is used to tell Tempest whether it should be using
keystone's v2 or v3 api for communicating with keystone. The two uri options are
Keystone's v2 or v3 api for communicating with Keystone. The two uri options are
used to tell Tempest the url of the keystone endpoint. The ``uri`` option is
used for keystone v2 request and ``uri_v3`` is used for keystone v3. You want to
used for Keystone v2 request and ``uri_v3`` is used for Keystone v3. You want to
ensure that which ever version you set for ``auth_version`` has its uri option
defined.
Credential Provider Mechanisms
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -95,7 +94,7 @@ configured to use dynamic credentials.
When the ``admin_domain_scope`` option is set to ``True``, provisioned admin
accounts will be assigned a role on domain configured in
``default_credentials_domain_name``. This will make the accounts provisioned
usable in a cloud where domain scoped tokens are required by keystone for
usable in a cloud where domain scoped tokens are required by Keystone for
admin operations. Note that the initial pre-provision admin accounts,
configured in tempest.conf, must have a role on the same domain as well, for
Dynamic Credentials to work.
@ -140,10 +139,10 @@ used do not have any pre-existing resources created. Tempest assumes all
tenants it's using are empty and may sporadically fail if there are unexpected
resources present.
When the keystone in the target cloud requires domain scoped tokens to
When the Keystone in the target cloud requires domain scoped tokens to
perform admin actions, all pre-provisioned admin users must have a role
assigned on the domain where test accounts a provisioned.
The option ``admin_domain_scope`` is used to tell tempest that domain scoped
The option ``admin_domain_scope`` is used to tell Tempest that domain scoped
tokens shall be used. ``default_credentials_domain_name`` is the domain where
test accounts are expected to be provisioned if no domain is specified.
@ -167,7 +166,7 @@ for doing this:
#. ``flavor_ref_alt``
Both of these options are in the ``compute`` section of the config file and take
in the flavor id (not the name) from nova. The ``flavor_ref`` option is what
in the flavor id (not the name) from Nova. The ``flavor_ref`` option is what
will be used for booting almost all of the guests; ``flavor_ref_alt`` is only
used in tests where two different-sized servers are required (for example, a
resize test).
@ -185,7 +184,7 @@ servers. There are two options in the compute section just like with flavors:
#. ``image_ref``
#. ``image_ref_alt``
Both options are expecting an image id (not name) from nova. The ``image_ref``
Both options are expecting an image id (not name) from Nova. The ``image_ref``
option is what will be used for booting the majority of servers in Tempest.
``image_ref_alt`` is used for tests that require two images such as rebuild. If
two images are not available you can set both options to the same image id and
@ -224,7 +223,7 @@ small footprint.
Networking
----------
OpenStack has a myriad of different networking configurations possible and
depending on which of the two network backends, nova-network or neutron, you are
depending on which of the two network backends, nova-network or Neutron, you are
using things can vary drastically. Due to this complexity Tempest has to provide
a certain level of flexibility in its configuration to ensure it will work
against any cloud. This ends up causing a large number of permutations in
@ -302,21 +301,21 @@ cases it shouldn't matter because nova-network should have no problem booting a
server with multiple networks. If this is not the case for your cloud then using
an accounts file is recommended because it provides the necessary flexibility to
describe your configuration. Dynamic credentials is not able to dynamically
allocate things as necessary if neutron is not enabled.
allocate things as necessary if Neutron is not enabled.
With neutron and dynamic credentials enabled there should not be any additional
With Neutron and dynamic credentials enabled there should not be any additional
configuration necessary to enable Tempest to create servers with working
networking, assuming you have properly configured the ``network`` section to
work for your cloud. Tempest will dynamically create the neutron resources
work for your cloud. Tempest will dynamically create the Neutron resources
necessary to enable using servers with that network. Also, just as with the
accounts file, if you specify a fixed network name while using neutron and
accounts file, if you specify a fixed network name while using Neutron and
dynamic credentials it will enable running tests which require a static network
and it will additionally be used as a fallback for server creation. However,
unlike accounts.yaml this should never be triggered.
However, there is an option ``create_isolated_networks`` to disable dynamic
credentials's automatic provisioning of network resources. If this option is set
to False you will have to either rely on there only being a single/default
to ``False`` you will have to either rely on there only being a single/default
network available for the server creation, or use ``fixed_network_name`` to
inform Tempest which network to use.
@ -336,16 +335,16 @@ To enable remote access to servers, there are 3 options at a minimum that are us
The ``run_validation`` is used to enable or disable ssh connectivity for
all tests (with the exception of scenario tests which do not have a flag for
enabling or disabling ssh) To enable ssh connectivity this needs be set to ``true``.
enabling or disabling ssh) To enable ssh connectivity this needs be set to ``True``.
The ``connect_method`` option is used to tell tempest what kind of IP to use for
The ``connect_method`` option is used to tell Tempest what kind of IP to use for
establishing a connection to the server. Two methods are available: ``fixed``
and ``floating``, the later being set by default. If this is set to floating
tempest will create a floating ip for the server before attempted to connect
Tempest will create a floating ip for the server before attempted to connect
to it. The IP for the floating ip is what is used for the connection.
For the ``auth_method`` option there is currently, only one valid option,
``keypair``. With this set to ``keypair`` tempest will create an ssh keypair
``keypair``. With this set to ``keypair`` Tempest will create an ssh keypair
and use that for authenticating against the created server.
Configuring Available Services
@ -359,8 +358,8 @@ on the available services.
The ``service_available`` section of the config file is used to set which
services are available. It contains a boolean option for each service (except
for keystone which is a hard requirement) set it to True if the service is
available or False if it is not.
for Keystone which is a hard requirement) set it to ``True`` if the service is
available or ``False`` if it is not.
Service Catalog
^^^^^^^^^^^^^^^
@ -382,17 +381,18 @@ has multiple regions available and you need to specify a particular one to use a
service you can set the ``region`` option in that service's section.
It should also be noted that the default values for these options are set
to what devstack uses (which is a de facto standard for service catalog
to what DevStack uses (which is a de facto standard for service catalog
entries). So often nothing actually needs to be set on these options to enable
communication to a particular service. It is only if you are either not using
the same ``catalog_type`` as devstack or you want Tempest to talk to a different
endpoint type instead of publicURL for a service that these need to be changed.
the same ``catalog_type`` as DevStack or you want Tempest to talk to a different
endpoint type instead of ``publicURL`` for a service that these need to be
changed.
.. note::
Tempest does not serve all kinds of fancy URLs in the service catalog. The
service catalog should be in a standard format (which is going to be
standardized at the keystone level).
standardized at the Keystone level).
Tempest expects URLs in the Service catalog in the following format:
* ``http://example.com:1234/<version-info>``
@ -420,7 +420,7 @@ for a feature (assuming one exists). Instead Tempest has to be explicitly
configured as to which optional features are enabled. This is in order to
prevent bugs in the discovery mechanisms from masking failures.
The service feature-enabled config sections are how Tempest addresses the
The service ``feature-enabled`` config sections are how Tempest addresses the
optional feature question. Each service that has tests for optional features
contains one of these sections. The only options in it are boolean options
with the name of a feature which is used. If it is set to false any test which
@ -431,7 +431,7 @@ options refer to the sample config file.
API Extensions
^^^^^^^^^^^^^^
The service feature-enabled sections often contain an ``api-extensions`` option
(or in the case of swift a ``discoverable_apis`` option). This is used to tell
(or in the case of Swift a ``discoverable_apis`` option). This is used to tell
Tempest which api extensions (or configurable middleware) is used in your
deployment. It has two valid config states: either it contains a single value
``all`` (which is the default) which means that every api extension is assumed

@ -6,13 +6,13 @@ Tempest Test Writing Guide
This guide serves as a starting point for developers working on writing new
Tempest tests. At a high level tests in Tempest are just tests that conform to
the standard python `unit test`_ framework. But there are several aspects of
that are unique to tempest and it's role as an integration test suite running
that are unique to Tempest and its role as an integration test suite running
against a real cloud.
.. _unit test: https://docs.python.org/3.6/library/unittest.html
.. note:: This guide is for writing tests in the tempest repository. While many
parts of this guide are also applicable to tempest plugins, not all
.. note:: This guide is for writing tests in the Tempest repository. While many
parts of this guide are also applicable to Tempest plugins, not all
the APIs mentioned are considered stable or recommended for use in
plugins. Please refer to :ref:`tempest_plugin` for details about
writing plugins
@ -24,8 +24,8 @@ Adding a New TestCase
The base unit of testing in Tempest is the `TestCase`_ (also called the test
class). Each TestCase contains test methods which are the individual tests that
will be executed by the test runner. But, the TestCase is the smallest self
contained unit for tests from the tempest perspective. It's also the level at
which tempest is parallel safe. In other words, multiple TestCases can be
contained unit for tests from the Tempest perspective. It's also the level at
which Tempest is parallel safe. In other words, multiple TestCases can be
executed in parallel, but individual test methods in the same TestCase can not.
Also, all test methods within a TestCase are assumed to be executed serially. As
such you can use the test case to store variables that are shared between
@ -151,7 +151,7 @@ for tests cases that need a second user/project.
You can also specify credentials with specific roles assigned. This is useful
for cases where there are specific RBAC requirements hard coded into an API.
The canonical example of this are swift tests which often want to test swift's
concepts of operator and reseller_admin. An actual example from tempest on how
concepts of operator and reseller_admin. An actual example from Tempest on how
to do this is::
class PublicObjectTest(base.BaseObjectTest):
@ -187,7 +187,7 @@ requested credentials you can reference:
| [$label, $role] | cls.os_roles_$label |
+-------------------+---------------------+
By default cls.os_primary is available since it is allocated in the base tempest test
By default cls.os_primary is available since it is allocated in the base Tempest test
class (located in tempest/test.py). If your TestCase inherits from a different
direct parent class (it'll still inherit from the BaseTestCase, just not
directly) be sure to check if that class overrides allocated credentials.
@ -195,8 +195,8 @@ directly) be sure to check if that class overrides allocated credentials.
Dealing with Network Allocation
'''''''''''''''''''''''''''''''
When neutron is enabled and a testing requires networking this isn't normally
automatically setup when a tenant is created. Since tempest needs isolated
When Neutron is enabled and a testing requires networking this isn't normally
automatically setup when a tenant is created. Since Tempest needs isolated
tenants to function properly it also needs to handle network allocation. By
default the base test class will allocate a network, subnet, and router
automatically (this depends on the configured credential provider, for more

@ -9,12 +9,13 @@ implementations for both correctness, as well as a burn in tool for
OpenStack clouds.
As such Tempest tests come in many flavors, each with their own rules
and guidelines. Below is the proposed Havana restructuring for Tempest
and guidelines. Below is the overview of the Tempest respository structure
to make this clear.
| tempest/
| api/ - API tests
| scenario/ - complex scenario tests
| tests/ - unit tests for Tempest internals
Each of these directories contains different types of tests. What
belongs in each directory, the rules and examples for good tests, are
@ -24,8 +25,8 @@ documented in a README.rst file in the directory.
----------------------
API tests are validation tests for the OpenStack API. They should not
use the existing python clients for OpenStack, but should instead use
the tempest implementations of clients. Having raw clients let us
use the existing Python clients for OpenStack, but should instead use
the Tempest implementations of clients. Having raw clients let us
pass invalid JSON to the APIs and see the results, something we could
not get with the native clients.
@ -41,14 +42,14 @@ Scenario tests are complex "through path" tests for OpenStack
functionality. They are typically a series of steps where complicated
state requiring multiple services is set up exercised, and torn down.
Scenario tests should not use the existing python clients for OpenStack,
but should instead use the tempest implementations of clients.
Scenario tests should not use the existing Python clients for OpenStack,
but should instead use the Tempest implementations of clients.
:ref:`unit_tests_field_guide`
-----------------------------
Unit tests are the self checks for Tempest. They provide functional
verification and regression checking for the internal components of tempest.
They should be used to just verify that the individual pieces of tempest are
verification and regression checking for the internal components of Tempest.
They should be used to just verify that the individual pieces of Tempest are
working as expected.

@ -13,7 +13,8 @@ portion of Tempest code is devoted to test cases that do exactly this.
It's also important to test not only the expected positive path on
APIs, but also to provide them with invalid data to ensure they fail
in expected and documented ways. Over the course of the OpenStack
in expected and documented ways. The latter type of tests is called
``negative tests`` in Tempest source code. Over the course of the OpenStack
project Tempest has discovered many fundamental bugs by doing just
this.
@ -22,7 +23,7 @@ enough data in the system. This means these tests might start by
spinning up a server, image, etc, then operating on it.
Why are these tests in tempest?
Why are these tests in Tempest?
-------------------------------
This is one of the core missions for the Tempest project, and where it

@ -21,9 +21,9 @@ Any scenario test should have a real-life use case. An example would be:
4. create a snapshot of the vm
Why are these tests in tempest?
Why are these tests in Tempest?
-------------------------------
This is one of tempests core purposes, testing the integration between
This is one of Tempest's core purposes, testing the integration between
projects.
@ -43,7 +43,7 @@ While we are looking for interaction of 2 or more services, be
specific in your interactions. A giant "this is my data center" smoke
test is hard to debug when it goes wrong.
A flow of interactions between glance and nova, like in the
A flow of interactions between Glance and Nova, like in the
introduction, is a good example. Especially if it involves a repeated
interaction when a resource is setup, modified, detached, and then
reused later again.

@ -7,16 +7,16 @@ What are these tests?
---------------------
Unit tests are the self checks for Tempest. They provide functional
verification and regression checking for the internal components of tempest.
They should be used to just verify that the individual pieces of tempest are
verification and regression checking for the internal components of Tempest.
They should be used to just verify that the individual pieces of Tempest are
working as expected. They should not require an external service to be running
and should be able to run solely from the tempest tree.
and should be able to run solely from the Tempest tree.
Why are these tests in tempest?
Why are these tests in Tempest?
-------------------------------
These tests exist to make sure that the mechanisms that we use inside of
tempest to are valid and remain functional. They are only here for self
validation of tempest.
Tempest are valid and remain functional. They are only here for self
validation of Tempest.
Scope of these tests

Loading…
Cancel
Save