Merge "Doc: fix markups, capitalization and add 2 REVIEWING advices"
This commit is contained in:
Jenkins
Gerrit Code Review
commit
c7a403debd
48
HACKING.rst
48
HACKING.rst
@ -102,20 +102,20 @@ to work even if just one ``test_method`` is selected for execution.
|
||||
Service Tagging
|
||||
---------------
|
||||
Service tagging is used to specify which services are exercised by a particular
|
||||
test method. You specify the services with the tempest.test.services decorator.
|
||||
For example:
|
||||
test method. You specify the services with the ``tempest.test.services``
|
||||
decorator. For example:
|
||||
|
||||
@services('compute', 'image')
|
||||
|
||||
Valid service tag names are the same as the list of directories in tempest.api
|
||||
that have tests.
|
||||
|
||||
For scenario tests having a service tag is required. For the api tests service
|
||||
tags are only needed if the test method makes an api call (either directly or
|
||||
For scenario tests having a service tag is required. For the API tests service
|
||||
tags are only needed if the test method makes an API call (either directly or
|
||||
indirectly through another service) that differs from the parent directory
|
||||
name. For example, any test that make an api call to a service other than nova
|
||||
in tempest.api.compute would require a service tag for those services, however
|
||||
they do not need to be tagged as compute.
|
||||
name. For example, any test that make an API call to a service other than Nova
|
||||
in ``tempest.api.compute`` would require a service tag for those services,
|
||||
however they do not need to be tagged as ``compute``.
|
||||
|
||||
Test fixtures and resources
|
||||
---------------------------
|
||||
@ -198,7 +198,7 @@ will expand the guideline based on our discussion and experience.
|
||||
Test skips because of Known Bugs
|
||||
--------------------------------
|
||||
If a test is broken because of a bug it is appropriate to skip the test until
|
||||
bug has been fixed. You should use the skip_because decorator so that
|
||||
bug has been fixed. You should use the ``skip_because`` decorator so that
|
||||
Tempest's skip tracking tool can watch the bug status.
|
||||
|
||||
Example::
|
||||
@ -229,7 +229,7 @@ parallel.
|
||||
require admin privileges are outside of projects.
|
||||
|
||||
- Races between methods in the same class are not a problem because
|
||||
parallelization in tempest is at the test class level, but if there is a json
|
||||
parallelization in Tempest is at the test class level, but if there is a json
|
||||
and xml version of the same test class there could still be a race between
|
||||
methods.
|
||||
|
||||
@ -238,8 +238,8 @@ parallel.
|
||||
avoided to prevent resource conflicts.
|
||||
|
||||
- If the execution of a set of tests is required to be serialized then locking
|
||||
can be used to perform this. See AggregatesAdminTest in
|
||||
tempest.api.compute.admin for an example of using locking.
|
||||
can be used to perform this. See usage of ``LockFixture`` for examples of
|
||||
using locking.
|
||||
|
||||
Sample Configuration File
|
||||
-------------------------
|
||||
@ -251,7 +251,7 @@ regenerated. This can be done running::
|
||||
|
||||
Unit Tests
|
||||
----------
|
||||
Unit tests are a separate class of tests in tempest. They verify tempest
|
||||
Unit tests are a separate class of tests in Tempest. They verify Tempest
|
||||
itself, and thus have a different set of guidelines around them:
|
||||
|
||||
1. They can not require anything running externally. All you should need to
|
||||
@ -321,8 +321,8 @@ format of the metadata looks like::
|
||||
|
||||
Tempest.lib includes a ``check-uuid`` tool that will test for the existence
|
||||
and uniqueness of idempotent_id metadata for every test. If you have
|
||||
tempest installed you run the tool against Tempest by calling from the
|
||||
tempest repo::
|
||||
Tempest installed you run the tool against Tempest by calling from the
|
||||
Tempest repo::
|
||||
|
||||
check-uuid
|
||||
|
||||
@ -337,7 +337,7 @@ one::
|
||||
|
||||
check-uuid --fix
|
||||
|
||||
The ``check-uuid`` tool is used as part of the tempest gate job
|
||||
The ``check-uuid`` tool is used as part of the Tempest gate job
|
||||
to ensure that all tests have an ``idempotent_id`` decorator.
|
||||
|
||||
Branchless Tempest Considerations
|
||||
@ -350,7 +350,7 @@ branches are also gated by the Tempest master branch, which also means that
|
||||
proposed commits to Tempest must work against both the master and all the
|
||||
currently supported stable branches of the projects. As such there are a few
|
||||
special considerations that have to be accounted for when pushing new changes
|
||||
to tempest.
|
||||
to Tempest.
|
||||
|
||||
1. New Tests for new features
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@ -370,14 +370,22 @@ be able to merge.
|
||||
When trying to land a bug fix which changes a tested API you'll have to use the
|
||||
following procedure::
|
||||
|
||||
- Propose change to the project, get a +2 on the change even with failing
|
||||
- Propose skip on Tempest which will only be approved after the
|
||||
1. Propose change to the project, get a +2 on the change even with failing
|
||||
2. Propose skip on Tempest which will only be approved after the
|
||||
corresponding change in the project has a +2 on change
|
||||
- Land project change in master and all open stable branches (if required)
|
||||
- Land changed test in Tempest
|
||||
3. Land project change in master and all open stable branches (if required)
|
||||
4. Land changed test in Tempest
|
||||
|
||||
Otherwise the bug fix won't be able to land in the project.
|
||||
|
||||
Handily, `Zuul’s cross-repository dependencies
|
||||
<https://docs.openstack.org/infra/zuul/gating.html#cross-repository-dependencies>`_.
|
||||
can be leveraged to do without step 2 and to have steps 3 and 4 happen
|
||||
"atomically". To do that, make the patch written in step 1 to depend (refer to
|
||||
Zuul's documentation above) on the patch written in step 4. The commit message
|
||||
for the Tempest change should have a link to the Gerrit review that justifies
|
||||
that change.
|
||||
|
||||
3. New Tests for existing features
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
54
README.rst
54
README.rst
@ -23,7 +23,7 @@ Design Principles
|
||||
Tempest Design Principles that we strive to live by.
|
||||
|
||||
- Tempest should be able to run against any OpenStack cloud, be it a
|
||||
one node devstack install, a 20 node lxc cloud, or a 1000 node kvm
|
||||
one node DevStack install, a 20 node LXC cloud, or a 1000 node KVM
|
||||
cloud.
|
||||
- Tempest should be explicit in testing features. It is easy to auto
|
||||
discover features of a cloud incorrectly, and give people an
|
||||
@ -65,13 +65,13 @@ as it is simpler, and quicker to work with.
|
||||
$ pip install tempest/
|
||||
|
||||
This can be done within a venv, but the assumption for this guide is that
|
||||
the Tempest cli entry point will be in your shell's PATH.
|
||||
the Tempest CLI entry point will be in your shell's PATH.
|
||||
|
||||
#. Installing Tempest may create a /etc/tempest dir, however if one isn't
|
||||
created you can create one or use ~/.tempest/etc or ~/.config/tempest in
|
||||
place of /etc/tempest. If none of these dirs are created tempest will create
|
||||
~/.tempest/etc when it's needed. The contents of this dir will always
|
||||
automatically be copied to all etc/ dirs in local workspaces as an initial
|
||||
#. Installing Tempest may create a ``/etc/tempest dir``, however if one isn't
|
||||
created you can create one or use ``~/.tempest/etc`` or ``~/.config/tempest`` in
|
||||
place of ``/etc/tempest``. If none of these dirs are created Tempest will create
|
||||
``~/.tempest/etc`` when it's needed. The contents of this dir will always
|
||||
automatically be copied to all ``etc/`` dirs in local workspaces as an initial
|
||||
setup step. So if there is any common configuration you'd like to be shared
|
||||
between local Tempest workspaces it's recommended that you pre-populate it
|
||||
before running ``tempest init``.
|
||||
@ -90,12 +90,12 @@ as it is simpler, and quicker to work with.
|
||||
is that you'll create a new working directory for each to maintain separate
|
||||
configuration files and local artifact storage for each.
|
||||
|
||||
#. Then cd into the newly created working dir and also modify the local
|
||||
config files located in the etc/ subdir created by the ``tempest init``
|
||||
command. Tempest is expecting a tempest.conf file in etc/ so if only a
|
||||
#. Then ``cd`` into the newly created working dir and also modify the local
|
||||
config files located in the ``etc/`` subdir created by the ``tempest init``
|
||||
command. Tempest is expecting a ``tempest.conf`` file in etc/ so if only a
|
||||
sample exists you must rename or copy it to tempest.conf before making
|
||||
any changes to it otherwise Tempest will not know how to load it. For
|
||||
details on configuring tempest refer to the :ref:`tempest-configuration`.
|
||||
details on configuring Tempest refer to the :ref:`tempest-configuration`.
|
||||
|
||||
#. Once the configuration is done you're now ready to run Tempest. This can
|
||||
be done using the :ref:`tempest_run` command. This can be done by either
|
||||
@ -124,8 +124,8 @@ Library
|
||||
Tempest exposes a library interface. This interface is a stable interface and
|
||||
should be backwards compatible (including backwards compatibility with the
|
||||
old tempest-lib package, with the exception of the import). If you plan to
|
||||
directly consume tempest in your project you should only import code from the
|
||||
tempest library interface, other pieces of tempest do not have the same
|
||||
directly consume Tempest in your project you should only import code from the
|
||||
Tempest library interface, other pieces of Tempest do not have the same
|
||||
stable interface and there are no guarantees on the Python API unless otherwise
|
||||
stated.
|
||||
|
||||
@ -137,7 +137,7 @@ Release Versioning
|
||||
shows what changes have been released on each version.
|
||||
|
||||
Tempest's released versions are broken into 2 sets of information. Depending on
|
||||
how you intend to consume tempest you might need
|
||||
how you intend to consume Tempest you might need
|
||||
|
||||
The version is a set of 3 numbers:
|
||||
|
||||
@ -146,12 +146,12 @@ X.Y.Z
|
||||
While this is almost `semver`_ like, the way versioning is handled is slightly
|
||||
different:
|
||||
|
||||
X is used to represent the supported OpenStack releases for tempest tests
|
||||
in-tree, and to signify major feature changes to tempest. It's a monotonically
|
||||
X is used to represent the supported OpenStack releases for Tempest tests
|
||||
in-tree, and to signify major feature changes to Tempest. It's a monotonically
|
||||
increasing integer where each version either indicates a new supported OpenStack
|
||||
release, the drop of support for an OpenStack release (which will coincide with
|
||||
the upstream stable branch going EOL), or a major feature lands (or is removed)
|
||||
from tempest.
|
||||
from Tempest.
|
||||
|
||||
Y.Z is used to represent library interface changes. This is treated the same
|
||||
way as minor and patch versions from `semver`_ but only for the library
|
||||
@ -166,16 +166,16 @@ Configuration
|
||||
|
||||
Detailed configuration of Tempest is beyond the scope of this
|
||||
document see :ref:`tempest-configuration` for more details on configuring
|
||||
Tempest. The etc/tempest.conf.sample attempts to be a self-documenting version
|
||||
of the configuration.
|
||||
Tempest. The ``etc/tempest.conf.sample`` attempts to be a self-documenting
|
||||
version of the configuration.
|
||||
|
||||
You can generate a new sample tempest.conf file, run the following
|
||||
command from the top level of the Tempest directory::
|
||||
|
||||
$ tox -e genconfig
|
||||
|
||||
The most important pieces that are needed are the user ids, openstack
|
||||
endpoint, and basic flavors and images needed to run tests.
|
||||
The most important pieces that are needed are the user ids, OpenStack
|
||||
endpoints, and basic flavors and images needed to run tests.
|
||||
|
||||
Unit Tests
|
||||
----------
|
||||
@ -190,13 +190,13 @@ should only be run on the unit test directory. The default value of OS_TEST_PATH
|
||||
is OS_TEST_PATH=./tempest/test_discover which will only run test discover on the
|
||||
Tempest suite.
|
||||
|
||||
Alternatively, there are the py27 and py34 tox jobs which will run the unit
|
||||
Alternatively, there are the py27 and py35 tox jobs which will run the unit
|
||||
tests with the corresponding version of python.
|
||||
|
||||
Python 2.6
|
||||
----------
|
||||
|
||||
Starting in the kilo release the OpenStack services dropped all support for
|
||||
Starting in the Kilo release the OpenStack services dropped all support for
|
||||
python 2.6. This change has been mirrored in Tempest, starting after the
|
||||
tempest-2 tag. This means that proposed changes to Tempest which only fix
|
||||
python 2.6 compatibility will be rejected, and moving forward more features not
|
||||
@ -208,8 +208,8 @@ that has python 2.7)
|
||||
Python 3.x
|
||||
----------
|
||||
|
||||
Starting during the Pike cycle Tempest has a gating CI job that runs tempest
|
||||
with Python 3. Any tempest release after 15.0.0 should fully support running
|
||||
Starting during the Pike cycle Tempest has a gating CI job that runs Tempest
|
||||
with Python 3. Any Tempest release after 15.0.0 should fully support running
|
||||
under Python 3 as well as Python 2.7.
|
||||
|
||||
Legacy run method
|
||||
@ -239,10 +239,10 @@ and reference data to be used in testing.
|
||||
|
||||
.. note::
|
||||
|
||||
If you have a running devstack environment, Tempest will be
|
||||
If you have a running DevStack environment, Tempest will be
|
||||
automatically configured and placed in ``/opt/stack/tempest``. It
|
||||
will have a configuration file already set up to work with your
|
||||
devstack installation.
|
||||
DevStack installation.
|
||||
|
||||
Tempest is not tied to any single test runner, but `testr`_ is the most commonly
|
||||
used tool. Also, the nosetests test runner is **not** recommended to run Tempest.
|
||||
|
||||
@ -21,6 +21,15 @@ manual/developers.html#cross-repository-dependencies>`_ or a simple link
|
||||
to a Gerrit review.
|
||||
|
||||
|
||||
Execution time
|
||||
--------------
|
||||
While checking in the job logs that a new test is actually executed, also
|
||||
pay attention to the execution time of that test. Keep in mind that each test
|
||||
is going to be executed hundreds of time each day, because Tempest tests
|
||||
run in many OpenStack projects. It's worth considering how important/critical
|
||||
the feature under test is with how costly the new test is.
|
||||
|
||||
|
||||
Unit Tests
|
||||
----------
|
||||
|
||||
@ -48,6 +57,17 @@ the maintenance burden. Such changes should not be approved if it is easy to
|
||||
abstract the duplicated code into a function or method.
|
||||
|
||||
|
||||
Tests overlap
|
||||
-------------
|
||||
When a new test is being proposed, question whether this feature is not already
|
||||
tested with Tempest. Tempest has more than 1200 tests, spread amongst many
|
||||
directories, so it's easy to introduce test duplication. For example, testing
|
||||
volume attachment to a server could be a compute test or a volume test, depending
|
||||
on how you see it. So one must look carefully in the entire code base for possible
|
||||
overlap. As a rule of thumb, the older a feature is, the more likely it's
|
||||
already tested.
|
||||
|
||||
|
||||
Being explicit
|
||||
--------------
|
||||
When tests are being added that depend on a configurable feature or extension,
|
||||
@ -60,8 +80,8 @@ whether to skip or not.
|
||||
|
||||
Configuration Options
|
||||
---------------------
|
||||
With the introduction of the tempest external test plugin interface we needed
|
||||
to provide a stable contract for tempest's configuration options. This means
|
||||
With the introduction of the Tempest external test plugin interface we needed
|
||||
to provide a stable contract for Tempest's configuration options. This means
|
||||
we can no longer simply remove a configuration option when it's no longer used.
|
||||
Patches proposed that remove options without a deprecation cycle should not
|
||||
be approved. Similarly when changing default values with configuration we need
|
||||
|
||||
@ -28,7 +28,7 @@ can be used to:
|
||||
- Run tests for admin APIs
|
||||
- Generate test credentials on the fly (see `Dynamic Credentials`_)
|
||||
|
||||
When keystone uses a policy that requires domain scoped tokens for admin
|
||||
When Keystone uses a policy that requires domain scoped tokens for admin
|
||||
actions, the flag ``admin_domain_scope`` must be set to ``True``.
|
||||
The admin user configured, if any, must have a role assigned to the domain to
|
||||
be usable.
|
||||
@ -39,7 +39,7 @@ This can be done using the accounts.yaml file (see
|
||||
number of users available to run tests with.
|
||||
You can specify the location of the file in the ``auth`` section in the
|
||||
tempest.conf file. To see the specific format used in the file please refer to
|
||||
the accounts.yaml.sample file included in Tempest.
|
||||
the ``accounts.yaml.sample`` file included in Tempest.
|
||||
|
||||
Keystone Connection Info
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@ -47,18 +47,17 @@ In order for Tempest to be able to talk to your OpenStack deployment you need
|
||||
to provide it with information about how it communicates with keystone.
|
||||
This involves configuring the following options in the ``identity`` section:
|
||||
|
||||
#. ``auth_version``
|
||||
#. ``uri``
|
||||
#. ``uri_v3``
|
||||
- ``auth_version``
|
||||
- ``uri``
|
||||
- ``uri_v3``
|
||||
|
||||
The ``auth_version`` option is used to tell Tempest whether it should be using
|
||||
keystone's v2 or v3 api for communicating with keystone. The two uri options are
|
||||
Keystone's v2 or v3 api for communicating with Keystone. The two uri options are
|
||||
used to tell Tempest the url of the keystone endpoint. The ``uri`` option is
|
||||
used for keystone v2 request and ``uri_v3`` is used for keystone v3. You want to
|
||||
used for Keystone v2 request and ``uri_v3`` is used for Keystone v3. You want to
|
||||
ensure that which ever version you set for ``auth_version`` has its uri option
|
||||
defined.
|
||||
|
||||
|
||||
Credential Provider Mechanisms
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
@ -95,7 +94,7 @@ configured to use dynamic credentials.
|
||||
When the ``admin_domain_scope`` option is set to ``True``, provisioned admin
|
||||
accounts will be assigned a role on domain configured in
|
||||
``default_credentials_domain_name``. This will make the accounts provisioned
|
||||
usable in a cloud where domain scoped tokens are required by keystone for
|
||||
usable in a cloud where domain scoped tokens are required by Keystone for
|
||||
admin operations. Note that the initial pre-provision admin accounts,
|
||||
configured in tempest.conf, must have a role on the same domain as well, for
|
||||
Dynamic Credentials to work.
|
||||
@ -140,10 +139,10 @@ used do not have any pre-existing resources created. Tempest assumes all
|
||||
tenants it's using are empty and may sporadically fail if there are unexpected
|
||||
resources present.
|
||||
|
||||
When the keystone in the target cloud requires domain scoped tokens to
|
||||
When the Keystone in the target cloud requires domain scoped tokens to
|
||||
perform admin actions, all pre-provisioned admin users must have a role
|
||||
assigned on the domain where test accounts a provisioned.
|
||||
The option ``admin_domain_scope`` is used to tell tempest that domain scoped
|
||||
The option ``admin_domain_scope`` is used to tell Tempest that domain scoped
|
||||
tokens shall be used. ``default_credentials_domain_name`` is the domain where
|
||||
test accounts are expected to be provisioned if no domain is specified.
|
||||
|
||||
@ -167,7 +166,7 @@ for doing this:
|
||||
#. ``flavor_ref_alt``
|
||||
|
||||
Both of these options are in the ``compute`` section of the config file and take
|
||||
in the flavor id (not the name) from nova. The ``flavor_ref`` option is what
|
||||
in the flavor id (not the name) from Nova. The ``flavor_ref`` option is what
|
||||
will be used for booting almost all of the guests; ``flavor_ref_alt`` is only
|
||||
used in tests where two different-sized servers are required (for example, a
|
||||
resize test).
|
||||
@ -185,7 +184,7 @@ servers. There are two options in the compute section just like with flavors:
|
||||
#. ``image_ref``
|
||||
#. ``image_ref_alt``
|
||||
|
||||
Both options are expecting an image id (not name) from nova. The ``image_ref``
|
||||
Both options are expecting an image id (not name) from Nova. The ``image_ref``
|
||||
option is what will be used for booting the majority of servers in Tempest.
|
||||
``image_ref_alt`` is used for tests that require two images such as rebuild. If
|
||||
two images are not available you can set both options to the same image id and
|
||||
@ -224,7 +223,7 @@ small footprint.
|
||||
Networking
|
||||
----------
|
||||
OpenStack has a myriad of different networking configurations possible and
|
||||
depending on which of the two network backends, nova-network or neutron, you are
|
||||
depending on which of the two network backends, nova-network or Neutron, you are
|
||||
using things can vary drastically. Due to this complexity Tempest has to provide
|
||||
a certain level of flexibility in its configuration to ensure it will work
|
||||
against any cloud. This ends up causing a large number of permutations in
|
||||
@ -302,21 +301,21 @@ cases it shouldn't matter because nova-network should have no problem booting a
|
||||
server with multiple networks. If this is not the case for your cloud then using
|
||||
an accounts file is recommended because it provides the necessary flexibility to
|
||||
describe your configuration. Dynamic credentials is not able to dynamically
|
||||
allocate things as necessary if neutron is not enabled.
|
||||
allocate things as necessary if Neutron is not enabled.
|
||||
|
||||
With neutron and dynamic credentials enabled there should not be any additional
|
||||
With Neutron and dynamic credentials enabled there should not be any additional
|
||||
configuration necessary to enable Tempest to create servers with working
|
||||
networking, assuming you have properly configured the ``network`` section to
|
||||
work for your cloud. Tempest will dynamically create the neutron resources
|
||||
work for your cloud. Tempest will dynamically create the Neutron resources
|
||||
necessary to enable using servers with that network. Also, just as with the
|
||||
accounts file, if you specify a fixed network name while using neutron and
|
||||
accounts file, if you specify a fixed network name while using Neutron and
|
||||
dynamic credentials it will enable running tests which require a static network
|
||||
and it will additionally be used as a fallback for server creation. However,
|
||||
unlike accounts.yaml this should never be triggered.
|
||||
|
||||
However, there is an option ``create_isolated_networks`` to disable dynamic
|
||||
credentials's automatic provisioning of network resources. If this option is set
|
||||
to False you will have to either rely on there only being a single/default
|
||||
to ``False`` you will have to either rely on there only being a single/default
|
||||
network available for the server creation, or use ``fixed_network_name`` to
|
||||
inform Tempest which network to use.
|
||||
|
||||
@ -336,16 +335,16 @@ To enable remote access to servers, there are 3 options at a minimum that are us
|
||||
|
||||
The ``run_validation`` is used to enable or disable ssh connectivity for
|
||||
all tests (with the exception of scenario tests which do not have a flag for
|
||||
enabling or disabling ssh) To enable ssh connectivity this needs be set to ``true``.
|
||||
enabling or disabling ssh) To enable ssh connectivity this needs be set to ``True``.
|
||||
|
||||
The ``connect_method`` option is used to tell tempest what kind of IP to use for
|
||||
The ``connect_method`` option is used to tell Tempest what kind of IP to use for
|
||||
establishing a connection to the server. Two methods are available: ``fixed``
|
||||
and ``floating``, the later being set by default. If this is set to floating
|
||||
tempest will create a floating ip for the server before attempted to connect
|
||||
Tempest will create a floating ip for the server before attempted to connect
|
||||
to it. The IP for the floating ip is what is used for the connection.
|
||||
|
||||
For the ``auth_method`` option there is currently, only one valid option,
|
||||
``keypair``. With this set to ``keypair`` tempest will create an ssh keypair
|
||||
``keypair``. With this set to ``keypair`` Tempest will create an ssh keypair
|
||||
and use that for authenticating against the created server.
|
||||
|
||||
Configuring Available Services
|
||||
@ -359,8 +358,8 @@ on the available services.
|
||||
|
||||
The ``service_available`` section of the config file is used to set which
|
||||
services are available. It contains a boolean option for each service (except
|
||||
for keystone which is a hard requirement) set it to True if the service is
|
||||
available or False if it is not.
|
||||
for Keystone which is a hard requirement) set it to ``True`` if the service is
|
||||
available or ``False`` if it is not.
|
||||
|
||||
Service Catalog
|
||||
^^^^^^^^^^^^^^^
|
||||
@ -382,17 +381,18 @@ has multiple regions available and you need to specify a particular one to use a
|
||||
service you can set the ``region`` option in that service's section.
|
||||
|
||||
It should also be noted that the default values for these options are set
|
||||
to what devstack uses (which is a de facto standard for service catalog
|
||||
to what DevStack uses (which is a de facto standard for service catalog
|
||||
entries). So often nothing actually needs to be set on these options to enable
|
||||
communication to a particular service. It is only if you are either not using
|
||||
the same ``catalog_type`` as devstack or you want Tempest to talk to a different
|
||||
endpoint type instead of publicURL for a service that these need to be changed.
|
||||
the same ``catalog_type`` as DevStack or you want Tempest to talk to a different
|
||||
endpoint type instead of ``publicURL`` for a service that these need to be
|
||||
changed.
|
||||
|
||||
.. note::
|
||||
|
||||
Tempest does not serve all kinds of fancy URLs in the service catalog. The
|
||||
service catalog should be in a standard format (which is going to be
|
||||
standardized at the keystone level).
|
||||
standardized at the Keystone level).
|
||||
Tempest expects URLs in the Service catalog in the following format:
|
||||
|
||||
* ``http://example.com:1234/<version-info>``
|
||||
@ -420,7 +420,7 @@ for a feature (assuming one exists). Instead Tempest has to be explicitly
|
||||
configured as to which optional features are enabled. This is in order to
|
||||
prevent bugs in the discovery mechanisms from masking failures.
|
||||
|
||||
The service feature-enabled config sections are how Tempest addresses the
|
||||
The service ``feature-enabled`` config sections are how Tempest addresses the
|
||||
optional feature question. Each service that has tests for optional features
|
||||
contains one of these sections. The only options in it are boolean options
|
||||
with the name of a feature which is used. If it is set to false any test which
|
||||
@ -431,7 +431,7 @@ options refer to the sample config file.
|
||||
API Extensions
|
||||
^^^^^^^^^^^^^^
|
||||
The service feature-enabled sections often contain an ``api-extensions`` option
|
||||
(or in the case of swift a ``discoverable_apis`` option). This is used to tell
|
||||
(or in the case of Swift a ``discoverable_apis`` option). This is used to tell
|
||||
Tempest which api extensions (or configurable middleware) is used in your
|
||||
deployment. It has two valid config states: either it contains a single value
|
||||
``all`` (which is the default) which means that every api extension is assumed
|
||||
|
||||
@ -6,13 +6,13 @@ Tempest Test Writing Guide
|
||||
This guide serves as a starting point for developers working on writing new
|
||||
Tempest tests. At a high level tests in Tempest are just tests that conform to
|
||||
the standard python `unit test`_ framework. But there are several aspects of
|
||||
that are unique to tempest and it's role as an integration test suite running
|
||||
that are unique to Tempest and its role as an integration test suite running
|
||||
against a real cloud.
|
||||
|
||||
.. _unit test: https://docs.python.org/3.6/library/unittest.html
|
||||
|
||||
.. note:: This guide is for writing tests in the tempest repository. While many
|
||||
parts of this guide are also applicable to tempest plugins, not all
|
||||
.. note:: This guide is for writing tests in the Tempest repository. While many
|
||||
parts of this guide are also applicable to Tempest plugins, not all
|
||||
the APIs mentioned are considered stable or recommended for use in
|
||||
plugins. Please refer to :ref:`tempest_plugin` for details about
|
||||
writing plugins
|
||||
@ -24,8 +24,8 @@ Adding a New TestCase
|
||||
The base unit of testing in Tempest is the `TestCase`_ (also called the test
|
||||
class). Each TestCase contains test methods which are the individual tests that
|
||||
will be executed by the test runner. But, the TestCase is the smallest self
|
||||
contained unit for tests from the tempest perspective. It's also the level at
|
||||
which tempest is parallel safe. In other words, multiple TestCases can be
|
||||
contained unit for tests from the Tempest perspective. It's also the level at
|
||||
which Tempest is parallel safe. In other words, multiple TestCases can be
|
||||
executed in parallel, but individual test methods in the same TestCase can not.
|
||||
Also, all test methods within a TestCase are assumed to be executed serially. As
|
||||
such you can use the test case to store variables that are shared between
|
||||
@ -151,7 +151,7 @@ for tests cases that need a second user/project.
|
||||
You can also specify credentials with specific roles assigned. This is useful
|
||||
for cases where there are specific RBAC requirements hard coded into an API.
|
||||
The canonical example of this are swift tests which often want to test swift's
|
||||
concepts of operator and reseller_admin. An actual example from tempest on how
|
||||
concepts of operator and reseller_admin. An actual example from Tempest on how
|
||||
to do this is::
|
||||
|
||||
class PublicObjectTest(base.BaseObjectTest):
|
||||
@ -187,7 +187,7 @@ requested credentials you can reference:
|
||||
| [$label, $role] | cls.os_roles_$label |
|
||||
+-------------------+---------------------+
|
||||
|
||||
By default cls.os_primary is available since it is allocated in the base tempest test
|
||||
By default cls.os_primary is available since it is allocated in the base Tempest test
|
||||
class (located in tempest/test.py). If your TestCase inherits from a different
|
||||
direct parent class (it'll still inherit from the BaseTestCase, just not
|
||||
directly) be sure to check if that class overrides allocated credentials.
|
||||
@ -195,8 +195,8 @@ directly) be sure to check if that class overrides allocated credentials.
|
||||
Dealing with Network Allocation
|
||||
'''''''''''''''''''''''''''''''
|
||||
|
||||
When neutron is enabled and a testing requires networking this isn't normally
|
||||
automatically setup when a tenant is created. Since tempest needs isolated
|
||||
When Neutron is enabled and a testing requires networking this isn't normally
|
||||
automatically setup when a tenant is created. Since Tempest needs isolated
|
||||
tenants to function properly it also needs to handle network allocation. By
|
||||
default the base test class will allocate a network, subnet, and router
|
||||
automatically (this depends on the configured credential provider, for more
|
||||
|
||||
@ -9,12 +9,13 @@ implementations for both correctness, as well as a burn in tool for
|
||||
OpenStack clouds.
|
||||
|
||||
As such Tempest tests come in many flavors, each with their own rules
|
||||
and guidelines. Below is the proposed Havana restructuring for Tempest
|
||||
and guidelines. Below is the overview of the Tempest respository structure
|
||||
to make this clear.
|
||||
|
||||
| tempest/
|
||||
| api/ - API tests
|
||||
| scenario/ - complex scenario tests
|
||||
| tests/ - unit tests for Tempest internals
|
||||
|
||||
Each of these directories contains different types of tests. What
|
||||
belongs in each directory, the rules and examples for good tests, are
|
||||
@ -24,8 +25,8 @@ documented in a README.rst file in the directory.
|
||||
----------------------
|
||||
|
||||
API tests are validation tests for the OpenStack API. They should not
|
||||
use the existing python clients for OpenStack, but should instead use
|
||||
the tempest implementations of clients. Having raw clients let us
|
||||
use the existing Python clients for OpenStack, but should instead use
|
||||
the Tempest implementations of clients. Having raw clients let us
|
||||
pass invalid JSON to the APIs and see the results, something we could
|
||||
not get with the native clients.
|
||||
|
||||
@ -41,14 +42,14 @@ Scenario tests are complex "through path" tests for OpenStack
|
||||
functionality. They are typically a series of steps where complicated
|
||||
state requiring multiple services is set up exercised, and torn down.
|
||||
|
||||
Scenario tests should not use the existing python clients for OpenStack,
|
||||
but should instead use the tempest implementations of clients.
|
||||
Scenario tests should not use the existing Python clients for OpenStack,
|
||||
but should instead use the Tempest implementations of clients.
|
||||
|
||||
|
||||
:ref:`unit_tests_field_guide`
|
||||
-----------------------------
|
||||
|
||||
Unit tests are the self checks for Tempest. They provide functional
|
||||
verification and regression checking for the internal components of tempest.
|
||||
They should be used to just verify that the individual pieces of tempest are
|
||||
verification and regression checking for the internal components of Tempest.
|
||||
They should be used to just verify that the individual pieces of Tempest are
|
||||
working as expected.
|
||||
|
||||
@ -13,7 +13,8 @@ portion of Tempest code is devoted to test cases that do exactly this.
|
||||
|
||||
It's also important to test not only the expected positive path on
|
||||
APIs, but also to provide them with invalid data to ensure they fail
|
||||
in expected and documented ways. Over the course of the OpenStack
|
||||
in expected and documented ways. The latter type of tests is called
|
||||
``negative tests`` in Tempest source code. Over the course of the OpenStack
|
||||
project Tempest has discovered many fundamental bugs by doing just
|
||||
this.
|
||||
|
||||
@ -22,7 +23,7 @@ enough data in the system. This means these tests might start by
|
||||
spinning up a server, image, etc, then operating on it.
|
||||
|
||||
|
||||
Why are these tests in tempest?
|
||||
Why are these tests in Tempest?
|
||||
-------------------------------
|
||||
|
||||
This is one of the core missions for the Tempest project, and where it
|
||||
|
||||
@ -21,9 +21,9 @@ Any scenario test should have a real-life use case. An example would be:
|
||||
4. create a snapshot of the vm
|
||||
|
||||
|
||||
Why are these tests in tempest?
|
||||
Why are these tests in Tempest?
|
||||
-------------------------------
|
||||
This is one of tempests core purposes, testing the integration between
|
||||
This is one of Tempest's core purposes, testing the integration between
|
||||
projects.
|
||||
|
||||
|
||||
@ -43,7 +43,7 @@ While we are looking for interaction of 2 or more services, be
|
||||
specific in your interactions. A giant "this is my data center" smoke
|
||||
test is hard to debug when it goes wrong.
|
||||
|
||||
A flow of interactions between glance and nova, like in the
|
||||
A flow of interactions between Glance and Nova, like in the
|
||||
introduction, is a good example. Especially if it involves a repeated
|
||||
interaction when a resource is setup, modified, detached, and then
|
||||
reused later again.
|
||||
|
||||
@ -7,16 +7,16 @@ What are these tests?
|
||||
---------------------
|
||||
|
||||
Unit tests are the self checks for Tempest. They provide functional
|
||||
verification and regression checking for the internal components of tempest.
|
||||
They should be used to just verify that the individual pieces of tempest are
|
||||
verification and regression checking for the internal components of Tempest.
|
||||
They should be used to just verify that the individual pieces of Tempest are
|
||||
working as expected. They should not require an external service to be running
|
||||
and should be able to run solely from the tempest tree.
|
||||
and should be able to run solely from the Tempest tree.
|
||||
|
||||
Why are these tests in tempest?
|
||||
Why are these tests in Tempest?
|
||||
-------------------------------
|
||||
These tests exist to make sure that the mechanisms that we use inside of
|
||||
tempest to are valid and remain functional. They are only here for self
|
||||
validation of tempest.
|
||||
Tempest are valid and remain functional. They are only here for self
|
||||
validation of Tempest.
|
||||
|
||||
|
||||
Scope of these tests
|
||||
|
||||
Loading…
Reference in New Issue
Block a user