As Tempest support the keystone's scope and new default roles
like reader, this commit document that how those can be
requested and used in Tempest or its plugins tests.
Change-Id: Iebacbeda231f82d6d16dbdcde635f19ae862181f
Generate documentation for each test from its docstring.
The sphinx-apidoc tool is used to generate the .rst
files necessary for the generation of the documentation.
When the documentation is being build only functions that
start with "test" are passed to the resulting documentation.
Change-Id: I1878f67e10df9bb1f8289beda8e59a56b5057fe4
As discussed in Victoria PTG[1], we need to start pining
the constraint in tox.ini on Tempest new release so that our
tox.ini can match with compatible constraints with Tempest version.
[1] https://etherpad.opendev.org/p/qa-victoria-ptg
Change-Id: Ib705f59ef3aea93b2778cc87fc44d390b5a62c94
Now we have more documents for plugins on stable release,
support and their testing policy. Let's prepare a
index page for it will be easy for plugins maintainer.
Also renaming the stable support policy section to have a clear
name.
Adding redirect links for moved documents.
Change-Id: I17a9a6e0e88605a0ee5047678d6ed168c31396ec
As Tempest and plugins master version is used to test the supported
stable branches, we need to test them on master gate.
Adding document for that so that we can test the stable branches
more consistently on plugins side too.
Change-Id: Ied3c6bb4032fd3740ae0d6d185e4a138fd335b02
Tempest and its plugin are very tightly coupled for testing
Upstream or Production Cloud. Due to its branchless model,
one of the challenges we faced/facing is to provide their
compatible version set.
Since Queens release and with operator request, we started
doing a coordinated release of plugins along with Tempest
which will be marked as the First compatible version for
that particular release.
For example https://releases.openstack.org/train/index.html#tempest-plugins
We test our stable branch with master version of Tempest &
plugins until it is moved to Extended Maintainance state.
One stable release is in EM state, Tempest officially declares
the end of support tag so that we can use that as the last compatible
version for that EM stable testing. But we do not have the correct
last compatible version from plugins which can be used for testing
the EM branch. It should be with hash from the master at the time
stable release is moved to EM not the hash when the stable release
was done otherwise it will be incompatible with Tempest.
We have faced and fixed many issue due to that during py2-drop work
where we have to pick and try all different versions from plugin to
make it work against EM testing.
Change-Id: I99a6df0a2c02e19efaf484fcd97b875887f08d8e
Tempest is branchless and support many OpenStack stable branches
and python versions. Users of Tempest should have a clear doc about
what all openstack releases and python versions are supported.
Change-Id: Idc76e07efdd3710a59c773c564ca532419989ed5
This commit add PDF building tox.ini environment and options for LaTeX
output. overview.rst is copied from README.rst. And, the original
README.rst file is shrunk because we don't need such a long information
in the README file. People can see the same contents in overview.rst
now.
Change-Id: Id654c814988e78704726d2ba8bea9a03ce8596f8
Current doc explaining the stable branches support policy is under
coding guide. But support policy should be at higher level which
can be useful for user or maintainer of Tempest.
Change-Id: Id46e14080559062b4d89e5f7a9acbcd5df09691e
This commit fixes minor header wordings which were mentioned in the
previous patch review[1].
[1] Ia1e8e16e5f5ddb06e4c4ccb4979b0750b88a4db4
Change-Id: I4db09bda5628371186e3548143c59d0841fcb7b0
This commit updates the document theme. It is clear, simple, up-to-date
and beautiful. And this commit also updates section headers to fix the
document index.
Change-Id: Ia1e8e16e5f5ddb06e4c4ccb4979b0750b88a4db4
All other rst files don't contain "-" in the file names instead of '_'.
This patch renames test-removal.rst for consistent format on rst files.
TrivialFix
Change-Id: I229870baa1954e1f6b2277318ae5bae332f660ad
This commit is the start of a new doc guide for helping people figure
out how to write good tempest tests. This is just a starting point for
the document and doesn't cover every aspect of writing a test case in
tempest. Right now it just covers the setupClass phase for writing
TestCases. It will be expanded upon in future commits to give a complete
guide to writing tests in Tempest.
Change-Id: Ie0db2079163f536ae72668c8fc9129ae4c6389e2
It's not used anymore. There was general consensus in Feb 2016
to deprecate it (see [1]) and remove it in Newton.
[1] [qa] deprecating Tempest stress framework
Change-Id: Ib229985ea2a1fee495c9492c9ce1781e6bac1dc6
Javelin CLI utilities has been deprecated in Jan 2016
- Ie4ab5258946271a16e96a070abeedfafa5423714
which is over 6 months of Tempest deprecation period.
This patch removes the javelin CLI and corresponding doc/sample
files etc.
Change-Id: Ia59d3e81f245f29a7bf0f56b690586dfc55f16a0
This commit tidies up the tempest document index page. The tempest docs
are increasing day by day. It's really good however I feel it's a little
messy these days. So let's tidy up :)
Change-Id: I4d34934b8d1a157bf38a8f2cafe5de1ba0e6098c
Adds new command subunit-describe-calls, documentation,
and unittests.
subunit-describe-calls is a parser for subunit v1 & v2 streams to
determine what REST API calls are made inside of a test and in what
order they are called.
Later commits will add additional functionality relating to request
& response headers & body data along with a stdout output option and
stdin input.
Change-Id: I468d0d3e3b6098da95a81cc86d9bdd1b47ee1f03
This commit copies the Tempest test removal procedure doc from wiki[1]
to the tempest repo. Because this document is tightly coupled with
tempest so we don't need to manage it in the wiki. And we can see and
review this through the gerrit when we would like to update it.
And I'll update the wiki page to be just a link after merging this.
[1] https://wiki.openstack.org/wiki/QA/Tempest-test-removal
Change-Id: I5f2322a5de7b73252e0162165eeaec120ec94948
This commit adds a new run command to the unified cli endpoint. The
intent here is for tempest to control it's own run story. This
implements the basic runner and selection functionality to use the
command, however it's not necessarily the end state of the command.
The functionality in this patch is just a starting point to add the
command and the basic functionality needed. It is starting with a
limited feature set with the intent to add additional, more complex
functionality in self contained patches after the command exists.
Co-Authored-by: David Paterson <davpat2112@yahoo.com>
Co-Authored-by: Stephen Lowrie <stephen.lowrie@rackspace.com>
Partially-Implements bp tempest-run-cmd
Depends-On: I09299043e536521d48dbe10632621138e3a366e0
Change-Id: I24588b5c00d005320e8719cf82b5dd95662572cf
Create a consistent means for creation and management of Tempest
workspaces.
Creates a file located at ~/.tempest/workspaces.yaml which stores existing
workspaces.
Available subcommands: list, register, rename, move, remove
bp centralized-workspaces
Change-Id: I9595e3ba809e457951a0ffdf4b15f641f2fec4f4
This is based on the devstack plugin detection code, modified
to check setup.cfg for a tempest_plugins string. It is to be
executed from a periodic proposal job, generating a list of
plugins in the same format.
Change-Id: I53d082319b6af6fb47083ea42c3ff6b109882e72
This commit adds the doc about how to implement microversion
tests in Tempest.
Also this doc will keep record of what all microversion tests
have been implemented in Tempest.
Partially implements blueprint api-microversions-testing-support
Change-Id: I68e6d2a4a681e6588e43e02378a111fd00391f08
This commit adds documentation for the library interface, formerly
known as tempest-lib, and the release versioning.
Partially implements bp tempest-lib-reintegration
Change-Id: I623222a17fe3563bbe767fa803e8f771a21687a1
With the introduction of tempest plugins we can now remove the third
party tests for the ec2 api using boto. They've always been the ugly
duckling in tempest (except that it never turned into a swan) where
they go against some of the fundamental principles of tempest tests.
For exaple, like having it's own client implementation, oh and testing
OpenStack APIs. This patch removes all the pieces of the third party
test dir and deprecates all the config options related to boto. A
plugin implementation is being worked [1] that can be used to fill
the coverage hole left by removing these from the tempest tree
[1] https://github.com/mtreinish/tempest_ec2
Change-Id: Ib5e24e19bcba9808a9f49fe7f328668df77fe4f9
This commit fixes a few things in the tempest docs. First it fixes all
of the sphinx warnings and enables fail on warn to ensure we're using
valid sphinx everywhere. It also adds a link from the configuration
guide to the sample config file.
Change-Id: I3ad645a6bbfa46b4498e1732410743e46d6eb0cb
This commit starts the docs section on creating and using tempest
plugins. It describes how to create a plugin class, an entrypoint, and
a recommended dir structure. As well notes how to use tempest and
tempest-lib in creating a plugin. It'll likely be expanded in the
future as the plugin interface evolves.
Partially Implements bp external-plugin-interface
Change-Id: If2e0c47d0a719d869a3301ddbca07495c81cc498
Since including the sample configuration in tree is
becoming difficult to maintain, lets try moving it to
the developer docs. We still have the ability to view it
easily for users, and it can be auto-generated on doc build,
so it should also always be up to date.
Change-Id: I7ec2d76448feef5ecb9af7c238bbd12619fa31d8
Added docstring to cmd/account_generator.py.
Fixed help messages.
Added sphinx documentation for tempest-account-generator utility.
Change-Id: I6640ba8a0eaaea187cf38582ab537394dec020b6
This commit adds a new doc page to the tempest documentation to
provide a guide to configuring tempest. It currently contains one
section to elaborate on the credentials and auth options in the config
file. This doc will be expanded in the future to add details around
other pieces of tempest configuration.
Change-Id: Iaf7cefc6a5e508437e4b4620ba5bfaa221f3611b
I followed what's done for the cleanup.py script and re-used parts of
what was described in the javelin spec.
Change-Id: I57adc3764e75784d4614b6c03996433a4ee4332f
This commit removes the whitebox tests from tempest. The proper place
for this type of testing is in nova unit tests not in tempest.
Fixes bug 1222852
Change-Id: I0fac7b030c51985f9d6d93129bf9bab75c18cd11
This commit adds a framework for running unittests on tempest to
verify that tempest works as expect. The first tests added are to
verify the response codes of the test runner wrapper scripts we use
on jenkins jobs.
Change-Id: If8e91238054593999e7b5bc34da499bd0ac02311
This change adds an rst file for every tempest module, with the
'automodule' macro.
The root of the API documetation will be in the 'API and test cases'
section on the main page.
https://review.openstack.org/39427
Change-Id: I4065a24328dc85bb158b15bdfddb10ee0839916b
bug 1192692
Sphinx isn't configured to load the tempest modules,
so the easiest fix is to remove the links to the
general and module indices (which don't exist).
Change-Id: Ic33e0b6ce56cb31fa51b4cdce25524d40191235d
adjust the index file to be more readable for the field guide
adjust titles to make more sense when displayed in top level index
change theme to 'nature', which is closer to the openstack default
style. Will eventually write our own style here.
Change-Id: I4f5fed65f9e26a4e4d333b1be4ed02c9ec246b9a
This is a first bit of experiment to get a doc tree for tempest
that will publish our doc content to the web as part of a larger
QA doc guide.
Update some of the base READMEs to be a little more accurate,
now that we're going to be pushing them to the web.
Added tox [venv] to ensure that docs get built
Change-Id: Id54e8fbc1d884525087630106e17b37dc1206cc9