openstack
/
deb-tempest
Code Issues Proposed changes

Retire Packaging Deb project repos 

This commit is part of a series to retire the Packaging Deb
project. Step 2 is to remove all content from the project
repos, replacing it with a README notification where to find
ongoing work, and how to recover the repo if needed at some
future point (as in
https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project).

Change-Id: I3cb50c6ca8f660389ba2af2d8e78702d806509e9
This commit is contained in:
Tony Breeds 2017-09-12 16:20:49 -06:00
parent 72312d9911
commit b0aca38a60
1026 changed files with 14 additions and 104060 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.coveragerc
View File

@ -1,4 +0,0 @@
[run]
branch = True
source = tempest
omit = tempest/tests/*,tempest/scenario/test_*.py,tempest/api/*

31
.gitignore vendored
View File

@ -1,31 +0,0 @@
AUTHORS
ChangeLog
*.pyc
__pycache__/
etc/accounts.yaml
etc/tempest.conf
etc/tempest.conf.sample
etc/logging.conf
include/swift_objects/swift_small
include/swift_objects/swift_medium
include/swift_objects/swift_large
*.log
*.swp
*.swo
*.egg*
.tox
.venv
dist
build
.testrepository
.idea
.project
.pydevproject
.coverage*
!.coveragerc
cover/
doc/source/_static/tempest.conf.sample
doc/source/plugin-registry.rst
# Files created by releasenotes build
releasenotes/build

4
.gitreview
View File

@ -1,4 +0,0 @@
[gerrit]
host=review.openstack.org
port=29418
project=openstack/tempest.git

27
.mailmap
View File

@ -1,27 +0,0 @@
<brian.waldon@rackspace.com> <bcwaldon@gmail.com>
<jeblair@hp.com> <corvus@inaugust.com>
<jeblair@hp.com> <james.blair@rackspace.com>
Adam Gandelman <adamg@ubuntu.com> <adamg@canonical.com>
Andrea Frittoli <andrea.frittoli@gmail.com> <andrea.frittoli@hp.com>
Andrea Frittoli <andrea.frittoli@gmail.com> <andrea.frittoli@hpe.com>
Daryl Walleck <daryl.walleck@rackspace.com> <daryl.walleck@rackspace.com>
David Kranz <dkranz@redhat.com> David Kranz <david.kranz@qrclab.com>
Ghanshyam <ghanshyam.mann@nectechnologies.in> <ghanshyam.mann@nectechnologies.in>
Ghanshyam <ghanshyam.mann@nectechnologies.in> <ghanshyam.mann@nectechnologies.in>
Jay Pipes <jaypipes@gmail.com> <jpipes@librebox.gateway.2wire.net>
Joe Gordon <joe.gordon0@gmail.com> <jogo@cloudscaling.com>
Ken'ichi Ohmichi <ken-oomichi@wx.jp.nec.com> <oomichi@mxs.nes.nec.co.jp>
Ken'ichi Ohmichi <ken-oomichi@wx.jp.nec.com> <ken1ohmichi@gmail.com>
Marc Koderer <marc@koderer.com> <m.koderer@telekom.de>
Masayuki Igawa <masayuki@igawa.me> <igawa@mxs.nes.nec.co.jp>
Masayuki Igawa <masayuki@igawa.me> <mas-igawa@ut.jp.nec.com>
Masayuki Igawa <masayuki@igawa.me> <masayuki.igawa@gmail.com>
Matthew Treinish <mtreinish@kortar.org> <treinish@linux.vnet.ibm.com>
Nayna Patel <nayna.patel@hp.com> <nayna.patel@hp.com>
ravikumar-venkatesan <ravikumar.venkatesan@hp.com> <ravikumar.venkatesan@hp.com>
ravikumar-venkatesan <ravikumar.venkatesan@hp.com> <ravikumar.venkatesan@hp.com>
Rohit Karajgi <rohit.karajgi@nttdata.com> <rohit.karajgi@vertex.co.in>
Sean Dague <sean@dague.net> <sdague@linux.vnet.ibm.com>
Sean Dague <sean@dague.net> <sean.dague@samsung.com>
Yuiko Takada <takada-yuiko@mxn.nes.nec.co.jp> <takada-yuiko@mxn.nes.nec.co.jp>
Zhi Kun Liu <zhikunli@cn.ibm.com> <zhikunli@cn.ibm.com>

9
.testr.conf
View File

@ -1,9 +0,0 @@
[DEFAULT]
test_command=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} $LISTOPT $IDOPTION
test_id_option=--load-list $IDFILE
test_list_option=--list
group_regex=([^\.]*\.)*

403
HACKING.rst
View File

@ -1,403 +0,0 @@
Tempest Coding Guide
====================
- Step 1: Read the OpenStack Style Commandments
https://docs.openstack.org/hacking/latest/
- Step 2: Read on
Tempest Specific Commandments
------------------------------
- [T102] Cannot import OpenStack python clients in tempest/api &
tempest/scenario tests
- [T104] Scenario tests require a services decorator
- [T105] Tests cannot use setUpClass/tearDownClass
- [T106] vim configuration should not be kept in source files.
- [T107] Check that a service tag isn't in the module path
- [T108] Check no hyphen at the end of rand_name() argument
- [T109] Cannot use testtools.skip decorator; instead use
decorators.skip_because from tempest.lib
- [T110] Check that service client names of GET should be consistent
- [T111] Check that service client names of DELETE should be consistent
- [T112] Check that tempest.lib should not import local tempest code
- [T113] Check that tests use data_utils.rand_uuid() instead of uuid.uuid4()
- [T114] Check that tempest.lib does not use tempest config
- [T115] Check that admin tests should exist under admin path
- [N322] Method's default argument shouldn't be mutable
Test Data/Configuration
-----------------------
- Assume nothing about existing test data
- Tests should be self contained (provide their own data)
- Clean up test data at the completion of each test
- Use configuration files for values that will vary by environment
Exception Handling
------------------
According to the ``The Zen of Python`` the
``Errors should never pass silently.``
Tempest usually runs in special environment (jenkins gate jobs), in every
error or failure situation we should provide as much error related
information as possible, because we usually do not have the chance to
investigate the situation after the issue happened.
In every test case the abnormal situations must be very verbosely explained,
by the exception and the log.
In most cases the very first issue is the most important information.
Try to avoid using ``try`` blocks in the test cases, as both the ``except``
and ``finally`` blocks could replace the original exception,
when the additional operations leads to another exception.
Just letting an exception to propagate, is not a bad idea in a test case,
at all.
Try to avoid using any exception handling construct which can hide the errors
origin.
If you really need to use a ``try`` block, please ensure the original
exception at least logged. When the exception is logged you usually need
to ``raise`` the same or a different exception anyway.
Use of ``self.addCleanup`` is often a good way to avoid having to catch
exceptions and still ensure resources are correctly cleaned up if the
test fails part way through.
Use the ``self.assert*`` methods provided by the unit test framework.
This signals the failures early on.
Avoid using the ``self.fail`` alone, its stack trace will signal
the ``self.fail`` line as the origin of the error.
Avoid constructing complex boolean expressions for assertion.
The ``self.assertTrue`` or ``self.assertFalse`` without a ``msg`` argument,
will just tell you the single boolean value, and you will not know anything
about the values used in the formula, the ``msg`` argument might be good enough
for providing more information.
Most other assert method can include more information by default.
For example ``self.assertIn`` can include the whole set.
It is recommended to use testtools `matcher`_ for the more tricky assertions.
You can implement your own specific `matcher`_ as well.
.. _matcher: http://testtools.readthedocs.org/en/latest/for-test-authors.html#matchers
If the test case fails you can see the related logs and the information
carried by the exception (exception class, backtrack and exception info).
This and the service logs are your only guide to finding the root cause of flaky
issues.
Test cases are independent
--------------------------
Every ``test_method`` must be callable individually and MUST NOT depends on,
any other ``test_method`` or ``test_method`` ordering.
Test cases MAY depend on commonly initialized resources/facilities, like
credentials management, testresources and so on. These facilities, MUST be able
to work even if just one ``test_method`` is selected for execution.
Service Tagging
---------------
Service tagging is used to specify which services are exercised by a particular
test method. You specify the services with the ``tempest.test.services``
decorator. For example:
@services('compute', 'image')
Valid service tag names are the same as the list of directories in tempest.api
that have tests.
For scenario tests having a service tag is required. For the API tests service
tags are only needed if the test method makes an API call (either directly or
indirectly through another service) that differs from the parent directory
name. For example, any test that make an API call to a service other than Nova
in ``tempest.api.compute`` would require a service tag for those services,
however they do not need to be tagged as ``compute``.
Test fixtures and resources
---------------------------
Test level resources should be cleaned-up after the test execution. Clean-up
is best scheduled using `addCleanup` which ensures that the resource cleanup
code is always invoked, and in reverse order with respect to the creation
order.
Test class level resources should be defined in the `resource_setup` method of
the test class, except for any credential obtained from the credentials
provider, which should be set-up in the `setup_credentials` method.
The test base class `BaseTestCase` defines Tempest framework for class level
fixtures. `setUpClass` and `tearDownClass` are defined here and cannot be
overwritten by subclasses (enforced via hacking rule T105).
Set-up is split in a series of steps (setup stages), which can be overwritten
by test classes. Set-up stages are:
- `skip_checks`
- `setup_credentials`
- `setup_clients`
- `resource_setup`
Tear-down is also split in a series of steps (teardown stages), which are
stacked for execution only if the corresponding setup stage had been
reached during the setup phase. Tear-down stages are:
- `clear_credentials` (defined in the base test class)
- `resource_cleanup`
Skipping Tests
--------------
Skipping tests should be based on configuration only. If that is not possible,
it is likely that either a configuration flag is missing, or the test should
fail rather than be skipped.
Using discovery for skipping tests is generally discouraged.
When running a test that requires a certain "feature" in the target
cloud, if that feature is missing we should fail, because either the test
configuration is invalid, or the cloud is broken and the expected "feature" is
not there even if the cloud was configured with it.
Negative Tests
--------------
Error handling is an important aspect of API design and usage. Negative
tests are a way to ensure that an application can gracefully handle
invalid or unexpected input. However, as a black box integration test
suite, Tempest is not suitable for handling all negative test cases, as
the wide variety and complexity of negative tests can lead to long test
runs and knowledge of internal implementation details. The bulk of
negative testing should be handled with project function tests.
All negative tests should be based on `API-WG guideline`_ . Such negative
tests can block any changes from accurate failure code to invalid one.
.. _API-WG guideline: http://specs.openstack.org/openstack/api-wg/guidelines/http.html#failure-code-clarifications
If facing some gray area which is not clarified on the above guideline, propose
a new guideline to the API-WG. With a proposal to the API-WG we will be able to
build a consensus across all OpenStack projects and improve the quality and
consistency of all the APIs.
In addition, we have some guidelines for additional negative tests.
- About BadRequest(HTTP400) case: We can add a single negative tests of
BadRequest for each resource and method(POST, PUT).
Please don't implement more negative tests on the same combination of
resource and method even if API request parameters are different from
the existing test.
- About NotFound(HTTP404) case: We can add a single negative tests of
NotFound for each resource and method(GET, PUT, DELETE, HEAD).
Please don't implement more negative tests on the same combination
of resource and method.
The above guidelines don't cover all cases and we will grow these guidelines
organically over time. Patches outside of the above guidelines are left up to
the reviewers' discretion and if we face some conflicts between reviewers, we
will expand the guideline based on our discussion and experience.
Test skips because of Known Bugs
--------------------------------
If a test is broken because of a bug it is appropriate to skip the test until
bug has been fixed. You should use the ``skip_because`` decorator so that
Tempest's skip tracking tool can watch the bug status.
Example::
@skip_because(bug="980688")
def test_this_and_that(self):
...
Guidelines
----------
- Do not submit changesets with only testcases which are skipped as
they will not be merged.
- Consistently check the status code of responses in testcases. The
earlier a problem is detected the easier it is to debug, especially
where there is complicated setup required.
Parallel Test Execution
-----------------------
Tempest by default runs its tests in parallel this creates the possibility for
interesting interactions between tests which can cause unexpected failures.
Dynamic credentials provides protection from most of the potential race
conditions between tests outside the same class. But there are still a few of
things to watch out for to try to avoid issues when running your tests in
parallel.
- Resources outside of a project scope still have the potential to conflict. This
is a larger concern for the admin tests since most resources and actions that
require admin privileges are outside of projects.
- Races between methods in the same class are not a problem because
parallelization in Tempest is at the test class level, but if there is a json
and xml version of the same test class there could still be a race between
methods.
- The rand_name() function from tempest.lib.common.utils.data_utils should be
used anywhere a resource is created with a name. Static naming should be
avoided to prevent resource conflicts.
- If the execution of a set of tests is required to be serialized then locking
can be used to perform this. See usage of ``LockFixture`` for examples of
using locking.
Sample Configuration File
-------------------------
The sample config file is autogenerated using a script. If any changes are made
to the config variables in tempest/config.py then the sample config file must be
regenerated. This can be done running::
tox -e genconfig
Unit Tests
----------
Unit tests are a separate class of tests in Tempest. They verify Tempest
itself, and thus have a different set of guidelines around them:
1. They can not require anything running externally. All you should need to
run the unit tests is the git tree, python and the dependencies installed.
This includes running services, a config file, etc.
2. The unit tests cannot use setUpClass, instead fixtures and testresources
should be used for shared state between tests.
.. _TestDocumentation:
Test Documentation
------------------
For tests being added we need to require inline documentation in the form of
docstrings to explain what is being tested. In API tests for a new API a class
level docstring should be added to an API reference doc. If one doesn't exist
a TODO comment should be put indicating that the reference needs to be added.
For individual API test cases a method level docstring should be used to
explain the functionality being tested if the test name isn't descriptive
enough. For example::
def test_get_role_by_id(self):
"""Get a role by its id."""
the docstring there is superfluous and shouldn't be added. but for a method
like::
def test_volume_backup_create_get_detailed_list_restore_delete(self):
pass
a docstring would be useful because while the test title is fairly descriptive
the operations being performed are complex enough that a bit more explanation
will help people figure out the intent of the test.
For scenario tests a class level docstring describing the steps in the scenario
is required. If there is more than one test case in the class individual
docstrings for the workflow in each test methods can be used instead. A good
example of this would be::
class TestVolumeBootPattern(manager.ScenarioTest):
"""
This test case attempts to reproduce the following steps:
* Create in Cinder some bootable volume importing a Glance image
* Boot an instance from the bootable volume
* Write content to the volume
* Delete an instance and Boot a new instance from the volume
* Check written content in the instance
* Create a volume snapshot while the instance is running
* Boot an additional instance from the new snapshot based volume
* Check written content in the instance booted from snapshot
"""
Test Identification with Idempotent ID
--------------------------------------
Every function that provides a test must have an ``idempotent_id`` decorator
that is a unique ``uuid-4`` instance. This ID is used to complement the fully
qualified test name and track test functionality through refactoring. The
format of the metadata looks like::
@decorators.idempotent_id('585e934c-448e-43c4-acbf-d06a9b899997')
def test_list_servers_with_detail(self):
# The created server should be in the detailed list of all servers
...
Tempest.lib includes a ``check-uuid`` tool that will test for the existence
and uniqueness of idempotent_id metadata for every test. If you have
Tempest installed you run the tool against Tempest by calling from the
Tempest repo::
check-uuid
It can be invoked against any test suite by passing a package name::
check-uuid --package <package_name>
Tests without an ``idempotent_id`` can be automatically fixed by running
the command with the ``--fix`` flag, which will modify the source package
by inserting randomly generated uuids for every test that does not have
one::
check-uuid --fix
The ``check-uuid`` tool is used as part of the Tempest gate job
to ensure that all tests have an ``idempotent_id`` decorator.
Branchless Tempest Considerations
---------------------------------
Starting with the OpenStack Icehouse release Tempest no longer has any stable
branches. This is to better ensure API consistency between releases because
the API behavior should not change between releases. This means that the stable
branches are also gated by the Tempest master branch, which also means that
proposed commits to Tempest must work against both the master and all the
currently supported stable branches of the projects. As such there are a few
special considerations that have to be accounted for when pushing new changes
to Tempest.
1. New Tests for new features
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When adding tests for new features that were not in previous releases of the
projects the new test has to be properly skipped with a feature flag. Whether
this is just as simple as using the @test.requires_ext() decorator to check
if the required extension (or discoverable optional API) is enabled or adding
a new config option to the appropriate section. If there isn't a method of
selecting the new **feature** from the config file then there won't be a
mechanism to disable the test with older stable releases and the new test won't
be able to merge.
2. Bug fix on core project needing Tempest changes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When trying to land a bug fix which changes a tested API you'll have to use the
following procedure::
1. Propose change to the project, get a +2 on the change even with failing
2. Propose skip on Tempest which will only be approved after the
corresponding change in the project has a +2 on change
3. Land project change in master and all open stable branches (if required)
4. Land changed test in Tempest
Otherwise the bug fix won't be able to land in the project.
Handily, `Zuul’s cross-repository dependencies
<https://docs.openstack.org/infra/zuul/gating.html#cross-repository-dependencies>`_.
can be leveraged to do without step 2 and to have steps 3 and 4 happen
"atomically". To do that, make the patch written in step 1 to depend (refer to
Zuul's documentation above) on the patch written in step 4. The commit message
for the Tempest change should have a link to the Gerrit review that justifies
that change.
3. New Tests for existing features
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If a test is being added for a feature that exists in all the current releases
of the projects then the only concern is that the API behavior is the same
across all the versions of the project being tested. If the behavior is not
consistent the test will not be able to merge.
API Stability
-------------
For new tests being added to Tempest the assumption is that the API being
tested is considered stable and adheres to the OpenStack API stability
guidelines. If an API is still considered experimental or in development then
it should not be tested by Tempest until it is considered stable.

176
LICENSE
View File

@ -1,176 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.

14
README Normal file
View File

@ -0,0 +1,14 @@
This project is no longer maintained.
The contents of this repository are still available in the Git
source code management system. To see the contents of this
repository before it reached its end of life, please check out the
previous commit with "git checkout HEAD^1".
For ongoing work on maintaining OpenStack packages in the Debian
distribution, please see the Debian OpenStack packaging team at
https://wiki.debian.org/OpenStack/.
For any further questions, please email
openstack-dev@lists.openstack.org or join #openstack-dev on
Freenode.

268
README.rst
View File

@ -1,268 +0,0 @@
========================
Team and repository tags
========================
.. image:: http://governance.openstack.org/badges/tempest.svg
:target: http://governance.openstack.org/reference/tags/index.html
.. Change things from this point on
Tempest - The OpenStack Integration Test Suite
==============================================
The documentation for Tempest is officially hosted at:
https://docs.openstack.org/tempest/latest/
This is a set of integration tests to be run against a live OpenStack
cluster. Tempest has batteries of tests for OpenStack API validation,
Scenarios, and other specific tests useful in validating an OpenStack
deployment.
Design Principles
-----------------
Tempest Design Principles that we strive to live by.
- Tempest should be able to run against any OpenStack cloud, be it a
one node DevStack install, a 20 node LXC cloud, or a 1000 node KVM
cloud.
- Tempest should be explicit in testing features. It is easy to auto
discover features of a cloud incorrectly, and give people an
incorrect assessment of their cloud. Explicit is always better.
- Tempest uses OpenStack public interfaces. Tests in Tempest should
only touch public OpenStack APIs.
- Tempest should not touch private or implementation specific
interfaces. This means not directly going to the database, not
directly hitting the hypervisors, not testing extensions not
included in the OpenStack base. If there are some features of
OpenStack that are not verifiable through standard interfaces, this
should be considered a possible enhancement.
- Tempest strives for complete coverage of the OpenStack API and
common scenarios that demonstrate a working cloud.
- Tempest drives load in an OpenStack cloud. By including a broad
array of API and scenario tests Tempest can be reused in whole or in
parts as load generation for an OpenStack cloud.
- Tempest should attempt to clean up after itself, whenever possible
we should tear down resources when done.
- Tempest should be self-testing.
Quickstart
----------
To run Tempest, you first need to create a configuration file that will tell
Tempest where to find the various OpenStack services and other testing behavior
switches. Where the configuration file lives and how you interact with it
depends on how you'll be running Tempest. There are 2 methods of using Tempest.
The first, which is a newer and recommended workflow treats Tempest as a system
installed program. The second older method is to run Tempest assuming your
working dir is the actually Tempest source repo, and there are a number of
assumptions related to that. For this section we'll only cover the newer method
as it is simpler, and quicker to work with.
#. You first need to install Tempest. This is done with pip after you check out
the Tempest repo::
$ git clone http://git.openstack.org/openstack/tempest
$ pip install tempest/
This can be done within a venv, but the assumption for this guide is that
the Tempest CLI entry point will be in your shell's PATH.
#. Installing Tempest may create a ``/etc/tempest dir``, however if one isn't
created you can create one or use ``~/.tempest/etc`` or ``~/.config/tempest`` in
place of ``/etc/tempest``. If none of these dirs are created Tempest will create
``~/.tempest/etc`` when it's needed. The contents of this dir will always
automatically be copied to all ``etc/`` dirs in local workspaces as an initial
setup step. So if there is any common configuration you'd like to be shared
between local Tempest workspaces it's recommended that you pre-populate it
before running ``tempest init``.
#. Setup a local Tempest workspace. This is done by using the tempest init
command::
$ tempest init cloud-01
which also works the same as::
$ mkdir cloud-01 && cd cloud-01 && tempest init
This will create a new directory for running a single Tempest configuration.
If you'd like to run Tempest against multiple OpenStack deployments the idea
is that you'll create a new working directory for each to maintain separate
configuration files and local artifact storage for each.
#. Then ``cd`` into the newly created working dir and also modify the local
config files located in the ``etc/`` subdir created by the ``tempest init``
command. Tempest is expecting a ``tempest.conf`` file in etc/ so if only a
sample exists you must rename or copy it to tempest.conf before making
any changes to it otherwise Tempest will not know how to load it. For
details on configuring Tempest refer to the :ref:`tempest-configuration`.
#. Once the configuration is done you're now ready to run Tempest. This can
be done using the :ref:`tempest_run` command. This can be done by either
running::
$ tempest run
from the Tempest workspace directory. Or you can use the ``--workspace``
argument to run in the workspace you created regardless of your current
working directory. For example::
$ tempest run --workspace cloud-01
There is also the option to use testr directly, or any `testr`_ based test
runner, like `ostestr`_. For example, from the workspace dir run::
$ ostestr --regex '(?!.*\[.*\bslow\b.*\])(^tempest\.(api|scenario))'
will run the same set of tests as the default gate jobs.
.. _testr: https://testrepository.readthedocs.org/en/latest/MANUAL.html
.. _ostestr: https://docs.openstack.org/os-testr/latest/
Library
-------
Tempest exposes a library interface. This interface is a stable interface and
should be backwards compatible (including backwards compatibility with the
old tempest-lib package, with the exception of the import). If you plan to
directly consume Tempest in your project you should only import code from the
Tempest library interface, other pieces of Tempest do not have the same
stable interface and there are no guarantees on the Python API unless otherwise
stated.
For more details refer to the library documentation here: :ref:`library`
Release Versioning
------------------
`Tempest Release Notes <http://docs.openstack.org/releasenotes/tempest>`_
shows what changes have been released on each version.
Tempest's released versions are broken into 2 sets of information. Depending on
how you intend to consume Tempest you might need
The version is a set of 3 numbers:
X.Y.Z
While this is almost `semver`_ like, the way versioning is handled is slightly
different:
X is used to represent the supported OpenStack releases for Tempest tests
in-tree, and to signify major feature changes to Tempest. It's a monotonically
increasing integer where each version either indicates a new supported OpenStack
release, the drop of support for an OpenStack release (which will coincide with
the upstream stable branch going EOL), or a major feature lands (or is removed)
from Tempest.
Y.Z is used to represent library interface changes. This is treated the same
way as minor and patch versions from `semver`_ but only for the library
interface. When Y is incremented we've added functionality to the library
interface and when Z is incremented it's a bug fix release for the library.
Also note that both Y and Z are reset to 0 at each increment of X.
.. _semver: http://semver.org/
Configuration
-------------
Detailed configuration of Tempest is beyond the scope of this
document see :ref:`tempest-configuration` for more details on configuring
Tempest. The ``etc/tempest.conf.sample`` attempts to be a self-documenting
version of the configuration.
You can generate a new sample tempest.conf file, run the following
command from the top level of the Tempest directory::
$ tox -e genconfig
The most important pieces that are needed are the user ids, OpenStack
endpoints, and basic flavors and images needed to run tests.
Unit Tests
----------
Tempest also has a set of unit tests which test the Tempest code itself. These
tests can be run by specifying the test discovery path::
$ OS_TEST_PATH=./tempest/tests testr run --parallel
By setting OS_TEST_PATH to ./tempest/tests it specifies that test discover
should only be run on the unit test directory. The default value of OS_TEST_PATH
is OS_TEST_PATH=./tempest/test_discover which will only run test discover on the
Tempest suite.
Alternatively, there are the py27 and py35 tox jobs which will run the unit
tests with the corresponding version of python.
Python 2.6
----------
Starting in the Kilo release the OpenStack services dropped all support for
python 2.6. This change has been mirrored in Tempest, starting after the
tempest-2 tag. This means that proposed changes to Tempest which only fix
python 2.6 compatibility will be rejected, and moving forward more features not
present in python 2.6 will be used. If you're running your OpenStack services
on an earlier release with python 2.6 you can easily run Tempest against it
from a remote system running python 2.7. (or deploy a cloud guest in your cloud
that has python 2.7)
Python 3.x
----------
Starting during the Pike cycle Tempest has a gating CI job that runs Tempest
with Python 3. Any Tempest release after 15.0.0 should fully support running
under Python 3 as well as Python 2.7.
Legacy run method
-----------------
The legacy method of running Tempest is to just treat the Tempest source code
as a python unittest repository and run directly from the source repo. When
running in this way you still start with a Tempest config file and the steps
are basically the same except that it expects you know where the Tempest code
lives on your system and requires a bit more manual interaction to get Tempest
running. For example, when running Tempest this way things like a lock file
directory do not get generated automatically and the burden is on the user to
create and configure that.
To start you need to create a configuration file. The easiest way to create a
configuration file is to generate a sample in the ``etc/`` directory ::
$ cd $TEMPEST_ROOT_DIR
$ oslo-config-generator --config-file \
tempest/cmd/config-generator.tempest.conf \
--output-file etc/tempest.conf
After that, open up the ``etc/tempest.conf`` file and edit the
configuration variables to match valid data in your environment.
This includes your Keystone endpoint, a valid user and credentials,
and reference data to be used in testing.
.. note::
If you have a running DevStack environment, Tempest will be
automatically configured and placed in ``/opt/stack/tempest``. It
will have a configuration file already set up to work with your
DevStack installation.
Tempest is not tied to any single test runner, but `testr`_ is the most commonly
used tool. Also, the nosetests test runner is **not** recommended to run Tempest.
After setting up your configuration file, you can execute the set of Tempest
tests by using ``testr`` ::
$ testr run --parallel
To run one single test serially ::
$ testr run tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server
Tox also contains several existing job configurations. For example::
$ tox -e full
which will run the same set of tests as the OpenStack gate. (it's exactly how
the gate invokes Tempest) Or::
$ tox -e smoke
to run the tests tagged as smoke.

128
REVIEWING.rst
View File

@ -1,128 +0,0 @@
Reviewing Tempest Code
======================
To start read the `OpenStack Common Review Checklist
<http://docs.openstack.org/infra/manual/developers.html#peer-review>`_
Ensuring code is executed
-------------------------
For any new or change to a test it has to be verified in the gate. This means
that the first thing to check with any change is that a gate job actually runs
it. Tests which aren't executed either because of configuration or skips should
not be accepted.
If a new test is added that depends on a new config option (like a feature
flag), the commit message must reference a change in DevStack or DevStack-Gate
that enables the execution of this newly introduced test. This reference could
either be a `Cross-Repository Dependency <http://docs.openstack.org/infra/
manual/developers.html#cross-repository-dependencies>`_ or a simple link
to a Gerrit review.
Execution time
--------------
While checking in the job logs that a new test is actually executed, also
pay attention to the execution time of that test. Keep in mind that each test
is going to be executed hundreds of time each day, because Tempest tests
run in many OpenStack projects. It's worth considering how important/critical
the feature under test is with how costly the new test is.
Unit Tests
----------
For any change that adds new functionality to either common functionality or an
out-of-band tool unit tests are required. This is to ensure we don't introduce
future regressions and to test conditions which we may not hit in the gate runs.
Tests, and service clients aren't required to have unit tests since they should
be self verifying by running them in the gate.
API Stability
-------------
Tests should only be added for a published stable APIs. If a patch contains
tests for an API which hasn't been marked as stable or for an API that which
doesn't conform to the `API stability guidelines
<https://wiki.openstack.org/wiki/Governance/Approved/APIStability>`_ then it
should not be approved.
Reject Copy and Paste Test Code
-------------------------------
When creating new tests that are similar to existing tests it is tempting to
simply copy the code and make a few modifications. This increases code size and
the maintenance burden. Such changes should not be approved if it is easy to
abstract the duplicated code into a function or method.
Tests overlap
-------------
When a new test is being proposed, question whether this feature is not already
tested with Tempest. Tempest has more than 1200 tests, spread amongst many
directories, so it's easy to introduce test duplication. For example, testing
volume attachment to a server could be a compute test or a volume test, depending
on how you see it. So one must look carefully in the entire code base for possible
overlap. As a rule of thumb, the older a feature is, the more likely it's
already tested.
Being explicit
--------------
When tests are being added that depend on a configurable feature or extension,
polling the API to discover that it is enabled should not be done. This will
just result in bugs being masked because the test can be skipped automatically.
Instead the config file should be used to determine whether a test should be
skipped or not. Do not approve changes that depend on an API call to determine
whether to skip or not.
Configuration Options
---------------------
With the introduction of the Tempest external test plugin interface we needed
to provide a stable contract for Tempest's configuration options. This means
we can no longer simply remove a configuration option when it's no longer used.
Patches proposed that remove options without a deprecation cycle should not
be approved. Similarly when changing default values with configuration we need
to similarly be careful that we don't break existing functionality. Also, when
adding options, just as before, we need to weigh the benefit of adding an
additional option against the complexity and maintenance overhead having it
costs.
Test Documentation
------------------
When a new test is being added refer to the :ref:`TestDocumentation` section in
hacking to see if the requirements are being met. With the exception of a class
level docstring linking to the API ref doc in the API tests and a docstring for
scenario tests this is up to the reviewers discretion whether a docstring is
required or not.
Release Notes
-------------
Release notes are how we indicate to users and other consumers of Tempest what
has changed in a given release. Since Tempest 10.0.0 we've been using `reno`_
to manage and build the release notes. There are certain types of changes that
require release notes and we should not approve them without including a release
note. These include but aren't limited to, any addition, deprecation or removal
from the lib interface, any change to configuration options (including
deprecation), CLI additions or deprecations, major feature additions, and
anything backwards incompatible or would require a user to take note or do
something extra.
.. _reno: https://docs.openstack.org/reno/latest/
Deprecated Code
---------------
Sometimes we have some bugs in deprecated code. Basically, we leave it. Because
we don't need to maintain it. However, if the bug is critical, we might need to
fix it. When it will happen, we will deal with it on a case-by-case basis.
When to approve
---------------
* Every patch needs two +2s before being approved.
* Its ok to hold off on an approval until a subject matter expert reviews it
* If a patch has already been approved but requires a trivial rebase to merge,
you do not have to wait for a second +2, since the patch has already had
two +2s.

13
bindep.txt
View File

@ -1,13 +0,0 @@
# This file contains runtime (non-python) dependencies
# More info at: http://docs.openstack.org/infra/bindep/readme.html
libffi-dev [platform:dpkg]
libffi-devel [platform:rpm]
gcc [platform:rpm]
gcc [platform:dpkg]
python-dev [platform:dpkg]
python-devel [platform:rpm]
python3-dev [platform:dpkg]
python3-devel [platform:rpm]
openssl-devel [platform:rpm]
libssl-dev [platform:dpkg]

23
data/tempest-plugins-registry.header
View File

@ -1,23 +0,0 @@
..
Note to patch submitters: this file is covered by a periodic proposal
job. You should edit the files data/tempest-plugins-registry.footer
and data/tempest-plugins-registry.header instead of this one.
==========================
Tempest Plugin Registry
==========================
Since we've created the external plugin mechanism, it's gotten used by
a lot of projects. The following is a list of plugins that currently
exist.
Detected Plugins
================
The following are plugins that a script has found in the openstack/
namespace, which includes but is not limited to official OpenStack
projects.
+----------------------------+-------------------------------------------------------------------------+
|Plugin Name |URL |
+----------------------------+-------------------------------------------------------------------------+

1
doc/source/HACKING.rst
View File

@ -1 +0,0 @@
../../HACKING.rst

1
doc/source/REVIEWING.rst
View File

@ -1 +0,0 @@
../../REVIEWING.rst

0
doc/source/_static/.keep
View File

5
doc/source/account_generator.rst
View File

@ -1,5 +0,0 @@
--------------------------------------
Tempest Test-Account Generator Utility
--------------------------------------
.. automodule:: tempest.cmd.account_generator

5
doc/source/cleanup.rst
View File

@ -1,5 +0,0 @@
--------------------------------
Post Tempest Run Cleanup Utility
--------------------------------
.. automodule:: tempest.cmd.cleanup

203
doc/source/conf.py
View File

@ -1,203 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Tempest documentation build configuration file, created by
# sphinx-quickstart on Tue May 21 17:43:32 2013.
#
# This file is execfile()d with the current directory set to its containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import sys
import os
import subprocess
import warnings
import openstackdocstheme
# Build the plugin registry
def build_plugin_registry(app):
root_dir = os.path.dirname(
os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
subprocess.call(['tools/generate-tempest-plugins-list.sh'], cwd=root_dir)
def setup(app):
if os.getenv('GENERATE_TEMPEST_PLUGIN_LIST', 'true').lower() == 'true':
app.connect('builder-inited', build_plugin_registry)
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#sys.path.insert(0, os.path.abspath('.'))
# -- General configuration -----------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['sphinx.ext.autodoc',
'sphinx.ext.todo',
'sphinx.ext.viewcode',
'openstackdocstheme',
'oslo_config.sphinxconfiggen',
]
config_generator_config_file = '../../tempest/cmd/config-generator.tempest.conf'
sample_config_basename = '_static/tempest'
todo_include_todos = True
# openstackdocstheme options
repository_name = 'openstack/tempest'
bug_project = 'tempest'
bug_tag = ''
# Must set this variable to include year, month, day, hours, and minutes.
html_last_updated_fmt = '%Y-%m-%d %H:%M'
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'Tempest'
copyright = u'2013, OpenStack QA Team'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['_build']
# The reST default role (used for this markup: `text`) to use for all documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
add_module_names = False
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
modindex_common_prefix = ['tempest.']
# -- Options for HTML output ---------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'openstackdocs'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
html_use_smartypants = False
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
html_domain_indices = False
# If false, no index is generated.
html_use_index = False
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# A list of warning types to suppress arbitrary warning messages.
suppress_warnings = ['image.nonlocal_uri']

439
doc/source/configuration.rst
View File

@ -1,439 +0,0 @@
.. _tempest-configuration:
Tempest Configuration Guide
===========================
This guide is a starting point for configuring Tempest. It aims to elaborate
on and explain some of the mandatory and common configuration settings and how
they are used in conjunction. The source of truth on each option is the sample
config file which explains the purpose of each individual option. You can see
the sample config file here: :ref:`tempest-sampleconf`
.. _tempest_cred_provider_conf:
Test Credentials
----------------
Tempest allows for configuring a set of admin credentials in the ``auth``
section, via the following parameters:
#. ``admin_username``
#. ``admin_password``
#. ``admin_project_name``
#. ``admin_domain_name``
Admin credentials are not mandatory to run Tempest, but when provided they
can be used to:
- Run tests for admin APIs
- Generate test credentials on the fly (see `Dynamic Credentials`_)
When Keystone uses a policy that requires domain scoped tokens for admin
actions, the flag ``admin_domain_scope`` must be set to ``True``.
The admin user configured, if any, must have a role assigned to the domain to
be usable.
Tempest allows for configuring pre-provisioned test credentials as well.
This can be done using the accounts.yaml file (see
`Pre-Provisioned Credentials`_). This file is used to specify an arbitrary
number of users available to run tests with.
You can specify the location of the file in the ``auth`` section in the
tempest.conf file. To see the specific format used in the file please refer to
the ``accounts.yaml.sample`` file included in Tempest.
Keystone Connection Info
^^^^^^^^^^^^^^^^^^^^^^^^
In order for Tempest to be able to talk to your OpenStack deployment you need
to provide it with information about how it communicates with keystone.
This involves configuring the following options in the ``identity`` section:
- ``auth_version``
- ``uri``
- ``uri_v3``
The ``auth_version`` option is used to tell Tempest whether it should be using
Keystone's v2 or v3 api for communicating with Keystone. The two uri options are
used to tell Tempest the url of the keystone endpoint. The ``uri`` option is
used for Keystone v2 request and ``uri_v3`` is used for Keystone v3. You want to
ensure that which ever version you set for ``auth_version`` has its uri option
defined.
Credential Provider Mechanisms
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Tempest currently has two different internal methods for providing authentication
to tests: dynamic credentials and pre-provisioned credentials.
Depending on which one is in use the configuration of Tempest is slightly different.
Dynamic Credentials
"""""""""""""""""""
Dynamic Credentials (formerly known as Tenant isolation) was originally created
to enable running Tempest in parallel. For each test class it creates a unique
set of user credentials to use for the tests in the class. It can create up to
three sets of username, password, and project names for a primary user,
an admin user, and an alternate user. To enable and use dynamic credentials you
only need to configure two things:
#. A set of admin credentials with permissions to create users and
projects. This is specified in the ``auth`` section with the
``admin_username``, ``admin_project_name``, ``admin_domain_name`` and
``admin_password`` options
#. To enable dynamic credentials in the ``auth`` section with the
``use_dynamic_credentials`` option.
This is also currently the default credential provider enabled by Tempest, due
to its common use and ease of configuration.
It is worth pointing out that depending on your cloud configuration you might
need to assign a role to each of the users created by Tempest's dynamic
credentials. This can be set using the ``tempest_roles`` option. It takes in a
list of role names each of which will be assigned to each of the users created
by dynamic credentials. This option will not have any effect when Tempest is not
configured to use dynamic credentials.
When the ``admin_domain_scope`` option is set to ``True``, provisioned admin
accounts will be assigned a role on domain configured in
``default_credentials_domain_name``. This will make the accounts provisioned
usable in a cloud where domain scoped tokens are required by Keystone for
admin operations. Note that the initial pre-provision admin accounts,
configured in tempest.conf, must have a role on the same domain as well, for
Dynamic Credentials to work.
Pre-Provisioned Credentials
"""""""""""""""""""""""""""
For a long time using dynamic credentials was the only method available if you
wanted to enable parallel execution of Tempest tests. However, this was
insufficient for certain use cases because of the admin credentials requirement
to create the credential sets on demand. To get around that the accounts.yaml
file was introduced and with that a new internal credential provider to enable
using the list of credentials instead of creating them on demand. With locking
test accounts each test class will reserve a set of credentials from the
accounts.yaml before executing any of its tests so that each class is isolated
like with dynamic credentials.
To enable and use locking test accounts you need do a few things:
#. Create an accounts.yaml file which contains the set of pre-existing
credentials to use for testing. To make sure you don't have a credentials
starvation issue when running in parallel make sure you have at least two
times the number of worker processes you are using to execute Tempest
available in the file. (If running serially the worker count is 1.)
You can check the accounts.yaml.sample file packaged in Tempest for the yaml
format.
#. Provide Tempest with the location of your accounts.yaml file with the
``test_accounts_file`` option in the ``auth`` section
*NOTE: Be sure to use a full path for the file; otherwise Tempest will
likely not find it.*
#. Set ``use_dynamic_credentials = False`` in the ``auth`` group
It is worth pointing out that each set of credentials in the accounts.yaml
should have a unique project. This is required to provide proper isolation
to the tests using the credentials, and failure to do this will likely cause
unexpected failures in some tests. Also, ensure that these projects and users
used do not have any pre-existing resources created. Tempest assumes all
tenants it's using are empty and may sporadically fail if there are unexpected
resources present.
When the Keystone in the target cloud requires domain scoped tokens to
perform admin actions, all pre-provisioned admin users must have a role
assigned on the domain where test accounts a provisioned.
The option ``admin_domain_scope`` is used to tell Tempest that domain scoped
tokens shall be used. ``default_credentials_domain_name`` is the domain where
test accounts are expected to be provisioned if no domain is specified.
Note that if credentials are pre-provisioned via ``tempest account-generator``
the role on the domain will be assigned automatically for you, as long as
``admin_domain_scope`` as ``default_credentials_domain_name`` are configured
properly in tempest.conf.
Pre-Provisioned Credentials are also known as accounts.yaml or accounts file.
Compute
-------
Flavors
^^^^^^^
For Tempest to be able to create servers you need to specify flavors that it
can use to boot the servers with. There are two options in the Tempest config
for doing this:
#. ``flavor_ref``
#. ``flavor_ref_alt``
Both of these options are in the ``compute`` section of the config file and take
in the flavor id (not the name) from Nova. The ``flavor_ref`` option is what
will be used for booting almost all of the guests; ``flavor_ref_alt`` is only
used in tests where two different-sized servers are required (for example, a
resize test).
Using a smaller flavor is generally recommended. When larger flavors are used,
the extra time required to bring up servers will likely affect total run time
and probably require tweaking timeout values to ensure tests have ample time to
finish.
Images
^^^^^^
Just like with flavors, Tempest needs to know which images to use for booting
servers. There are two options in the compute section just like with flavors:
#. ``image_ref``
#. ``image_ref_alt``
Both options are expecting an image id (not name) from Nova. The ``image_ref``
option is what will be used for booting the majority of servers in Tempest.
``image_ref_alt`` is used for tests that require two images such as rebuild. If
two images are not available you can set both options to the same image id and
those tests will be skipped.
There are also options in the ``scenario`` section for images:
#. ``img_file``
#. ``img_dir``
#. ``aki_img_file``
#. ``ari_img_file``
#. ``ami_img_file``
#. ``img_container_format``
#. ``img_disk_format``
However, unlike the other image options, these are used for a very small subset
of scenario tests which are uploading an image. These options are used to tell
Tempest where an image file is located and describe its metadata for when it is
uploaded.
The behavior of these options is a bit convoluted (which will likely be fixed in
future versions). You first need to specify ``img_dir``, which is the directory
in which Tempest will look for the image files. First it will check if the
filename set for ``img_file`` could be found in ``img_dir``. If it is found then
the ``img_container_format`` and ``img_disk_format`` options are used to upload
that image to glance. However, if it is not found, Tempest will look for the
three uec image file name options as a fallback. If neither is found, the tests
requiring an image to upload will fail.
It is worth pointing out that using `cirros`_ is a very good choice for running
Tempest. It's what is used for upstream testing, they boot quickly and have a
small footprint.
.. _cirros: https://launchpad.net/cirros
Networking
----------
OpenStack has a myriad of different networking configurations possible and
depending on which of the two network backends, nova-network or Neutron, you are
using things can vary drastically. Due to this complexity Tempest has to provide
a certain level of flexibility in its configuration to ensure it will work
against any cloud. This ends up causing a large number of permutations in
Tempest's config around network configuration.
Enabling Remote Access to Created Servers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. _tempest_conf_network_allocation:
Network Creation/Usage for Servers
""""""""""""""""""""""""""""""""""
When Tempest creates servers for testing, some tests require being able to
connect those servers. Depending on the configuration of the cloud, the methods
for doing this can be different. In certain configurations it is required to
specify a single network with server create calls. Accordingly, Tempest provides
a few different methods for providing this information in configuration to try
and ensure that regardless of the cloud's configuration it'll still be able to
run. This section covers the different methods of configuring Tempest to provide
a network when creating servers.
Fixed Network Name
''''''''''''''''''
This is the simplest method of specifying how networks should be used. You can
just specify a single network name/label to use for all server creations. The
limitation with this is that all projects and users must be able to see
that network name/label if they are to perform a network list and be able to use
it.
If no network name is assigned in the config file and none of the below
alternatives are used, then Tempest will not specify a network on server
creations, which depending on the cloud configuration might prevent them from
booting.
To set a fixed network name simply:
#. Set the ``fixed_network_name`` option in the ``compute`` group
In the case that the configured fixed network name can not be found by a user
network list call, it will be treated like one was not provided except that a
warning will be logged stating that it couldn't be found.
Accounts File
'''''''''''''
If you are using an accounts file to provide credentials for running Tempest
then you can leverage it to also specify which network should be used with
server creations on a per project and user pair basis. This provides
the necessary flexibility to work with more intricate networking configurations
by enabling the user to specify exactly which network to use for which
projects. You can refer to the accounts.yaml.sample file included in
the Tempest repo for the syntax around specifying networks in the file.
However, specifying a network is not required when using an accounts file. If
one is not specified you can use a fixed network name to specify the network to
use when creating servers just as without an accounts file. However, any network
specified in the accounts file will take precedence over the fixed network name
provided. If no network is provided in the accounts file and a fixed network
name is not set then no network will be included in create server requests.
If a fixed network is provided and the accounts.yaml file also contains networks
this has the benefit of enabling a couple more tests which require a static
network to perform operations like server lists with a network filter. If a
fixed network name is not provided these tests are skipped. Additionally, if a
fixed network name is provided it will serve as a fallback in case of a
misconfiguration or a missing network in the accounts file.
With Dynamic Credentials
''''''''''''''''''''''''
With dynamic credentials enabled and using nova-network, your only option for
configuration is to either set a fixed network name or not. However, in most
cases it shouldn't matter because nova-network should have no problem booting a
server with multiple networks. If this is not the case for your cloud then using
an accounts file is recommended because it provides the necessary flexibility to
describe your configuration. Dynamic credentials is not able to dynamically
allocate things as necessary if Neutron is not enabled.
With Neutron and dynamic credentials enabled there should not be any additional
configuration necessary to enable Tempest to create servers with working
networking, assuming you have properly configured the ``network`` section to
work for your cloud. Tempest will dynamically create the Neutron resources
necessary to enable using servers with that network. Also, just as with the
accounts file, if you specify a fixed network name while using Neutron and
dynamic credentials it will enable running tests which require a static network
and it will additionally be used as a fallback for server creation. However,
unlike accounts.yaml this should never be triggered.
However, there is an option ``create_isolated_networks`` to disable dynamic
credentials's automatic provisioning of network resources. If this option is set
to ``False`` you will have to either rely on there only being a single/default
network available for the server creation, or use ``fixed_network_name`` to
inform Tempest which network to use.
SSH Connection Configuration
""""""""""""""""""""""""""""
There are also several different ways to actually establish a connection and
authenticate/login on the server. After a server is booted with a provided
network there are still details needed to know how to actually connect to
the server. The ``validation`` group gathers all the options regarding
connecting to and remotely accessing the created servers.
To enable remote access to servers, there are 3 options at a minimum that are used:
#. ``run_validation``
#. ``connect_method``
#. ``auth_method``
The ``run_validation`` is used to enable or disable ssh connectivity for
all tests (with the exception of scenario tests which do not have a flag for
enabling or disabling ssh) To enable ssh connectivity this needs be set to ``True``.
The ``connect_method`` option is used to tell Tempest what kind of IP to use for
establishing a connection to the server. Two methods are available: ``fixed``
and ``floating``, the later being set by default. If this is set to floating
Tempest will create a floating ip for the server before attempted to connect
to it. The IP for the floating ip is what is used for the connection.
For the ``auth_method`` option there is currently, only one valid option,
``keypair``. With this set to ``keypair`` Tempest will create an ssh keypair
and use that for authenticating against the created server.
Configuring Available Services
------------------------------
OpenStack is really a constellation of several different projects which
are running together to create a cloud. However which projects you're running
is not set in stone, and which services are running is up to the deployer.
Tempest however needs to know which services are available so it can figure
out which tests it is able to run and certain setup steps which differ based
on the available services.
The ``service_available`` section of the config file is used to set which
services are available. It contains a boolean option for each service (except
for Keystone which is a hard requirement) set it to ``True`` if the service is
available or ``False`` if it is not.
Service Catalog
^^^^^^^^^^^^^^^
Each project which has its own REST API contains an entry in the service
catalog. Like most things in OpenStack this is also completely configurable.
However, for Tempest to be able to figure out which endpoints should get REST
API calls for each service, it needs to know how that project is defined in the
service catalog. There are three options for each service section to accomplish
this:
#. ``catalog_type``
#. ``endpoint_type``
#. ``region``
Setting ``catalog_type`` and ``endpoint_type`` should normally give Tempest
enough information to determine which endpoint it should pull from the service
catalog to use for talking to that particular service. However, if your cloud
has multiple regions available and you need to specify a particular one to use a
service you can set the ``region`` option in that service's section.
It should also be noted that the default values for these options are set
to what DevStack uses (which is a de facto standard for service catalog
entries). So often nothing actually needs to be set on these options to enable
communication to a particular service. It is only if you are either not using
the same ``catalog_type`` as DevStack or you want Tempest to talk to a different
endpoint type instead of ``publicURL`` for a service that these need to be
changed.
.. note::
Tempest does not serve all kinds of fancy URLs in the service catalog. The
service catalog should be in a standard format (which is going to be
standardized at the Keystone level).
Tempest expects URLs in the Service catalog in the following format:
* ``http://example.com:1234/<version-info>``
Examples:
* Good - ``http://example.com:1234/v2.0``
* Wouldn’t work - ``http://example.com:1234/xyz/v2.0/``
(adding prefix/suffix around version etc)
Service Feature Configuration
-----------------------------
OpenStack provides its deployers a myriad of different configuration options to
enable anyone deploying it to create a cloud tailor-made for any individual use
case. It provides options for several different backend types, databases,
message queues, etc. However, the downside to this configurability is that
certain operations and features aren't supported depending on the configuration.
These features may or may not be discoverable from the API so the burden is
often on the user to figure out what is supported by the cloud they're talking
to. Besides the obvious interoperability issues with this it also leaves
Tempest in an interesting situation trying to figure out which tests are
expected to work. However, Tempest tests do not rely on dynamic API discovery
for a feature (assuming one exists). Instead Tempest has to be explicitly
configured as to which optional features are enabled. This is in order to
prevent bugs in the discovery mechanisms from masking failures.
The service ``feature-enabled`` config sections are how Tempest addresses the
optional feature question. Each service that has tests for optional features
contains one of these sections. The only options in it are boolean options
with the name of a feature which is used. If it is set to false any test which
depends on that functionality will be skipped. For a complete list of all these
options refer to the sample config file.
API Extensions
^^^^^^^^^^^^^^
The service feature-enabled sections often contain an ``api-extensions`` option
(or in the case of Swift a ``discoverable_apis`` option). This is used to tell
Tempest which api extensions (or configurable middleware) is used in your
deployment. It has two valid config states: either it contains a single value
``all`` (which is the default) which means that every api extension is assumed
to be enabled, or it is set to a list of each individual extension that is
enabled for that service.

1
doc/source/field_guide/api.rst
View File

@ -1 +0,0 @@
../../../tempest/api/README.rst

1
doc/source/field_guide/index.rst
View File

@ -1 +0,0 @@
../../../tempest/README.rst

1
doc/source/field_guide/scenario.rst
View File

@ -1 +0,0 @@
../../../tempest/scenario/README.rst

1
doc/source/field_guide/unit_tests.rst
View File

@ -1 +0,0 @@
../../../tempest/tests/README.rst

86
doc/source/index.rst
View File

@ -1,86 +0,0 @@
=======================
Tempest Testing Project
=======================
Overview
========
.. toctree::
:maxdepth: 2
overview
Field Guides
============
Tempest contains tests of many different types, the field guides
attempt to explain these in a way that makes it easy to understand
where your test contributions should go.
.. toctree::
:maxdepth: 1
field_guide/index
field_guide/api
field_guide/scenario
field_guide/unit_tests
Users Guide
===========
Tempest Configuration Guide
---------------------------
.. toctree::
:maxdepth: 2
configuration
sampleconf
Command Documentation
---------------------
.. toctree::
:maxdepth: 1
account_generator
cleanup
subunit_describe_calls
workspace
run
Developers Guide
================
Development
-----------
.. toctree::
:maxdepth: 2
HACKING
REVIEWING
microversion_testing
test_removal
write_tests
Plugins
-------
.. toctree::
:maxdepth: 2
plugin
plugin-registry
Library
-------
.. toctree::
:maxdepth: 2
library
Indices and tables
==================
* :ref:`search`

71
doc/source/library.rst
View File

@ -1,71 +0,0 @@
.. _library:
Tempest Library Documentation
=============================
Tempest provides a stable library interface that provides external tools or
test suites an interface for reusing pieces of tempest code. Any public
interface that lives in tempest/lib in the tempest repo is treated as a stable
public interface and it should be safe to external consume that. Every effort
goes into maintaining backwards compatibility with any change.
The library is self contained and doesn't have any dependency on
other tempest internals outside of lib (including no usage of tempest
configuration).
Stability
---------
Any code that lives in tempest/lib will be treated as a stable interface.
This means that any public interface under the tempest/lib directory is
expected to be a stable interface suitable for public consumption. However, for
any interfaces outside of tempest/lib in the tempest tree (unless otherwise
noted) or any private interfaces the same stability guarantees don't apply.
Adding Interfaces
'''''''''''''''''
When adding an interface to tempest/lib we have to make sure there are no
dependencies on any pieces of tempest outside of tempest/lib. This means if
for example there is a dependency on the configuration file we need remove that.
The other aspect when adding an interface is to make sure it's really an
interface ready for external consumption and something we want to commit to
supporting.
Making changes
''''''''''''''
When making changes to tempest/lib you have to be conscious of the effect of
any changes on external consumers. If your proposed changeset will change the
default behaviour of any interface, or make something which previously worked
not after your change, then it is not acceptable. Every effort needs to go into
preserving backwards compatibility in changes.
Reviewing
'''''''''
When reviewing a proposed change to tempest/lib code we need to be careful to
ensure that we don't break backwards compatibility. For patches that change
existing interfaces we have to be careful to make sure we don't break any
external consumers. Some common red flags are:
* a change to an existing API requires a change outside the library directory
where the interface is being consumed
* a unit test has to be significantly changed to make the proposed change pass
Testing
'''''''
When adding a new interface to the library we need to at a minimum have unit
test coverage. A proposed change to add an interface to tempest/lib that
doesn't have unit tests shouldn't be accepted. Ideally these unit tests will
provide sufficient coverage to ensure a stable interface moving forward.
Current Library APIs
--------------------
.. toctree::
:maxdepth: 2
library/cli
library/decorators
library/rest_client
library/utils
library/api_microversion_testing
library/auth
library/clients
library/credential_providers

29
doc/source/library/api_microversion_testing.rst
View File

@ -1,29 +0,0 @@
.. _api_microversion_testing:
API Microversion Testing Support in Tempest
===========================================
---------------------------------------------
Framework to support API Microversion testing
---------------------------------------------
Many of the OpenStack components have implemented API microversions.
It is important to test those microversions in Tempest or external plugins.
Tempest now provides stable interfaces to support to test the API microversions.
Based on the microversion range coming from the combination of both configuration
and each test case, APIs request will be made with selected microversion.
This document explains the interfaces needed for microversion testing.
The api_version_request module
""""""""""""""""""""""""""""""
.. automodule:: tempest.lib.common.api_version_request
:members:
The api_version_utils module
""""""""""""""""""""""""""""
.. automodule:: tempest.lib.common.api_version_utils
:members:

11
doc/source/library/auth.rst
View File

@ -1,11 +0,0 @@
.. _auth:
Authentication Framework Usage
==============================
---------------
The auth module
---------------
.. automodule:: tempest.lib.auth
:members:

18
doc/source/library/cli.rst
View File

@ -1,18 +0,0 @@
.. _cli:
CLI Testing Framework Usage
===========================
-------------------
The cli.base module
-------------------
.. automodule:: tempest.lib.cli.base
:members:
----------------------------
The cli.output_parser module
----------------------------
.. automodule:: tempest.lib.cli.output_parser
:members:

33
doc/source/library/clients.rst
View File

@ -1,33 +0,0 @@
.. _clients:
Service Clients Usage
=====================
Tests make requests against APIs using service clients. Service clients are
specializations of the ``RestClient`` class. The service clients that cover the
APIs exposed by a service should be grouped in a service clients module.
A service clients module is python module where all service clients are
defined. If major API versions are available, submodules should be defined,
one for each version.
The ``ClientsFactory`` class helps initializing all clients of a specific
service client module from a set of shared parameters.
The ``ServiceClients`` class provides a convenient way to get access to all
available service clients initialized with a provided set of credentials.
-----------------------------
The clients management module
-----------------------------
.. automodule:: tempest.lib.services.clients
:members:
------------------------------
Compute service client modules
------------------------------
.. toctree::
:maxdepth: 2
service_clients/compute_clients

148
doc/source/library/credential_providers.rst
View File

@ -1,148 +0,0 @@
.. _cred_providers:
Credential Providers
====================
These library interfaces are used to deal with allocating credentials on demand
either dynamically by calling keystone to allocate new credentials, or from
a list of preprovisioned credentials. These 2 modules are implementations of
the same abstract credential providers class and can be used interchangably.
However, each implementation has some additional parameters that are used to
influence the behavior of the modules. The API reference at the bottom of this
doc shows the interface definitions for both modules, however that may be a bit
opaque. You can see some examples of how to leverage this interface below.
Initialization Example
----------------------
This example is from Tempest itself (from tempest/common/credentials_factory.py
just modified slightly) and is how it initializes the credential provider based
on config::
from tempest import config
from tempest.lib.common import dynamic_creds
from tempest.lib.common import preprov_creds
CONF = config.CONF
def get_credentials_provider(name, network_resources=None,
force_tenant_isolation=False,
identity_version=None):
# If a test requires a new account to work, it can have it via forcing
# dynamic credentials. A new account will be produced only for that test.
# In case admin credentials are not available for the account creation,
# the test should be skipped else it would fail.
identity_version = identity_version or CONF.identity.auth_version
if CONF.auth.use_dynamic_credentials or force_tenant_isolation:
admin_creds = get_configured_admin_credentials(
fill_in=True, identity_version=identity_version)
return dynamic_creds.DynamicCredentialProvider(
name=name,
network_resources=network_resources,
identity_version=identity_version,
admin_creds=admin_creds,
identity_admin_domain_scope=CONF.identity.admin_domain_scope,
identity_admin_role=CONF.identity.admin_role,
extra_roles=CONF.auth.tempest_roles,
neutron_available=CONF.service_available.neutron,
project_network_cidr=CONF.network.project_network_cidr,
project_network_mask_bits=CONF.network.project_network_mask_bits,
public_network_id=CONF.network.public_network_id,
create_networks=(CONF.auth.create_isolated_networks and not
CONF.network.shared_physical_network),
resource_prefix=CONF.resources_prefix,
credentials_domain=CONF.auth.default_credentials_domain_name,
admin_role=CONF.identity.admin_role,
identity_uri=CONF.identity.uri_v3,
identity_admin_endpoint_type=CONF.identity.v3_endpoint_type)
else:
if CONF.auth.test_accounts_file:
# Most params are not relevant for pre-created accounts
return preprov_creds.PreProvisionedCredentialProvider(
name=name, identity_version=identity_version,
accounts_lock_dir=lockutils.get_lock_path(CONF),
test_accounts_file=CONF.auth.test_accounts_file,
object_storage_operator_role=CONF.object_storage.operator_role,
object_storage_reseller_admin_role=reseller_admin_role,
credentials_domain=CONF.auth.default_credentials_domain_name,
admin_role=CONF.identity.admin_role,
identity_uri=CONF.identity.uri_v3,
identity_admin_endpoint_type=CONF.identity.v3_endpoint_type)
else:
raise exceptions.InvalidConfiguration(
'A valid credential provider is needed')
This function just returns an initialized credential provider class based on the
config file. The consumer of this function treats the output as the same
regardless of whether it's a dynamic or preprovisioned provider object.
Dealing with Credentials
------------------------
Once you have a credential provider object created the access patterns for
allocating and removing credentials are the same across both the dynamic
and preprovisioned credentials. These are defined in the abstract
CredentialProvider class. At a high level the credentials provider enables
you to get 3 basic types of credentials at once (per object): a primary, alt,
and admin. You're also able to allocate a credential by role. These credentials
are tracked by the provider object and delete must manually be called otherwise
the created resources will not be deleted (or returned to the pool in the case
of preprovisioned creds)
Examples
''''''''
Continuing from the example above, to allocate credentials by the 3 basic types
you can do the following::
provider = get_credentials_provider('my_tests')
primary_creds = provider.get_primary_creds()
alt_creds = provider.get_alt_creds()
admin_creds = provider.get_admin_creds()
# Make sure to delete the credentials when you're finished
provider.clear_creds()
To create and interact with credentials by role you can do the following::
provider = get_credentials_provider('my_tests')
my_role_creds = provider.get_creds_by_role({'roles': ['my_role']})
# provider.clear_creds() will clear all creds including those allocated by
# role
provider.clear_creds()
When multiple roles are specified a set of creds with all the roles assigned
will be allocated::
provider = get_credentials_provider('my_tests')
my_role_creds = provider.get_creds_by_role({'roles': ['my_role',
'my_other_role']})
# provider.clear_creds() will clear all creds including those allocated by
# role
provider.clear_creds()
If you need multiple sets of credentials with the same roles you can also do
this by leveraging the ``force_new`` kwarg::
provider = get_credentials_provider('my_tests')
my_role_creds = provider.get_creds_by_role({'roles': ['my_role']})
my_role_other_creds = provider.get_creds_by_role({'roles': ['my_role']},
force_new=True)
# provider.clear_creds() will clear all creds including those allocated by
# role
provider.clear_creds()
API Reference
=============
------------------------------
The dynamic credentials module
------------------------------
.. automodule:: tempest.lib.common.dynamic_creds
:members:
--------------------------------------
The pre-provisioned credentials module
--------------------------------------
.. automodule:: tempest.lib.common.preprov_creds
:members:

13
doc/source/library/decorators.rst
View File

@ -1,13 +0,0 @@
.. _decorators:
Decorators Usage Guide
======================
---------------------
The decorators module
---------------------
.. automodule:: tempest.lib.decorators
:members:

11
doc/source/library/rest_client.rst
View File

@ -1,11 +0,0 @@
.. _rest_client:
Rest Client Usage
=================
----------------------
The rest_client module
----------------------
.. automodule:: tempest.lib.common.rest_client
:members:

7
doc/source/library/service_clients/compute_clients.rst
View File

@ -1,7 +0,0 @@
.. _servers_client:
Compute Client Usage
====================
.. automodule:: tempest.lib.services.compute.servers_client
:members:

11
doc/source/library/utils.rst
View File

@ -1,11 +0,0 @@
.. _utils:
Utils Usage
===========
---------------
The misc module
---------------
.. automodule:: tempest.lib.common.utils.misc
:members:

341
doc/source/microversion_testing.rst
View File

@ -1,341 +0,0 @@
=================================
Microversion Testing With Tempest
=================================
Many OpenStack Services provide their APIs with `microversion`_
support and want to test them in Tempest.
.. _microversion: http://specs.openstack.org/openstack/api-wg/guidelines/microversion_specification.html
This document covers how to test microversions for each project and
whether tests should live in Tempest or on project side.
Tempest Scope For Microversion Testing
""""""""""""""""""""""""""""""""""""""
APIs microversions for any OpenStack service grow rapidly and
testing each and every microversion in Tempest is not feasible and
efficient way.
Also not every API microversion changes the complete system behavior
and many of them only change the API or DB layer to accept and return more
data on API.
Tempest is an integration test suite, but not all API microversion testing fall under this category.
As a result, Tempest mainly covers integration test cases for microversions, Other testing coverage
for microversion should be hosted on project side as functional tests or via Tempest plugin as per
project guidelines.
.. note:: Integration tests are those tests which involve more than one service to
verify the expected behavior by single or combination of API requests.
If a test is just to verify the API behavior as success and failure cases
or verify its expected response object, then it does not fall under integration
tests.
Tempest will cover only integration testing of applicable microversions with
below exceptions:
#. Test covers a feature which is important for interoperability. This covers tests requirement
from Defcore.
#. Test needed to fill Schema gaps.
Tempest validates API responses with defined JSON schema. API responses can be different on
each microversion and the JSON schemas need to be defined separately for the microversion.
While implementing new integration tests for a specific microversion, there
may be a gap in the JSON schemas (caused by previous microversions) implemented
in Tempest.
Filling that gap while implementing the new integration test cases is not efficient due to
many reasons:
* Hard to review
* Sync between multiple integration tests patches which try to fill the same schema gap at same
time
* Might delay the microversion change on project side where project team wants Tempest
tests to verify the results.
Tempest will allow to fill the schema gaps at the end of each cycle, or more
often if required.
Schema gap can be filled with testing those with a minimal set of tests. Those
tests might not be integration tests and might be already covered on project
side also.
This exception is needed because:
* Allow to create microversion response schema in Tempest at the same time that projects are
implementing their API microversions. This will make implementation easier for adding
required tests before a new microversion change can be merged in the corresponding project
and hence accelerate the development of microversions.
* New schema must be verified by at least one test case which exercises such schema.
For example:
If any projects implemented 4 API microversion say- v2.3, v2.4, v2.5, v2.6
Assume microversion v2.3, v2.4, v2.6 change the API Response which means Tempest
needs to add JSON schema for v2.3, v2.4, v2.6.
In that case if only 1 or 2 tests can verify all new schemas then we do not need
separate tests for each new schemas. In worst case, we have to add 3 separate tests.
#. Test covers service behavior at large scale with involvement of more deep layer like hypervisor
etc not just API/DB layer. This type of tests will be added case by case basis and
with project team consultation about why it cannot be covered on project side and worth to test
in Tempest.
Project Scope For Microversion Testing
""""""""""""""""""""""""""""""""""""""
All microversions testing which are not covered under Tempest as per above section, should be
tested on project side as functional tests or as Tempest plugin as per project decision.
Configuration options for Microversion
""""""""""""""""""""""""""""""""""""""
* Add configuration options for specifying test target Microversions.
We need to specify test target Microversions because the supported
Microversions may be different between OpenStack clouds. For operating
multiple Microversion tests in a single Tempest operation, configuration
options should represent the range of test target Microversions.
New configuration options are:
* min_microversion
* max_microversion
Those should be defined under respective section of each service.
For example:
.. code-block:: ini
[compute]
min_microversion = None
max_microversion = latest
How To Implement Microversion Tests
"""""""""""""""""""""""""""""""""""
Tempest provides stable interfaces to test API Microversion.
For Details, see: `API Microversion testing Framework`_
This document explains how to implement Microversion tests using those
interfaces.
.. _API Microversion testing Framework: https://docs.openstack.org/tempest/latest/library/api_microversion_testing.html
Step1: Add skip logic based on configured Microversion range
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Add logic to skip the tests based on Tests class and configured Microversion
range.
api_version_utils.check_skip_with_microversion function can be used
to automatically skip the tests which do not fall under configured
Microversion range.
For example:
.. code-block:: python
class BaseTestCase1(api_version_utils.BaseMicroversionTest):
[..]
@classmethod
def skip_checks(cls):
super(BaseTestCase1, cls).skip_checks()
api_version_utils.check_skip_with_microversion(cls.min_microversion,
cls.max_microversion,
CONF.compute.min_microversion,
CONF.compute.max_microversion)
Skip logic can be added in tests base class or any specific test class depends on
tests class structure.
Step2: Selected API request microversion
''''''''''''''''''''''''''''''''''''''''
Select appropriate Microversion which needs to be used
to send with API request.
api_version_utils.select_request_microversion function can be used
to select the appropriate Microversion which will be used for API request.
For example:
.. code-block:: python
@classmethod
def resource_setup(cls):
super(BaseTestCase1, cls).resource_setup()
cls.request_microversion = (
api_version_utils.select_request_microversion(
cls.min_microversion,
CONF.compute.min_microversion))
Step3: Set Microversion on Service Clients
''''''''''''''''''''''''''''''''''''''''''
Microversion selected by Test Class in previous step needs to be set on
service clients so that APIs can be requested with selected Microversion.
Microversion can be defined as global variable on service clients which
can be set using fixture.
Also Microversion header name needs to be defined on service clients which
should be constant because it is not supposed to be changed by project
as per API contract.
For example:
.. code-block:: python
COMPUTE_MICROVERSION = None
class BaseClient1(rest_client.RestClient):
api_microversion_header_name = 'X-OpenStack-Nova-API-Version'
Now test class can set the selected Microversion on required service clients
using fixture which can take care of resetting the same once tests is completed.
For example:
.. code-block:: python
def setUp(self):
super(BaseTestCase1, self).setUp()
self.useFixture(api_microversion_fixture.APIMicroversionFixture(
self.request_microversion))
Service clients needs to add set Microversion in API request header which
can be done by overriding the get_headers() method of rest_client.
For example:
.. code-block:: python
COMPUTE_MICROVERSION = None
class BaseClient1(rest_client.RestClient):
api_microversion_header_name = 'X-OpenStack-Nova-API-Version'
def get_headers(self):
headers = super(BaseClient1, self).get_headers()
if COMPUTE_MICROVERSION:
headers[self.api_microversion_header_name] = COMPUTE_MICROVERSION
return headers
Step4: Separate Test classes for each Microversion
''''''''''''''''''''''''''''''''''''''''''''''''''
This is last step to implement Microversion test class.
For any Microversion tests, basically we need to implement a
separate test class. In addition, each test class defines its
Microversion range with class variable like min_microversion
and max_microversion. Tests will be valid for that defined range.
If that range is out of configured Microversion range then, test
will be skipped.
.. note:: Microversion testing is supported at test class level not at
individual test case level.
For example:
Below test is applicable for Microversion from 2.2 till 2.9:
.. code-block:: python
class BaseTestCase1(api_version_utils.BaseMicroversionTest,
tempest.test.BaseTestCase):
[..]
class Test1(BaseTestCase1):
min_microversion = '2.2'
max_microversion = '2.9'
[..]
Below test is applicable for Microversion from 2.10 till latest:
.. code-block:: python
class Test2(BaseTestCase1):
min_microversion = '2.10'
max_microversion = 'latest'
[..]
Notes about Compute Microversion Tests
""""""""""""""""""""""""""""""""""""""
Some of the compute Microversion tests have been already implemented
with the Microversion testing framework. So for further tests only
step 4 is needed.
Along with that JSON response schema might need versioning if needed.
Compute service clients strictly validate the response against defined JSON
schema and does not allow additional elements in response.
So if that Microversion changed the API response then schema needs to be versioned.
New JSON schema file needs to be defined with new response attributes and service
client methods will select the schema based on requested microversion.
If Microversion tests are implemented randomly meaning not
in sequence order(v2.20 tests added and previous Microversion tests are not yet added)
then, still schema might need to be version for older Microversion if they changed
the response.
This is because Nova Microversion includes all the previous Microversions behavior.
For Example:
Implementing the v2.20 Microversion tests before v2.9 and 2.19-
v2.20 API request will respond as latest behavior of Nova till v2.20,
and in v2.9 and 2.19, server response has been changed so response schema needs
to be versioned accordingly.
That can be done by using the get_schema method in below module:
The base_compute_client module
''''''''''''''''''''''''''''''
.. automodule:: tempest.lib.services.compute.base_compute_client
:members:
Microversion tests implemented in Tempest
"""""""""""""""""""""""""""""""""""""""""
* Compute
* `2.1`_
.. _2.1: https://docs.openstack.org/nova/latest/api_microversion_history.html#id1
* `2.2`_
.. _2.2: http://docs.openstack.org/nova/latest/api_microversion_history.html#id2
* `2.10`_
.. _2.10: http://docs.openstack.org/nova/latest/api_microversion_history.html#id9
* `2.20`_
.. _2.20: http://docs.openstack.org/nova/latest/api_microversion_history.html#id18
* `2.25`_
.. _2.25: http://docs.openstack.org/nova/latest/api_microversion_history.html#maximum-in-mitaka
* `2.32`_
.. _2.32: http://docs.openstack.org/nova/latest/api_microversion_history.html#id29
* `2.37`_
.. _2.37: http://docs.openstack.org/nova/latest/api_microversion_history.html#id34
* `2.42`_
.. _2.42: http://docs.openstack.org/nova/latest/api_microversion_history.html#maximum-in-ocata
* `2.47`_
.. _2.47: http://docs.openstack.org/nova/latest/api_microversion_history.html#id42
* `2.48`_
.. _2.48: http://docs.openstack.org/nova/latest/api_microversion_history.html#id43
* Volume
* `3.3`_
.. _3.3: https://docs.openstack.org/cinder/latest/devref/api_microversion_history.html#id4

1
doc/source/overview.rst
View File

@ -1 +0,0 @@
../../README.rst

346
doc/source/plugin.rst
View File

@ -1,346 +0,0 @@
.. _tempest_plugin:
=============================
Tempest Test Plugin Interface
=============================
Tempest has an external test plugin interface which enables anyone to integrate
an external test suite as part of a tempest run. This will let any project
leverage being run with the rest of the tempest suite while not requiring the
tests live in the tempest tree.
Creating a plugin
=================
Creating a plugin is fairly straightforward and doesn't require much additional
effort on top of creating a test suite using tempest.lib. One thing to note with
doing this is that the interfaces exposed by tempest are not considered stable
(with the exception of configuration variables which ever effort goes into
ensuring backwards compatibility). You should not need to import anything from
tempest itself except where explicitly noted.
Stable Tempest APIs plugins may use
-----------------------------------
As noted above, several tempest APIs are acceptable to use from plugins, while
others are not. A list of stable APIs available to plugins is provided below:
* tempest.lib.*
* tempest.config
* tempest.test_discover.plugins
If there is an interface from tempest that you need to rely on in your plugin
which is not listed above, it likely needs to be migrated to tempest.lib. In
that situation, file a bug, push a migration patch, etc. to expedite providing
the interface in a reliable manner.
Plugin Cookiecutter
-------------------
In order to create the basic structure with base classes and test directories
you can use the tempest-plugin-cookiecutter project::
> pip install -U cookiecutter && cookiecutter https://git.openstack.org/openstack/tempest-plugin-cookiecutter
Cloning into 'tempest-plugin-cookiecutter'...
remote: Counting objects: 17, done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 17 (delta 1), reused 14 (delta 1)
Unpacking objects: 100% (17/17), done.
Checking connectivity... done.
project (default is "sample")? foo
testclass (default is "SampleTempestPlugin")? FooTempestPlugin
This would create a folder called ``foo_tempest_plugin/`` with all necessary
basic classes. You only need to move/create your test in
``foo_tempest_plugin/tests``.
Entry Point
-----------
Once you've created your plugin class you need to add an entry point to your
project to enable tempest to find the plugin. The entry point must be added
to the "tempest.test_plugins" namespace.
If you are using pbr this is fairly straightforward, in the setup.cfg just add
something like the following:
.. code-block:: ini
[entry_points]
tempest.test_plugins =
plugin_name = module.path:PluginClass
Standalone Plugin vs In-repo Plugin
-----------------------------------
Since all that's required for a plugin to be detected by tempest is a valid
setuptools entry point in the proper namespace there is no difference from the
tempest perspective on either creating a separate python package to
house the plugin or adding the code to an existing python project. However,
there are tradeoffs to consider when deciding which approach to take when
creating a new plugin.
If you create a separate python project for your plugin this makes a lot of
things much easier. Firstly it makes packaging and versioning much simpler, you
can easily decouple the requirements for the plugin from the requirements for
the other project. It lets you version the plugin independently and maintain a
single version of the test code across project release boundaries (see the
`Branchless Tempest Spec`_ for more details on this). It also greatly
simplifies the install time story for external users. Instead of having to
install the right version of a project in the same python namespace as tempest
they simply need to pip install the plugin in that namespace. It also means
that users don't have to worry about inadvertently installing a tempest plugin
when they install another package.
.. _Branchless Tempest Spec: http://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/branchless-tempest.html
The sole advantage to integrating a plugin into an existing python project is
that it enables you to land code changes at the same time you land test changes
in the plugin. This reduces some of the burden on contributors by not having
to land 2 changes to add a new API feature and then test it and doing it as a
single combined commit.
Plugin Class
============
To provide tempest with all the required information it needs to be able to run
your plugin you need to create a plugin class which tempest will load and call
to get information when it needs. To simplify creating this tempest provides an
abstract class that should be used as the parent for your plugin. To use this
you would do something like the following:
.. code-block:: python
from tempest.test_discover import plugins
class MyPlugin(plugins.TempestPlugin):
Then you need to ensure you locally define all of the mandatory methods in the
abstract class, you can refer to the api doc below for a reference of what that
entails.
Abstract Plugin Class
---------------------
.. autoclass:: tempest.test_discover.plugins.TempestPlugin
:members:
Plugin Structure
================
While there are no hard and fast rules for the structure a plugin, there are
basically no constraints on what the plugin looks like as long as the 2 steps
above are done. However, there are some recommended patterns to follow to make
it easy for people to contribute and work with your plugin. For example, if you
create a directory structure with something like::
plugin_dir/
config.py
plugin.py
tests/
api/
scenario/
services/
client.py
That will mirror what people expect from tempest. The file
* **config.py**: contains any plugin specific configuration variables
* **plugin.py**: contains the plugin class used for the entry point
* **tests**: the directory where test discovery will be run, all tests should
be under this dir
* **services**: where the plugin specific service clients are
Additionally, when you're creating the plugin you likely want to follow all
of the tempest developer and reviewer documentation to ensure that the tests
being added in the plugin act and behave like the rest of tempest.
Dealing with configuration options
----------------------------------
Historically Tempest didn't provide external guarantees on its configuration
options. However, with the introduction of the plugin interface this is no
longer the case. An external plugin can rely on using any configuration option
coming from Tempest, there will be at least a full deprecation cycle for any
option before it's removed. However, just the options provided by Tempest
may not be sufficient for the plugin. If you need to add any plugin specific
configuration options you should use the ``register_opts`` and
``get_opt_lists`` methods to pass them to Tempest when the plugin is loaded.
When adding configuration options the ``register_opts`` method gets passed the
CONF object from tempest. This enables the plugin to add options to both
existing sections and also create new configuration sections for new options.
Service Clients
---------------
If a plugin defines a service client, it is beneficial for it to implement the
``get_service_clients`` method in the plugin class. All service clients which
are exposed via this interface will be automatically configured and be
available in any instance of the service clients class, defined in
``tempest.lib.services.clients.ServiceClients``. In case multiple plugins are
installed, all service clients from all plugins will be registered, making it
easy to write tests which rely on multiple APIs whose service clients are in
different plugins.
Example implementation of ``get_service_clients``:
.. code-block:: python
def get_service_clients(self):
# Example implementation with two service clients
my_service1_config = config.service_client_config('my_service')
params_my_service1 = {
'name': 'my_service_v1',
'service_version': 'my_service.v1',
'module_path': 'plugin_tempest_tests.services.my_service.v1',
'client_names': ['API1Client', 'API2Client'],
}
params_my_service1.update(my_service_config)
my_service2_config = config.service_client_config('my_service')
params_my_service2 = {
'name': 'my_service_v2',
'service_version': 'my_service.v2',
'module_path': 'plugin_tempest_tests.services.my_service.v2',
'client_names': ['API1Client', 'API2Client'],
}
params_my_service2.update(my_service2_config)
return [params_my_service1, params_my_service2]
Parameters:
* **name**: Name of the attribute used to access the ``ClientsFactory`` from
the ``ServiceClients`` instance. See example below.
* **service_version**: Tempest enforces a single implementation for each
service client. Available service clients are held in a ``ClientsRegistry``
singleton, and registered with ``service_version``, which means that
``service_version`` must be unique and it should represent the service API
and version implemented by the service client.
* **module_path**: Relative to the service client module, from the root of the
plugin.
* **client_names**: Name of the classes that implement service clients in the
service clients module.
Example usage of the service clients in tests:
.. code-block:: python
# my_creds is instance of tempest.lib.auth.Credentials
# identity_uri is v2 or v3 depending on the configuration
from tempest.lib.services import clients
my_clients = clients.ServiceClients(my_creds, identity_uri)
my_service1_api1_client = my_clients.my_service_v1.API1Client()
my_service2_api1_client = my_clients.my_service_v2.API1Client(my_args='any')
Automatic configuration and registration of service clients imposes some extra
constraints on the structure of the configuration options exposed by the
plugin.
First ``service_version`` should be in the format `service_config[.version]`.
The `.version` part is optional, and should only be used if there are multiple
versions of the same API available. The `service_config` must match the name of
a configuration options group defined by the plugin. Different versions of one
API must share the same configuration group.
Second the configuration options group `service_config` must contain the
following options:
* `catalog_type`: corresponds to `service` in the catalog
* `endpoint_type`
The following options will be honoured if defined, but they are not mandatory,
as they do not necessarily apply to all service clients.
* `region`: default to identity.region
* `build_timeout` : default to compute.build_timeout
* `build_interval`: default to compute.build_interval
Third the service client classes should inherit from ``RestClient``, should
accept generic keyword arguments, and should pass those arguments to the
``__init__`` method of ``RestClient``. Extra arguments can be added. For
instance:
.. code-block:: python
class MyAPIClient(rest_client.RestClient):
def __init__(self, auth_provider, service, region,
my_arg, my_arg2=True, **kwargs):
super(MyAPIClient, self).__init__(
auth_provider, service, region, **kwargs)
self.my_arg = my_arg
self.my_args2 = my_arg
Finally the service client should be structured in a python module, so that all
service client classes are importable from it. Each major API version should
have its own module.
The following folder and module structure is recommended for a single major
API version::
plugin_dir/
services/
__init__.py
client_api_1.py
client_api_2.py
The content of __init__.py module should be:
.. code-block:: python
from client_api_1.py import API1Client
from client_api_2.py import API2Client
__all__ = ['API1Client', 'API2Client']
The following folder and module structure is recommended for multiple major
API version::
plugin_dir/
services/
v1/
__init__.py
client_api_1.py
client_api_2.py
v2/
__init__.py
client_api_1.py
client_api_2.py
The content each of __init__.py module under vN should be:
.. code-block:: python
from client_api_1.py import API1Client
from client_api_2.py import API2Client
__all__ = ['API1Client', 'API2Client']
Using Plugins
=============
Tempest will automatically discover any installed plugins when it is run. So by
just installing the python packages which contain your plugin you'll be using
them with tempest, nothing else is really required.
However, you should take care when installing plugins. By their very nature
there are no guarantees when running tempest with plugins enabled about the
quality of the plugin. Additionally, while there is no limitation on running
with multiple plugins it's worth noting that poorly written plugins might not
properly isolate their tests which could cause unexpected cross interactions
between plugins.
Notes for using plugins with virtualenvs
----------------------------------------
When using a tempest inside a virtualenv (like when running under tox) you have
to ensure that the package that contains your plugin is either installed in the
venv too or that you have system site-packages enabled. The virtualenv will
isolate the tempest install from the rest of your system so just installing the
plugin package on your system and then running tempest inside a venv will not
work.
Tempest also exposes a tox job, all-plugin, which will setup a tox virtualenv
with system site-packages enabled. This will let you leverage tox without
requiring to manually install plugins in the tox venv before running tests.

7
doc/source/run.rst
View File

@ -1,7 +0,0 @@
.. _tempest_run:
-----------
Tempest Run
-----------
.. automodule:: tempest.cmd.run

13
doc/source/sampleconf.rst
View File

@ -1,13 +0,0 @@
.. _tempest-sampleconf:
Sample Configuration File
==========================
The following is a sample Tempest configuration for adaptation and use. It is
auto-generated from Tempest when this documentation is built, so
if you are having issues with an option, please compare your version of
Tempest with the version of this documentation.
The sample configuration can also be viewed in `file form <_static/tempest.conf.sample>`_.
.. literalinclude:: _static/tempest.conf.sample

5
doc/source/subunit_describe_calls.rst
View File

@ -1,5 +0,0 @@
------------------------------
Subunit Describe Calls Utility
------------------------------
.. automodule:: tempest.cmd.subunit_describe_calls

188
doc/source/test_removal.rst
View File

@ -1,188 +0,0 @@
Tempest Test Removal Procedure
==============================
Historically tempest was the only way of doing functional testing and
integration testing in OpenStack. This was mostly only an artifact of tempest
being the only proven pattern for doing this, not an artifact of a design
decision. However, moving forward as functional testing is being spun up in
each individual project we really only want tempest to be the integration test
suite it was intended to be; testing the high level interactions between
projects through REST API requests. In this model there are probably existing
tests that aren't the best fit living in tempest. However, since tempest is
largely still the only gating test suite in this space we can't carelessly rip
out everything from the tree. This document outlines the procedure which was
developed to ensure we minimize the risk for removing something of value from
the tempest tree.
This procedure might seem overly conservative and slow paced, but this is by
design to try and ensure we don't remove something that is actually providing
value. Having potential duplication between testing is not a big deal
especially compared to the alternative of removing something which is actually
providing value and is actively catching bugs, or blocking incorrect patches
from landing.
Proposing a test removal
------------------------
3 prong rule for removal
^^^^^^^^^^^^^^^^^^^^^^^^
In the proposal etherpad we'll be looking for answers to 3 questions
#. The tests proposed for removal must have equiv. coverage in a different
project's test suite (whether this is another gating test project, or an in
tree functional test suite). For API tests preferably the other project will
have a similar source of friction in place to prevent breaking api changes
so that we don't regress and let breaking api changes slip through the
gate.
#. The test proposed for removal has a failure rate < 0.50% in the gate over
the past release (the value and interval will likely be adjusted in the
future)
.. _`prong #3`:
#. There must not be an external user/consumer of tempest
that depends on the test proposed for removal
The answers to 1 and 2 are easy to verify. For 1 just provide a link to the new
test location. If you are linking to the tempest removal patch please also put
a Depends-On in the commit message for the commit which moved the test into
another repo.
For prong 2 you can use OpenStack-Health:
Using OpenStack-Health
""""""""""""""""""""""
Go to: http://status.openstack.org/openstack-health and then navigate to a per
test page for six months. You'll end up with a page that will graph the success
and failure rates on the bottom graph. For example, something like `this URL`_.
.. _this URL: http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern?groupKey=project&resolutionKey=day&duration=P6M
The Old Way using subunit2sql directly
""""""""""""""""""""""""""""""""""""""
SELECT * from tests where test_id like "%test_id%";
(where $test_id is the full test_id, but truncated to the class because of
setUpClass or tearDownClass failures)
You can access the infra mysql subunit2sql db w/ read-only permissions with:
* hostname: logstash.openstack.org
* username: query
* password: query
* db_name: subunit2sql
For example if you were trying to remove the test with the id:
tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON.test_get_flavor_details_for_deleted_flavor
you would run the following:
#. run: "mysql -u query -p -h logstash.openstack.org subunit2sql" to connect
to the subunit2sql db
#. run the query: MySQL [subunit2sql]> select * from tests where test_id like
"tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON%";
which will return a table of all the tests in the class (but it will also
catch failures in setUpClass and tearDownClass)
#. paste the output table with numbers and the mysql command you ran to
generate it into the etherpad.
Eventually a cli interface will be created to make that a bit more friendly.
Also a dashboard is in the works so we don't need to manually run the command.
The intent of the 2nd prong is to verify that moving the test into a project
specific testing is preventing bugs (assuming the tempest tests were catching
issues) from bubbling up a layer into tempest jobs. If we're seeing failure
rates above a certain threshold in the gate checks that means the functional
testing isn't really being effective in catching that bug (and therefore
blocking it from landing) and having the testing run in tempest still has
value.
However for the 3rd prong verification is a bit more subjective. The original
intent of this prong was mostly for refstack/defcore and also for things that
running on the stable branches. We don't want to remove any tests if that
would break our api consistency checking between releases, or something that
defcore/refstack is depending on being in tempest. It's worth pointing out
that if a test is used in defcore as part of interop testing then it will
probably have continuing value being in tempest as part of the
integration/integrated tests in general. This is one area where some overlap
is expected between testing in projects and tempest, which is not a bad thing.
Discussing the 3rd prong
""""""""""""""""""""""""
There are 2 approaches to addressing the 3rd prong. Either it can be raised
during a qa meeting during the tempest discussion. Please put it on the agenda
well ahead of the scheduled meeting. Since the meeting time will be well known
ahead of time anyone who depends on the tests will have ample time beforehand
to outline any concerns on the before the meeting. To give ample time for
people to respond to removal proposals please add things to the agenda by the
Monday before the meeting.
The other option is to raise the removal on the openstack-dev mailing list.
(for example see: http://lists.openstack.org/pipermail/openstack-dev/2016-February/086218.html )
This will raise the issue to the wider community and attract at least the same
(most likely more) attention than discussing it during the irc meeting. The
only downside is that it might take more time to get a response, given the
nature of ML.
Exceptions to this procedure
----------------------------
For the most part all tempest test removals have to go through this procedure
there are a couple of exceptions though:
#. The class of testing has been decided to be outside the scope of tempest.
#. A revert for a patch which added a broken test, or testing which didn't
actually run in the gate (basically any revert for something which
shouldn't have been added)
#. Tests that would become out of scope as a consequence of an API change,
as described in `API Compatibility`_.
Such tests cannot live in Tempest because of the branchless nature of
Tempest. Such test must still honor `prong #3`_.
For the first exception type the only types of testing in tree which have been
declared out of scope at this point are:
* The CLI tests (which should be completely removed at this point)
* Neutron Adv. Services testing (which should be completely removed at this
point)
* XML API Tests (which should be completely removed at this point)
* EC2 API/boto tests (which should be completely removed at this point)
For tests that fit into this category the only criteria for removal is that
there is equivalent testing elsewhere.
Tempest Scope
^^^^^^^^^^^^^
Starting in the liberty cycle tempest has defined a set of projects which
are defined as in scope for direct testing in tempest. As of today that list
is:
* Keystone
* Nova
* Glance
* Cinder
* Neutron
* Swift
anything that lives in tempest which doesn't test one of these projects can be
removed assuming there is equivalent testing elsewhere. Preferably using the
`tempest plugin mechanism`_
to maintain continuity after migrating the tests out of tempest.
.. _tempest plugin mechanism: https://docs.openstack.org/tempest/latest/plugin.html
API Compatibility
"""""""""""""""""
If an API introduces a non-discoverable, backward incompatible change, and
such change is not backported to all versions supported by Tempest, tests for
that API cannot live in Tempest anymore.
This is because tests would not be able to know or control which API response
to expect, and thus would not be able to enforce a specific behavior.
If a test exists in Tempest that would meet this criteria as consequence of a
change, the test must be removed according to the procedure discussed into
this document. The API change should not be merged until all conditions
required for test removal can be met.

5
doc/source/workspace.rst
View File

@ -1,5 +0,0 @@
-----------------
Tempest Workspace
-----------------
.. automodule:: tempest.cmd.workspace

311
doc/source/write_tests.rst
View File

@ -1,311 +0,0 @@
.. _tempest_test_writing:
Tempest Test Writing Guide
##########################
This guide serves as a starting point for developers working on writing new
Tempest tests. At a high level tests in Tempest are just tests that conform to
the standard python `unit test`_ framework. But there are several aspects of
that are unique to Tempest and its role as an integration test suite running
against a real cloud.
.. _unit test: https://docs.python.org/3.6/library/unittest.html
.. note:: This guide is for writing tests in the Tempest repository. While many
parts of this guide are also applicable to Tempest plugins, not all
the APIs mentioned are considered stable or recommended for use in
plugins. Please refer to :ref:`tempest_plugin` for details about
writing plugins
Adding a New TestCase
=====================
The base unit of testing in Tempest is the `TestCase`_ (also called the test
class). Each TestCase contains test methods which are the individual tests that
will be executed by the test runner. But, the TestCase is the smallest self
contained unit for tests from the Tempest perspective. It's also the level at
which Tempest is parallel safe. In other words, multiple TestCases can be
executed in parallel, but individual test methods in the same TestCase can not.
Also, all test methods within a TestCase are assumed to be executed serially. As
such you can use the test case to store variables that are shared between
methods.
.. _TestCase: https://docs.python.org/3.6/library/unittest.html#unittest.TestCase
In standard unittest the lifecycle of a TestCase can be described in the
following phases:
#. setUpClass
#. setUp
#. Test Execution
#. tearDown
#. doCleanups
#. tearDownClass
setUpClass
----------
The setUpClass phase is the first phase executed by the test runner and is used
to perform any setup required for all the test methods to be executed. In
Tempest this is a very important step and will automatically do the necessary
setup for interacting with the configured cloud.
To accomplish this you do **not** define a setUpClass function, instead there
are a number of predefined phases to setUpClass that are used. The phases are:
* skip_checks
* setup_credentials
* setup_clients
* resource_setup
which is executed in that order. An example of a TestCase which defines all
of these would be::
from tempest import config
from tempest import test
CONF = config.CONF
class TestExampleCase(test.BaseTestCase):
@classmethod
def skip_checks(cls):
"""This section is used to evaluate config early and skip all test
methods based on these checks
"""
super(TestExampleCase, cls).skip_checks()
if not CONF.section.foo
cls.skip('A helpful message')
@classmethod
def setup_credentials(cls):
"""This section is used to do any manual credential allocation and also
in the case of dynamic credentials to override the default network
resource creation/auto allocation
"""
# This call is used to tell the credential allocator to not create any
# network resources for this test case. It also enables selective
# creation of other neutron resources. NOTE: it must go before the
# super call
cls.set_network_resources()
super(TestExampleCase, cls).setup_credentials()
@classmethod
def setup_clients(cls):
"""This section is used to setup client aliases from the manager object
or to initialize any additional clients. Except in a few very
specific situations you should not need to use this.
"""
super(TestExampleCase, cls).setup_clients()
cls.servers_client = cls.os_primary.servers_client
@classmethod
def resource_setup(cls):
"""This section is used to create any resources or objects which are
going to be used and shared by **all** test methods in the
TestCase. Note then anything created in this section must also be
destroyed in the corresponding resource_cleanup() method (which will
be run during tearDownClass())
"""
super(TestExampleCase, cls).resource_setup()
cls.shared_server = cls.servers_client.create_server(...)
.. _credentials:
Allocating Credentials
''''''''''''''''''''''
Since Tempest tests are all about testing a running cloud, every test will need
credentials to be able to make API requests against the cloud. Since this is
critical to operation and, when running in parallel, easy to make a mistake,
the base TestCase class will automatically allocate a regular user for each
TestCase during the setup_credentials() phase. During this process it will also
initialize a client manager object using those credentials, which will be your
entry point into interacting with the cloud. For more details on how credentials
are allocated the :ref:`tempest_cred_provider_conf` section of the Tempest
Configuration Guide provides more details on the operation of this.
There are some cases when you need more than a single set of credentials, or
credentials with a more specialized set of roles. To accomplish this you have
to set a class variable ``credentials`` on the TestCase directly. For example::
from tempest import test
class TestExampleAdmin(test.BaseTestCase):
credentials = ['primary', 'admin']
@classmethod
def skip_checks(cls):
...
In this example the ``TestExampleAdmin`` TestCase will allocate 2 sets of
credentials, one regular user and one admin user. The corresponding manager
objects will be set as class variables ``cls.os_primary`` and ``cls.os_admin``
respectively. You can also allocate a second user by putting **'alt'** in the
list too. A set of alt credentials are the same as primary but can be used
for tests cases that need a second user/project.
You can also specify credentials with specific roles assigned. This is useful
for cases where there are specific RBAC requirements hard coded into an API.
The canonical example of this are swift tests which often want to test swift's
concepts of operator and reseller_admin. An actual example from Tempest on how
to do this is::
class PublicObjectTest(base.BaseObjectTest):
credentials = [['operator', CONF.object_storage.operator_role],
['operator_alt', CONF.object_storage.operator_role]]
@classmethod
def setup_credentials(cls):
super(PublicObjectTest, cls).setup_credentials()
...
In this case the manager objects will be set to ``cls.os_roles_operator`` and
``cls.os_roles_operator_alt`` respectively.
There is no limit to how many credentials you can allocate in this manner,
however in almost every case you should **not** need more than 3 sets of
credentials per test case.
To figure out the mapping of manager objects set on the TestCase and the
requested credentials you can reference:
+-------------------+---------------------+
| Credentials Entry | Manager Variable |
+===================+=====================+
| primary | cls.os_primary |
+-------------------+---------------------+
| admin | cls.os_admin |
+-------------------+---------------------+
| alt | cls.os_alt |
+-------------------+---------------------+
| [$label, $role] | cls.os_roles_$label |
+-------------------+---------------------+
By default cls.os_primary is available since it is allocated in the base Tempest test
class (located in tempest/test.py). If your TestCase inherits from a different
direct parent class (it'll still inherit from the BaseTestCase, just not
directly) be sure to check if that class overrides allocated credentials.
Dealing with Network Allocation
'''''''''''''''''''''''''''''''
When Neutron is enabled and a testing requires networking this isn't normally
automatically setup when a tenant is created. Since Tempest needs isolated
tenants to function properly it also needs to handle network allocation. By
default the base test class will allocate a network, subnet, and router
automatically (this depends on the configured credential provider, for more
details see: :ref:`tempest_conf_network_allocation`). However, there are
situations where you do no need all of these resources allocated (or your
TestCase inherits from a class that overrides the default in tempest/test.py).
There is a class level mechanism to override this allocation and specify which
resources you need. To do this you need to call `cls.set_network_resources()`
in the `setup_credentials()` method before the `super()`. For example::
from tempest import test
class TestExampleCase(test.BaseTestCase):
@classmethod
def setup_credentials(cls):
cls.set_network_resources(network=True, subnet=True, router=False)
super(TestExampleCase, cls).setup_credentials()
There are 2 quirks with the usage here. First for the set_network_resources
function to work properly it **must be called before super()**. This is so
that children classes' settings are always used instead of a parent classes'.
The other quirk here is that if you do not want to allocate any network
resources for your test class simply call `set_network_resources()` without
any arguments. For example::
from tempest import test
class TestExampleCase(test.BaseTestCase):
@classmethod
def setup_credentials(cls):
cls.set_network_resources()
super(TestExampleCase, cls).setup_credentials()
This will not allocate any networking resources. This is because by default all
the arguments default to False.
It's also worth pointing out that it is common for base test classes for
different services (and scenario tests) to override this setting. When
inheriting from classes other than the base TestCase in tempest/test.py it is
worth checking the immediate parent for what is set to determine if your
class needs to override that setting.
Interacting with Credentials and Clients
========================================
Once you have your basic TestCase setup you'll want to start writing tests. To
do that you need to interact with an OpenStack deployment. This section will
cover how credentials and clients are used inside of Tempest tests.
Manager Objects
---------------
The primary interface with which you interact with both credentials and
API clients is the client manager object. These objects are created
automatically by the base test class as part of credential setup (for more
details see the previous :ref:`credentials` section). Each manager object is
initialized with a set of credentials and has each client object already setup
to use that set of credentials for making all the API requests. Each client is
accessible as a top level attribute on the manager object. So to start making
API requests you just access the client's method for making that call and the
credentials are already setup for you. For example if you wanted to make an API
call to create a server in Nova::
from tempest import test
class TestExampleCase(test.BaseTestCase):
def test_example_create_server(self):
self.os_primary.servers_client.create_server(...)
is all you need to do. As described previously, in the above example the
``self.os_primary`` is created automatically because the base test class sets the
``credentials`` attribute to allocate a primary credential set and initializes
the client manager as ``self.os_primary``. This same access pattern can be used
for all of the clients in Tempest.
Credentials Objects
-------------------
In certain cases you need direct access to the credentials (the most common
use case would be an API request that takes a user or project id in the request
body). If you're in a situation where you need to access this you'll need to
access the ``credentials`` object which is allocated from the configured
credential provider in the base test class. This is accessible from the manager
object via the manager's ``credentials`` attribute. For example::
from tempest import test
class TestExampleCase(test.BaseTestCase):
def test_example_create_server(self):
credentials = self.os_primary.credentials
The credentials object provides access to all of the credential information you
would need to make API requests. For example, building off the previous
example::
from tempest import test
class TestExampleCase(test.BaseTestCase):
def test_example_create_server(self):
credentials = self.os_primary.credentials
username = credentials.username
user_id = credentials.user_id
password = credentials.password
tenant_id = credentials.tenant_id

62
etc/accounts.yaml.sample
View File

@ -1,62 +0,0 @@
# The number of accounts required can be estimated as CONCURRENCY x 2
# It is expected that each user provided here will be in a different tenant.
# This is required to provide isolation between test for running in parallel
#
# Valid fields for credentials are defined in the descendants of
# lib.auth.Credentials - see KeystoneV[2|3]Credentials.ATTRIBUTES
#
# The fields in KeystoneV3Credentials behave as follows:
#
# tenant_[id|name] also sets project_[id|name].
#
# project_[id|name] also sets tenant_[id|name].
#
# Providing distinct values for both tenant_[id|name] and project_[id|name]
# will result in an InvalidCredentials exception.
#
# The value of project_domain_[id|name] is used for user_domain_[id|name] if
# the latter is not specified.
#
# The value of user_domain_[id|name] is used for project_domain_[id|name] if
# the latter is not specified.
#
# The value of domain_[id|name] is used for project_domain_[id|name] if not
# specified and user_domain_[id|name] if not specified.
- username: 'user_1'
tenant_name: 'test_tenant_1'
password: 'test_password'
- username: 'user_2'
tenant_name: 'test_tenant_2'
password: 'test_password'
# To specify which roles a user has list them under the roles field
- username: 'multi_role_user'
tenant_name: 'test_tenant_42'
password: 'test_password'
roles:
- 'fun_role'
- 'not_an_admin'
- 'an_admin'
# To specify a user has a role specified in the config file you can use the
# type field to specify it, valid values are admin, operator, and reseller_admin
- username: 'swift_pseudo_admin_user_1'
tenant_name: 'admin_tenant_1'
password: 'test_password'
types:
- 'reseller_admin'
- 'operator'
# Networks can be specified to tell tempest which network it should use when
# creating servers with an account
- username: 'admin_user_1'
tenant_name: 'admin_tenant_1'
password: 'test_password'
types:
- 'admin'
resources:
network: 'public'
router: 'admin_tenant_1-router'

35
etc/logging.conf.sample
View File

@ -1,35 +0,0 @@
[loggers]
keys=root
[handlers]
keys=file,devel,syslog
[formatters]
keys=simple,tests
[logger_root]
level=DEBUG
handlers=file
[handler_file]
class=FileHandler
level=DEBUG
args=('tempest.log', 'w+')
formatter=tests
[handler_syslog]
class=handlers.SysLogHandler
level=ERROR
args = ('/dev/log', handlers.SysLogHandler.LOG_USER)
[handler_devel]
class=StreamHandler
level=DEBUG
args=(sys.stdout,)
formatter=simple
[formatter_tests]
class = oslo_log.formatters.ContextFormatter
[formatter_simple]
format=%(asctime)s.%(msecs)03d %(process)d %(levelname)s: %(message)s

0
etc/whitelist.yaml
View File

13
releasenotes/notes/10/10.0-supported-openstack-releases-b88db468695348f6.yaml
View File

@ -1,13 +0,0 @@
---
other:
- OpenStack Releases Supported at this time are the same as in the
previous release 9,
**Kilo** and
**Liberty**.
The release under current development as of this tag is Mitaka,
meaning that every Tempest commit is also tested against master during
the Mitaka cycle. However, this does not necessarily mean that using
Tempest as of this tag will work against a Mitaka (or future releases)
cloud.

11
releasenotes/notes/10/10.0.0-Tempest-library-interface-0eb680b810139a50.yaml
View File

@ -1,11 +0,0 @@
---
prelude: |
This release includes the addition of the stable library interface for
tempest. This behaves just as tempest-lib did prior to this, but instead
it lives directly in the tempest project. For more information refer to
the `library docs`_.
.. _library docs: https://docs.openstack.org/tempest/latest/library.html#current-library-apis
features:
- Tempest library interface

3
releasenotes/notes/10/10.0.0-start-using-reno-ed9518126fd0e1a3.yaml
View File

@ -1,3 +0,0 @@
---
other:
- Start using reno for managing release notes.

3
releasenotes/notes/11/11.0.0-api-microversion-testing-support-2ceddd2255670932.yaml
View File

@ -1,3 +0,0 @@
---
features:
- Tempest library interface addition(API Microversion testing interfaces).

3
releasenotes/notes/11/11.0.0-compute-microversion-support-e0b23f960f894b9b.yaml
View File

@ -1,3 +0,0 @@
---
features:
- Compute Microversion testing support in Service Clients.

12
releasenotes/notes/11/11.0.0-supported-openstack-releases-1e5d7295d939d439.yaml
View File

@ -1,12 +0,0 @@
---
prelude: >
This release is marking the start of Mitaka release support in tempest
other:
- OpenStack Releases Supported at this time are **Kilo**, **Liberty**,
**Mitaka**
The release under current development as of this tag is Newton,
meaning that every Tempest commit is also tested against master during
the Newton cycle. However, this does not necessarily mean that using
Tempest as of this tag will work against a Newton (or future releases)
cloud.

12
releasenotes/notes/12/12.0.0-supported-openstack-releases-f10aac381d933dd1.yaml
View File

@ -1,12 +0,0 @@
---
prelude: >
This release is marking the end of Kilo release support in Tempest
other:
- OpenStack Releases Supported after this release are **Liberty**
and **Mitaka**
The release under current development as of this tag is Newton,
meaning that every Tempest commit is also tested against master during
the Newton cycle. However, this does not necessarily mean that using
Tempest as of this tag will work against a Newton (or future releases)
cloud.

4
releasenotes/notes/12/12.1.0-add-network-versions-client-d90e8334e1443f5c.yaml
View File

@ -1,4 +0,0 @@
---
features:
- Adds a network version client for querying
Neutron's API version discovery URL ("GET /").

7
releasenotes/notes/12/12.1.0-add-scope-to-auth-b5a82493ea89f41e.yaml
View File

@ -1,7 +0,0 @@
---
features:
- Tempest library auth interface now supports scope. Scope allows to control
the scope of tokens requested via the identity API. Identity V2 supports
unscoped and project scoped tokens, but only the latter are implemented.
Identity V3 supports unscoped, project and domain scoped token, all three
are available.

4
releasenotes/notes/12/12.1.0-add-tempest-run-3d0aaf69c2ca4115.yaml
View File

@ -1,4 +0,0 @@
---
features:
- Adds the tempest run command to the unified tempest CLI. This new command
is used for running tempest tests.

5
releasenotes/notes/12/12.1.0-add-tempest-workspaces-228a2ba4690b5589.yaml
View File

@ -1,5 +0,0 @@
---
features:
- Adds tempest workspaces command and WorkspaceManager.
This is used to have a centralized repository for managing
different tempest configurations.

8
releasenotes/notes/12/12.1.0-add_subunit_describe_calls-5498a37e6cd66c4b.yaml
View File

@ -1,8 +0,0 @@
---
features:
- |
Adds subunit-describe-calls. A parser for subunit streams to determine what
REST API calls are made inside of a test and in what order they are called.
* Input can be piped in or a file can be specified
* Output is shortened for stdout, the output file has more information

7
releasenotes/notes/12/12.1.0-bug-1486834-7ebca15836ae27a9.yaml
View File

@ -1,7 +0,0 @@
---
features:
- |
Tempest library auth interface now supports
filtering with catalog name. Note that filtering by
name is only successful if a known service type is
provided.

13
releasenotes/notes/12/12.1.0-identity-clients-as-library-e663c6132fcac6c2.yaml
View File

@ -1,13 +0,0 @@
---
features:
- |
Define identity service clients as libraries
The following identity service clients are defined as library interface,
so the other projects can use these modules as stable libraries without
any maintenance changes.
* endpoints_client(v2)
* roles_client(v2)
* services_client(v2)
* tenants_client(v2)
* users_client(v2)

15
releasenotes/notes/12/12.1.0-image-clients-as-library-86d17caa26ce3961.yaml
View File

@ -1,15 +0,0 @@
---
features:
- |
Define image service clients as libraries
The following image service clients are defined as library interface,
so the other projects can use these modules as stable libraries
without any maintenance changes.
* image_members_client(v1)
* images_client(v1)
* image_members_client(v2)
* images_client(v2)
* namespaces_client(v2)
* resource_types_client(v2)
* schemas_client(v2)

11
releasenotes/notes/12/12.1.0-new-test-utils-module-adf34468c4d52719.yaml
View File

@ -1,11 +0,0 @@
---
features:
- A new `test_utils` module has been added to tempest.lib.common.utils. It
should hold any common utility functions that help writing Tempest tests.
- A new utility function called `call_and_ignore_notfound_exc` has been
added to the `test_utils` module. That function call another function
passed as parameter and ignore the NotFound exception if it raised.
deprecations:
- tempest.lib.common.utils.misc.find_test_caller has been moved into the
tempest.lib.common.utils.test_utils module. Calling the find_test_caller
function with its old location is deprecated.

11
releasenotes/notes/12/12.1.0-remove-input-scenarios-functionality-01308e6d4307f580.yaml
View File

@ -1,11 +0,0 @@
---
upgrade:
- The input scenarios functionality no longer exists in tempest. This caused
a large number of issues for limited benefit and was only used by a single
test, test_server_basic_ops. If you were using this functionality you'll
now have to do it manually with a script and/or tempest workspaces
deprecations:
- All the options in the input-scenario group are now deprecated. These were
only used in tree by the now removed input scenarios functionality in
test_server_basic_ops. They were only deprecated because there could be
external consumers via plugins. They will be removed during the Ocata cycle.

7
releasenotes/notes/12/12.1.0-remove-integrated-horizon-bb57551c1e5f5be3.yaml
View File

@ -1,7 +0,0 @@
---
upgrade:
- The integrated dashboard scenario test has been
removed and is now in a separate tempest plugin
tempest-horizon. The removed test coverage can be
used by installing tempest-horizon on the server
where you run tempest.

5
releasenotes/notes/12/12.1.0-remove-legacy-credential-providers-3d653ac3ba1ada2b.yaml
View File

@ -1,5 +0,0 @@
---
upgrade:
- The deprecated legacy credential provider has been removed. The only way to
configure credentials in tempest now is to use the dynamic or preprovisioned
credential providers

4
releasenotes/notes/12/12.1.0-remove-trove-tests-666522e9113549f9.yaml
View File

@ -1,4 +0,0 @@
---
upgrade:
- All tests for the Trove project have been removed from tempest. They now
live as a tempest plugin in the trove project.

6
releasenotes/notes/12/12.1.0-routers-client-as-library-25a363379da351f6.yaml
View File

@ -1,6 +0,0 @@
---
features:
- Define routers_client as stable library interface.
The routers_client module is defined as library interface,
so the other projects can use the module as stable library
without any maintenance changes.

9
releasenotes/notes/12/12.1.0-support-chunked-encoding-d71f53225f68edf3.yaml
View File

@ -1,9 +0,0 @@
---
features:
- The RestClient (in tempest.lib.common.rest_client) now supports POSTing
and PUTing data with chunked transfer encoding. Just pass an `iterable`
object as the `body` argument and set the `chunked` argument to `True`.
- A new generator called `chunkify` is added in
tempest.lib.common.utils.data_utils that yields fixed-size chunks (slices)
from a Python sequence.

12
releasenotes/notes/12/12.1.0-tempest-init-global-config-dir-location-changes-12260255871d3a2b.yaml
View File

@ -1,12 +0,0 @@
---
upgrade:
- The location on disk that the *tempest init* command looks for has changed.
Previously it would attempt to use python packaging's data files to guess
where setuptools/distutils were installing data files, which was incredibly
unreliable and depended on how you installed tempest and which versions of
setuptools, distutils, and python you had installed. Instead, now it will
use either /etc/tempest, $XDG_CONFIG_PATH/.config/tempest, or
~/.tempest/etc (attempted in that order). If none of these exist it will
create an empty ~/.tempest/etc directory. If you were relying on the
previous behavior and none of these directories were being used you will
need to move the files to live in one of these directories.

7
releasenotes/notes/12/12.2.0-add-httptimeout-in-restclient-ax78061900e3f3d7.yaml
View File

@ -1,7 +0,0 @@
---
features:
- RestClient now supports setting timeout in urllib3.poolmanager.
Clients will use CONF.service_clients.http_timeout for timeout
value to wait for http request to response.
- KeystoneAuthProvider will accept http_timeout and will use it in
get_credentials.

13
releasenotes/notes/12/12.2.0-add-new-identity-clients-3c3afd674a395bde.yaml
View File

@ -1,13 +0,0 @@
---
features:
- |
Define identity service clients as libraries.
The following identity service clients are defined as library interface,
so the other projects can use these modules as stable libraries without
any maintenance changes.
* endpoints_client(v3)
* policies_client (v3)
* regions_client(v3)
* services_client(v3)
* projects_client(v3)

18
releasenotes/notes/12/12.2.0-clients_module-16f3025f515bf9ec.yaml
View File

@ -1,18 +0,0 @@
---
features:
- The Tempest plugin interface contains a new optional method, which allows
plugins to declare and automatically register any service client defined
in the plugin.
- tempest.lib exposes a new stable interface, the clients module and
ServiceClients class, which provides a convenient way for plugin tests to
access service clients defined in Tempest as well as service clients
defined in all loaded plugins.
The new ServiceClients class only exposes for now the service clients
which are in tempest.lib, i.e. compute, network and image. The remaining
service clients (identity, volume and object-storage) will be added in
future updates.
deprecations:
- The new clients module provides a stable alternative to tempest classes
manager.Manager and clients.Manager. manager.Manager only exists now
to smoothen the transition of plugins to the new interface, but it will
be removed shortly without further notice.

8
releasenotes/notes/12/12.2.0-nova_cert_default-90eb7c1e3cde624a.yaml
View File

@ -1,8 +0,0 @@
---
upgrade:
- The ``nova_cert`` option default is changed to ``False``. The nova
certification management APIs were a hold over from ec2, and are
not used by any other parts of nova. They are deprecated for
removal in nova after the newton release. This makes false a more
sensible default going forward.

12
releasenotes/notes/12/12.2.0-plugin-service-client-registration-00b19a2dd4935ba0.yaml
View File

@ -1,12 +0,0 @@
---
features:
- A new optional interface `TempestPlugin.get_service_clients`
is available to plugins. It allows them to declare
any service client they implement. For now this is used by
tempest only, for auto-registration of service clients
in the new class `ServiceClients`.
- A new singleton class `clients.ClientsRegistry` is
available. It holds the service clients registration data
from all plugins. It is used by `ServiceClients` for
auto-registration of the service clients implemented
in plugins.

5
releasenotes/notes/12/12.2.0-remove-javelin-276f62d04f7e4a1d.yaml
View File

@ -1,5 +0,0 @@
---
upgrade:
- The previously deprecated Javelin utility has been removed from Tempest.
As an alternative Ansible can be used to construct similar yaml workflows
to what Javelin used to provide.

6
releasenotes/notes/12/12.2.0-service_client_config-8a1d7b4de769c633.yaml
View File

@ -1,6 +0,0 @@
---
features:
- A new helper method `service_client_config` has been added
to the stable module config.py that returns extracts from
configuration into a dictionary the configuration settings
relevant for the initisialisation of a service client.

18
releasenotes/notes/12/12.2.0-volume-clients-as-library-9a3444dd63c134b3.yaml
View File

@ -1,18 +0,0 @@
---
features:
- |
Define volume service clients as libraries
The following volume service clients are defined as library interface,
so the other projects can use these modules as stable libraries
without any maintenance changes.
* availability_zone_client(v1)
* availability_zone_client(v2)
* extensions_client(v1)
* extensions_client(v2)
* hosts_client(v1)
* hosts_client(v2)
* quotas_client(v1)
* quotas_client(v2)
* services_client(v1)
* services_client(v2)

15
releasenotes/notes/13/13.0.0-add-new-identity-clients-as-library-5f7ndha733nwdsn9.yaml
View File

@ -1,15 +0,0 @@
---
features:
- |
Define identity service clients as libraries.
Add new service clients to the library interface so the other projects can use these modules as stable libraries without
any maintenance changes.
* identity_client(v2)
* groups_client(v3)
* trusts_client(v3)
* users_client(v3)
* identity_client(v3)
* roles_client(v3)
* inherited_roles_client(v3)
* credentials_client(v3)

16
releasenotes/notes/13/13.0.0-add-volume-clients-as-a-library-d05b6bc35e66c6ef.yaml
View File

@ -1,16 +0,0 @@
---
features:
- |
Define volume service clients as libraries.
The following volume service clients are defined as library interface,
so the other projects can use these modules as stable libraries without
any maintenance changes.
* backups_client
* encryption_types_client (v1)
* encryption_types_client (v2)
* qos_clients (v1)
* qos_clients (v2)
* snapshots_client (v1)
* snapshots_client (v2)

4
releasenotes/notes/13/13.0.0-deprecate-get_ipv6_addr_by_EUI64-4673f07677289cf6.yaml
View File

@ -1,4 +0,0 @@
---
deprecations:
- Oslo.utils provides same method get_ipv6_addr_by_EUI64,
so deprecate it in Newton and remove it in Ocata.

5
releasenotes/notes/13/13.0.0-move-call-until-true-to-tempest-lib-c9ea70dd6fe9bd15.yaml
View File

@ -1,5 +0,0 @@
---
deprecations:
- The ``call_until_true`` function is moved from the ``tempest.test`` module
to the ``tempest.lib.common.utils.test_utils`` module. Backward
compatibility is preserved until Ocata.

13
releasenotes/notes/13/13.0.0-start-of-newton-support-3ebb274f300f28eb.yaml
View File

@ -1,13 +0,0 @@
---
prelude: >
This release is marking the start of Newton release support in Tempest
other:
- |
OpenStack releases supported at this time are **Liberty**, **Mitaka**,
and **Newton**.
The release under current development as of this tag is Ocata,
meaning that every Tempest commit is also tested against master during
the Ocata cycle. However, this does not necessarily mean that using
Tempest as of this tag will work against a Ocata (or future releases)
cloud.

5
releasenotes/notes/13/13.0.0-tempest-cleanup-nostandalone-39df2aafb2545d35.yaml
View File

@ -1,5 +0,0 @@
---
upgrade:
- the already deprecated tempest-cleanup standalone command has been
removed. The corresponding functionalities can be accessed through
the unified `tempest` command (`tempest cleanup`).

6
releasenotes/notes/13/13.0.0-volume-clients-as-library-660811011be29d1a.yaml
View File

@ -1,6 +0,0 @@
---
features:
- |
Define the v1 and v2 types_client clients for the volume service as
library interfaces, allowing other projects to use these modules as
stable libraries without maintenance changes.

6
releasenotes/notes/14/14.0.0-add-cred-provider-abstract-class-to-lib-70ff513221f8a871.yaml
View File

@ -1,6 +0,0 @@
---
features:
- The cred_provider abstract class which serves as the basis for both
of tempest's cred providers, pre-provisioned credentials and dynamic
credentials, is now a library interface. This provides the common signature
required for building a credential provider.

5
releasenotes/notes/14/14.0.0-add-cred_client-to-tempest.lib-4d4af33f969c576f.yaml
View File

@ -1,5 +0,0 @@
---
features:
- The cred_client module was added to tempest.lib. This module provides a
wrapper to the keystone services client which provides a uniform
interface that abstracts out the differences between keystone api versions.

6
releasenotes/notes/14/14.0.0-add-error-code-translation-to-versions-clients-acbc78292e24b014.yaml
View File

@ -1,6 +0,0 @@
---
upgrade:
- Add an error translation to list_versions() of versions_client of both
compute and network. This can affect users who are expecting that these
clients return error status code instead of the exception. It is needed
to change the code for handling the exception like the other clients code.

9
releasenotes/notes/14/14.0.0-add-image-clients-af94564fb34ddca6.yaml
View File

@ -1,9 +0,0 @@
---
features:
- |
As in the [doc]:
http://developer.openstack.org/api-ref/image/v2/metadefs-index.html,
there are some apis are not included, add them.
* namespace_properties_client(v2)

6
releasenotes/notes/14/14.0.0-add-role-assignments-client-as-a-library-d34b4fdf376984ad.yaml
View File

@ -1,6 +0,0 @@
---
features:
- Define the identity service role_assignments_client as a library.
Add role_assignments_client to the library interface so the other
projects can use this module as a stable library without any
maintenance changes.

4
releasenotes/notes/14/14.0.0-add-service-provider-client-cbba77d424a30dd3.yaml
View File

@ -1,4 +0,0 @@
---
features:
- A Neutron Service Providers client is added to deal with resources
of the '/service-providers' route.

4
releasenotes/notes/14/14.0.0-add-ssh-port-parameter-to-client-6d16c374ac4456c1.yaml
View File

@ -1,4 +0,0 @@
---
features:
- A new optional parameter `port` for ssh client (`tempest.lib.common.ssh.Client`)
to specify destination port for a host. The default value is 22.

7
releasenotes/notes/14/14.0.0-deprecate-nova-api-extensions-df16b02485dae203.yaml
View File

@ -1,7 +0,0 @@
---
deprecations:
- The *api_extensions* config option in the *compute-feature-enabled* group is
now deprecated. This option will be removed from tempest when all the
OpenStack releases supported by tempest no longer support the API extensions
mechanism. This was removed from Nova during the Newton cycle, so this will
be removed at the Mitaka EOL.

7
releasenotes/notes/14/14.0.0-move-cinder-v3-to-lib-service-be3ba0c20753b594.yaml
View File

@ -1,7 +0,0 @@
features:
- |
Define the Volume v3 service clients as library interfaces,
allowing other projects to use these modules as stable
libraries without maintenance changes.
* messages_client(v3)

3
releasenotes/notes/14/14.0.0-new-volume-limit-client-517c17d9090f4df4.yaml
View File

@ -1,3 +0,0 @@
---
features:
- The volume_limits client was added to tempest.lib.

13
releasenotes/notes/14/14.0.0-remo-stress-tests-81052b211ad95d2e.yaml
View File

@ -1,13 +0,0 @@
---
prelude: >
This release is marking the end of Liberty release support in Tempest
upgrade:
- The Stress tests framework and all the stress tests have been removed.
other:
- |
OpenStack releases supported at this time are **Mitaka** and **Newton**.
The release under current development as of this tag is Ocata, meaning that
every Tempest commit is also tested against master during the Ocata cycle.
However, this does not necessarily mean that using Tempest as of this tag
will work against a Ocata (or future releases) cloud.

4
releasenotes/notes/14/14.0.0-remove-baremetal-tests-65186d9e15d5b8fb.yaml
View File

@ -1,4 +0,0 @@
---
upgrade:
- All tests for the Ironic project have been removed from Tempest. Those
exist as a Tempest plugin in the Ironic project.

6
releasenotes/notes/14/14.0.0-remove-bootable-option-024f8944c056a3e0.yaml
View File

@ -1,6 +0,0 @@
---
deprecations:
- The *bootable* config option in the *volume_feature_enabled* group is
removed because the corresponding feature os-set_bootable has been
implemented 2.5 years ago and all OpenStack versions which are supported
by Tempest should support the feature.

4
releasenotes/notes/14/14.0.0-remove-negative-test-generator-1653f4c0f86ccf75.yaml
View File

@ -1,4 +0,0 @@
---
upgrade:
- The Negative Tests Generator has been removed (it was not used by any
Tempest tests).

4
releasenotes/notes/14/14.0.0-remove-sahara-tests-1532c47c7df80e3a.yaml
View File

@ -1,4 +0,0 @@
---
upgrade:
- All tests for the Sahara project have been removed from Tempest. They now
live as a Tempest plugin in the ``openstack/sahara-tests`` repository.

12
releasenotes/notes/14/14.0.0-volume-clients-as-library-309030c7a16e62ab.yaml
View File

@ -1,12 +0,0 @@
---
features:
- |
Define volume service clients as libraries.
The following volume service clients are defined as library interface,
so the other projects can use these modules as stable libraries without
any maintenance changes.
* volumes_client(v1)
* volumes_client(v2)
* capabilities_client(v2)
* scheduler_stats_client(v2)

Some files were not shown because too many files have changed in this diff Show More