History

Sean Dague 1937d09fb0 rename tests -> api Now that all the other tests are moved out of the tests directory we can rename tests -> api to reflect that these tests are api testing, and need to use only the internal clients. Clean up references from other parts of OpenStack to the new api namespace. Reorder the imports as required with this naming change. Added README.rst Change-Id: I19203957f917b59e7c8a3838c590937752461a2f		2013-05-20 17:20:54 -04:00
..
api	rename tests -> api	2013-05-20 17:20:54 -04:00
cli	move cli directory into new tree structure	2013-05-13 11:28:11 -04:00
common	Add a sleep with back-off to retries	2013-05-16 14:00:39 +01:00
openstack	Migrate to pbr from openstack.common.setup.	2013-05-01 14:04:27 -04:00
scenario	rename tests -> api	2013-05-20 17:20:54 -04:00
services	Merge "Enhance the validation of the quotas update"	2013-05-17 16:16:15 +00:00
stress	Simplified stress tests.	2013-05-08 10:21:47 -04:00
thirdparty	move boto tests to thirdparty directory - part 1	2013-05-17 08:52:25 -04:00
whitebox	rename tests -> api	2013-05-20 17:20:54 -04:00
README.rst	move boto tests to thirdparty directory - part 1	2013-05-17 08:52:25 -04:00
__init__.py	Changes the namespace from storm to tempest, as well as adding addition tests and improvements	2011-12-06 16:48:03 -06:00
clients.py	Add tests list the services to compute	2013-05-16 12:14:23 +09:00
config.py	Merge "Adding new test for iSCSI live block migration"	2013-05-18 18:54:11 +00:00
exceptions.py	A Heat client which does basic stack operations.	2013-05-15 12:55:38 +12:00
manager.py	add scenario directory	2013-05-17 08:52:22 -04:00
test.py	Smoke attribute implies gate attribute	2013-05-19 15:59:00 +09:30

README.rst

Tempest Field Guide -----------

Tempest is designed to be useful for a large number of different environments. This includes being useful for gating commits to OpenStack core projects, being used to validate OpenStack cloud implementations for both correctness, as well as a burn in tool for OpenStack clouds.

As such Tempest tests come in many flavors, each with their own rules and guidelines. Below is the proposed Havana restructuring for Tempest to make this clear.

tempest/: api/ - API tests cli/ - CLI tests scenario/ - complex scenario tests stress/ - stress tests thirdparty/ - 3rd party api tests whitebox/ - white box testing

Each of these directories contains different types of tests. What belongs in each directory, the rules and examples for good tests, are documented in a README.rst file in the directory.

api

API tests are validation tests for the OpenStack API. They should not use the existing python clients for OpenStack, but should instead use the tempest implementations of clients. This allows us to test both XML and JSON. Having raw clients also lets us pass invalid JSON and XML to the APIs and see the results, something we could not get with the native clients.

When it makes sense, API testing should be moved closer to the projects themselves, possibly as functional tests in their unit test frameworks.

TODO: The bulk of tempest/tests should move to tempest/api

cli

CLI tests use the openstack CLI to interact with the OpenStack cloud. CLI testing in unit tests is somewhat difficult because unlike server testing, there is no access to server code to instantiate. Tempest seems like a logical place for this, as it prereqs having a running OpenStack cloud.

TODO: the top level cli directory moves to tempest/cli

scenario

Scenario tests are complex "through path" tests for OpenStack functionality. They are typically a series of steps where complicated state requiring multiple services is set up exercised, and torn down.

Scenario tests can and should use the OpenStack python clients.

TODO: tests/network/test_network_basic_ops.py, tests/compute/servers/*_ops.py should move to tempest/scenario (others)

stress

Stress tests are designed to stress an OpenStack environment by running a high workload against it and seeing what breaks. Tools may be provided to help detect breaks (stack traces in the logs).

TODO: old stress tests deleted, new_stress that david is working on moves into here.

thirdparty

Many openstack components include 3rdparty API support. It is completely legitmate for Tempest to include tests of 3rdparty APIs, but those should be kept seperate from the normal OpenStack validation.

TODO: tempest/tests/boto should become tempest/3rdparty/boto

whitebox

Whitebox tests are tests which require access to the database of the target OpenStack machine to verify internal state after opperations are made. White box tests are allowed to use the python clients.

TODO: collect out whitebox tests to this location.