interop/working_materials/DefCoreSpec.rst

Feature Description

This test specification describes interoperability testing as it is understood by the DefCore Committee. An important goal of interoperability is that end users of OpenStack should be able to expect certain APIs to be reliable across clouds.

The spec aims to define how interoperability should be tested for OpenStack. It provides guidelines for evaluating current tests and the need for new test cases.

This document is a declaration of intent and looks at where interoperability testing is heading, not just where it is today.

What is tested

The focus of this test specification is functional and behavioral testing of the OpenStack REST API of the OpenStack projects.

What is not tested

• OpenStack clients
• Internal APIs
• Python APIs
• Performance testing

Interoperability Tests

Goals

1. The only requirement to run interop functional tests should be an endpoint and credentials necessary to run the tests. Tests should ideally discover the methods it needs to use to set up whatever is necessary to exercise the API under test. For example: a test for image-delete needs to have an image in the cloud to delete. Ideally, the test should discover what method it needs to use to put an image in the cloud (image upload, image import, etc). Tests must favor direct API calls where they can. Discoverability for multiple implementations remains an open issue.
2. The test should favor minimization. Test only the capability under test with as little set up as possible, using as few other resources as possible. For example, tests that don't actually need multiple user credentials or admin privileges shouldn't require them.

Guidelines

1. Interop API functional tests should target and validate one capability. Some capabilities may require discovery calls and these should be included also in the test. Each capability should have at least one interop API functional test.
2. Interop API functional tests should not use non-required capabilities.
3. Interop scenario tests may use several capabilities to test end to end functionality and none of the capabilities used may be non-required capabilities.
4. Tests must be reliable; race conditions must be avoided; tests should not be time dependent. If any of these problems occur the test will be flagged and either fixed or removed from the guidelines.
5. Tests must be able to run consistently across all the releases covered by the Guidelines which are considered valid by the OpenStack Foundation for the OpenStack Powered program.
6. Tests should not require more credentials than required to test the capability itself.
7. Tests must not be harmful to the end user's environment, that is no changing credentials nor destroying user's data nor altering any other user resource other than those specifically created as part of the test.
8. Test cases must setup any data they need to run and clean up after themselves. It is not acceptable to corrupt the user's environment or leave any test data or test configuration behind.