DefCore test specification proposal.

Change-Id: Icd318f3583950375264fc91a925071a74f73d661

working_materials/DefCoreSpec.rst

@@ -0,0 +1,79 @@
+======================
+Feature Description
+======================
+
+This test specification describes interoperability testing as it is understood
+by the DefCore Committee. An important goal of interoperability is that end users
+of OpenStack should be able to expect certain APIs to be reliable across clouds.
+
+The spec aims to define how interoperability should be tested for OpenStack.
+It provides guidelines for evaluating current tests and the need for new test
+cases.
+
+This document is a declaration of intent and looks at where interoperability
+testing is heading, not just where it is today.
+
+What is tested
+==============
+
+The focus of this test specification is functional and behavioral testing
+of the OpenStack REST API of the OpenStack projects.
+
+What is not tested
+==================
+
+- OpenStack clients
+- Internal APIs
+- Python APIs
+- Performance testing
+
+Interoperability Tests
+======================
+
+Goals
+ #. The only requirement to run interop functional tests should be an endpoint
+    and credentials necessary to run the tests. Tests should ideally discover
+    the methods it needs to use to set up whatever is necessary to exercise the
+    API under test. For example: a test for image-delete needs to have an image
+    in the cloud to delete.  Ideally, the test should discover what method it needs
+    to use to put an image in the cloud (image upload, image import, etc). Tests
+    must favor direct API calls where they can. Discoverability for multiple
+    implementations remains an open issue.
+
+ #. The test should favor minimization. Test only the capability under
+    test with as little set up as possible, using as few other resources
+    as possible. For example, tests that don't actually need multiple user
+    credentials or admin privileges shouldn't require them.
+
+Guidelines
+ #. Interop API functional tests should target and validate one
+    capability. Some capabilities may require discovery calls
+    and these should be included also in the test. Each capability
+    should have at least one interop API functional test.
+
+ #. Interop API functional tests should not use non-required
+    capabilities.
+
+ #. Interop scenario tests may use several capabilities to test end
+    to end functionality and none of the capabilities used may be
+    non-required capabilities.
+
+ #. Tests must be reliable; race conditions must be avoided; tests
+    should not be time dependent. If any of these problems occur the
+    test will be flagged and either fixed or removed from the guidelines.
+
+ #. Tests must be able to run consistently across all the releases
+    covered by the Guidelines which are considered valid by the OpenStack
+    Foundation for the OpenStack Powered program.
+
+ #. Tests should not require more credentials than required to test the
+    capability itself.
+
+ #. Tests must not be harmful to the end user's environment, that is no
+    changing credentials nor destroying user's data nor altering any
+    other user resource other than those specifically created as part
+    of the test.
+
+ #. Test cases must setup any data they need to run and clean up after
+    themselves. It is not acceptable to corrupt the user's environment
+    or leave any test data or test configuration behind.