openinfra/interop
Code Issues Proposed changes
9d71066870
interop/working_materials/interop_test_spec.rst
Mark T. Voelker d8fb682a2b Change doc references to DefCore 
At the Board of Directors meeting on April 24, 2016 the Board of
Directors indicated that as DefCore has evolved it's focus on
interoperability and it's working structure, it's name may be a source
of some confusion to those outside of the community or who are new to
the community.  The Board made an informal request that the DefCore
Committee consider changing it's name to more clearly reflect it's focus
and structure.

This patch is the first step in that process.  It updates references
in various documents to change the name "DefCore Committee" to
"Interop Working Group" as agreed at the summer 2016 DefCore Committee
Sprint [1].

It should be noted that this patch should be considered a work in
progress to generate discussion until the new name is approved by
the DefCore Committee and the Board of Directors.  Should we elect
to go forward with the new name, some other actions will also need
to be taken, including but not limited to:

1. We will need to consider updating references on the OpenStack wiki.
2. We will need to consider updating the name of our IRC channel,
   mailing list, Launchpad project, and git repository.  Most of these
   changes will need to be carefully coordinated with the OpenStack
   Infrastructure team.
3. We will need to take into account external resources that point to
   DefCore artifacts, such as Foundation-maintained websites (such as:
   http://www.openstack.org/interop ).
4. We will need to coordinate with RefStack to minimize impact.
5. We will need to clearly communicate the name change to the rest of the
   community.

Note also that I've intentionally left many historical documents that
have been superceded (such as the 2015A process docs, Guidelines that
are no longer used, etc) in tact.  There seemed little value in
spending time on them and cluttering the patch with them since they're
now obsolete.

[1] https://etherpad.openstack.org/p/DefCoreSummer2016Sprint

Change-Id: I79d337c193e75c54d49f1d847468f6347e2ef2b3
2017-01-23 21:25:05 -05:00

3.2 KiB
Raw Blame History

Feature Description

This test specification describes interoperability testing as it is understood by the Interop Working Group. An important goal of interoperability is that end users of OpenStack should be able to expect certain APIs to be reliable across clouds.

The spec aims to define how interoperability should be tested for OpenStack. It provides guidelines for evaluating current tests and the need for new test cases.

This document is a declaration of intent and looks at where interoperability testing is heading, not just where it is today.

What is tested

The focus of this test specification is functional and behavioral testing of the OpenStack REST API of the OpenStack projects.

What is not tested

  • OpenStack clients
  • Internal APIs
  • Python APIs
  • Performance testing

Interoperability Tests

Goals
  1. The only requirement to run interop functional tests should be an endpoint and credentials necessary to run the tests. Tests should ideally discover the methods it needs to use to set up whatever is necessary to exercise the API under test. For example: a test for image-delete needs to have an image in the cloud to delete. Ideally, the test should discover what method it needs to use to put an image in the cloud (image upload, image import, etc). Tests must favor direct API calls where they can. Discoverability for multiple implementations remains an open issue.
  2. The test should favor minimization. Test only the capability under test with as little set up as possible, using as few other resources as possible. For example, tests that don't actually need multiple user credentials or admin privileges shouldn't require them.
Guidelines
  1. Interop API functional tests should target and validate one capability. Some capabilities may require discovery calls and these should be included also in the test. Each capability should have at least one interop API functional test.
  2. Interop API functional tests should not use non-required capabilities.
  3. Interop scenario tests may use several capabilities to test end to end functionality and none of the capabilities used may be non-required capabilities.
  4. Tests must be reliable; race conditions must be avoided; tests should not be time dependent. If any of these problems occur the test will be flagged and either fixed or removed from the guidelines.
  5. Tests must be able to run consistently across all the releases covered by the Guidelines which are considered valid by the OpenStack Foundation for the OpenStack Powered program.
  6. Tests should not require more credentials than required to test the capability itself.
  7. Tests must not be harmful to the end user's environment, that is no changing credentials nor destroying user's data nor altering any other user resource other than those specifically created as part of the test.
  8. Test cases must setup any data they need to run and clean up after themselves. It is not acceptable to corrupt the user's environment or leave any test data or test configuration behind.