governance/resolutions/20160504-defcore-test-location.rst

2016-05-04 Recommendation on Location of Tests for DefCore

At the Newton summit discussion about QA, DefCore, and interoperability testing the DefCore committee asked for direction about where they should look for tests to consider for verifying capabilities that are selected as criteria for interoperability testing. Two options were discussed.

1. Use the tests within the Tempest git repository by themselves.
2. Add to those Tempest tests by allowing projects to host tests in their tree using Tempest's plugin feature.

Many projects do want to host their own functional tests, and the QA team is understandably hesitant to host all functional and integration tests for all projects. However, there are several reasons for maintaining a centralized set of the tests we use for trademark enforcement.

First, as interop tests are adopted for DefCore, an additional set of review requirements for changes becomes especially important. For example, API and behavioral changes must be carefully managed, as must mundane aspects such as test and module naming and location within the test suite. Even changes that leave tests functionally equivalent may cause unexpected consequences for their use in DefCore processes and validation. Placing the tests in a central repository will make it easier to maintain consistency and avoid breaking the trademark enforcement tool.

Centralizing the tests also makes it easier for anyone running the validation tool against their cloud or cloud distribution to use the tests. It is easier to install the test suite and its dependencies, and it is easier to read and understand a set of tests following a consistent implementation pattern. Finally, having the tests in a central location makes it easier to ensure that all members of the community have equal input into what the tests do and how they are implemented and maintained.

For all of these reasons, the OpenStack community will benefit from having the interoperability tests used by DefCore in a central location. The TC therefore encourages the DefCore committee to consider it an indication of future technical direction that we do not want tests outside of the Tempest repository used for trademark enforcement, and that any new or existing tests that cover capabilities they want to consider for trademark enforcement should be placed in Tempest.

Project teams should work with the DefCore committee to move any existing tests that need to move as a result of this policy. Project team contributors should also be prepared to work with the QA and DefCore teams on reviews and updates to tests that cover capabilties needed for DefCore, but where the test implementation is not suitable. Using a central repository for interoperability tests does not remove the responsibility for helping to maintain them.

Adopting this policy may also require the QA team to expand the scope of what they consider suitable for Tempest to allow tests that cover projects that the DefCore committee wants to have available for trademark status. Only tests related to capabilities would need to be moved into Tempest, and the DefCore committee, QA team, and project teams should collaborate to ensure that those tests are identified and implemented in a suitable way.