diff --git a/resolutions/20160504-defcore-test-location.rst b/resolutions/20160504-defcore-test-location.rst new file mode 100644 index 000000000..b77227ebf --- /dev/null +++ b/resolutions/20160504-defcore-test-location.rst @@ -0,0 +1,67 @@ +============================================================ + 2016-05-04 Recommendation on Location of Tests for DefCore +============================================================ + +At the Newton summit discussion about `QA, DefCore, and +interoperability testing`_ the DefCore committee asked for direction +about where they should look for tests to consider for verifying +capabilities that are selected as criteria for interoperability +testing. Two options were discussed. + +1. Use the tests within the Tempest git repository by themselves. +2. Add to those Tempest tests by allowing projects to host tests in + their tree using Tempest's plugin feature. + +Many projects do want to host their own functional tests, and the QA +team is understandably hesitant to host all functional and integration +tests for all projects. However, there are several reasons for +maintaining a centralized set of the tests we use for trademark +enforcement. + +First, as interop tests are adopted for DefCore, an additional set of +review requirements for changes becomes especially important. For +example, API and behavioral changes must be carefully managed, as must +mundane aspects such as test and module naming and location within the +test suite. Even changes that leave tests functionally equivalent may +cause unexpected consequences for their use in DefCore processes and +validation. Placing the tests in a central repository will make it +easier to maintain consistency and avoid breaking the trademark +enforcement tool. + +Centralizing the tests also makes it easier for anyone running the +validation tool against their cloud or cloud distribution to use the +tests. It is easier to install the test suite and its dependencies, +and it is easier to read and understand a set of tests following a +consistent implementation pattern. Finally, having the tests in a +central location makes it easier to ensure that all members of the +community have equal input into what the tests do and how they are +implemented and maintained. + +For all of these reasons, the OpenStack community will benefit from +having the interoperability tests used by DefCore in a central +location. The TC therefore encourages the DefCore committee to +consider it an indication of future technical direction that we do not +want tests outside of the `Tempest repository`_ used for trademark +enforcement, and that any new or existing tests that cover +capabilities they want to consider for trademark enforcement should be +placed in Tempest. + +Project teams should work with the DefCore committee to move any +existing tests that need to move as a result of this policy. Project +team contributors should also be prepared to work with the QA and +DefCore teams on reviews and updates to tests that cover capabilties +needed for DefCore, but where the test implementation is not +suitable. Using a central repository for interoperability tests does +not remove the responsibility for helping to maintain them. + +Adopting this policy may also require the QA team to expand the `scope +of what they consider suitable for Tempest`_ to allow tests that cover +projects that the DefCore committee wants to have available for +trademark status. Only tests related to capabilities would need to be +moved into Tempest, and the DefCore committee, QA team, and project +teams should collaborate to ensure that those tests are identified and +implemented in a suitable way. + +.. _QA, DefCore, and interoperability testing: https://etherpad.openstack.org/p/newton-qa-defcore-and-interoperability +.. _Tempest repository: http://git.openstack.org/cgit/openstack/tempest +.. _scope of what they consider suitable for Tempest: https://wiki.openstack.org/wiki/QA/Tempest-test-removal#Tempest_Scope