From 971446f5abbfd63f0188dd98db59ce9828b9936c Mon Sep 17 00:00:00 2001 From: Joe Gordon Date: Fri, 20 Dec 2013 10:40:13 -0800 Subject: [PATCH] Explicitly document requirements for 3rd party testing As we are moving towards requiring more and more third party testing, we should make sure we have a central and explicit list of requirements. These are the requirements to have a voting third party testing system, not the requirements of what the system should be running (to count a driver as fully tested). Change-Id: Ic1e1cdf9af9cd21e29a0e768d5d78ae4f84108fb --- doc/source/third_party.rst | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/doc/source/third_party.rst b/doc/source/third_party.rst index 69b45df55e..29c2db0a1c 100644 --- a/doc/source/third_party.rst +++ b/doc/source/third_party.rst @@ -14,6 +14,38 @@ Smokestack reads the Gerrit event stream and runs it's own tests on the commits. If one of the tests fails it will publish information and links to the failure on the review in Gerrit. +Requirements +------------ + +* Until a third party testing system operates in a stable fashion, third + party tests can comment on patches but not vote on them. + + * A system can also be set up to only do '+1' reviews and leave all the + '-1's to be manually confirmed. +* The maintainers are responsible for re-triggering tests when their third + party testing system breaks. +* Support recheck to request re-running a test. + + * Support the following syntaxes ``recheck no bug`` and ``recheck bug ###``. + * Recheck means recheck everything. A single recheck comment should + re-trigger all testing systems. +* Publish who the maintainers of the third party testing system are, and make + them available for support as needed. Maintainers are encouraged to be + in IRC regularly to make it faster to contact them. + + * All CI comments must contain a link to a contact page with the details. +* Include a public link to all test artifacts to make debugging failed tests + easier. This should include: + + * Environment details + * Test configuration + + * Skipped tests + * logs should include a trace of the commands used + * OpenStack logs + * Tempest logs (including ``testr_results.html.gz``) + + Reading the Event Stream ------------------------