From 2b47841c4d563bd2cc2a380853e05b0447cf5dd4 Mon Sep 17 00:00:00 2001 From: Chris Hoge Date: Thu, 23 Jun 2016 16:03:28 -0700 Subject: [PATCH] Add guidance on negative tests in HACKING.rst Adds guidance on negative tests based on work items from the Newton design summit in Austin[1]. [1] https://etherpad.openstack.org/p/newton-qa-negative-testing Change-Id: Ic44b0e17540a5e984006a4a74c8318ad7e74db31 --- HACKING.rst | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/HACKING.rst b/HACKING.rst index 480650c296..432db7d03b 100644 --- a/HACKING.rst +++ b/HACKING.rst @@ -160,11 +160,17 @@ not there even if the cloud was configured with it. Negative Tests -------------- -TODO: Write the guideline related to negative tests. +Error handling is an important aspect of API design and usage. Negative +tests are a way to ensure that an application can gracefully handle +invalid or unexpected input. However, as a black box integration test +suite, Tempest is not suitable for handling all negative test cases, as +the wide variety and complexity of negative tests can lead to long test +runs and knowledge of internal implementation details. The bulk of +negative testing should be handled with project function tests. The +exception to this rule is API tests used for interoperability testing. Test skips because of Known Bugs -------------------------------- - If a test is broken because of a bug it is appropriate to skip the test until bug has been fixed. You should use the skip_because decorator so that Tempest's skip tracking tool can watch the bug status.