Adds a wip decorator for tests

This decorator can be used to commit failing tests while they are still
in development. It can also be used by people to show how an issue can
be reproduced without them having to make the code changes necessary to
make the test pass.

This is nicer than just raising TestSkipped because there is a built in
reminder to remove the decorator when the test starts passing.

Implements: blueprint failing-tests
Change-Id: I9ded266b368e7955b1e295950df394823b1a4088

doc/source/developing.rst

@@ -390,6 +390,44 @@ file ``keystone/tests/config_files/backend_liveldap.conf`` and
     to a non-negative value.
 
 
+"Work in progress" Tests
+========================
+
+Work in progress (WIP) tests are very useful in a variety of situations
+including:
+
+* During a TDD process they can be used to add tests to a review while
+  they are not yet working and will not cause test failures. (They should
+  be removed before the final merge.)
+* Often bug reports include small snippets of code to show broken
+  behaviors. Some of these can be converted into WIP tests that can later
+  be worked on by a developer. This allows us to take code that can be
+  used to catch bug regressions and commit it before any code is
+  written.
+
+The ``keystone.tests.util.wip`` decorator can be used to mark a test as
+WIP. A WIP test will always be run. If the test fails then a TestSkipped
+exception is raised because we expect the test to fail. We do not pass
+the test in this case so that it doesn't count toward the number of
+successfully run tests. If the test passes an AssertionError exception is
+raised so that the developer knows they made the test pass. This is a
+reminder to remove the decorator.
+
+The ``wip`` decorator requires that the author provides a message. This
+message is important because it will tell other developers why this test
+is marked as a work in progress. Reviewers will require that these
+messages are descriptive and accurate.
+
+.. NOTE::
+    The ``wip`` decorator is not a replacement for skipping tests.
+
+.. code-block:: python
+
+    @wip('waiting on bug #000000')
+    def test():
+        pass
+
+
 Generating Updated Sample Config File
 -------------------------------------
 

keystone/tests/unit/tests/test_utils.py

@@ -0,0 +1,37 @@
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+from testtools import matchers
+from testtools import testcase
+
+from keystone.tests import utils
+
+
+class TestWipDecorator(testcase.TestCase):
+
+    def test_raises_SkipError_when_broken_test_fails(self):
+
+        @utils.wip('waiting on bug #000000')
+        def test():
+            raise Exception('i expected a failure - this is a WIP')
+
+        e = self.assertRaises(testcase.TestSkipped, test)
+        self.assertThat(str(e), matchers.Contains('#000000'))
+
+    def test_raises_AssertionError_when_test_passes(self):
+
+        @utils.wip('waiting on bug #000000')
+        def test():
+            pass  # literally
+
+        e = self.assertRaises(AssertionError, test)
+        self.assertThat(str(e), matchers.Contains('#000000'))

keystone/tests/utils.py

@@ -14,6 +14,9 @@
 
 import uuid
 
+import six
+from testtools import testcase
+
 from keystone.common import environment
 from keystone.openstack.common import log
 
@@ -65,3 +68,40 @@ def check_output(*popenargs, **kwargs):
 
 def git(*args):
     return check_output(['git'] + list(args))
+
+
+def wip(message):
+    """Mark a test as work in progress.
+
+    Based on code by Nat Pryce:
+    https://gist.github.com/npryce/997195#file-wip-py
+
+    The test will always be run. If the test fails then a TestSkipped
+    exception is raised. If the test passes an AssertionError exception
+    is raised so that the developer knows they made the test pass. This
+    is a reminder to remove the decorator.
+
+    :param message: a string message to help clarify why the test is
+                    marked as a work in progress
+
+    usage:
+      >>> @wip('waiting on bug #000000')
+      >>> def test():
+      >>>     pass
+
+    """
+
+    def _wip(f):
+        @six.wraps(f)
+        def run_test(*args, **kwargs):
+            try:
+                f(*args, **kwargs)
+            except Exception:
+                raise testcase.TestSkipped('work in progress test failed: ' +
+                                           message)
+
+            raise AssertionError('work in progress test passed: ' + message)
+
+        return run_test
+
+    return _wip