225 lines
7.7 KiB
Plaintext
225 lines
7.7 KiB
Plaintext
|
|
subunit: A streaming protocol for test results
|
|
Copyright (C) 2005 Robert Collins <robertc@robertcollins.net>
|
|
|
|
This program is free software; you can redistribute it and/or modify
|
|
it under the terms of the GNU General Public License as published by
|
|
the Free Software Foundation; either version 2 of the License, or
|
|
(at your option) any later version.
|
|
|
|
This program is distributed in the hope that it will be useful,
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
GNU General Public License for more details.
|
|
|
|
You should have received a copy of the GNU General Public License
|
|
along with this program; if not, write to the Free Software
|
|
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
|
|
|
|
|
|
Subunit
|
|
-------
|
|
|
|
Subunit is a streaming protocol for test results. The protocol is human
|
|
readable and easily generated and parsed. By design all the components of
|
|
the protocol conceptually fit into the xUnit TestCase->TestResult interaction.
|
|
|
|
Subunit comes with command line filters to process a subunit stream and
|
|
language bindings for python, C, C++ and shell. Bindings are easy to write
|
|
for other languages.
|
|
|
|
A number of useful things can be done easily with subunit:
|
|
* Test aggregation: Tests run separately can be combined and then
|
|
reported/displayed together. For instance, tests from different languages
|
|
can be shown as a seamless whole.
|
|
* Test archiving: A test run may be recorded and replayed later.
|
|
* Test isolation: Tests that may crash or otherwise interact badly with each
|
|
other can be run seperately and then aggregated, rather than interfering
|
|
with each other.
|
|
* Grid testing: subunit can act as the necessary serialisation and
|
|
deserialiation to get test runs on distributed machines to be reported in
|
|
real time.
|
|
|
|
Subunit supplies the following filters:
|
|
* tap2subunit - convert perl's TestAnythingProtocol to subunit.
|
|
* subunit2pyunit - convert a subunit stream to pyunit test results.
|
|
* subunit-filter - filter out tests from a subunit stream.
|
|
* subunit-ls - list the tests present in a subunit stream.
|
|
* subunit-stats - generate a summary of a subunit stream.
|
|
* subunit-tags - add or remove tags from a stream.
|
|
|
|
Integration with other tools
|
|
----------------------------
|
|
|
|
Subunit's language bindings act as integration with various test runners like
|
|
'check', 'cppunit', Python's 'unittest'. Beyond that a small amount of glue
|
|
(typically a few lines) will allow Subunit to be used in more sophisticated
|
|
ways.
|
|
|
|
Python
|
|
======
|
|
|
|
As a TestResult, Subunit can translate method calls from a test run into a
|
|
Subunit stream::
|
|
|
|
# Get a TestSuite or TestCase to run
|
|
suite = make_suite()
|
|
# Create a stream (any object with a 'write' method)
|
|
stream = file('tests.log', 'wb')
|
|
# Create a subunit result object which will output to the stream
|
|
result = subunit.TestProtocolClient(stream)
|
|
# Run the test suite reporting to the subunit result object
|
|
suite.run(result)
|
|
# Close the stream.
|
|
stream.close()
|
|
|
|
As a TestCase, subunit can read from a stream and inform a TestResult
|
|
of the activity from the stream::
|
|
|
|
# Get a stream (any object with a readline() method), in this case example the
|
|
# stream output by the example before.
|
|
stream = file('tests.log', 'rb')
|
|
# Create a subunit ProtocolTestCase which will read from the stream and emit
|
|
# activity to a result when run() is called.
|
|
suite = subunit.ProtocolTestCase(stream)
|
|
# Create a result object to show the contents of the stream.
|
|
result = unittest._TextTestResult(sys.stdout)
|
|
# 'run' the tests - process the stream and feed its contents to result.
|
|
suite.run(result)
|
|
stream.close()
|
|
|
|
Subunit has support for non-blocking usage too, for use with asyncore or
|
|
Twisted. See the TestProtocolServer class for more details.
|
|
|
|
Building on these foundations, Subunit also offers some convenience tools.
|
|
|
|
The ``IsolatedTestSuite`` class is a decorator that will fork() before running
|
|
the decorated item, and gather the results from the child process via subunit.
|
|
This is useful for handlings tests that mutate global state, or are testing C
|
|
extensions that could crash the VM.
|
|
|
|
Similarly, ``IsolatedTestCase`` is a base class which can be subclassed to get
|
|
tests that will fork() before the test is run.
|
|
|
|
Finally, ``ExecTestCase`` is a convenience wrapper for running an external
|
|
program to get a subunit stream and then report that back to an arbitrary
|
|
result object::
|
|
|
|
class AggregateTests(subunit.ExecTestCase):
|
|
|
|
def test_script_one(self):
|
|
"""./bin/script_one"""
|
|
|
|
def test_script_two(self):
|
|
"""./bin/script_two"""
|
|
|
|
# Normally your normal test loading would take of this automatically,
|
|
# It is only spelt out in detail here for clarity.
|
|
suite = unittest.TestSuite([AggregateTests("test_script_one"),
|
|
AggregateTests("test_script_two")])
|
|
# Create any TestResult class you like.
|
|
result = unittest._TextTestResult(sys.stdout)
|
|
# And run your suite as normal, subunit will exec each external script as
|
|
# needed and report to your result object.
|
|
suite.run(result)
|
|
|
|
C
|
|
=
|
|
|
|
Subunit has C bindings to emit the protocol, and comes with a patch for 'check'
|
|
which has been nominally accepted by the 'check' developers. See 'c/README' for
|
|
more details.
|
|
|
|
C++
|
|
===
|
|
|
|
C++ uses the C bindings and includes a patch for cppunit. See 'c++/README' for
|
|
details.
|
|
|
|
shell
|
|
=====
|
|
|
|
Similar to C, the shell bindings consist of simple functions to output protocol
|
|
elements, and a patch for adding subunit output to the 'ShUnit' shell test
|
|
runner. See 'shell/README' for details.
|
|
|
|
|
|
The protocol
|
|
------------
|
|
|
|
Sample subunit wire contents
|
|
----------------------------
|
|
|
|
The following::
|
|
test: test foo works
|
|
success: test foo works.
|
|
test: tar a file.
|
|
failure: tar a file. [
|
|
..
|
|
].. space is eaten.
|
|
foo.c:34 WARNING foo is not defined.
|
|
]
|
|
a writeln to stdout
|
|
|
|
When run through subunit2pyunit::
|
|
.F
|
|
a writeln to stdout
|
|
|
|
========================
|
|
FAILURE: tar a file.
|
|
-------------------
|
|
..
|
|
].. space is eaten.
|
|
foo.c:34 WARNING foo is not defined.
|
|
|
|
|
|
Subunit protocol description
|
|
============================
|
|
test|testing|test:|testing: test label
|
|
success|success:|successful|successful: test label
|
|
success|success:|successful|successful: test label [
|
|
...
|
|
]
|
|
failure test label
|
|
failure: test label
|
|
failure test label [
|
|
...
|
|
]
|
|
failure: test label [
|
|
...
|
|
]
|
|
error: test label
|
|
error: test label [
|
|
]
|
|
skip[:] test label
|
|
skip[:] test label [
|
|
]
|
|
xfail[:] test label
|
|
xfail[:] test label [
|
|
]
|
|
tags: [-]TAG ...
|
|
time: YYYY-MM-DD HH:MM:SSZ
|
|
unexpected output on stdout -> stdout.
|
|
exit w/0 or last test -> error
|
|
Tags given outside a test are applied to all following tests
|
|
Tags given after a test: line and before the result line for the same test
|
|
apply only to that test, and inheric the current global tags.
|
|
A '-' before a tag is used to remove tags - e.g. to prevent a global tag
|
|
applying to a single test, or to cancel a global tag.
|
|
In Python, tags are assigned to the .tags attribute on the RemoteTest objects
|
|
created by the TestProtocolServer.
|
|
|
|
The time element acts as a clock event - it sets the time for all future events.
|
|
Currently this is not exposed at the python API layer.
|
|
|
|
The skip result is used to indicate a test that was found by the runner but not
|
|
fully executed due to some policy or dependency issue. This is represented in
|
|
python using the addSkip interface that testtools
|
|
(https://edge.launchpad.net/testtools) defines. When communicating with a non
|
|
skip aware test result, the test is reported as an error.
|
|
The xfail result is used to indicate a test that was expected to fail failing
|
|
in the expected manner. As this is a normal condition for such tests it is
|
|
represented as a successful test in Python.
|
|
In future, skip and xfail results will be represented semantically in Python,
|
|
but some discussion is underway on the right way to do this.
|