This commit adds another utility to os-testr, subunit2html, which is
used to generate html output from a subunit stream. This utility is
currently being used after OpenStack test jobs to generate the
testr_results.html page. The previous home for this file was in
the openstack-infra project config repo as a jenkins slave script.
This commit adds a missing option to calling subunit trace inside the
until-failure loop. Previously, when a test failed it would not print
the failure output which would make it impossible to debug why it had
failed. This fixes that oversight.
This commit adds an --until-failure option to ostestr which basically
calls through to --until-failure on testr. The key difference is
because of open issues with testr you can't use --until-failure with
subunit output enabled. This would break just a straight passthrough
if either pretty output or subunit output are enabled. This
works around this by manually reimplementing this functionality by
generating a test list and looping over it and running the tests with
the desired output forever until a failure is encountered. The tradeoff
here is that to do this the test operations are serialized.
As part of this to make the pretty output not overly verbose an option
is added to subunit-trace to disable the summary view from being
printed.
This commit adds some basic unit tests which runs ostestr in some
basic configurations against a fake test suite to ensure that the
ostestr always exits with a 0 on success and 1 on errors when running
tests. This is invaluable for using ostestr in ci systems.
This commit adds a percent change to the duration on individual test
output lines. This is conditionally displayed based on the presence
of a testrepository times.dbm file and data in that file for the test
being run. If there is useable data subunit-trace will now use the
runtimes from there to display how the current run has changed from
the dbm file. A new threshold option is added to optionally specify a
minimum percent change to be used to determine whether to display the
value or not.
Change-Id: I3d68425f48114531a78cab08c353111648ce3911
This commit switches from using subprocess to call testtools.run and
subunit.run with python -m to directly calling the methods being run.
This should make the non-default cases when using subunit.run and
testtools.run faster, and it simplifies the code. As part of this, the
code around call_subunit is fixed to make sure the it works as expected
given different argument combinations.
ostestr will still call subprocess to run testr, because the interface
is more complex, and when subunit.run is used with subunit-trace,
because the stdin handling is tricky. The subunit.run with subunit-trace
case will be handled in a later patch.
This commit adds options to ostestr to control the parallelism and
concurrency of testr. By default it will run in parallel with
concurrency equal to the number of cpus on the system. This commit
also modifies the metavar values on the no-discover and pdb options
to make them more descriptive.
The subprocess.call() to run testr init if the .testrepository dir
hasn't been created yet was setting passing the actual call assuming
shell=True, which it wasn't. This commit fixes this oversight to make
the call actually work.
This commit fixes a couple of issues from running flake8 for the first
time. Mostly minor whitespace fixes, but we'll need this for when we
move this repo over into openstack's ci system.
This commit adds a --pdb flag which is similar to --no-discover except
that instead of running subunit.run test_id to bypass test discovery,
testtools.run test_id is used so that pdb traces set in the code are
useable. As part of this several fixes in the --no-discover path were
added since the codes is similar.
This commit migrates the current subunit trace commit from tempest-lib
with the commits:
d7c3f6b Merge "Summarize expected failures"
e29ec71 Summarize expected failures
21e3f6a Enable stdout passthrough for subunit-trace
d588748 Default the worker number to 0 not NaN
87c1442 Fix subunit-trace on python < 2.7
b73b9eb bring over fail only functionality from nova
5715fd6 Switch to elapsed time in subunit-trace summary
d2e4040 Setup subunit-trace as an entry point
and also adds the start of the ostestr command to wrap testr.
