This change makes use of argparse groups to generate error
messages when the user specifies invalid command line option
combinations. For example if the user specified --pretty
and --no-pretty before it would not complain, but now it
will.
Change-Id: I07e1edb5c43ff7b0a81879f388770fb732fed43b
Add the capability to the --no-discover option to convert file
names into regular sexpressions. Also add the --path option that
converts a file or directory to a regular expression. Create a
mutually exclusive argparse group for --regex, --path, and
--no-discover.
This change will allow the user to specify a file name instead of a
regular expression to match a particular set of tests e.g:
tox -e py27 -- --path os_testr/tests/test_os_testr.py
Will run os_testr.tests.test_os_testr but you can use tab complete
to generate the name.
Change-Id: Ibfca2bc023aed44b1b87a0444559ab2a00303a70
If a subunit stream is aborted in the middle (like in the case of a
segfault) this causes subunit-trace to emit a stacktrace. This commit
attempts to handle this edge case gracefully.
Related-Bug: #1482230
Change-Id: I1a8a0a8e2ab65e637c6a5212e324670b7d95d28d
This commit adds a minimum version on the setuptools setup_requires
for pbr. This is required to enable markers and also to ensure a
consistent version when installing with other packages.
Change-Id: I2168d91896ba82581929a2faeebace7419b4539e
This commit adds a new entry to the os-testr TODO file to indicate the
long term goal for migrating away from using subprocess in ostestr.
Everything ostestr uses is a python project so we shouldn't need to
call out for everything, it was just faster to bootstrap the project
this way, but we should probably move away from doing this.
Change-Id: I3b2e54d72514ad92079a893fd7a835e6d446d4bc
This commit adds real docs for using the 3 currently packaged commands
in the os-testr project. As part of this change it also updates the
sphinx config to generate proper man pages for all the tooling. At a
future stage we need to ensure that these get installed properly.
Change-Id: I0055c7961203a094590f6cfeb136a2236a2f65cc
So it turns out we weren't actually using ostestr for running the
os-testr unit tests. We probably should use the test runner runner
wrapper we're developing to run the unit tests for the test runner
runner wrapper and the other utilities in the package.
Change-Id: I88af9104352163f2412c2a3cbaf6c88d0a937988
This commit disables printing the percent change in run time by
default. This was a cool experiment, but isn't really useful in
practice. Especially in the gate, things are so noisy and there
is so much variance looking at the change for a single doesn't
actually mean anything. It makes more sense to make this feature
opt-in because it's not generally useful.
Change-Id: Iecb153452edfe1d7b55757d022ae0331ac563b35
Python 3 renames StringIO -> io. Use six to deal
with this change.
Introduces new test dependency on six for StringIO.
Change-Id: Ia875b7fcbb976599053970ef79ed3f3474626bad
The database I am getting back is a gdm database and it does
not have a get method. Catch the exception and try something
else. If that blows up, ignore because we'd rather see the
results of our tests.
Change-Id: I2882e19d49f2fb3471669f5eb8a017c5d5ac98c2
When a subunit stream for a testcase doesn't contain start end
enddate, the duration can't be calculated which leads to a:
ValueError: could not convert string to float
Check now if the duration is an empty string and add basic test
coverage based on ddt for the subunit_trace command.
Change-Id: I9953019794ba53fcfcb20e32fecbe94da22c9565
This commit adds the support to have comments in the exclude file.
After this commit anything after a '#' will be treated as a comment
and ignored for matching. A new option --print-exclude is added which
will print out what is being skipped by the exclude file with any
comments in the file.
Change-Id: Ia357730f918e0a57cf2ac51cd8195e2721667511
This commit adds a TODO file to the project to indicate both short and
long term work items for the project.
Change-Id: I90c19330eb5d8216a88f40411ed883e6f64c750d
This commit fixes pep8 issues that were added when subunit2html was
copied into the repo. As part of this the subunit2html was missing a
license header for the additions made to it on top of the original
HTMLTestRunner script it was based on. The addition of the copyright
header mirrors project-config change:
I6fc16d316012fd3e1c196f74df25d725a310f6dc
Change-Id: Ieb42d5baddb3e1446fcd50197136f605324323aa
This commit adds some details the README about the included utils and
starts the Release Notes section in prepartion for pushing the first
release.
Change-Id: I45dd057bdfc6b5d8810b2f15fedabbecccdd7478
This commit adds another utility to os-testr, subunit2html, which is
used to generate html output from a subunit stream. This utility is
currently being used after OpenStack test jobs to generate the
testr_results.html page. The previous home for this file was in
the openstack-infra project config repo as a jenkins slave script.
This commit adds a missing option to calling subunit trace inside the
until-failure loop. Previously, when a test failed it would not print
the failure output which would make it impossible to debug why it had
failed. This fixes that oversight.
This commit adds an --until-failure option to ostestr which basically
calls through to --until-failure on testr. The key difference is
because of open issues with testr you can't use --until-failure with
subunit output enabled. This would break just a straight passthrough
if either pretty output or subunit output are enabled. This
works around this by manually reimplementing this functionality by
generating a test list and looping over it and running the tests with
the desired output forever until a failure is encountered. The tradeoff
here is that to do this the test operations are serialized.
As part of this to make the pretty output not overly verbose an option
is added to subunit-trace to disable the summary view from being
printed.
This commit adds some basic unit tests which runs ostestr in some
basic configurations against a fake test suite to ensure that the
ostestr always exits with a 0 on success and 1 on errors when running
tests. This is invaluable for using ostestr in ci systems.
This commit adds a percent change to the duration on individual test
output lines. This is conditionally displayed based on the presence
of a testrepository times.dbm file and data in that file for the test
being run. If there is useable data subunit-trace will now use the
runtimes from there to display how the current run has changed from
the dbm file. A new threshold option is added to optionally specify a
minimum percent change to be used to determine whether to display the
value or not.
Change-Id: I3d68425f48114531a78cab08c353111648ce3911
This commit switches from using subprocess to call testtools.run and
subunit.run with python -m to directly calling the methods being run.
This should make the non-default cases when using subunit.run and
testtools.run faster, and it simplifies the code. As part of this, the
code around call_subunit is fixed to make sure the it works as expected
given different argument combinations.
ostestr will still call subprocess to run testr, because the interface
is more complex, and when subunit.run is used with subunit-trace,
because the stdin handling is tricky. The subunit.run with subunit-trace
case will be handled in a later patch.
This commit adds options to ostestr to control the parallelism and
concurrency of testr. By default it will run in parallel with
concurrency equal to the number of cpus on the system. This commit
also modifies the metavar values on the no-discover and pdb options
to make them more descriptive.
The subprocess.call() to run testr init if the .testrepository dir
hasn't been created yet was setting passing the actual call assuming
shell=True, which it wasn't. This commit fixes this oversight to make
the call actually work.
This commit fixes a couple of issues from running flake8 for the first
time. Mostly minor whitespace fixes, but we'll need this for when we
move this repo over into openstack's ci system.
This commit adds a --pdb flag which is similar to --no-discover except
that instead of running subunit.run test_id to bypass test discovery,
testtools.run test_id is used so that pdb traces set in the code are
useable. As part of this several fixes in the --no-discover path were
added since the codes is similar.
This commit migrates the current subunit trace commit from tempest-lib
with the commits:
d7c3f6b Merge "Summarize expected failures"
e29ec71 Summarize expected failures
21e3f6a Enable stdout passthrough for subunit-trace
d588748 Default the worker number to 0 not NaN
87c1442 Fix subunit-trace on python < 2.7
b73b9eb bring over fail only functionality from nova
5715fd6 Switch to elapsed time in subunit-trace summary
d2e4040 Setup subunit-trace as an entry point
and also adds the start of the ostestr command to wrap testr.
