Improved docs for os-testr commands
This commit adds real docs for using the 3 currently packaged commands in the os-testr project. As part of this change it also updates the sphinx config to generate proper man pages for all the tooling. At a future stage we need to ensure that these get installed properly. Change-Id: I0055c7961203a094590f6cfeb136a2236a2f65cc
This commit is contained in:
parent
16330c0526
commit
2f29ca51be
@ -21,6 +21,14 @@ Features
|
|||||||
Release Notes
|
Release Notes
|
||||||
=============
|
=============
|
||||||
|
|
||||||
|
0.2.0
|
||||||
|
-----
|
||||||
|
* Adds support for comments in a blacklist file and printing comments for each
|
||||||
|
exclude
|
||||||
|
* Several bugfixes for subunit-trace
|
||||||
|
* Switched subunit-trace behavior to disable printing percent change
|
||||||
|
in run time by default and make it optional
|
||||||
|
|
||||||
0.1.0
|
0.1.0
|
||||||
-----
|
-----
|
||||||
* First release which includes: ostestr, subunit-trace, and subunit2html
|
* First release which includes: ostestr, subunit-trace, and subunit2html
|
||||||
|
@ -38,7 +38,7 @@ master_doc = 'index'
|
|||||||
|
|
||||||
# General information about the project.
|
# General information about the project.
|
||||||
project = u'os-testr'
|
project = u'os-testr'
|
||||||
copyright = u'2013, OpenStack Foundation'
|
copyright = u'2015, Matthew Treinish'
|
||||||
|
|
||||||
# If true, '()' will be appended to :func: etc. cross-reference text.
|
# If true, '()' will be appended to :func: etc. cross-reference text.
|
||||||
add_function_parentheses = True
|
add_function_parentheses = True
|
||||||
@ -68,8 +68,15 @@ latex_documents = [
|
|||||||
('index',
|
('index',
|
||||||
'%s.tex' % project,
|
'%s.tex' % project,
|
||||||
u'%s Documentation' % project,
|
u'%s Documentation' % project,
|
||||||
u'OpenStack Foundation', 'manual'),
|
u'Matthew Treinish', 'manual'),
|
||||||
]
|
]
|
||||||
|
|
||||||
|
man_pages = [('ostestr', 'ostestr', 'tooling to run OpenStack tests',
|
||||||
|
['Matthew Treinish'], 1),
|
||||||
|
('subunit_trace', 'subunit-trace', 'pretty output filter for '
|
||||||
|
'subunit streams', ['Matthew Treinish'], 1),
|
||||||
|
('subunit2html', 'subunit2html', 'generate a html results page '
|
||||||
|
'from a subunit stream', ['Matthew Treinish'], 1)]
|
||||||
|
|
||||||
# Example configuration for intersphinx: refer to the Python standard library.
|
# Example configuration for intersphinx: refer to the Python standard library.
|
||||||
# intersphinx_mapping = {'http://docs.python.org/': None}
|
# intersphinx_mapping = {'http://docs.python.org/': None}
|
||||||
|
217
doc/source/ostestr.rst
Normal file
217
doc/source/ostestr.rst
Normal file
@ -0,0 +1,217 @@
|
|||||||
|
.. _ostestr:
|
||||||
|
|
||||||
|
ostestr
|
||||||
|
=======
|
||||||
|
|
||||||
|
The ostestr command provides a wrapper around the testr command included in
|
||||||
|
the testrepository package. It's designed to build on the functionality
|
||||||
|
included in testr and workaround several UI bugs in the short term. By default
|
||||||
|
it also has output that is much more useful for OpenStack's test suites which
|
||||||
|
are lengthy in both runtime and number of tests. Please note that the CLI
|
||||||
|
semantics are still a work in progress as the project is quite young, so
|
||||||
|
default behavior might change in future version.
|
||||||
|
|
||||||
|
Summary
|
||||||
|
-------
|
||||||
|
ostestr [-b|--blacklist_file <blacklist_file>] [-r|--regex REGEX]
|
||||||
|
[-p|--pretty] [--no-pretty] [-s|--subunit] [-l|--list]
|
||||||
|
[-n|--no-discover <test_id>] [--slowest] [--no-slowest]
|
||||||
|
[--pdb <test_id>] [--parallel] [--serial]
|
||||||
|
[-c|--concurrency <workers>] [--until-failure] [--print-exclude]
|
||||||
|
|
||||||
|
Options
|
||||||
|
-------
|
||||||
|
|
||||||
|
--blacklist_file BLACKLIST_FILE, -b BLACKLIST_FILE
|
||||||
|
Path to a blacklist file, this file contains a
|
||||||
|
separate regex exclude on each newline
|
||||||
|
--regex REGEX, -r REGEX
|
||||||
|
A normal testr selection regex. If a blacklist file is
|
||||||
|
specified, the regex will be appended to the end of
|
||||||
|
the generated regex from that file
|
||||||
|
--pretty, -p
|
||||||
|
Print pretty output from subunit-trace. This is
|
||||||
|
mutually exclusive with --subunit
|
||||||
|
--no-pretty
|
||||||
|
Disable the pretty output with subunit-trace
|
||||||
|
--subunit, -s
|
||||||
|
output the raw subunit v2 from the test run this is
|
||||||
|
mutuall exclusive with --pretty
|
||||||
|
--list, -l
|
||||||
|
List all the tests which will be run.
|
||||||
|
--no-discover TEST_ID, -n TEST_ID
|
||||||
|
Takes in a single test to bypasses test discover and
|
||||||
|
just excute the test specified
|
||||||
|
--slowest
|
||||||
|
After the test run print the slowest tests
|
||||||
|
--no-slowest
|
||||||
|
After the test run don't print the slowest tests
|
||||||
|
--pdb TEST_ID
|
||||||
|
Run a single test that has pdb traces added
|
||||||
|
--parallel
|
||||||
|
Run tests in parallel (this is the default)
|
||||||
|
--serial
|
||||||
|
Run tests serially
|
||||||
|
--concurrency WORKERS, -c WORKERS
|
||||||
|
The number of workers to use when running in parallel.
|
||||||
|
By default this is the number of cpus
|
||||||
|
--until-failure
|
||||||
|
Run the tests in a loop until a failure is
|
||||||
|
encountered. Running with subunit or prettyoutput
|
||||||
|
enable will force the loop to run testsserially
|
||||||
|
--print-exclude
|
||||||
|
If an exclude file is used this option will prints the
|
||||||
|
comment from the same line and all skipped tests
|
||||||
|
before the test run
|
||||||
|
|
||||||
|
Running Tests
|
||||||
|
-------------
|
||||||
|
|
||||||
|
os-testr is primarily for running tests at it's basic level you just invoke
|
||||||
|
ostestr to run a test suite for a project. (assuming it's setup to run tests
|
||||||
|
using testr already) For example::
|
||||||
|
|
||||||
|
$ ostestr
|
||||||
|
|
||||||
|
This will run tests in parallel (with the number of workers matching the number
|
||||||
|
of CPUs) and with subunit-trace output. If you need to run tests in serial you
|
||||||
|
can use the serial option::
|
||||||
|
|
||||||
|
$ ostestr --serial
|
||||||
|
|
||||||
|
Or if you need to adjust the concurrency but still run in parallel you can use
|
||||||
|
-c/--concurrency::
|
||||||
|
|
||||||
|
$ ostestr --concurrency 2
|
||||||
|
|
||||||
|
If you only want to run an individual test module or more specific (a single
|
||||||
|
class, or test) and parallel execution doesn't matter, you can use the
|
||||||
|
-n/--no-discover to skip test discovery and just directly calls subunit.run on
|
||||||
|
the tests under the covers. Bypassing discovery is desirable when running a
|
||||||
|
small subset of tests in a larger test suite because the discovery time can
|
||||||
|
often far exceed the total run time of the tests.
|
||||||
|
|
||||||
|
For example::
|
||||||
|
|
||||||
|
$ ostestr --no-discover test.test_thing.TestThing.test_thing_method
|
||||||
|
|
||||||
|
Additionally, if you need to run a single test module, class, or single test
|
||||||
|
with pdb enabled you can use --pdb to directly call testtools.run under the
|
||||||
|
covers which works with pdb. For example::
|
||||||
|
|
||||||
|
$ ostestr --pdb tests.test_thing.TestThing.test_thing_method
|
||||||
|
|
||||||
|
|
||||||
|
Test Selection
|
||||||
|
--------------
|
||||||
|
|
||||||
|
ostestr is designed to build on top of the test selection in testr. testr only
|
||||||
|
exposed a regex option to select tests. This equivalent is functionality is
|
||||||
|
exposed via the --regex option. For example::
|
||||||
|
|
||||||
|
$ ostestr --regex 'magic\.regex'
|
||||||
|
|
||||||
|
This will do a straight passthrough of the provided regex to testr.
|
||||||
|
Additionally, ostestr allows you to specify a a blacklist file to define a set
|
||||||
|
of regexes to exclude. You can specify a blacklist file with the
|
||||||
|
--blacklist-file/-b option, for example::
|
||||||
|
|
||||||
|
$ ostestr --blacklist_file $path_to_file
|
||||||
|
|
||||||
|
The format for the file is line separated regex, with '#' used to signify the
|
||||||
|
start of a comment on a line. For example::
|
||||||
|
|
||||||
|
# Blacklist File
|
||||||
|
^regex1 # Excludes these tests
|
||||||
|
.*regex2 # exclude those tests
|
||||||
|
|
||||||
|
Will generate a regex to pass to testr which will exclude both any tests
|
||||||
|
matching '^regex1' and '.*regex2'. If a blacklist file is used in conjunction
|
||||||
|
with the --regex option the regex specified with --regex will be appended to
|
||||||
|
the generated output from the --blacklist_file. Also it's worth noting that the
|
||||||
|
regex test selection options can not be used in conjunction with the
|
||||||
|
--no-discover or --pdb options described in the previous section. This is
|
||||||
|
because the regex selection requires using testr under the covers to actually
|
||||||
|
do the filtering, and those 2 options do not use testr.
|
||||||
|
|
||||||
|
It's also worth noting that you can use the test list option to dry run any
|
||||||
|
selection arguments you are using. You just need to use --list/-l with your
|
||||||
|
selection options to do this, for example::
|
||||||
|
|
||||||
|
$ ostestr --regex 'regex3.*' --blacklist_file blacklist.txt --list
|
||||||
|
|
||||||
|
This will list all the tests which will be run by ostestr using that combination
|
||||||
|
of arguments.
|
||||||
|
|
||||||
|
Please not that all of this selection functionality will be expanded on in the
|
||||||
|
future and a default grammar for selecting multiple tests will be chosen in a
|
||||||
|
future release. However as of right now all current arguments (which have
|
||||||
|
guarantees on always remaining in place) are still required to perform any
|
||||||
|
selection logic while this functionality is still under development.
|
||||||
|
|
||||||
|
|
||||||
|
Output Options
|
||||||
|
--------------
|
||||||
|
|
||||||
|
By default ostestr will use subunit-trace as the output filter on the test
|
||||||
|
run. It will also print the slowest tests from the run after the run is
|
||||||
|
concluded. You can disable the printing the slowest tests with the --no-slowest
|
||||||
|
flag, for example::
|
||||||
|
|
||||||
|
$ ostestr --no-slowest
|
||||||
|
|
||||||
|
If you'd like to disable the subunit-trace output you can do this using
|
||||||
|
--no-pretty::
|
||||||
|
|
||||||
|
$ ostestr --no-pretty
|
||||||
|
|
||||||
|
ostestr also provides the option to just output the raw subunit stream on
|
||||||
|
STDOUT with --subunit/-s. Note if you want to use this you also have to
|
||||||
|
specify --no-pretty as the subunit-trace output and the raw subunit output
|
||||||
|
are mutually exclusive. For example, to get raw subunit output the arguments
|
||||||
|
would be::
|
||||||
|
|
||||||
|
$ ostestr --no-pretty --subunit
|
||||||
|
|
||||||
|
An additional option on top of the blacklist file is --print-exclude option.
|
||||||
|
When this option is specified when using a blacklist file before the tests are
|
||||||
|
run ostestr will print all the tests it will be excluding from the blacklist
|
||||||
|
file. If a line in the blacklist file has a comment that will be printed before
|
||||||
|
listing the tests which will be excluded by that line's regex. If no comment is
|
||||||
|
present on a line the regex from that line will be used instead. For example,
|
||||||
|
if you were using the example blacklist file from the previous section the
|
||||||
|
output before the regular test run output would be::
|
||||||
|
|
||||||
|
$ ostestr -b blacklist-file blacklist.txt --print-exclude
|
||||||
|
Excludes these tests
|
||||||
|
regex1_match
|
||||||
|
regex1_exclude
|
||||||
|
|
||||||
|
exclude those tests
|
||||||
|
regex2_match
|
||||||
|
regex2_exclude
|
||||||
|
|
||||||
|
...
|
||||||
|
|
||||||
|
Notes for running with tox
|
||||||
|
--------------------------
|
||||||
|
|
||||||
|
If you use `tox`_ for running your tests and call ostestr as the test command
|
||||||
|
.. _tox: https://tox.readthedocs.org/en/latest/
|
||||||
|
it's recommended that you set a posargs following ostestr on the commands
|
||||||
|
stanza. For example::
|
||||||
|
|
||||||
|
[testenv]
|
||||||
|
commands = ostestr {posargs}
|
||||||
|
|
||||||
|
this will enable end users to pass args to configure the output, use the
|
||||||
|
selection logic, or any other options directly from the tox cli. This will let
|
||||||
|
tox take care of the venv management and the environment separation but enable
|
||||||
|
direct access to all of the ostestr options to easily customize your test run.
|
||||||
|
For example, assuming the above posargs usage you would be to do::
|
||||||
|
|
||||||
|
$ tox -epy34 -- --regex ^regex1
|
||||||
|
|
||||||
|
or to skip discovery::
|
||||||
|
|
||||||
|
$ tox -epy34 -- -n test.test_thing.TestThing.test_thing_method
|
33
doc/source/subunit2html.rst
Normal file
33
doc/source/subunit2html.rst
Normal file
@ -0,0 +1,33 @@
|
|||||||
|
.. _subunit2html:
|
||||||
|
|
||||||
|
subunit2html
|
||||||
|
============
|
||||||
|
|
||||||
|
subunit2html is a tool that takes in a subunit stream file and will output an
|
||||||
|
html page
|
||||||
|
|
||||||
|
Summary
|
||||||
|
-------
|
||||||
|
|
||||||
|
subunit2html subunit_stream [output]
|
||||||
|
|
||||||
|
Usage
|
||||||
|
-----
|
||||||
|
|
||||||
|
subunit2html takes in 1 mandatory argument. This is used to specify the location
|
||||||
|
of the subunit stream file. For example::
|
||||||
|
|
||||||
|
$ subunit2html subunit_stream
|
||||||
|
|
||||||
|
By default subunit2html will store the generated html results file at
|
||||||
|
results.html file in the current working directory.
|
||||||
|
|
||||||
|
An optional second argument can be provided to set the output path of the html
|
||||||
|
results file that is generated. If it is provided this will be the output path
|
||||||
|
for saving the generated file, otherwise results.html in the current working
|
||||||
|
directory will be used. For example::
|
||||||
|
|
||||||
|
$ subunit2html subunit_stream test_results.html
|
||||||
|
|
||||||
|
will write the generated html results file to test_results.html in the current
|
||||||
|
working directory
|
110
doc/source/subunit_trace.rst
Normal file
110
doc/source/subunit_trace.rst
Normal file
@ -0,0 +1,110 @@
|
|||||||
|
.. _subunit_trace:
|
||||||
|
|
||||||
|
subunit-trace
|
||||||
|
=============
|
||||||
|
|
||||||
|
subunit-trace is an output filter for subunit streams. It is often used in
|
||||||
|
conjunction with test runners that emit subunit to enable a consistent and
|
||||||
|
useful realtime output from a test run.
|
||||||
|
|
||||||
|
Summary
|
||||||
|
-------
|
||||||
|
|
||||||
|
subunit-trace [--fails|-f] [--failonly] [--perc-diff|-d] [--no-summary]
|
||||||
|
[--diff-threshold|-t <threshold>]
|
||||||
|
|
||||||
|
Options
|
||||||
|
-------
|
||||||
|
|
||||||
|
--no-failure-debug, -n
|
||||||
|
Disable printing failure debug information in realtime
|
||||||
|
--fails, -f
|
||||||
|
Print failure debug information after the stream is
|
||||||
|
proccesed
|
||||||
|
--failonly
|
||||||
|
Don't print success items
|
||||||
|
--perc-diff, -d
|
||||||
|
Print percent change in run time on each test
|
||||||
|
--diff-threshold THRESHOLD, -t THRESHOLD
|
||||||
|
Threshold to use for displaying percent change from the
|
||||||
|
avg run time. If one is not specified the percent
|
||||||
|
change will always be displayed.
|
||||||
|
--no-summary
|
||||||
|
Don't print the summary of the test run after completes
|
||||||
|
|
||||||
|
Usage
|
||||||
|
-----
|
||||||
|
subunit-trace will take a subunit stream in via STDIN. This is the only input
|
||||||
|
into the tool. It will then print on STDOUT the formatted test result output
|
||||||
|
for the test run information contained in the stream.
|
||||||
|
|
||||||
|
A subunit v2 stream must be passed into subunit-trace. If only a subunit v1
|
||||||
|
stream is available you must use the subunit-1to2 utility to convert it before
|
||||||
|
passing the stream into subunit-trace. For example this can be done by chaining
|
||||||
|
pipes::
|
||||||
|
|
||||||
|
$ cat subunit_v1 | subunit-1to2 | subunit-trace
|
||||||
|
|
||||||
|
Adjusting per test output
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
subunit-trace provides several options to customize it's output. This allows
|
||||||
|
users to customize the output from subunit-trace to suit their needs. The output
|
||||||
|
from subunit-trace basically comes in 2 parts, the per test output, and the
|
||||||
|
summary at the end. By default subunit-trace will print failure messages during
|
||||||
|
the per test output, meaning when a test fails it will also print the message
|
||||||
|
and any traceback and other attachments at that time. However this can be
|
||||||
|
disabled by using --no-failure-debug, -n. For example::
|
||||||
|
|
||||||
|
$ testr run --subunit | subunit-trace --no-failure-debug
|
||||||
|
|
||||||
|
Rhere is also the option to print all failures together at the end of the test
|
||||||
|
run before the summary view. This is done using the --fails/-f option. For
|
||||||
|
example::
|
||||||
|
|
||||||
|
$ testr run --subunit | subunit-trace --fails
|
||||||
|
|
||||||
|
Often the --fails and --no-failure-debug options are used in conjunction to
|
||||||
|
only print failures at the end of a test run. This is useful for large test
|
||||||
|
suites where an error message might be lost in the noise. To do this ::
|
||||||
|
|
||||||
|
$ testr run --subunit | subunit-trace --fails --no-failure-debug
|
||||||
|
|
||||||
|
By default subunit-trace will print a line for each test after it completes with
|
||||||
|
the test status. However, if you only want to see the run time output for
|
||||||
|
failures and not any other test status you can use the --failonly option. For
|
||||||
|
example::
|
||||||
|
|
||||||
|
$ testr run --subunit | subunit-trace --failonly
|
||||||
|
|
||||||
|
The last output option provided by subunit-trace is to diable the summary view
|
||||||
|
of the test run which is normally displayed at the end of a run. You can do
|
||||||
|
this using the --no-summary option. For example::
|
||||||
|
|
||||||
|
$ testr run --subunit | subunit-trace --no-summary
|
||||||
|
|
||||||
|
|
||||||
|
Show per test run time percent change
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
subunit-trace provides an option to display the percent change in run time
|
||||||
|
from the previous run. To do this subunit-trace leverages the testr internals
|
||||||
|
a bit. It uses the times.dbm database which, the file repository type in
|
||||||
|
testrepository will create, to get the previous run time for a test. If testr
|
||||||
|
hasn't ever been used before or for whatever reason subunit-trace is unable to
|
||||||
|
find the times.dbm file from testr no percentages will be displayed even if it's
|
||||||
|
enabled. Additionally, if a test is run which does not have an entry in the
|
||||||
|
times.dbm file will not have a percentage printed for it.
|
||||||
|
|
||||||
|
To enable this feature you use --perc-diff/-d, for example::
|
||||||
|
|
||||||
|
$ testr run --subunit | subunit-trace --perc-diff
|
||||||
|
|
||||||
|
There is also the option to set a threshold value for this option. If used it
|
||||||
|
acts as an absolute value and only percentage changes that exceed it will be
|
||||||
|
printed. Use the --diff-threshold/-t option to set a threshold, for example::
|
||||||
|
|
||||||
|
$ testr run --subunit | subunit-trace --perc-diff --threshold 45
|
||||||
|
|
||||||
|
This will only display percent differences when the change in run time is either
|
||||||
|
>=45% faster or <=45% slower.
|
@ -1,7 +1,12 @@
|
|||||||
========
|
=====
|
||||||
Usage
|
Usage
|
||||||
========
|
=====
|
||||||
|
|
||||||
To use os-testr in a project::
|
This section contains the documentation for each of tools packaged in os-testr
|
||||||
|
|
||||||
import os_testr
|
.. toctree::
|
||||||
|
:maxdepth: 2
|
||||||
|
|
||||||
|
ostestr
|
||||||
|
subunit_trace
|
||||||
|
subunit2html
|
||||||
|
Loading…
Reference in New Issue
Block a user