diff --git a/README.rst b/README.rst index ebfd8f7..87dc74c 100644 --- a/README.rst +++ b/README.rst @@ -21,6 +21,14 @@ Features Release Notes ============= +0.2.0 +----- + * Adds support for comments in a blacklist file and printing comments for each + exclude + * Several bugfixes for subunit-trace + * Switched subunit-trace behavior to disable printing percent change + in run time by default and make it optional + 0.1.0 ----- * First release which includes: ostestr, subunit-trace, and subunit2html diff --git a/doc/source/conf.py b/doc/source/conf.py index 251d672..45a8546 100755 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -22,7 +22,7 @@ sys.path.insert(0, os.path.abspath('../..')) # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = [ 'sphinx.ext.autodoc', - #'sphinx.ext.intersphinx', + # 'sphinx.ext.intersphinx', 'oslosphinx' ] @@ -38,7 +38,7 @@ master_doc = 'index' # General information about the project. project = u'os-testr' -copyright = u'2013, OpenStack Foundation' +copyright = u'2015, Matthew Treinish' # If true, '()' will be appended to :func: etc. cross-reference text. add_function_parentheses = True @@ -68,8 +68,15 @@ latex_documents = [ ('index', '%s.tex' % project, u'%s Documentation' % project, - u'OpenStack Foundation', 'manual'), + u'Matthew Treinish', 'manual'), ] +man_pages = [('ostestr', 'ostestr', 'tooling to run OpenStack tests', + ['Matthew Treinish'], 1), + ('subunit_trace', 'subunit-trace', 'pretty output filter for ' + 'subunit streams', ['Matthew Treinish'], 1), + ('subunit2html', 'subunit2html', 'generate a html results page ' + 'from a subunit stream', ['Matthew Treinish'], 1)] + # Example configuration for intersphinx: refer to the Python standard library. -#intersphinx_mapping = {'http://docs.python.org/': None} +# intersphinx_mapping = {'http://docs.python.org/': None} diff --git a/doc/source/ostestr.rst b/doc/source/ostestr.rst new file mode 100644 index 0000000..6c36bac --- /dev/null +++ b/doc/source/ostestr.rst @@ -0,0 +1,217 @@ +.. _ostestr: + +ostestr +======= + +The ostestr command provides a wrapper around the testr command included in +the testrepository package. It's designed to build on the functionality +included in testr and workaround several UI bugs in the short term. By default +it also has output that is much more useful for OpenStack's test suites which +are lengthy in both runtime and number of tests. Please note that the CLI +semantics are still a work in progress as the project is quite young, so +default behavior might change in future version. + +Summary +------- + ostestr [-b|--blacklist_file ] [-r|--regex REGEX] + [-p|--pretty] [--no-pretty] [-s|--subunit] [-l|--list] + [-n|--no-discover ] [--slowest] [--no-slowest] + [--pdb ] [--parallel] [--serial] + [-c|--concurrency ] [--until-failure] [--print-exclude] + +Options +------- + + --blacklist_file BLACKLIST_FILE, -b BLACKLIST_FILE + Path to a blacklist file, this file contains a + separate regex exclude on each newline + --regex REGEX, -r REGEX + A normal testr selection regex. If a blacklist file is + specified, the regex will be appended to the end of + the generated regex from that file + --pretty, -p + Print pretty output from subunit-trace. This is + mutually exclusive with --subunit + --no-pretty + Disable the pretty output with subunit-trace + --subunit, -s + output the raw subunit v2 from the test run this is + mutuall exclusive with --pretty + --list, -l + List all the tests which will be run. + --no-discover TEST_ID, -n TEST_ID + Takes in a single test to bypasses test discover and + just excute the test specified + --slowest + After the test run print the slowest tests + --no-slowest + After the test run don't print the slowest tests + --pdb TEST_ID + Run a single test that has pdb traces added + --parallel + Run tests in parallel (this is the default) + --serial + Run tests serially + --concurrency WORKERS, -c WORKERS + The number of workers to use when running in parallel. + By default this is the number of cpus + --until-failure + Run the tests in a loop until a failure is + encountered. Running with subunit or prettyoutput + enable will force the loop to run testsserially + --print-exclude + If an exclude file is used this option will prints the + comment from the same line and all skipped tests + before the test run + +Running Tests +------------- + +os-testr is primarily for running tests at it's basic level you just invoke +ostestr to run a test suite for a project. (assuming it's setup to run tests +using testr already) For example:: + + $ ostestr + +This will run tests in parallel (with the number of workers matching the number +of CPUs) and with subunit-trace output. If you need to run tests in serial you +can use the serial option:: + + $ ostestr --serial + +Or if you need to adjust the concurrency but still run in parallel you can use +-c/--concurrency:: + + $ ostestr --concurrency 2 + +If you only want to run an individual test module or more specific (a single +class, or test) and parallel execution doesn't matter, you can use the +-n/--no-discover to skip test discovery and just directly calls subunit.run on +the tests under the covers. Bypassing discovery is desirable when running a +small subset of tests in a larger test suite because the discovery time can +often far exceed the total run time of the tests. + +For example:: + + $ ostestr --no-discover test.test_thing.TestThing.test_thing_method + +Additionally, if you need to run a single test module, class, or single test +with pdb enabled you can use --pdb to directly call testtools.run under the +covers which works with pdb. For example:: + + $ ostestr --pdb tests.test_thing.TestThing.test_thing_method + + +Test Selection +-------------- + +ostestr is designed to build on top of the test selection in testr. testr only +exposed a regex option to select tests. This equivalent is functionality is +exposed via the --regex option. For example:: + + $ ostestr --regex 'magic\.regex' + +This will do a straight passthrough of the provided regex to testr. +Additionally, ostestr allows you to specify a a blacklist file to define a set +of regexes to exclude. You can specify a blacklist file with the +--blacklist-file/-b option, for example:: + + $ ostestr --blacklist_file $path_to_file + +The format for the file is line separated regex, with '#' used to signify the +start of a comment on a line. For example:: + + # Blacklist File + ^regex1 # Excludes these tests + .*regex2 # exclude those tests + +Will generate a regex to pass to testr which will exclude both any tests +matching '^regex1' and '.*regex2'. If a blacklist file is used in conjunction +with the --regex option the regex specified with --regex will be appended to +the generated output from the --blacklist_file. Also it's worth noting that the +regex test selection options can not be used in conjunction with the +--no-discover or --pdb options described in the previous section. This is +because the regex selection requires using testr under the covers to actually +do the filtering, and those 2 options do not use testr. + +It's also worth noting that you can use the test list option to dry run any +selection arguments you are using. You just need to use --list/-l with your +selection options to do this, for example:: + + $ ostestr --regex 'regex3.*' --blacklist_file blacklist.txt --list + +This will list all the tests which will be run by ostestr using that combination +of arguments. + +Please not that all of this selection functionality will be expanded on in the +future and a default grammar for selecting multiple tests will be chosen in a +future release. However as of right now all current arguments (which have +guarantees on always remaining in place) are still required to perform any +selection logic while this functionality is still under development. + + +Output Options +-------------- + +By default ostestr will use subunit-trace as the output filter on the test +run. It will also print the slowest tests from the run after the run is +concluded. You can disable the printing the slowest tests with the --no-slowest +flag, for example:: + + $ ostestr --no-slowest + +If you'd like to disable the subunit-trace output you can do this using +--no-pretty:: + + $ ostestr --no-pretty + +ostestr also provides the option to just output the raw subunit stream on +STDOUT with --subunit/-s. Note if you want to use this you also have to +specify --no-pretty as the subunit-trace output and the raw subunit output +are mutually exclusive. For example, to get raw subunit output the arguments +would be:: + + $ ostestr --no-pretty --subunit + +An additional option on top of the blacklist file is --print-exclude option. +When this option is specified when using a blacklist file before the tests are +run ostestr will print all the tests it will be excluding from the blacklist +file. If a line in the blacklist file has a comment that will be printed before +listing the tests which will be excluded by that line's regex. If no comment is +present on a line the regex from that line will be used instead. For example, +if you were using the example blacklist file from the previous section the +output before the regular test run output would be:: + + $ ostestr -b blacklist-file blacklist.txt --print-exclude + Excludes these tests + regex1_match + regex1_exclude + + exclude those tests + regex2_match + regex2_exclude + + ... + +Notes for running with tox +-------------------------- + +If you use `tox`_ for running your tests and call ostestr as the test command +.. _tox: https://tox.readthedocs.org/en/latest/ +it's recommended that you set a posargs following ostestr on the commands + stanza. For example:: + + [testenv] + commands = ostestr {posargs} + +this will enable end users to pass args to configure the output, use the +selection logic, or any other options directly from the tox cli. This will let +tox take care of the venv management and the environment separation but enable +direct access to all of the ostestr options to easily customize your test run. +For example, assuming the above posargs usage you would be to do:: + + $ tox -epy34 -- --regex ^regex1 + +or to skip discovery:: + + $ tox -epy34 -- -n test.test_thing.TestThing.test_thing_method diff --git a/doc/source/subunit2html.rst b/doc/source/subunit2html.rst new file mode 100644 index 0000000..f0ebbc9 --- /dev/null +++ b/doc/source/subunit2html.rst @@ -0,0 +1,33 @@ +.. _subunit2html: + +subunit2html +============ + +subunit2html is a tool that takes in a subunit stream file and will output an +html page + +Summary +------- + + subunit2html subunit_stream [output] + +Usage +----- + +subunit2html takes in 1 mandatory argument. This is used to specify the location +of the subunit stream file. For example:: + + $ subunit2html subunit_stream + +By default subunit2html will store the generated html results file at +results.html file in the current working directory. + +An optional second argument can be provided to set the output path of the html +results file that is generated. If it is provided this will be the output path +for saving the generated file, otherwise results.html in the current working +directory will be used. For example:: + + $ subunit2html subunit_stream test_results.html + +will write the generated html results file to test_results.html in the current +working directory diff --git a/doc/source/subunit_trace.rst b/doc/source/subunit_trace.rst new file mode 100644 index 0000000..1338fe6 --- /dev/null +++ b/doc/source/subunit_trace.rst @@ -0,0 +1,110 @@ +.. _subunit_trace: + +subunit-trace +============= + +subunit-trace is an output filter for subunit streams. It is often used in +conjunction with test runners that emit subunit to enable a consistent and +useful realtime output from a test run. + +Summary +------- + +subunit-trace [--fails|-f] [--failonly] [--perc-diff|-d] [--no-summary] + [--diff-threshold|-t ] + +Options +------- + + --no-failure-debug, -n + Disable printing failure debug information in realtime + --fails, -f + Print failure debug information after the stream is + proccesed + --failonly + Don't print success items + --perc-diff, -d + Print percent change in run time on each test + --diff-threshold THRESHOLD, -t THRESHOLD + Threshold to use for displaying percent change from the + avg run time. If one is not specified the percent + change will always be displayed. + --no-summary + Don't print the summary of the test run after completes + +Usage +----- +subunit-trace will take a subunit stream in via STDIN. This is the only input +into the tool. It will then print on STDOUT the formatted test result output +for the test run information contained in the stream. + +A subunit v2 stream must be passed into subunit-trace. If only a subunit v1 +stream is available you must use the subunit-1to2 utility to convert it before +passing the stream into subunit-trace. For example this can be done by chaining +pipes:: + + $ cat subunit_v1 | subunit-1to2 | subunit-trace + +Adjusting per test output +^^^^^^^^^^^^^^^^^^^^^^^^^ + +subunit-trace provides several options to customize it's output. This allows +users to customize the output from subunit-trace to suit their needs. The output +from subunit-trace basically comes in 2 parts, the per test output, and the +summary at the end. By default subunit-trace will print failure messages during +the per test output, meaning when a test fails it will also print the message +and any traceback and other attachments at that time. However this can be +disabled by using --no-failure-debug, -n. For example:: + + $ testr run --subunit | subunit-trace --no-failure-debug + +Rhere is also the option to print all failures together at the end of the test +run before the summary view. This is done using the --fails/-f option. For +example:: + + $ testr run --subunit | subunit-trace --fails + +Often the --fails and --no-failure-debug options are used in conjunction to +only print failures at the end of a test run. This is useful for large test +suites where an error message might be lost in the noise. To do this :: + + $ testr run --subunit | subunit-trace --fails --no-failure-debug + +By default subunit-trace will print a line for each test after it completes with +the test status. However, if you only want to see the run time output for +failures and not any other test status you can use the --failonly option. For +example:: + + $ testr run --subunit | subunit-trace --failonly + +The last output option provided by subunit-trace is to diable the summary view +of the test run which is normally displayed at the end of a run. You can do +this using the --no-summary option. For example:: + + $ testr run --subunit | subunit-trace --no-summary + + +Show per test run time percent change +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +subunit-trace provides an option to display the percent change in run time +from the previous run. To do this subunit-trace leverages the testr internals +a bit. It uses the times.dbm database which, the file repository type in +testrepository will create, to get the previous run time for a test. If testr +hasn't ever been used before or for whatever reason subunit-trace is unable to +find the times.dbm file from testr no percentages will be displayed even if it's +enabled. Additionally, if a test is run which does not have an entry in the +times.dbm file will not have a percentage printed for it. + +To enable this feature you use --perc-diff/-d, for example:: + + $ testr run --subunit | subunit-trace --perc-diff + +There is also the option to set a threshold value for this option. If used it +acts as an absolute value and only percentage changes that exceed it will be +printed. Use the --diff-threshold/-t option to set a threshold, for example:: + + $ testr run --subunit | subunit-trace --perc-diff --threshold 45 + +This will only display percent differences when the change in run time is either +>=45% faster or <=45% slower. diff --git a/doc/source/usage.rst b/doc/source/usage.rst index a5b21da..b1ab10a 100644 --- a/doc/source/usage.rst +++ b/doc/source/usage.rst @@ -1,7 +1,12 @@ -======== +===== Usage -======== +===== -To use os-testr in a project:: +This section contains the documentation for each of tools packaged in os-testr - import os_testr +.. toctree:: + :maxdepth: 2 + + ostestr + subunit_trace + subunit2html