Improved docs for os-testr commands

This commit adds real docs for using the 3 currently packaged commands
in the os-testr project. As part of this change it also updates the
sphinx config to generate proper man pages for all the tooling. At a
future stage we need to ensure that these get installed properly.

Change-Id: I0055c7961203a094590f6cfeb136a2236a2f65cc

README.rst

@@ -21,6 +21,14 @@ Features
 Release Notes
 =============
 
+0.2.0
+-----
+ * Adds support for comments in a blacklist file and printing comments for each
+   exclude
+ * Several bugfixes for subunit-trace
+ * Switched subunit-trace behavior to disable printing percent change
+   in run time by default and make it optional
+
 0.1.0
 -----
  * First release which includes: ostestr, subunit-trace, and subunit2html

doc/source/conf.py

@@ -22,7 +22,7 @@ sys.path.insert(0, os.path.abspath('../..'))
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = [
     'sphinx.ext.autodoc',
-    #'sphinx.ext.intersphinx',
+    # 'sphinx.ext.intersphinx',
     'oslosphinx'
 ]
 
@@ -38,7 +38,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'os-testr'
-copyright = u'2013, OpenStack Foundation'
+copyright = u'2015, Matthew Treinish'
 
 # If true, '()' will be appended to :func: etc. cross-reference text.
 add_function_parentheses = True
@@ -68,8 +68,15 @@ latex_documents = [
     ('index',
      '%s.tex' % project,
      u'%s Documentation' % project,
-     u'OpenStack Foundation', 'manual'),
+     u'Matthew Treinish', 'manual'),
 ]
 
+man_pages = [('ostestr', 'ostestr', 'tooling to run OpenStack tests',
+             ['Matthew Treinish'], 1),
+             ('subunit_trace', 'subunit-trace', 'pretty output filter for '
+              'subunit streams', ['Matthew Treinish'], 1),
+             ('subunit2html', 'subunit2html', 'generate a html results page '
+              'from a subunit stream', ['Matthew Treinish'], 1)]
+
 # Example configuration for intersphinx: refer to the Python standard library.
-#intersphinx_mapping = {'http://docs.python.org/': None}
+# intersphinx_mapping = {'http://docs.python.org/': None}

doc/source/ostestr.rst

@@ -0,0 +1,217 @@
+.. _ostestr:
+
+ostestr
+=======
+
+The ostestr command provides a wrapper around the testr command included in
+the testrepository package. It's designed to build on the functionality
+included in testr and workaround several UI bugs in the short term. By default
+it also has output that is much more useful for OpenStack's test suites which
+are lengthy in both runtime and number of tests. Please note that the CLI
+semantics are still a work in progress as the project is quite young, so
+default behavior might change in future version.
+
+Summary
+-------
+    ostestr [-b|--blacklist_file <blacklist_file>] [-r|--regex REGEX]
+            [-p|--pretty] [--no-pretty] [-s|--subunit] [-l|--list]
+            [-n|--no-discover <test_id>] [--slowest] [--no-slowest]
+            [--pdb <test_id>] [--parallel] [--serial]
+            [-c|--concurrency <workers>] [--until-failure] [--print-exclude]
+
+Options
+-------
+
+  --blacklist_file BLACKLIST_FILE, -b BLACKLIST_FILE
+                        Path to a blacklist file, this file contains a
+                        separate regex exclude on each newline
+  --regex REGEX, -r REGEX
+                        A normal testr selection regex. If a blacklist file is
+                        specified, the regex will be appended to the end of
+                        the generated regex from that file
+  --pretty, -p
+                        Print pretty output from subunit-trace. This is
+                        mutually exclusive with --subunit
+  --no-pretty
+                        Disable the pretty output with subunit-trace
+  --subunit, -s
+                        output the raw subunit v2 from the test run this is
+                        mutuall exclusive with --pretty
+  --list, -l
+                        List all the tests which will be run.
+  --no-discover TEST_ID, -n TEST_ID
+                        Takes in a single test to bypasses test discover and
+                        just excute the test specified
+  --slowest
+                        After the test run print the slowest tests
+  --no-slowest
+                        After the test run don't print the slowest tests
+  --pdb TEST_ID
+                        Run a single test that has pdb traces added
+  --parallel
+                        Run tests in parallel (this is the default)
+  --serial
+                        Run tests serially
+  --concurrency WORKERS, -c WORKERS
+                        The number of workers to use when running in parallel.
+                        By default this is the number of cpus
+  --until-failure
+                        Run the tests in a loop until a failure is
+                        encountered. Running with subunit or prettyoutput
+                        enable will force the loop to run testsserially
+  --print-exclude
+                        If an exclude file is used this option will prints the
+                        comment from the same line and all skipped tests
+                        before the test run
+
+Running Tests
+-------------
+
+os-testr is primarily for running tests at it's basic level you just invoke
+ostestr to run a test suite for a project. (assuming it's setup to run tests
+using testr already) For example::
+
+    $ ostestr
+
+This will run tests in parallel (with the number of workers matching the number
+of CPUs) and with subunit-trace output. If you need to run tests in serial you
+can use the serial option::
+
+    $ ostestr --serial
+
+Or if you need to adjust the concurrency but still run in parallel you can use
+-c/--concurrency::
+
+    $ ostestr --concurrency 2
+
+If you only want to run an individual test module or more specific (a single
+class, or test) and parallel execution doesn't matter, you can use the
+-n/--no-discover to skip test discovery and just directly calls subunit.run on
+the tests under the covers. Bypassing discovery is desirable when running a
+small subset of tests in a larger test suite because the discovery time can
+often far exceed the total run time of the tests.
+
+For example::
+
+    $ ostestr --no-discover test.test_thing.TestThing.test_thing_method
+
+Additionally, if you need to run a single test module, class, or single test
+with pdb enabled you can use --pdb to directly call testtools.run under the
+covers which works with pdb. For example::
+
+    $ ostestr --pdb tests.test_thing.TestThing.test_thing_method
+
+
+Test Selection
+--------------
+
+ostestr is designed to build on top of the test selection in testr. testr only
+exposed a regex option to select tests. This equivalent is functionality is
+exposed via the --regex option. For example::
+
+    $ ostestr --regex 'magic\.regex'
+
+This will do a straight passthrough of the provided regex to testr.
+Additionally, ostestr allows you to specify a a blacklist file to define a set
+of regexes to exclude. You can specify a blacklist file with the
+--blacklist-file/-b option, for example::
+
+    $ ostestr --blacklist_file $path_to_file
+
+The format for the file is line separated regex, with '#' used to signify the
+start of a comment on a line. For example::
+
+    # Blacklist File
+    ^regex1 # Excludes these tests
+    .*regex2 # exclude those tests
+
+Will generate a regex to pass to testr which will exclude both any tests
+matching '^regex1' and '.*regex2'. If a blacklist file is used in conjunction
+with the --regex option the regex specified with --regex will be appended to
+the generated output from the --blacklist_file. Also it's worth noting that the
+regex test selection options can not be used in conjunction with the
+--no-discover or --pdb options described in the previous section. This is
+because the regex selection requires using testr under the covers to actually
+do the filtering, and those 2 options do not use testr.
+
+It's also worth noting that you can use the test list option to dry run any
+selection arguments you are using. You just need to use --list/-l with your
+selection options to do this, for example::
+
+    $ ostestr --regex 'regex3.*' --blacklist_file blacklist.txt --list
+
+This will list all the tests which will be run by ostestr using that combination
+of arguments.
+
+Please not that all of this selection functionality will be expanded on in the
+future and a default grammar for selecting multiple tests will be chosen in a
+future release. However as of right now all current arguments (which have
+guarantees on always remaining in place) are still required to perform any
+selection logic while this functionality is still under development.
+
+
+Output Options
+--------------
+
+By default ostestr will use subunit-trace as the output filter on the test
+run. It will also print the slowest tests from the run after the run is
+concluded. You can disable the printing the slowest tests with the --no-slowest
+flag, for example::
+
+    $ ostestr --no-slowest
+
+If you'd like to disable the subunit-trace output you can do this using
+--no-pretty::
+
+    $ ostestr --no-pretty
+
+ostestr also provides the option to just output the raw subunit stream on
+STDOUT with --subunit/-s. Note if you want to use this you also have to
+specify --no-pretty as the subunit-trace output and the raw subunit output
+are mutually exclusive. For example, to get raw subunit output the arguments
+would be::
+
+    $ ostestr --no-pretty --subunit
+
+An additional option on top of the blacklist file is --print-exclude option.
+When this option is specified when using a blacklist file before the tests are
+run ostestr will print all the tests it will be excluding from the blacklist
+file. If a line in the blacklist file has a comment that will be printed before
+listing the tests which will be excluded by that line's regex. If no comment is
+present on a line the regex from that line will be used instead. For example,
+if you were using the example blacklist file from the previous section the
+output before the regular test run output would be::
+
+    $ ostestr -b blacklist-file blacklist.txt --print-exclude
+    Excludes these tests
+    regex1_match
+    regex1_exclude
+
+    exclude those tests
+    regex2_match
+    regex2_exclude
+
+    ...
+
+Notes for running with tox
+--------------------------
+
+If you use `tox`_ for running your tests and call ostestr as the test command
+.. _tox: https://tox.readthedocs.org/en/latest/
+it's recommended that you set a posargs following ostestr on the commands
+ stanza. For example::
+
+    [testenv]
+    commands = ostestr {posargs}
+
+this will enable end users to pass args to configure the output, use the
+selection logic, or any other options directly from the tox cli. This will let
+tox take care of the venv management and the environment separation but enable
+direct access to all of the ostestr options to easily customize your test run.
+For example, assuming the above posargs usage you would be to do::
+
+    $ tox -epy34 -- --regex ^regex1
+
+or to skip discovery::
+
+    $ tox -epy34 -- -n test.test_thing.TestThing.test_thing_method

doc/source/subunit2html.rst

@@ -0,0 +1,33 @@
+.. _subunit2html:
+
+subunit2html
+============
+
+subunit2html is a tool that takes in a subunit stream file and will output an
+html page
+
+Summary
+-------
+
+    subunit2html subunit_stream [output]
+
+Usage
+-----
+
+subunit2html takes in 1 mandatory argument. This is used to specify the location
+of the subunit stream file. For example::
+
+    $ subunit2html subunit_stream
+
+By default subunit2html will store the generated html results file at
+results.html file in the current working directory.
+
+An optional second argument can be provided to set the output path of the html
+results file that is generated. If it is provided this will be the output path
+for saving the generated file, otherwise results.html in the current working
+directory will be used. For example::
+
+    $ subunit2html subunit_stream test_results.html
+
+will write the generated html results file to test_results.html in the current
+working directory

doc/source/subunit_trace.rst

@@ -0,0 +1,110 @@
+.. _subunit_trace:
+
+subunit-trace
+=============
+
+subunit-trace is an output filter for subunit streams. It is often used in
+conjunction with test runners that emit subunit to enable a consistent and
+useful realtime output from a test run.
+
+Summary
+-------
+
+subunit-trace [--fails|-f] [--failonly] [--perc-diff|-d] [--no-summary]
+              [--diff-threshold|-t <threshold>]
+
+Options
+-------
+
+  --no-failure-debug, -n
+                        Disable printing failure debug information in realtime
+  --fails, -f
+                        Print failure debug information after the stream is
+                        proccesed
+  --failonly
+                        Don't print success items
+  --perc-diff, -d
+                        Print percent change in run time on each test
+  --diff-threshold THRESHOLD, -t THRESHOLD
+                        Threshold to use for displaying percent change from the
+                        avg run time. If one is not specified the percent
+                        change will always be displayed.
+  --no-summary
+                        Don't print the summary of the test run after completes
+
+Usage
+-----
+subunit-trace will take a subunit stream in via STDIN. This is the only input
+into the tool. It will then print on STDOUT the formatted test result output
+for the test run information contained in the stream.
+
+A subunit v2 stream must be passed into subunit-trace. If only a subunit v1
+stream is available you must use the subunit-1to2 utility to convert it before
+passing the stream into subunit-trace. For example this can be done by chaining
+pipes::
+
+    $ cat subunit_v1 | subunit-1to2 | subunit-trace
+
+Adjusting per test output
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+subunit-trace provides several options to customize it's output. This allows
+users to customize the output from subunit-trace to suit their needs. The output
+from subunit-trace basically comes in 2 parts, the per test output, and the
+summary at the end. By default subunit-trace will print failure messages during
+the per test output, meaning when a test fails it will also print the message
+and any traceback and other attachments at that time. However this can be
+disabled by using --no-failure-debug, -n. For example::
+
+    $ testr run --subunit | subunit-trace --no-failure-debug
+
+Rhere is also the option to print all failures together at the end of the test
+run before the summary view. This is done using the --fails/-f option. For
+example::
+
+    $ testr run --subunit | subunit-trace --fails
+
+Often the --fails and --no-failure-debug options are used in conjunction to
+only print failures at the end of a test run. This is useful for large test
+suites where an error message might be lost in the noise. To do this ::
+
+    $ testr run --subunit | subunit-trace --fails --no-failure-debug
+
+By default subunit-trace will print a line for each test after it completes with
+the test status. However, if you only want to see the run time output for
+failures and not any other test status you can use the --failonly option. For
+example::
+
+     $ testr run --subunit | subunit-trace --failonly
+
+The last output option provided by subunit-trace is to diable the summary view
+of the test run which is normally displayed at the end of a run. You can do
+this using the --no-summary option. For example::
+
+    $ testr run --subunit | subunit-trace --no-summary
+
+
+Show per test run time percent change
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+subunit-trace provides an option to display the percent change in run time
+from the previous run. To do this subunit-trace leverages the testr internals
+a bit. It uses the times.dbm database which, the file repository type in
+testrepository will create, to get the previous run time for a test. If testr
+hasn't ever been used before or for whatever reason subunit-trace is unable to
+find the times.dbm file from testr no percentages will be displayed even if it's
+enabled. Additionally, if a test is run which does not have an entry in the
+times.dbm file will not have a percentage printed for it.
+
+To enable this feature you use --perc-diff/-d, for example::
+
+    $ testr run --subunit | subunit-trace --perc-diff
+
+There is also the option to set a threshold value for this option. If used it
+acts as an absolute value and only percentage changes that exceed it will be
+printed. Use the --diff-threshold/-t option to set a threshold, for example::
+
+    $ testr run --subunit | subunit-trace --perc-diff --threshold 45
+
+This will only display percent differences when the change in run time is either
+>=45% faster or <=45% slower.

doc/source/usage.rst

@@ -1,7 +1,12 @@
-========
+=====
 Usage
-========
+=====
 
-To use os-testr in a project::
+This section contains the documentation for each of tools packaged in os-testr
 
-    import os_testr
+.. toctree::
+   :maxdepth: 2
+
+   ostestr
+   subunit_trace
+   subunit2html