x/deb-python-tempest-lib
Code Issues Proposed changes

Summarize expected failures

Browse Source 
The incoming stream has the xfail data but count_tests was not
matching on the xfail status. Instead it was matching on the
fragment 'fail' meaning that expected failures were counted as
such. By bounding the regular expressions it is possible to get more
specific results will still leaving the count_tests method flexible
for other users where fragments would be useful.

Counts of 'uxsuccess' are also summarized as "Unexpected Success".

It's unclear how to effectively automate testing of this. Manual
testing returns the expected results.

Change-Id: I5b1458f9a98712ea3e424d2c9610b915055138af
This commit is contained in:
Chris Dent
2015-02-11 14:39:10 +00:00
parent c70c26d05c
commit e29ec710ef
1 changed files with 6 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
tempest_lib/cmd/subunit_trace.py
View File

@@ -205,9 +205,12 @@ def print_summary(stream, elapsed_time):
stream.write("\n======\nTotals\n======\n")
stream.write("Ran: %s tests in %.4f sec.\n" % (
count_tests('status', '.*'), total_seconds(elapsed_time)))
stream.write(" - Passed: %s\n" % count_tests('status', 'success'))
stream.write(" - Skipped: %s\n" % count_tests('status', 'skip'))
stream.write(" - Failed: %s\n" % count_tests('status', 'fail'))
stream.write(" - Passed: %s\n" % count_tests('status', '^success$'))
stream.write(" - Skipped: %s\n" % count_tests('status', '^skip$'))
stream.write(" - Expected Fail: %s\n" % count_tests('status', '^xfail$'))
stream.write(" - Unexpected Success: %s\n" % count_tests('status',
'^uxsuccess$'))
stream.write(" - Failed: %s\n" % count_tests('status', '^fail$'))
stream.write("Sum of execute time for each test: %.4f sec.\n" % run_time())
# we could have no results, especially as we filter out the process-codes