Merge "Report Visualization Spec and Mocks"

specs/proposed/refstack-org-result-visualization.rst

@@ -0,0 +1,167 @@
+Intuitive visualization for test comparison
+==========================================
+
+Blueprint: https://blueprints.launchpad.net/refstack/+spec/results-visualization
+Storyboard: https://storyboard.openstack.org/#!/story/111
+
+Result comparison is the very essence of refstack.  This spec lays out the
+basic design objectives, comparison matrix and wire-frames for the initial
+visualization between cloud test runs.  The display of this information
+must make it simple for users to map their cloud interoperability to other
+clouds.
+
+  Note: Positive results only for refstack public site.
+  We will still handle negative/skip use cases.
+
+Problem description
+-------------------
+
+Refstack collects substantial amounts of detailed raw data in the form
+of passed test results.  Individually, these tests provide little insight
+about cloud interoperability; consequently, restack must provide a way to
+group results (likely capabilities) and contract with other posted test
+results.
+
+Specific Items that Need Visualization
+
+Comparison of results against:
+  * Core Test List ("am I core?")
+  * Universe of Tests ("close to 100%?")
+  * Other's runs ("do I interoperate?")
+  * Previous runs ("did I improve?")
+
+To make it easier to determine, results should follow the capabilities
+groups rather than individual tests.  Users should be able to drill down
+into a capability to look at the test detail.
+
+Note about Capabilities versus Tests: In DefCore, capabilities are
+tracked as the definition of "core."   Each capability has a defined
+set of tests.  For a core capability to be considered as "passed,"
+all of the tests in that capability must pass.  Since we do not
+track "failed" as a state, a non-passing test simply makes the whole
+capability not passing.
+
+General Visualization: Tristate
+----------------------------
+
+The general approach is to focus on _deltas_ from other sets rather
+than showing the actual results.  This means that visualizations
+will be more about what's different or changed.  The preferred tool
+will be the "tristate" graph: http://omnipotent.net/jquery.sparkline/#s-about.
+
+For consistency, users should expect that:
+  * +1 = good, match
+  * 0  = ok, no match (run is advantaged over reference/core)
+  * -1 = bad, missing functionality
+
+.. image:: https://wiki.openstack.org/w/images/1/19/Refstack_mock_tristate.png
+   :width: 700px
+   :alt: Tristate Graph Sample
+
+There are two consistent but slightly different ways that we will use tri-state:
+
+1) comparing to core tests with a goal of showing compliance
+  * +1 = passed a core test or capability
+  * 0 = passed a non-core test or capability
+  * -1 = did not pass a core test or capability (this is the same as "not-reported")
+2) compare to other tests with a goal of showing interoperability
+  * +1 = passed in both samples
+  * 0 = passed in subject but not in reference (subject is advantaged)
+  * -1 = not passed in subject but did in reference (subject is disadvantaged)
+
+An example rendering would lock like this:
+
+.. image:: https://wiki.openstack.org/w/images/5/5e/Refstack_mock_comparison.png
+   :width: 700px
+   :alt: Comparison Mock Up
+
+Important Design Note: All tristate graphs must use the same ordered capability/test list 
+to ensure that results are easily to compare visually.  The purpose of the tristate is
+to help quickly find outliers not perform detailed comparison.  Drill downs will be used
+to resolve specific differences.
+
+Detailed Visualization: Drill Down
+----------------------------
+
+We will expand the capabilities level tristate in the detailed visualization but
+still retain the tristate meanings with specific tests.  In the drill down, the
+user will see the original tristate graph above a table with the capabilities
+list (order preserved) by rows.  In each row, the following columns:
+* the name of the capability
+* a tristate will visualize the individual test results using the same +1/0/-1 semantics
+* a simple list of the -1 tests
+
+Usability Note: The name of the test/capability should be included as a hover.
+
+Alternatives
+----------------------------
+
+There are several other approaches to visualize this information including shaded table
+cells and spider charts.  This would be acceptable alternatives; however, the tristate
+chart is compact, very simple to use and highly intuitive for comparing result sets.
+
+It would be possible to use tristate shapes (circle, open circle, square) to reflect the same 
+data on tables.
+
+Data model impact
+
+Likely none; however, depending on the complexity of the queries,
+it may be necessary to create intermediate tables to to summarize
+capabilities from test results per run to improve performance.
+
+If new models are needed, this spec should be updated with the design.
+At this time, we assume that the collection does not require an
+intermediate model.
+
+Specification for the method 
+
+These are read-only reports and should use GETs. 
+
+The URL path should match the other UI paths with then following pattern: 
+
+Compare against previous results:
+HTML response: GET /[refstack base]/compare/[release]/[cloud id]
+
+Compare against other clouds:
+HTML response: GET /[refstack base]/compare/[release]/[cloud id]?to=[other 1]|[other 2]
+
+JSON response same as HTML but with .json
+
+Security impact
+
+None.  These are open reports.
+
+Notifications impact
+
+None.
+
+Other end user impact
+
+None.
+
+Developer impact
+
+None.
+
+Assignee(s)
+
+TBD
+
+Work Items
+
+* Spec & Mock
+* CSS & HTML Frame
+* Data Collection
+* Connect Data into UI Page
+
+Dependencies
+
+Sparklines JS libraries: http://omnipotent.net/jquery.sparkline/#s-about 
+
+Documentation Impact
+
+Need to document screen and drill down expectation.
+
+References
+
+http://wiki.openstack.org/wiki/Governance/DefCoreCommittee