diff --git a/specs/proposed/refstack-org-result-visualization.rst b/specs/proposed/refstack-org-result-visualization.rst new file mode 100644 index 00000000..a3e12e55 --- /dev/null +++ b/specs/proposed/refstack-org-result-visualization.rst @@ -0,0 +1,167 @@ +Intuitive visualization for test comparison +========================================== + +Blueprint: https://blueprints.launchpad.net/refstack/+spec/results-visualization +Storyboard: https://storyboard.openstack.org/#!/story/111 + +Result comparison is the very essence of refstack. This spec lays out the +basic design objectives, comparison matrix and wire-frames for the initial +visualization between cloud test runs. The display of this information +must make it simple for users to map their cloud interoperability to other +clouds. + + Note: Positive results only for refstack public site. + We will still handle negative/skip use cases. + +Problem description +------------------- + +Refstack collects substantial amounts of detailed raw data in the form +of passed test results. Individually, these tests provide little insight +about cloud interoperability; consequently, restack must provide a way to +group results (likely capabilities) and contract with other posted test +results. + +Specific Items that Need Visualization + +Comparison of results against: + * Core Test List ("am I core?") + * Universe of Tests ("close to 100%?") + * Other's runs ("do I interoperate?") + * Previous runs ("did I improve?") + +To make it easier to determine, results should follow the capabilities +groups rather than individual tests. Users should be able to drill down +into a capability to look at the test detail. + +Note about Capabilities versus Tests: In DefCore, capabilities are +tracked as the definition of "core." Each capability has a defined +set of tests. For a core capability to be considered as "passed," +all of the tests in that capability must pass. Since we do not +track "failed" as a state, a non-passing test simply makes the whole +capability not passing. + +General Visualization: Tristate +---------------------------- + +The general approach is to focus on _deltas_ from other sets rather +than showing the actual results. This means that visualizations +will be more about what's different or changed. The preferred tool +will be the "tristate" graph: http://omnipotent.net/jquery.sparkline/#s-about. + +For consistency, users should expect that: + * +1 = good, match + * 0 = ok, no match (run is advantaged over reference/core) + * -1 = bad, missing functionality + +.. image:: https://wiki.openstack.org/w/images/1/19/Refstack_mock_tristate.png + :width: 700px + :alt: Tristate Graph Sample + +There are two consistent but slightly different ways that we will use tri-state: + +1) comparing to core tests with a goal of showing compliance + * +1 = passed a core test or capability + * 0 = passed a non-core test or capability + * -1 = did not pass a core test or capability (this is the same as "not-reported") +2) compare to other tests with a goal of showing interoperability + * +1 = passed in both samples + * 0 = passed in subject but not in reference (subject is advantaged) + * -1 = not passed in subject but did in reference (subject is disadvantaged) + +An example rendering would lock like this: + +.. image:: https://wiki.openstack.org/w/images/5/5e/Refstack_mock_comparison.png + :width: 700px + :alt: Comparison Mock Up + +Important Design Note: All tristate graphs must use the same ordered capability/test list +to ensure that results are easily to compare visually. The purpose of the tristate is +to help quickly find outliers not perform detailed comparison. Drill downs will be used +to resolve specific differences. + +Detailed Visualization: Drill Down +---------------------------- + +We will expand the capabilities level tristate in the detailed visualization but +still retain the tristate meanings with specific tests. In the drill down, the +user will see the original tristate graph above a table with the capabilities +list (order preserved) by rows. In each row, the following columns: +* the name of the capability +* a tristate will visualize the individual test results using the same +1/0/-1 semantics +* a simple list of the -1 tests + +Usability Note: The name of the test/capability should be included as a hover. + +Alternatives +---------------------------- + +There are several other approaches to visualize this information including shaded table +cells and spider charts. This would be acceptable alternatives; however, the tristate +chart is compact, very simple to use and highly intuitive for comparing result sets. + +It would be possible to use tristate shapes (circle, open circle, square) to reflect the same +data on tables. + +Data model impact + +Likely none; however, depending on the complexity of the queries, +it may be necessary to create intermediate tables to to summarize +capabilities from test results per run to improve performance. + +If new models are needed, this spec should be updated with the design. +At this time, we assume that the collection does not require an +intermediate model. + +Specification for the method + +These are read-only reports and should use GETs. + +The URL path should match the other UI paths with then following pattern: + +Compare against previous results: +HTML response: GET /[refstack base]/compare/[release]/[cloud id] + +Compare against other clouds: +HTML response: GET /[refstack base]/compare/[release]/[cloud id]?to=[other 1]|[other 2] + +JSON response same as HTML but with .json + +Security impact + +None. These are open reports. + +Notifications impact + +None. + +Other end user impact + +None. + +Developer impact + +None. + +Assignee(s) + +TBD + +Work Items + +* Spec & Mock +* CSS & HTML Frame +* Data Collection +* Connect Data into UI Page + +Dependencies + +Sparklines JS libraries: http://omnipotent.net/jquery.sparkline/#s-about + +Documentation Impact + +Need to document screen and drill down expectation. + +References + +http://wiki.openstack.org/wiki/Governance/DefCoreCommittee \ No newline at end of file