From 821e4e6b3277ba456416c9a663def7e693fdc6fe Mon Sep 17 00:00:00 2001
From: Alexander Maretskiy <amaretskiy@mirantis.com>
Date: Fri, 8 Jan 2016 18:30:52 +0200
Subject: [PATCH] [Spec] Update and move spec about new output format
Spec 'improve_scenario_output_format.rst' is implemented,
so it is moved to doc/specs/implemented/
Also there were some updates to this spec to conform
improvements that are actually done:
* Each output data set has 'description' text passed to
Scenario.add_output() while adding the output.
Description is displayed in HTML report under chart title.
* Method Scenario.add_output() has changed (schema
verification is added)
* Subtab 'Detailed' is renamed to 'Per iteration'
Change-Id: Iddf0b0f5edea8002493d47131994f46202363ef5
---
.../improve_scenario_output_format.rst | 324 ++++++++++++++++++
1 file changed, 324 insertions(+)
create mode 100644 doc/specs/implemented/improve_scenario_output_format.rst
diff --git a/doc/specs/implemented/improve_scenario_output_format.rst b/doc/specs/implemented/improve_scenario_output_format.rst
new file mode 100644
index 00000000..98ca6918
--- /dev/null
+++ b/doc/specs/implemented/improve_scenario_output_format.rst
@@ -0,0 +1,324 @@
+..
+ This work is licensed under a Creative Commons Attribution 3.0 Unported
+ License.
+
+ http://creativecommons.org/licenses/by/3.0/legalcode
+
+..
+ This template should be in ReSTructured text. The filename in the git
+ repository should match the launchpad URL, for example a URL of
+ https://blueprints.launchpad.net/heat/+spec/awesome-thing should be named
+ awesome-thing.rst . Please do not delete any of the sections in this
+ template. If you have nothing to say for a whole section, just write: None
+ For help with syntax, see http://sphinx-doc.org/rest.html
+ To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html
+
+=======================================
+Improvements for scenario output format
+=======================================
+
+Current implementation of how scenario saves output data is limited and
+does not meet the needs - it neither allows having more than one data set,
+nor saving custom data structures by each iteration. There is simply a dict
+with int values.
+
+This specification proposes how this can be significantly improved.
+
+Problem description
+===================
+
+At first, let's clarify types of desired output.
+
+Output divides on two main types: additive and complete.
+
+*Additive output* requires processing and representation for the whole
+scenario. For example each iteration has duration - this additive data can
+be taken from each iteration and analyzed how it changes during the
+scenario execution.
+
+*Complete output* data is completely created by iteration and does not require
+extra processing. It is related to this specific iteration only.
+
+Currently scenario can just return a single dict with int values - this is an
+additive data only, and it is stored in iteration results according to
+this schema:
+
+.. code-block::
+
+ "result": {
+ ...
+ "scenario_output": {
+ "type": "object",
+ "properties": {
+ "data": {
+ "type": "object"
+ },
+ "errors": {
+ "type": "string"
+ },
+ },
+ "required": ["data", "errors"]
+ }
+ }
+
+Here are main issues:
+
+ * single data set - this does not allow to split data (if required) among
+ different sources. For example scenario runs two (or more) third-party
+ tools or scripts but has to put all data into single dict
+
+ * output is additive only - so its representation makes sense only after
+ putting data from all iterations together. Scenario iteration can not
+ save its own data list that can be processed independently from another
+ iterations.
+
+ * there is no specific data for HTML report generation like chart title
+ and chart type, so report uses hardcoded values.
+
+As result, HTML report can represent output by a single chart of single type:
+
+.. code-block::
+
+ .--------.
+ | Output |
+ -----' '-----------
+ Scenario output
+ --------------------
+ | |
+ | SINGLE StackedArea |
+ | |
+ --------------------
+
+Proposed change
+===============
+
+Scenario should have ability to save arbitrary number of both additive
+and complete output data. This data should include titles and instructions
+how to be processed and displayed in HTML report.
+
+Here is proposed iterations results structure for output data:
+
+.. code-block::
+
+
+ "result": {
+ ...
+ "output": {
+ "additive": [
+ # Each iteration duplicates "title", "description", "chart" and
+ # items keys, however this seems to be less evil than keeping
+ # aggregated metadata on upper level of task results schema.
+ # "chart" is required by HTML report and should be a name of
+ # existent Chart subclass that is responsible for processing
+ # and displaying the data
+ {"title": "How some durations changes during the scenario",
+ "description": "Some details explaind here",
+ "chart": "OutputStackedAreaChart",
+ "items": [[<str key>, <float value>], ...] # Additive data
+ },
+ ... # More data if required
+ ],
+ "complete": [
+ # Complete data from this specific iteration.
+ # "widget" is required by HTML report and should be a name
+ # of chart widget (see details below) that responsible for
+ # displaying data. We do not need to specify "chart" here
+ # because this data does not require processing - it is
+ # already processed and represents a result of Chart.render()
+ {"title": "Interesting data from specific iteration",
+ "description": "Some details explaind here",
+ "widget": "StackedArea",
+ "data": [
+ [
+ <str key>,
+ [[<float X pos>, <float Y value>], ...]
+ ],
+ ...
+ ]
+ },
+ ... # More data if required
+ ]
+ }
+ }
+
+**NOTES**:
+
+ * for backward compatibility, data from deprecated "scenario_output" should
+ be transformed into "output/data/additive[0]" on-the-fly (for example
+ if we load task results from file)
+
+ * as you can see, there is no container *output/errors* - that is because
+ value of *errors* is not used at all and not required (there is another
+ container for errors in iteration results)
+
+How scenario saves output data
+------------------------------
+
+Scenario should be extended with method *add_output()*:
+
+.. code-block::
+
+ class Scenario(...):
+
+ def __init__(self, context=None):
+ ...
+ self._output = {"additive": [], "complete": []}
+
+ ...
+
+ def add_output(self, additive=None, complete=None):
+ """Add iteration values for additive output.
+
+ :param additive: dict with additive output
+ :param complete: dict with complete output
+ :raises RallyException: When additive or complete has wrong format
+ """
+ for key, value in (("additive", additive), ("complete", complete)):
+ if value:
+ try:
+ jsonschema.validate(
+ value, task.OUTPUT_SCHEMA["properties"][key]["items"])
+ self._output[key].append(value)
+ except jsonschema.ValidationError:
+ raise exceptions.RallyException(
+ "%s output has wrong format" % key.capitalize())
+
+
+Here is an example how scenario can save different output:
+
+.. code-block::
+
+ class SomePlugin(Scenario):
+
+ def specific_scenario(self):
+ ...
+
+ self.add_output(additive={"title": "Foo data",
+ "description": "Some words about Foo",
+ "chart": "OutputStackedAreaChart",
+ "items": [["foo 1", 12], ["foo 2", 34]]})
+ self.add_output(additive={"title": "Bar data",
+ "description": "Some words about Bar",
+ "chart": "OutputAvgChart",
+ "items": [["bar 1", 56], ["bar 2", 78]]})
+ self.add_output(complete={"title": "Complete data",
+ "description": "Some details here",
+ "widget": "StackedArea",
+ "data": [["foo key", [ ... ]], ... ]})
+ self.add_output(complete={"title": "Another data",
+ "description": "Some details here",
+ "widget": "Pie",
+ "data": [["bar key", [ ... ]], ... ]})
+ self.add_output(complete={"title": "Yet another data",
+ "description": "Some details here",
+ "widget": "Table",
+ "data": [["spam key", [ ... ]], ... ]})
+
+Displaying scenario output in HTML report
+-----------------------------------------
+
+The following changes are planned for HTML report and charts classes:
+
+ * rename tab *Output* to *Scenario Data*
+ * implement subtabs under *Scenario Data*: *Aggregated* and *Per iteration*
+ * *Aggregated* subtab shows charts with additive data
+ * *Per iteration* subtab shows charts with complete data, for each iteration
+ * Both subtabs (as well as parent tab) are shown only if there is
+ something to display
+ * add base class OutputChart and generic charts classes for processing
+ output data: OutputStackedAreaChart, OutputAvgChart, OutputStatsTable
+ * add optional *title* and *description* arguments to OutputChart.__init__()
+ so title and description - this is important for custom charts
+ * add *WIDGET* property to each OutputChart subclass to bind it to specific
+ chart widget (StackedArea, Pie, Table). For example, AvgChart will be
+ bound to "Pie". This will allow defining both how to process and how
+ to display some data simply by single class name
+ * update return value format of OutputChart.render() with title and widget:
+ {"title": <str>, "description": <str>, "widget": <str>, "data": [...]}
+
+UI sketch for active "Aggregated" subtab:
+
+.. code-block::
+
+ .---------------.
+ | Scenario Data |
+ ----' '-------------------
+ Aggregated Per iteration
+ -------------
+ <Custom chart title>
+ <Here is a description text>
+ ----------------------------
+ | |
+ | Any available chart widget |
+ | |
+ ----------------------------
+
+ <Custom chart title>
+ <Here is a description text>
+ ----------------------------
+ | |
+ | Any available chart widget |
+ | |
+ ----------------------------
+
+ [... more charts]
+
+UI sketch for active "Per iteration" subtab, let it be iteration 5
+selected by dropdown:
+
+.. code-block::
+
+ .---------------.
+ | Scenario Data |
+ ----' '-------------------
+ Aggregated Per iteration
+ ----------
+
+ [iteration 5]
+
+ <Custom chart title>
+ <Here is a description text>
+ ----------------------------
+ | |
+ | Any available chart widget |
+ | |
+ ----------------------------
+
+ <Custom chart title>
+ <Here is a description text>
+ ----------------------------
+ | |
+ | Any available chart widget |
+ | |
+ ----------------------------
+
+ [... more charts]
+
+Alternatives
+------------
+
+None
+
+Implementation
+==============
+
+Assignee(s)
+-----------
+
+Primary assignee:
+ * amaretskiy <amaretskiy@mirantis.com>
+
+Work Items
+----------
+
+ * Update task results schema with *output* container
+ * Extend Scenario with method *add_output()*
+ * Bound Chart subclasses to specific charts widgets
+ * Add generic Charts subclasses for output data
+ * Changes in HTML report related to *Output* tab
+ * Add scenario with example output data
+
+Dependencies
+============
+
+None