Add to stackviz-export and stackviz-front docs

Adds additional information to the stackviz-export and stackviz-front
technical documents. Info about `stackviz-export` output files is
now included in the corresponding document. Directive and service
descriptions were added to stackviz-front, alongside inline comments
in the appropriate .js files.

Due to the unique structure of GitHub's RST renderer, includes will
no longer be used in the main README.

Change-Id: Iaebf1f1c3b5e4cbb4ea5e262f85672bc082bbe2f

CONTRIBUTING.rst

@@ -1 +1,19 @@
-.. include:: ./doc/source/contributing.rst
+Contributing
+============
+If you would like to contribute to the development of OpenStack, you must
+follow the steps in this page:
+
+   http://docs.openstack.org/infra/manual/developers.html
+
+If you already have a good understanding of how the system works and your
+OpenStack accounts are set up, you can skip to the development workflow
+section of this documentation to learn how changes to OpenStack should be
+submitted for review via the Gerrit tool:
+
+   http://docs.openstack.org/infra/manual/developers.html#development-workflow
+
+Pull requests submitted through GitHub will be ignored.
+
+Bugs should be filed on Launchpad, not GitHub:
+
+   https://bugs.launchpad.net/stackviz

README.rst

@@ -1 +1,90 @@
-.. include:: ./doc/source/readme.rst
+========
+StackViz
+========
+A visualization utility to help analyze the performance of DevStack setup and
+Tempest executions. This repository can be cloned and built to use Stackviz
+with local run data. Stackviz is currently in the process of being implemented
+upstream (see Roadmap and Planning). To use Stackviz with upstream gate runs,
+please see the server deployment project at:
+
+    https://github.com/timothyb89/stackviz-deployer
+
+Installation
+============
+Installation - Frontend
+-----------------------
+Installation of the frontend requires Node.js and Gulp. On Ubuntu::
+    sudo apt-get install nodejs npm nodejs-legacy
+    sudo npm install -g gulp
+
+Then, install the Node modules by running, from the project directory::
+
+    npm install
+
+Installation - Processing
+-------------------------
+The data processor is a small Python module located in the same source tree. To
+install, run::
+
+    sudo pip install .
+
+Usage
+========
+Usage - Development
+-------------------
+A development server can be run as follows::
+
+    gulp dev
+
+This will open a web browser and reload code automatically as it changes on the
+filesystem.
+
+If you have subunit and dstat logs, you can create a config.json to display
+your runs::
+
+    stackviz-export -f <path/to/subunit> --dstat <path/to/dstat> app/data/
+
+During :code:`gulp dev`, files written to :code:`app/data/` will be
+automatically synchronized with the browser. Note that these files will *not* be
+copied to :code:`build/` during :code:`gulp prod`, but you can copy them
+manually using :code:`gulp data`.
+
+Usage - Production
+------------------
+The production application can be build using::
+
+    gulp prod
+
+The result will be written to :code:`./build` and should be appropriate for
+distribution. Note that all files are not required:
+
+- Directory structure (:code:`js/`, :code:`css/`, :code:`fonts/`,
+  :code:`images/`): required.
+- Static resources (:code:`fonts/`, :code:`images/`): required.
+- Core files (:code:`index.html`, :code:`js/main.js`, :code:`css/main.css`):
+  required unless gzipped versions are used.
+- Gzipped versions of core files (:code:`*.gz`): not required, but preferred.
+  Use instead of plain core files to save on disk usage and bandwidth.
+- Source maps (:code:`js/main.js.map`, :code:`js/main.js.map.gz`): only required
+  for debugging purposes.
+
+Data should be written to :code:`build/data/` using :code:`stackviz-export` like
+above. Note that the static production code generated above is portable, and can
+be generated anywhere and copied to another host to be combined with exported
+data.
+
+Testing
+=======
+* Python tests: :code:`tox -epy27`
+* JavaScript unit tests: :code:`gulp unit`
+* JavaScript E2E tests: :code:`gulp e2e`
+
+Manuals & Developer Docs
+========================
+For more detailed information on how Stackviz works, please see the manuals
+located at doc/source/man/
+
+Roadmap and Planning
+====================
+- Planning: https://etherpad.openstack.org/p/stackviz
+- Gate integration planning: https://etherpad.openstack.org/p/BKgWlKIjgQ

app/js/controllers/test-details.js

@@ -6,6 +6,14 @@ var controllersModule = require('./_index');
  * @ngInject
  */
 var TestDetailsCtrl =
+/**
+ * Responsible for making three calls to the dataset service. First, the
+ * dataset corresponding to the given int id is loaded, then the raw and details
+ * JSON files are loaded and placed into state variables. Also note that a copy
+ * of the details JSON is kept in `originalDetails` so that information is not
+ * lost when parsing. Progress of the dataset service calls is recorded and
+ * displayed in a progress bar on `test-details.html`.
+*/
 function($scope, $location, $stateParams, $log, datasetService, progressService) {
   var vm = this;
   vm.datasetId = $stateParams.datasetId;
@@ -43,7 +51,18 @@ function($scope, $location, $stateParams, $log, datasetService, progressService)
       progressService.done();
     });
 
-  vm.parsePythonLogging = function(showINFO, showDEBUG, showWARNING, showERROR) {
+  vm.parsePythonLogging =
+  /**
+   * This function changes the `itemDetails.pythonlogging` variable to only
+   * show lines with the log levels specified by the four boolean parameters.
+   * EX: If the `showINFO` parameter is set to true, `itemDetails.pythonlogging`
+   * will display lines that contain the text `INFO`.
+   * @param {boolean} showINFO
+   * @param {boolean} showDEBUG
+   * @param {boolean} showWARNING
+   * @param {boolean} showERROR
+   */
+  function(showINFO, showDEBUG, showWARNING, showERROR) {
     if (vm.originalDetails && vm.originalDetails.pythonlogging) {
       var log = vm.originalDetails.pythonlogging;
       var ret = [];

app/js/directives/tempest-summary.js

@@ -10,7 +10,13 @@ function tempestSummary() {
   /**
    * @ngInject
    */
-  var controller = function($scope, $attrs, datasetService) {
+  var controller =
+  /**
+   * Responsible for getting the basic run summary stats via the dataset service.
+   * Also calculates the duration of the run - `timeDiff` - by subtracting the
+   * run's start and end timestamps.
+   */
+  function($scope, $attrs, datasetService) {
     $scope.$watch('dataset', function(dataset) {
       var stats = dataset.stats;
       $scope.stats = stats;

app/js/directives/test-details-search.js

@@ -10,7 +10,14 @@ function testDetailsSearch() {
   /**
    * @ngInject
    */
-  var controller = function($scope, $element) {
+  var controller =
+  /**
+   * Responsible for calling the `parsePythonLogging` filter function in
+   * `TestDetailsController` when the log level buttons change state. The
+   * `filter` function is passed from `test-details` to `test-details-search`
+   * when the directive is initially instantiated.
+   */
+  function($scope, $element) {
     var self = this;
     this.open = false;
     this.showINFO = true;
@@ -18,10 +25,12 @@ function testDetailsSearch() {
     this.showWARNING = true;
     this.showERROR = true;
 
+    // Wrapper for parent controller's filter function.
     var update = function() {
       $scope.filter(self.showINFO, self.showDEBUG, self.showWARNING, self.showERROR);
     };
 
+    // Watchers to signal update function upon button state change.
     $scope.$watch(function() { return self.query; }, update);
     $scope.$watch(function() { return self.showINFO; }, update);
     $scope.$watch(function() { return self.showDEBUG; }, update);

doc/source/contributing.rst

@@ -1,19 +1 @@
-Contributing
+.. include:: ../../CONTRIBUTING.rst
-============
-If you would like to contribute to the development of OpenStack, you must
-follow the steps in this page:
-
-   http://docs.openstack.org/infra/manual/developers.html
-
-If you already have a good understanding of how the system works and your
-OpenStack accounts are set up, you can skip to the development workflow
-section of this documentation to learn how changes to OpenStack should be
-submitted for review via the Gerrit tool:
-
-   http://docs.openstack.org/infra/manual/developers.html#development-workflow
-
-Pull requests submitted through GitHub will be ignored.
-
-Bugs should be filed on Launchpad, not GitHub:
-
-   https://bugs.launchpad.net/stackviz

doc/source/man/stackviz-export.rst

@@ -14,7 +14,6 @@ is as simple as::
 
 Usage
 =====
-
 :code:`stackviz-export [options] <DEST>`
 
 Where DEST is the output directory of the module. If DEST does not exist, a new
@@ -45,3 +44,40 @@ Additional options:
 
   **-z --gzip**
       Enables gzip compression for data files.
+
+Output
+======
+:code:`stackviz-export` outputs the following files to the destination directory.
+Note that <source> in the details, raw, and tree logs refer to what stream
+source the
+
+**config.json**
+    Contains all the basic information about a dataset that the front-end needs.
+    There will be one `tempest` entry for every dataset that was generated in
+    :code:`stackviz-export`. Each `tempest` entry has general information about
+    each run, as well as the locations of the details, raw, and tree JSON files.
+
+**dstat_log.csv**
+    This file will only be present if a dstat log was used in the corresponding
+    :code:`stackviz-export` run. Has a wide variety of system statistics
+    including CPU, memory, and disk utilization. This information is displayed
+    on the timeline graph.
+
+**tempest_<source>_<id>_details.json**
+    The details log contains timestamp and status information for tests in
+    addition to all of the logs associated with the test (e.g. tracebacks).
+    These artifacts are displayed in the test details page.
+
+**tempest_<source>_<id>_raw.json**
+    Contains nearly all information available about tests:
+        - :code:`status`: pass, fail, or skipped
+        - :code:`name`: full name of test
+        - :code:`tags`: which worker the test was run on
+        - :code:`details`: empty, this info is available in the details JSON
+        - :code:`duration`: how long the test took, in seconds
+        - :code:`timestamps`: timestamps at test begin and test end
+    This file is used in the timeline and test details page.
+
+**tempest_<source>_<id>_tree.json**
+    Stores test names in a hierarchy for display on the deprecated
+    sunburst diagram. Not currently used by any page in Stackviz.

doc/source/man/stackviz-front.rst

@@ -49,7 +49,7 @@ Test Details
 ------------
 :Path: <host>/#/<run>/test-details/<test>/
 :Directive: :code:`./app/views/test-details.html`
-:Controller: :code:`./app/js/controllers/home.js`
+:Controller: :code:`./app/js/controllers/test-details.js`
 
 The test details page consists of one panel that displays various log info
 from one test. The first tab contains summary information similar to the info
@@ -80,17 +80,62 @@ additional information to aid debugging. The most common tabs include:
 When enough information has been gleaned from more detailed logs, the button
 in the panel filter can be used to quickly navigate back to the timeline page.
 
-Controllers
-===========
-
 Directives
 ==========
 
-Filters
+**tempestSummary**
-=======
+    The tempest summary directive consists of one panel that shows stats for
+    one run: Duration of the run, number of tests run, number of tests skipped,
+    and number of tests failed. :code:`timeDiff` (the duration of the run) is
+    calculated from the start and end timestamps contained in summary data.
+    All other fields are populated directly from the summary data, via a call
+    to the dataset service.
+
+**testDetailsSearch**
+    :code:`testDetailsSearch` uses two HTML pages to search the test details
+    page: :code:`test-details-search-popover.html` and :code:`test-details-search.html`.
+    The popover contains the filter levels for the :code:`pythonlogging` tab:
+    INFO, DEBUG, WARNING, ERROR. This directive is used as the template for
+    :code:`test-details-search`, per AngularJS popover convention. The function
+    used to parse the logs, :code:`parsePythonLogging` actually lives in the
+    controller for testDetailsSearch, and is passed through both the prior
+    directives' scopes. This function reads in the :code:`pythonLogging` tab
+    as one text object, then splits it by :code:`\n` to create an array of
+    lines. Each line is then added back to the :code:`pythonLogging` tab if it
+    contains the specific log level somewhere in the line.
+
+**timeline**
+    The timeline directive is a container for the actual timeline components,
+    detailed below.
+
+**timelineDetails**
+
+**timelineDstat**
+
+**timelineOverview**
+
+**timelineSearch**
+
+**timelineViewport**
+
 
 Services
 ========
 
-Util
+**dataset**
-====
+    The dataset service is an API that provides the front-end with all of the
+    data generated by :code:`stackviz-export`. All data processed by
+    :code:`stackviz-export` ends up in the `./app/data/` directory to be called
+    by dataset service with :code:`$http` and :code:`$q` directives. Below is
+    the list of calls:
+        - :code:`list` returns `config.json` using GET.
+        - :code:`get(id)` calls :code:`list`, then iterates through all the
+          available datasets for the requested id number. Rejects if not found.
+        - :code:`raw(dataset)` returns `<dataset>_raw.json` file using GET.
+        - :code:`details(dataset)` returns `<dataset>_details.json` file using GET.
+        - :code:`tree(dataset)` returns `<dataset>_tree.json` file using GET.
+        - :code:`dstat(dataset)` returns `dstat_log.csv` file using GET, if available.
+
+**progress**
+    A wrapper for :code:`nprogress`, a progress bar library. Used in the timeline
+    and test details pages to show progress in loading datasets.

doc/source/readme.rst

@@ -1,30 +1 @@
-========
+.. include:: ../../README.rst
-StackViz
-========
-A visualization utility to help analyze the performance of DevStack setup and
-Tempest executions. This repository can be cloned and built to use Stackviz
-with local run data. Stackviz is currently in the process of being implemented
-upstream (see Roadmap and Planning). To use Stackviz with upstream gate runs,
-please see the server deployment project at:
-
-    https://github.com/timothyb89/stackviz-deployer
-
-.. include:: ./installation.rst
-
-.. include:: ./usage.rst
-
-Testing
-=======
-* Python tests: :code:`tox -epy27`
-* JavaScript unit tests: :code:`gulp unit`
-* JavaScript E2E tests: :code:`gulp e2e`
-
-Manuals & Developer Docs
-========================
-For more detailed information on how Stackviz works, please see the manuals
-located at doc/source/man/
-
-Roadmap and Planning
-====================
-- Planning: https://etherpad.openstack.org/p/stackviz
-- Gate integration planning: https://etherpad.openstack.org/p/BKgWlKIjgQ