[docs][4] Re-design docs to cover all user-groups
First pack of changes in upcoming chain to redesign Rally docs.
All information related to Rally CLI and Rally components
separated and refactored. Modified files fit 80 symbols margin
where possible.
[NOTE] Rally Verify Component documentation is referencing to
already existing part of Rally step-by-step tutorial.
This documentation is going to be changed soon after
work on this component refactoring will be completed.
[TODO] continue with other parts of the docs:
- Rally Plugins, Rally Plugins Reference
- Contribute to Rally
- Request New Features
- Project Info
[TODO] add 80 symbols margin check similar to what
Performance Documentation has
Change-Id: I64f7f78c9036aedca9f20d0770e491bef8183386
This commit is contained in:
Dina Belova
parent
8f249cd571
commit
bd96f85a4e
doc/source
@ -4,4 +4,8 @@
|
||||
Command Line Interface
|
||||
======================
|
||||
|
||||
.. contents::
|
||||
:depth: 1
|
||||
:local:
|
||||
|
||||
.. make_cli_reference::
|
||||
|
||||
@ -1,26 +1,30 @@
|
||||
:tocdepth: 1
|
||||
.. contents::
|
||||
:depth: 3
|
||||
:local:
|
||||
|
||||
==============
|
||||
Task Component
|
||||
==============
|
||||
|
||||
This section describes Rally Task Component (including feature presented since
|
||||
Rally v0.5.0, allowing to analyze statistics trends for the given tasks).
|
||||
|
||||
============
|
||||
HTML Reports
|
||||
============
|
||||
|
||||
HTML reports provide comprehensive analysis.
|
||||
Data is structured and displayed interactively, with charts and tables.
|
||||
|
||||
.. contents::
|
||||
:depth: 2
|
||||
:local:
|
||||
|
||||
Task Report
|
||||
===========
|
||||
-----------
|
||||
|
||||
Get the whole information about task workloads results, in pretty
|
||||
and convenient format!
|
||||
|
||||
.. image:: ./images/Report-Collage.png
|
||||
.. image:: ../images/Report-Collage.png
|
||||
|
||||
Generate report for single task, using task UUID
|
||||
------------------------------------------------
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Having a finished task, generate report with command:
|
||||
|
||||
@ -35,7 +39,7 @@ Example:
|
||||
$ rally task report 6f63d9ec-eecd-4696-8e9c-2ba065c68535 --out report.html
|
||||
|
||||
Generate report for single task, using JSON file
|
||||
------------------------------------------------
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Report can be generated from a task results JSON file.
|
||||
This file can be generated with command *rally task results*:
|
||||
@ -46,7 +50,7 @@ This file can be generated with command *rally task results*:
|
||||
$ rally task report results.json --out report.html
|
||||
|
||||
Generate report for many tasks
|
||||
------------------------------
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Report can be generated from many tasks. All workloads from specified
|
||||
tasks results will be composed into an entire report.
|
||||
@ -60,58 +64,61 @@ Example:
|
||||
$ rally task report --tasks 6f63d9ec-eecd-4696-8e9c-2ba065c68535 20ae7e95-7395-4be4-aec2-b89220adee60 a5737eba-a204-43d6-a262-d5ea4b0065da results.json another_results.json --out report.html
|
||||
|
||||
Task Overview
|
||||
-------------
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
This is a table with brief summary of all workloads results.
|
||||
All columns are sortable and clickable.
|
||||
|
||||
.. image:: ./images/Report-Task-Overview.png
|
||||
.. image:: ../images/Report-Task-Overview.png
|
||||
|
||||
Load duration
|
||||
~~~~~~~~~~~~~
|
||||
+++++++++++++
|
||||
|
||||
Time from first iteration start to last iteration end.
|
||||
In other words, this is a time of all workload iterations execution.
|
||||
|
||||
Full duration
|
||||
~~~~~~~~~~~~~
|
||||
+++++++++++++
|
||||
|
||||
This time includes iterations time (`Load duration <#load-duration>`_)
|
||||
plus time taken by another actions related to the task, mostly Contexts
|
||||
execution time.
|
||||
|
||||
Iterations
|
||||
~~~~~~~~~~
|
||||
++++++++++
|
||||
|
||||
How many times the workload has run. This comes from the value of *runner.times* in task input file.
|
||||
How many times the workload has run. This comes from the value of
|
||||
*runner.times* in task input file.
|
||||
|
||||
Failures
|
||||
~~~~~~~~
|
||||
++++++++
|
||||
|
||||
Number of failed iterations.
|
||||
Failure means that there was an Exception raised.
|
||||
Number of failed iterations. Failure means that there was an Exception raised.
|
||||
|
||||
Success (SLA)
|
||||
~~~~~~~~~~~~~
|
||||
+++++++++++++
|
||||
|
||||
This is a boolean result of workload SLA. See `Service-level agreement explanation <#id2>`_ below.
|
||||
This is a boolean result of workload SLA. See
|
||||
`Service-level agreement explanation <#id2>`_ below.
|
||||
|
||||
Input file
|
||||
----------
|
||||
~~~~~~~~~~
|
||||
|
||||
This shows JSON which can be used to run a task with exactly the same workloads list and configuration.
|
||||
This is not an exact copy (neither concatenation) of actually used input files (in command *rally task start*), however this is exactly what is needed to run workloads given in the report.
|
||||
This shows JSON which can be used to run a task with exactly the same workloads
|
||||
list and configuration. This is not an exact copy (neither concatenation) of
|
||||
actually used input files (in command *rally task start*), however this is
|
||||
exactly what is needed to run workloads given in the report.
|
||||
|
||||
.. image:: ./images/Report-Task-Input-file.png
|
||||
.. image:: ../images/Report-Task-Input-file.png
|
||||
|
||||
Tab «Overview»
|
||||
--------------
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
Service-level agreement
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
+++++++++++++++++++++++
|
||||
|
||||
`SLA <https://en.wikipedia.org/wiki/Service-level_agreement>`_ results appear
|
||||
in task report only if *"sla"* section is defined in task input file.
|
||||
`SLA`_ results appear in task report only if *"sla"* section is defined in task
|
||||
input file.
|
||||
|
||||
For example, having this in task input file:
|
||||
|
||||
@ -135,128 +142,132 @@ For example, having this in task input file:
|
||||
|
||||
will result SLA section similar to the following:
|
||||
|
||||
.. image:: ./images/Report-Task-SLA.png
|
||||
.. image:: ../images/Report-Task-SLA.png
|
||||
|
||||
|
||||
What if workload has no "sla" configuration in input file?
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
If *"sla"* section is missed in input file, then block *Service-level agreement*
|
||||
is not displayed and its result is assumed to be always passed
|
||||
If *"sla"* section is missed in input file, then block *Service-level
|
||||
agreement* is not displayed and its result is assumed to be always passed
|
||||
(no matter how many failures occurred).
|
||||
|
||||
Total durations
|
||||
~~~~~~~~~~~~~~~
|
||||
+++++++++++++++
|
||||
|
||||
There is a durations analysis, which is is represented by statistics table and duration StackedArea chart.
|
||||
There is a durations analysis, which is is represented by statistics table and
|
||||
duration StackedArea chart.
|
||||
|
||||
.. image:: ./images/Report-Task-Total-durations.png
|
||||
.. image:: ../images/Report-Task-Total-durations.png
|
||||
|
||||
Table with statistics data
|
||||
++++++++++++++++++++++++++
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
**Action**
|
||||
Name of the workload metric that has some duration saved.
|
||||
This is either an atomic action name or *Total* which points to workload `load duration <#load-duration>`_.
|
||||
This is either an atomic action name or *Total* which points to workload
|
||||
`load duration <#load-duration>`_.
|
||||
|
||||
**Min (sec)**
|
||||
`Minimal <https://en.wikipedia.org/wiki/Maxima_and_minima>`_ duration value
|
||||
`Minimal`_ duration value
|
||||
|
||||
**Median (sec)**
|
||||
`Median <https://en.wikipedia.org/wiki/Median>`_ duration value
|
||||
`Median`_ duration value
|
||||
|
||||
**90%ile (sec)**
|
||||
`Percentile <https://en.wikipedia.org/wiki/Percentile>`_ for 90% durations
|
||||
`Percentile`_ for 90% durations
|
||||
|
||||
**95%ile (sec)**
|
||||
`Percentile <https://en.wikipedia.org/wiki/Percentile>`_ for 95% durations
|
||||
`Percentile`_ for 95% durations
|
||||
|
||||
**Max (sec)**
|
||||
`Maximal <https://en.wikipedia.org/wiki/Maxima_and_minima>`_ duration value
|
||||
`Maximal`_ duration value
|
||||
|
||||
**Avg (sec)**
|
||||
`Average <https://en.wikipedia.org/wiki/Average>`_ duration value
|
||||
`Average`_ duration value
|
||||
|
||||
**Success**
|
||||
Percent of successful runs. This is how many percent of this action runs
|
||||
(number of runs is given in *Count* column) were successful.
|
||||
|
||||
**Count**
|
||||
Number of actually run atomic actions. This can differ from `iterations count <#iterations>`_
|
||||
because some atomic actions do not start if some exception is raised
|
||||
before in the workload runtime (for example in previous atomic action).
|
||||
Number of actually run atomic actions. This can differ from
|
||||
`iterations count <#iterations>`_ because some atomic actions do not start if
|
||||
some exception is raised before in the workload runtime (for example in
|
||||
previous atomic action).
|
||||
|
||||
StackedArea with durations per iteration
|
||||
++++++++++++++++++++++++++++++++++++++++
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This chart shows `load_duration <#load-duration>`_ and `idle_duration <#id5>`_ values per iteration.
|
||||
If there is only one iteration, then chart is useless so it is hidden.
|
||||
This chart shows `load_duration <#load-duration>`_ and `idle_duration <#id5>`_
|
||||
values per iteration. If there is only one iteration, then chart is useless so
|
||||
it is hidden.
|
||||
|
||||
Idle duration
|
||||
+++++++++++++
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
Sometimes workload does nothing for some reason (waiting for something or just making a dummy load).
|
||||
This is achieved by calling *time.sleep()* and spent time is called *idle duration*.
|
||||
Sometimes workload does nothing for some reason (waiting for something or just
|
||||
making a dummy load). This is achieved by calling *time.sleep()* and spent time
|
||||
is called *idle duration*.
|
||||
|
||||
Load Profile
|
||||
~~~~~~~~~~~~
|
||||
++++++++++++
|
||||
|
||||
`Load profile <https://en.wikipedia.org/wiki/Load_profile>`_ chart shows number
|
||||
of iterations running in parallel for each workload moment
|
||||
`Load profile`_ chart shows number of iterations running in parallel for each
|
||||
workload moment:
|
||||
|
||||
.. image:: ./images/Report-Task-Load-profile.png
|
||||
.. image:: ../images/Report-Task-Load-profile.png
|
||||
|
||||
Distribution
|
||||
~~~~~~~~~~~~
|
||||
++++++++++++
|
||||
|
||||
Pie chart shows percent of successful and failed `iterations <#iterations>`_.
|
||||
|
||||
Histogram shows durations distribution with the following
|
||||
`methods <https://en.wikipedia.org/wiki/Histogram>`_ (selected in dropdown list):
|
||||
**Square Root Choice**, **Sturges Formula**, **Rise Rule**
|
||||
Histogram shows durations distribution with the following `methods`_ (selected
|
||||
in dropdown list): **Square Root Choice**, **Sturges Formula**, **Rise Rule**
|
||||
|
||||
.. image:: ./images/Report-Task-Distribution.png
|
||||
.. image:: ../images/Report-Task-Distribution.png
|
||||
|
||||
Tab «Details»
|
||||
-------------
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Atomic Action Durations
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
+++++++++++++++++++++++
|
||||
|
||||
There is a StackedArea chart that shows atomic actions durations per iteration.
|
||||
If there is only one iteration, then chart is useless so it is hidden.
|
||||
|
||||
.. image:: ./images/Report-Task-Actions-durations.png
|
||||
.. image:: ../images/Report-Task-Actions-durations.png
|
||||
|
||||
Distribution
|
||||
~~~~~~~~~~~~
|
||||
++++++++++++
|
||||
|
||||
`Distribution <#distribution>`_ for atomic actions durations
|
||||
|
||||
Tab «Scenario Data»
|
||||
-------------------
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This tab only appears if workload provides some custom output via method
|
||||
*Scenario.add_output()*.
|
||||
|
||||
Aggregated
|
||||
~~~~~~~~~~
|
||||
++++++++++
|
||||
|
||||
This shows charts with data aggregated from all iterations.
|
||||
This means that each X axis point represents an iteration, so each iteration
|
||||
provided some values that are aggregated into charts or tables.
|
||||
|
||||
.. image:: ./images/Report-Task-Scenario-Data-Aggregated.png
|
||||
.. image:: ../images/Report-Task-Scenario-Data-Aggregated.png
|
||||
|
||||
Per iteration
|
||||
~~~~~~~~~~~~~
|
||||
+++++++++++++
|
||||
|
||||
Each iteration can create its own, complete charts and tables.
|
||||
|
||||
.. image:: ./images/Report-Task-Scenario-Data-Per-iteration.png
|
||||
.. image:: ../images/Report-Task-Scenario-Data-Per-iteration.png
|
||||
|
||||
Tab «Failures»
|
||||
--------------
|
||||
++++++++++++++
|
||||
|
||||
Complete information about exceptions raised during the workload run
|
||||
|
||||
@ -271,26 +282,27 @@ Complete information about exceptions raised during the workload run
|
||||
|
||||
Click on a row expands it with exception traceback.
|
||||
|
||||
.. image:: ./images/Report-Task-Failures.png
|
||||
.. image:: ../images/Report-Task-Failures.png
|
||||
|
||||
Tab «Input Task»
|
||||
----------------
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
This shows JSON for input file which can be used to run current workload.
|
||||
|
||||
.. image:: ./images/Report-Task-Subtask-configuration.png
|
||||
.. image:: ../images/Report-Task-Subtask-configuration.png
|
||||
|
||||
Trends Report
|
||||
=============
|
||||
-------------
|
||||
|
||||
If same workload is run several times, some results of these runs can be compared.
|
||||
Compared metrics are ssuccess rate (percent of successful iterations) and statistics for durations.
|
||||
If same workload is run several times, some results of these runs can be
|
||||
compared. Compared metrics are ssuccess rate (percent of successful iterations)
|
||||
and statistics for durations.
|
||||
|
||||
How to generate trends report
|
||||
-----------------------------
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Use command *rally task trends* with given tasks UUIDs and/or tasks results JSON files
|
||||
and the name of desired output file.
|
||||
Use command *rally task trends* with given tasks UUIDs and/or tasks results
|
||||
JSON files and the name of desired output file.
|
||||
|
||||
Example:
|
||||
|
||||
@ -299,51 +311,68 @@ Example:
|
||||
$ rally task trends --tasks 6f63d9ec-eecd-4696-8e9c-2ba065c68535 a5737eba-a204-43d6-a262-d5ea4b0065da --out trends.html
|
||||
|
||||
What is an order of workload runs?
|
||||
----------------------------------
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Workload run number in shown on charts X axis, the order of runs is exactly as
|
||||
it comes from tasks data in the moment of report generation.
|
||||
|
||||
Trends overview
|
||||
---------------
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
.. image:: ./images/Report-Trends-Overview.png
|
||||
.. image:: ../images/Report-Trends-Overview.png
|
||||
|
||||
If workload has been actually run only once
|
||||
-------------------------------------------
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
That is obvious that it is not possible to have trend for a single value.
|
||||
There should be at least two workload runs to make results comparison possible.
|
||||
So in this case there is only a help message displayed.
|
||||
|
||||
.. image:: ./images/Report-Trends-single-run.png
|
||||
.. image:: ../images/Report-Trends-single-run.png
|
||||
|
||||
Tab «Total»
|
||||
-----------
|
||||
~~~~~~~~~~~
|
||||
|
||||
Total durations
|
||||
~~~~~~~~~~~~~~~
|
||||
+++++++++++++++
|
||||
|
||||
Shows workload `load_duration <#load-duration>`_ statistics trends.
|
||||
|
||||
Total success rate
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
++++++++++++++++++
|
||||
|
||||
Shows trends for percent of successful iterations
|
||||
|
||||
.. image:: ./images/Report-Trends-Total.png
|
||||
.. image:: ../images/Report-Trends-Total.png
|
||||
|
||||
Tab «Atomic actions»
|
||||
--------------------
|
||||
++++++++++++++++++++
|
||||
|
||||
Statistics trends for atomic actions durations.
|
||||
Charts are same as for total durations.
|
||||
|
||||
.. image:: ./images/Report-Trends-Atomic-actions.png
|
||||
.. image:: ../images/Report-Trends-Atomic-actions.png
|
||||
|
||||
Tab «Configuration»
|
||||
-------------------
|
||||
+++++++++++++++++++
|
||||
|
||||
Here is a configuration JSON for current workload.
|
||||
|
||||
.. image:: ./images/Report-Trends-Configuration.png
|
||||
.. image:: ../images/Report-Trends-Configuration.png
|
||||
|
||||
CLI References
|
||||
==============
|
||||
|
||||
For more information regarding Rally Task Component CLI please proceed
|
||||
to `CLI reference <../cli/cli_reference.html#category-task>`_
|
||||
|
||||
.. references:
|
||||
|
||||
.. _SLA: https://en.wikipedia.org/wiki/Service-level_agreement
|
||||
.. _Minimal: https://en.wikipedia.org/wiki/Maxima_and_minima
|
||||
.. _Median: https://en.wikipedia.org/wiki/Median
|
||||
.. _Percentile: https://en.wikipedia.org/wiki/Percentile
|
||||
.. _Maximal: https://en.wikipedia.org/wiki/Maxima_and_minima
|
||||
.. _Average: https://en.wikipedia.org/wiki/Average
|
||||
.. _Load profile: https://en.wikipedia.org/wiki/Load_profile
|
||||
.. _methods: https://en.wikipedia.org/wiki/Histogram
|
||||
38
doc/source/components/verification.rst
Normal file
38
doc/source/components/verification.rst
Normal file
@ -0,0 +1,38 @@
|
||||
..
|
||||
Copyright 2015 Mirantis Inc. All Rights Reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
not use this file except in compliance with the License. You may obtain
|
||||
a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
License for the specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
.. contents::
|
||||
:depth: 2
|
||||
:local:
|
||||
|
||||
======================
|
||||
Verification Component
|
||||
======================
|
||||
|
||||
This section describes Rally Verification Component.
|
||||
|
||||
HTML Reports
|
||||
============
|
||||
|
||||
Right now this section is referencing to
|
||||
:ref:`Rally step-by-step: verify <tutorial_step_10_verifying_cloud_via_tempest>` -
|
||||
documentation will be changed in future, once this component refactoring will
|
||||
be completed.
|
||||
|
||||
CLI References
|
||||
==============
|
||||
|
||||
For more information regarding Rally Task Component CLI please proceed
|
||||
to `CLI reference <../cli/cli_reference.html#category-verify>`_
|
||||
@ -40,7 +40,8 @@ Contents
|
||||
install_and_upgrade/index
|
||||
quick_start/index
|
||||
cli/cli_reference
|
||||
reports
|
||||
components/task
|
||||
components/verification
|
||||
plugins
|
||||
plugin/plugin_reference
|
||||
contribute
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user