Hook results order can differ depending on which hook has
finished first, so functional test can fail sometimes.
This is fixed by sorting results.
Change-Id: Ie6dc1d85e6107dddaa464019b53465c287d0f345
This adds support of hooks output (both "additive"
and "complete") to HTML report.
Tab "Hooks" with output charts appears in case if
hook has saved some output.
Change-Id: I9f14cec23da51ff1603d04338b8f95e6ae2a6801
- Triggers should care about triggering(launching) hooks, not only about
checking when hooks should be launched. This logic is moved from
HookExecutor to Triggers
- Remove "n/a" status of Hooks. There is no possible situation when such
status can be configured. Also, "success" status is used now as default
status.
- `sys.call` saves output for success call too.
- `get_configured_event_type` method of Trigger is renamed to
`get_listening_event`.
- Task Engine should not launch hook executor and timers if hooks section
is not configured
- HookExecutor doesn't create and start timer thread if "time" unit is not
configured in any hook.
- Results of Hooks relate to each particular trigger, so they are stored in
trigger object and HookExecutor obtains results from there.
Co-Authored-By: Andrey Kurilin <andr.kurilin@gmail.com>
Change-Id: I2ce6132e101790af75ed0899bc4d5e3ddb00d0d7
This adds SLA plugin that finds minimum and maximum duration of
iterations completed without errors during Rally task execution.
Assuming that minimum duration is 100%, it calculates
performance degradation against maximum duration.
Example config:
sla:
performance_degradation:
max_degradation: 75
Spec: sla_pd_plugin
Change-Id: Ieedba7be72364f5599a3c0cf79f5f494a7391ea0
New HTML report for statistics trends for given tasks
which is generated by cli command "rally task trends".
Blueprint: trends-report
Change-Id: I0805058e8bd225796b02516fad094d73037d5495
This is the first task exporter plugin. It used to export rally task
results to the file in json format.
Change-Id: I25117933ac0881453a001d96600d4c95e6064fff
The patch introduces Workload model
instead of regular dict that was used to describe
scenario configuration in TaskEngine.
As a side effect it allows to fix subtask representaion
in generated reports and other rally outputs.
Change-Id: I62148707c89f575f83da11e34d0d193e716e8249
This switches keystone scenarios to using the new consistent random
name generator.
Implements blueprint: consistent-resource-names
Change-Id: Ie753b6c6b570ce40bd17df545df4a2cde2c4bc50
Task and SubTask classes are introduced that
incapsulate differences in formats of task
configuration and perform corresponding conversion.
Change-Id: I93ccdcd51296ace64c1fb99059080d47e1853d93
When run with the --soft key, the 'rally task abort' command should
wait until the currently running benchmark is finished, otherwise
the command interrupts benchmarking immediately after running
scenario iterations are finished.
Add Task get_status(uuid) method to db sqlalchemy api.
NOTE: db schema is updated with new Task statuses. You should execute
`rally-manage db recreate` to fix existing databases.
Co-Authored-By: Mikhail Dubov <mdubov@mirantis.com>
Co-Authored-By: Andrey Kurilin <andr.kurilin@gmail.com>
Co-Authored-By: Roman Vasilets <rvasilets@mirantis.com>
Change-Id: Ia07d165037eb98a9542ffdc91b7279ef3c3d496f
This allows users to feed reports to tools such as Jenkins.
Change-Id: I55f5fa282af8a8130be5c07bcdf8030fc591f6b7
Implements: blueprint task-report-junit
There is module 'cmd' in standard library, and unittest's discover
does modify sys.path when running tests. This cause errors when running
unit tests in some environments (E.g. pycharm)
Change-Id: I4fd2d271da1b7fd300fbbeb6107aa958e716fd2b
Closes-Bug: 1457162
Changes:
- list() declarations replaced with []
- dict() declarations replaced with {}
- Added N351 to hacking for literal rule along with checks, test hacking, and to readme
- Minor fix to incorrect path in hacking guidelines pointing to test_hacking.py
Change-Id: Ie7b9201d61b6f2024217b6f4117605eb52fa916a
Closes-Bug: #1437515
"aborted" flag will be used for saving information about situation when
task was stopped due to "abort-on-sla" option.
"broken" flag will be used for saving information about situtaion when
something gone wrong in context or runner.
Also, this patch removes two unstable scenarios.
Change-Id: I9e0c16c48af3643b15c8f5a241535bfb4540332c
This option prints the list of UUIDs only.
The motivation for this option is scripting, for example:
for x in $(rally task list --status passed --uuids-only); \
do \
rally task report --tasks $x --out $x.html; \
done
Change-Id: Ia967071bc3e4454116adbc59a790ab684efff15c
Signed-off-by: Andrew McDermott <andrew.mcdermott@linaro.org>
Rally code is python 3.4 compatible(required several small fixes, see a list
below for more details), so we can mention this great information in
setup.cfg file.
Fixed py34 incompatibe issues:
- usage of OrderedDict for storing task results. Order of such results as
AtomicActions is important.
- printed sla results are sorted by criterea name, so the results are equal
in all envs
Change-Id: I5f031170218ef4c2daf0ce0c2c391822218643dd
The reason why we need to refactor constant runner are as follows:
1) using processes to genereate load is to expensive
The optimal way to do thing is to create N processes
(where N is amount of cpu) and use threads.
2) using multiprocessing.Pool() doesn't allow us to
make proper timeouts
3) using multprocessing.Pool() doesn't allow us to
stop imidiatelly load (when we run runner.abort())
What has also been done in this patch:
1) Unification of common part between rps and new constant
runner
2) Using RAMInt for iterations counting in the RPS runner as well
3) Added a "sleep" parameter in Dummy.dummy_exception. This will allow us
a better functional testing for the "--abort-on-sla-failure" feature.
4) Minor fix in SLA
ToDo in next patches:
* Implement timeouts
* Work on the constant for duration runner
Co-Authored-By: Boris Pavlovic <boris@pavlovic.me>
Co-Authored-By: Mikhail Dubov <mdubov@mirantis.com>
Change-Id: I49bcc8694e10b3bbb8dcf5c9fe52e29e48a985f3
* Change the api of the SLA classes:
- add_iteration() - processes a single iteration result from the queue
- result() - return the SLAResult based on the data processed so far.
It now also returns a successful result in case there was no iterations
data.
* Add a new SLAChecker class:
- add_iteration() - processes a single iteration result with different SLAs
- results() - return cumulative SLA results for all SLAs
* Change the benchmark engine so that it supports scenario runner aborts
on SLA failure
* Support the "abort on SLA failure" feature in the CLI:
rally task start ... --abort-on-sla-failure
* Modify the SLA detailed messages (cleaner text and less decimal places)
* Update unit and functional tests correspondingly. Also remove the usage of the
deprecated "max_failure_percent" SLA from the functional tests.
Change-Id: I91894a81649815428fd1ac7afcfce9cf47160fc9
In case an input scenario had no atomic actions, the "--iterations-data"
table contained "n/a" values. Here we fix this and enhance the corresponding
functional tests.
Change-Id: Ic4d3809f64bcfcbce804023de7ce0c2d1f6e965f
Closes-Bug: 1420703
Add alias "rally task use" for "rally use task",
"rally deployment use" for "rally use deployment"
and "rally verify use" for "rally use verification"
Change-Id: I6e63f4b60e1faf794f050f98be127744fe8011a4
Implement task templates based on jinja2.
This allow us to pass as a task jinja2 template
and it's argument via arguments --task-args and
--task-args-file that should be dict in JSON or YAML
presentations.
So now command looks like:
rally task start <file> --task-args <template-args-json-or-yaml> \
--task-args-file <file-with-args-in-json-yaml>
If both --task-args and --task-args-file then file dict is updated
by task args file.
Extend rally CI performance job. Now we can set template args
via file with name: ${TASK}_args.yaml
Bonus:
* Better message on InvalidTask format
* Remove redudant catch of "keyboardinterrupt"
it should be implement in different way.
* Replace ' -> " in rally.cmd.commands.task
and tests.unit.cmd.commands.task
* Imporve a bit CLI messages on rally task start
* Remove old plot2html command (it's enough deprecated)
* Improve test coverage of rally/cmd/commands/task
* Fix rally/cmd/commands/validate return 1 if bad format
* Write errors to stderr (in whole cmd/commands/task.py)
Change-Id: I7dadf2986bb10407865bc73bb2fb8c96a5162d9a
Added ability to build one report from
results of multiple tasks. Tasks may be passed as UUID or as previously
saved json results. Also it may be mixed list of UUIDs and file pathes.
Closes bug 1406585
Change-Id: I11f2ca3ee6b868b79f91a4fa95e6a9ea918a79b0
In Rally, we have functional tests that run different CLI
commands and check that the CLI output contains all the expected
data.
We need to ensure that every single subcommand of “rally task”,
“rally deployment” etc. (for example, “rally task start”, “rally
task report”) is covered in these functional test with all possible
parameters usage (e.g. the “--deployment” parameter in “rally task
start --deployment <name>” didn’t get tested, which has lead to
errors recently).
Change-Id: I654ad4317b2d061dd899f5fdfb028cda50af9b8b
In some common cases (like detecting absense of requested object),
CLI shows exception traceback instead of just a simple message
that "<something> is not found".
This traceback is redundant and confusing users - they think that
rally went into trouble, but all that actually happened is wrong
uuid specified.
The traceback must be shown only with `--debug' option explicitly
specified.
The unit test and functional test are both provided.
Change-Id: I32811974d88f414c4508dfed19a7c5dc5e80fd98
Closes-Bug: #1392354
The overview page shows all scenarios and their summarized results
in single table, so it is easier to understand the whole result
and compare specific scenarios.
Also, there are some minor features and fixes in html report.
Changes:
* Benchmark overview page
* Task input file is available now
* Scenario full duration is added to overview
* Scenario duration is renamed to `Load duration'
* Overview is sortable
* Fix: control from browser history: back/forward buttons now work
* Fix: durations are reset for iterations with errors
* AngularJS version is updated to release 1.3.3
* task sla_check output: column `success' with options True/False
is renamed to `status' with options PASS/FAIL
Change-Id: I0eb7af01432c9c10e4ec55720bb53417478a5789
This patch adds UI templates directory and utils,
which serve and unify HTML generation.
Also, there are some fixes and improvements for HTML task report.
In this patch:
* Base mako templates (package rally.ui)
* Rework tests/ci/rally-gate templates
in order to use base template with generic header and styles
* Show scenario errors (if any) in task report tab
* Show scenario output (if any) in task report tab
* Show SLA data in Overview tab
* Show total scenario duration value (after the scenario name, above tabs)
* If got some iteration error, save exception class name in the database
instead of its repr()
* Prevent layout from breaking and show proper message if JS libs
can not be loaded for some reason
* Fix bug 1387661 - the cause of the bug is wrong input json data,
generated by plot.py. This happens when some atomic actions data
missed (which is a result of scenario errors) - and we have different
atomic actions sets between iterations. The fix saves atomic actions
integrity by adding missed atomic actions (with 0 value).
* Fix: if unexistend task uuid is specified in `task report' command,
then proper exception is raised
Closes-Bug: 1387661
Change-Id: I4bcbf86e6fad844e6752306eb6c1ccfefa6c6909
CLI command `task plot2html' is renamed to `task report',
however old name is still available for compatibility.
HTML output of benchmark results is reworked:
* scenario select dropdown is changed to aside menu,
with scenarios methods grouped by classes
* initially select scenario from URL hash
* scenario results are divided into tabs
* new layout and styling, from bootstrap framework
* jQuery is removed
* AngularJS is used for rendering all the data
Change-Id: I00db7016900952f397681045ff99d07a0714fafe
Find and show argument name instead of showing argument of 'dest'
Add functional test for this
Remove test_cli_utils.py -> utils.py because this actually utils,
not tests.
Change-Id: Icc138ecf40b4929826efcad52c135c8809348d20
Closes-Bug: #1380562
