137 Commits

Author SHA1 Message Date
Jenkins
e2f30013af Merge "Add task command rally task import" 2017-05-31 15:51:36 +00:00
msimonin
755af67a02 OSprofiler support
The OSprofiler is a distributed trace toolkit library.
It helps to trace internal calls of Openstack services including RPC, DB
and WSGI.

This patch integrates OSprofiler in Rally. Rally can trigger the
profiling on a per-iteration basis. To do so a secret key
(profiler_hmac_key) is stored alongside the credentials and used to
initialize the profiler in the constructor of the scenarios. A
configuration parameter (enable_profiler) can disabled the profiling.

Note that in this patch we don't embed the full osprofiler report but
only a trace id. This trace id can be used to retrieve the full trace
from the osprofiler tool later.

Change-Id: I7602856d094e073fde80d287b4d92b5750aacc3c
Co-Authored-By: rcherrueau <Ronan-Alexandre.Cherrueau@inria.fr>
Implements: spec osprofiler
2017-05-27 15:09:42 +02:00
chenhb
7dadd45501 Add task command rally task import
This command could be used to import json results
of a test into rally database.
example:
rally task import --file <json_file> [--deployment <uuid>] [--tag <tag>]

Change-Id: Ie2db6ef19d73ca49e0bf0a4e888f65c07cec3a6c
2017-05-16 10:12:45 +00:00
astaroverov
f75ba846bb Remove OPTs from 'utils.py'
Change-Id: I686887ecb75078c80bb060fcb07718c223eeccc2
2017-04-10 16:36:04 +03:00
Andrey Kurilin
f488f8652c [verify] Extend rerun cmd with tag and concurrency args
CLI method rerun should look as similar as `rally verify start`

Change-Id: Iad93e344c8f9d9d27a972ba5b7314788beb91580
2017-03-16 13:36:57 +02:00
Andrey Kurilin
5a465f9057 [verify] Extend rerun cmd with detailed and no-use args
CLI method rerun should look as similar as `rally verify start`

Change-Id: I8489b2abb48f581eea80fcdc5849cc00b4545ba1
2017-03-14 20:11:28 +02:00
Anton Studenov
24fdc1a6a2 [CI] Fix image regexp and version
Default image in devstack was changed recently from uec to qcow2.
All rally jobs except rally-mos now need to be updated accordingly.

devstack changes:
* Id65ebae73b28da7185cb349b714b659af51ef77f
* I5c154ec25555e768954538fc22b4f5d5975b2deb


Change-Id: Ia4d4afb661d8bcc7c38b0a16b508294f789eef14
2017-02-16 14:29:06 +03:00
Anton Studenov
5ee6a18df2 [CLI] Allow changing deployment config on recreate
* Added --filename option to deployment recreate that
  will update configuration from provided file.

Change-Id: Icefca03fd86ab8977d0910bf192187c713cb355f
2017-01-25 21:06:06 +03:00
Andrey Kurilin
6267f58df7 [verification] Introduce rally verify show-verifier command
We have a lot of information about verifiers and main horizontal table from
``rally verify list-verifiers`` command could not include all of it.

Co-Authored-By: Yaroslav Lobankov <ylobankov@mirantis.com>

Change-Id: If425e9c373483fa092ebbcefdaf9a46173275c8a
2017-01-24 14:51:22 +04:00
Yaroslav Lobankov
088a5b16d5 [Verify] Add --detailed flag to rally verify start cmd
This flag allows us to see test failures after the verification finished.

Closes-Bug: #1655098

Change-Id: Ief2d7c0366eded9a8ac7c07df16d6943df9243b8
2017-01-23 13:29:39 +00:00
Yaroslav Lobankov
1e5aa0df77 [Verify] Add tags to verifications
This patch allows us to add some tags to verifications when starting them.
When we have a lot of verifications, it is not a simple task to find the
desired verification from the list. But now we can filter verifications by
tags and it should help us.

Change-Id: I7c6c6a54f31100100b67a750d0381d88fa98dc96
2017-01-23 16:40:46 +04:00
Yaroslav Lobankov
1f9e6ceb2c [Verify] The latest cleanup before new release
1. Small bugs were fixed.
2. Missing doc strings were added.
3. Now verifier cannot have 'configuring' and 'configured' statuses.
   It was done to support work with a few deployments.

Change-Id: I064e863f7597c015e8a8f7c64a4d3049d6baac95
2017-01-21 15:43:28 +04:00
Yaroslav Lobankov
13c90c8047 [Verify] Improve names of args for configure_verifier method
The following changes were done:

    recreate -> reconfigure
    replace-by -> override
    add-options -> extend

Also, some printed messages in CLI interface were improved.

Change-Id: I47bfe88fabc22992b2bcde4c7ea03961c34bc1b6
2017-01-13 14:25:54 +00:00
Jenkins
5835c8cfd7 Merge "[verification] add force flag to configure verifier" 2017-01-10 21:12:32 +00:00
Andrey Kurilin
03b606f6b4 [plugins][cli] Include plugin base in plugins list
Two or more plugins can have the same name if they have different plugin
base classes. In the table we did not mention relations to base classes, so
it is hard to identify right plugins.

Change-Id: I31e5633af9a34d59216c281113e32c1839a6172f
2017-01-10 19:00:57 +02:00
Andrey Kurilin
73e9d68507 Make verification reporter pluggable
Some time ago we added new entity "exporters". For task we created a command
"rally task export" which allows to export task results to external systems.
Exporters can "easily" extende by plugins.

In case of verification component, proper command wasn't created yet. While I
thing idea of exporters is good enough, implementation can be improved.
To simplify usage, entity "exporters" was renamed to "reporters" and
integrated with "rally verify report" command. Generation of regular rally
reports (like html, json) is done in the same way as in plugabble reporters.

Proposed interface:

  rally verify report --uuid <uuid-1> [<uuid-2>...<uuid-n>] \
      --type <reporter-name> --to <destination>

Examples of usage:

  rally verify report --uuid some-uuid --type html --to /path/to/save

  # such exporter is not implemented yet, but we expect it soon
  rally verify report --uuids some-uuid --type elasticsearch \
      --to https://username@passwd:example.com

Change-Id: I4fb38984a73f92503bf2988e509477b10b308cac
2017-01-10 19:00:55 +02:00
Andrey Kurilin
95a4535c34 [verification] add force flag to configure verifier
There is a possible case when something went wrong and verifier stuck in
configuring state. In such situation there is not way to do un-stuck it,
just full recreation of verifier.

New flag --force should help in such situations.

Change-Id: Icf41f53ec3f6a6a8952a2f5922e1f396ca705ffc
2017-01-10 18:28:40 +02:00
Yaroslav Lobankov
ef2cd31a65 [Verify] Introduce rally verify rerun cmd
This command can re-run either all tests or only failed ones
from the specified verification.

Change-Id: I89ebfb3fa3ab87f66cdffdf02d204efa0ac96fee
2017-01-10 13:28:14 +04:00
Jenkins
e54d78d486 Merge "Save raw task results in chunks" 2016-12-29 15:34:19 +00:00
Jenkins
3ea91105d6 Merge "Change "a SQL" to "an SQL"" 2016-12-29 13:37:08 +00:00
Anton Studenov
1587efbcc1 Save raw task results in chunks
* Split task results into chunks with size
  configured by raw_result_chunk_size parameter
* Removed load_finished event as not used anywhere

Change-Id: I09841d02c60ab66eebd851bdf1c0a6d5f9e5a7be
2016-12-28 21:07:24 +03:00
Jenkins
968133f7b1 Merge "[verification] Add CLI interface" 2016-12-28 13:13:59 +00:00
Andrey Kurilin
f51604b264 [verification] Add CLI interface
Co-Authored-By: Yaroslav Lobankov <ylobankov@mirantis.com>

Change-Id: Ibc66ca3cdc6715363e4490d052398030b421c15a
2016-12-28 13:42:10 +04:00
Cady_Chen
9c3ae9fc34 Change "a SQL" to "an SQL"
According to the word choice convention in
http://docs.openstack.org/contributor-guide/writing-style/word-choice.html
We should use "an SQL" instead of "a SQL".

Change-Id: Icd42fc4e3d078725adca2bf464164111bfae1a2d
2016-12-28 14:37:01 +08:00
haishi
f2fc3015be [cli]remove deprecated rally show command
rally show command deprecated from Rally 0.2.0
relatednote:
http://rally.readthedocs.io/en/0.7.0/cli/cli_reference.html#category-show

Change-Id: I54b907d0b6a88a9ed42847ade2f7ac091241837c
2016-12-27 21:39:55 +08:00
Andrey Kurilin
837f967068 [verification] Remove outdated stuff
We are planning to rewrite the whole Verification component. It will be done
from scratch. To reduce diff in following patches, let's remove all redundant
"outdated" code now.

Co-Authored-By: Yaroslav Lobankov <ylobankov@mirantis.com>

Change-Id: I64dd3851e381324a8d5274029690b5cfd2333911
2016-12-20 13:18:47 +04:00
Andrey Kurilin
f7929d0f6d [CLI] group commands in help message
In subcommand(for example in verification) we have several groups of
methods(managements, launchers, results). It would be nice to split
these groups in help message.

Before:

   compare           Deprecated. Use `rally verify results' instead.
   detailed          Display results table of a verification with detailed errors.
   discover          Show a list of discovered tests.
   genconfig         Generate Tempest configuration file.
   import            Import Tempest tests results into the Rally database.
   install           Install Tempest.
   installplugin     Install Tempest plugin.
   list              List verification runs.
   listplugins       List all installed Tempest plugins.
   reinstall         Uninstall Tempest and install again.
   results           Display results of verifications.
   show              Display results table of a verification.
   showconfig        Show Tempest configuration file.
   start             Start verification (run Tempest tests).
   uninstall         Remove the deployment's local Tempest installation.
   uninstallplugin   Uninstall Tempest plugin.
   use               Set active verification.

After:

   genconfig         Generate Tempest configuration file.
   install           Install Tempest.
   installplugin     Install Tempest plugin.
   listplugins       List all installed Tempest plugins.
   reinstall         Uninstall Tempest and install again.
   showconfig        Show Tempest configuration file.
   uninstall         Remove the deployment's local Tempest installation.
   uninstallplugin   Uninstall Tempest plugin.

   discover          Show a list of discovered tests.
   start             Start verification (run Tempest tests).

   compare           Deprecated. Use `rally verify results' instead.
   detailed          Display results table of a verification with detailed errors.
   import-results    Import Tempest tests results into the Rally database.
   list              List verification runs.
   results           Display results of verifications.
   show              Display results table of a verification.
   use               Set active verification.

Also this change transforms all _ to - in cli methods names.

Change-Id: I292e71d159ee35e933119f7fb57209f071aa37d4
2016-11-28 22:58:14 +03:00
Oleksandr Savatieiev
70901577ab [Reports] Significant improvements in verification report
* ability to show several verifications results (this
   deprecates command `rally verify compare')
 * command `rally verify compare' is removed - new report
   compares results in better way
 * new, unified, AngularJS/Jinja2-based template
 * new verifications results processing
 * new module rally.ui.report which cares about
   report generation (this is a place where code from
   rally.task.processing.plot should be also moved)

Examples:

  rally verify results\
    --uuid <uuid> --html > single_verification_result.html

  rally verify results\
    --uuid <uuid1> <uuid2> <uuid3> --html > compare_3.html

Co-Authored-By: Oleksandr Savatieiev <osavatieiev@mirantis.com>
Co-Authored-By: Alexander Maretskiy <amaretskiy@mirantis.com>
Change-Id: I942e0d9bf2094f3254dbeccbaa76dbbc3a3ca40e
2016-10-18 11:47:04 +03:00
Michal Gershenzon
546bed4c6d Add option to create mistral execution from workbook
Change-Id: I637ff8cd73c788c519a1af5962fd923a6a12d0da
2016-10-12 17:21:40 +00:00
Spyros Trigazis
316a14c633 [magnum] Rename to cluster and cluster_template
Magnum renamed baymodels and bays to cluster_templates and clusters
respectively [1].

[1] https://blueprints.launchpad.net/magnum/+spec/rename-bay-to-cluster

Change-Id: I4390406d45a705cf93f8eb6522ebe4a65e468d2c
Partially-Implements: blueprint benchmark-scenarios-for-magnum
2016-09-27 11:23:12 +00:00
Jenkins
b6ef3f1305 Merge "Update config sample file with latest config options" 2016-09-23 16:58:36 +00:00
ChangBo Guo(gcb)
c2d8b93be4 Update config sample file with latest config options
Some config options have changed their help text like 'sqlite_db',
so run 'tox -e genconfig' to generate latest config sample file.

Change-Id: Ifad41b2e1203308c02392ad63e3e90853dbbca68
2016-09-23 08:32:30 +00:00
Winnie Tsang
c221c348de [Magnum] Scenario to create a Magnum bay
Measure time to create a bay

In this scenario, a single bay is created.  This can be used to establish
the baseline in a test environment.

The job yaml file describes the attribute of the bay to be created in
the scenario, such as node count.  The baymodel should already be
created by the context: Kubernetes, Swarm, Mesos.

Change-Id: I182abf34212d693d2bb246d195a5daba61e0efe4
Co-Authored-By: Spyros Trigazis <strigazi@gmail.com>
Co-Authored-By: Ton Ngo <ton@us.ibm.com>
Partially-Implements: blueprint benchmark-scenarios-for-magnum
2016-09-22 16:49:06 +02:00
Jenkins
bfc657b7d0 Merge "[Verify] Adding 'add-options' arg to rally verify genconfig cmd" 2016-09-01 14:29:11 +00:00
Jenkins
cb191759e8 Merge "Add watcher audit template context and 2 scenarios" 2016-08-29 10:32:58 +00:00
Alexander Chadin
27604462c4 Add watcher audit template context and 2 scenarios
This patch set adds audit template context with two scenarios:
list_audit_templates
create_audit_and_delete

Change-Id: I2d3adef7893fbef1b90e69f56818fbe4d3ce4366
2016-08-25 00:23:13 +03:00
Yaroslav Lobankov
2d9ceb5a34 [Verify] Adding 'add-options' arg to rally verify genconfig cmd
Tempest supports a mechanism that allows us to plug in various tests as
a Tempest plugin. Each plugin needs to be configured in the proper way.
Rally cannot configure all plugins on the planet. We should add some
option that will allow us to add extra configuration for plugins to
Tempest config file.

For example, if we want to run Ceilometer tests, the workflow will be
as follows:

$ rally verify install
$ rally verify installplugin --source https://github.com/openstack/ceilometer.git
$ rally verify genconfig --add-options /path/to/config/file/for/ceilometer/tests
$ rally verify start --regex ceilometer

where config file for Ceilometer tests is something like this:

...
[service_available]

ceilometer = True
event_enabled = True
...

Change-Id: Ida02a36e2757a742ca91919cc756846f137a23d3
2016-08-19 10:32:38 +00:00
Yaroslav Lobankov
423740ae9f [Verify] Deprecating 'tests-file' and 'xfails-file' args
This patch deprecates the following args for `rally verify start` cmd:

     Old           New
 tests-file  -> load-list
 xfails-file -> xfail-list

So now we have consistent args that allow us to manage the workflow for
a certain list of tests:

 1. Arg 'load-list' - Run tests listed in the named file
 2. Arg 'skip-list' - Skip run of tests listed in the named file
 3. Arg 'xfail-list' - Consider tests listed in the named file as
                       expected failures

Change-Id: Ida1252172c7edf7e7dd5350123a52c085867ba0f
2016-08-11 10:40:56 +00:00
Yaroslav Lobankov
72a4f74529 [Verify] Discover tests for system-wide Tempest installation
This patch allows us to discover tests for system-wide Tempest installation.

Change-Id: Icc7272f1589dcf2656f3d970859b590da64061ab
2016-08-03 23:38:41 +03:00
Yaroslav Lobankov
652a984bf4 [Verify] New command to uninstall Tempest plugins
This command allows us to uninstall a Tempest plugin.

E.g.
  $ rally verify uninstallplugin --repo-name hello-world-tempest-plugin

Also, the patch has the following changes:
1. Now Tempest is installed in the local env (without deps) when providing
   the --sistem-wide argument. It is needed to list plugins installed in the
   local env, not in Tempest virtual env.
2. Now plugins are installed in the 'plugins/system-wide' directory when
   providing the --system-wide argument. It is needed to separate plugins
   installed in the local env and in Tempest virtual env.

Change-Id: Ic30a2705edb321a8d5af70e9d2bf5d5d5d9e8781
2016-08-02 23:51:00 +03:00
Jenkins
40bb832700 Merge "[Verify] New command to list Tempest plugins" 2016-08-01 13:19:38 +00:00
Yaroslav Lobankov
e33ef2575d [Verify] New command to list Tempest plugins
This command allows us to list all Tempest plugins.

E.g.
  $ rally verify listplugins
+----------------------------+--------------------------------------------+
|            Name            |                 EntryPoint                 |
+----------------------------+--------------------------------------------+
| hello-world-tempest-plugin | hello_world_tempest_plugin.plugin:MyPlugin |
+----------------------------+--------------------------------------------+

Change-Id: Ifd53afd749ac47893b0fa11168b30d2536bf89fa
2016-07-29 18:49:53 +03:00
Yaroslav Lobankov
d936627f66 [Verify] Adding '--skip-list' arg to rally verify start cmd
Sometimes we need to run the full test suite without some tests.
Now the new '--skip-list' argument allows us to do it. In order to skip
the desired tests we just need to list the tests in some file and provide
the '--skip-list' argument with the path to the file:

    rally verify start --skip-list /path/to/the/file

Also, missing messages about incompatibility of arguments were added.

Change-Id: I882bfacd2e39bed2b54af4092752f642c964975a
2016-07-25 13:57:18 +03:00
Marcela Bonell
f6a02afcda [Verify] New command to install Tempest plugins
This command allows us to install a Tempest plugin from a repository.

E.g.
  $ rally verify installplugin --source https://github.com/MBonell/hello-world-tempest-plugin
  $ rally verify installplugin --source /home/ubuntu/my-plugin

Implements Blueprint: install-tempest-plugins

Change-Id: I03ac6063b6b9d7f860379b99f3b524adedba6df9
2016-07-13 00:24:42 +03:00
Jenkins
39dc633d41 Merge "[Verify] Add tempest configs and move to new section" 2016-06-09 17:16:36 +00:00
Ciprian Barbu
48b9975582 [Verify] Add tempest configs and move to new section
Moved the old image and role options to the [tempest] section and added a
few flavor related config options. With these new options it is possible
to configure the RAM size of the two flavors used when running tempest
and the RAM size of the orchestration instance type flavor.

These were previously hardcoded to 64M, 128M and 64M respectively which
may cause the test image to fail under certain conditions, due to lack of
memory (e.g. AArch64 VMs seem to fall into this category).

After applying this fix, the three ram sizes and all other tempest related
options can be configured in the [tempest] section of /etc/rally/rally.conf.

Closes-Bug: #1580695

Change-Id: I33b8682310a42e66b408802a0d955fc886f6c8ed
Signed-off-by: Ciprian Barbu <ciprian.barbu@gmail.com>
2016-06-09 13:19:12 +00:00
Alexander Maretskiy
b7fa1422f6 [Reports][CLI] Introduce Trends report
New HTML report for statistics trends for given tasks
which is generated by cli command "rally task trends".

Blueprint: trends-report
Change-Id: I0805058e8bd225796b02516fad094d73037d5495
2016-05-20 18:25:20 +03:00
Yaroslav Lobankov
b549a0c196 [Verify] Add 'version' arg to install/reinstall cmd
For now Rally always installs Tempest from the master branch. It is not
very convenient because Tempest may be broken in the master branch and
it would be very useful to use some commit or tag for Tempest.
So this commit adds a possibility to checkout the specified commit or
tag before Tempest installation.

Change-Id: Icfbb4221005466139c14d3f4aae34bbb3e013982
2016-05-07 10:51:34 +00:00
Yaroslav Lobankov
78ab9d22f9 [Verify] Fixing issue with discovering images
When we request a list of images to choose one of them for tests,
we should make sure all images are active, they have names and they
are PUBLIC. If images are not public, we will have failures of Tempest
tests as described in the bug.

Closes-Bug: #1564431

Change-Id: Ia2cc1586956dcf1fff4c7859faf39fdd98ba35e2
2016-04-15 13:02:59 +03:00
Roman Vasilets
e6234c3013 Add task exporter to the file system
This is the first task exporter plugin. It used to export rally task
results to the file in json format.

Change-Id: I25117933ac0881453a001d96600d4c95e6064fff
2016-03-28 17:00:52 +03:00