openstack/rally-openstack
Code Issues Proposed changes

[docs][3] Re-design docs to cover all user-groups

Browse Source 
First pack of changes in upcoming chain to redesign Rally docs.
All information related to Rally step by step and its usage
in the gates separated and refactored . Modified files fit 80
symbols margin where possible.

[TODO] continue with other parts of the docs:
       - Command Line Interface
       - Rally Task Component
       - Rally Verification Component
       - Rally Plugins, Rally Plugins Reference
       - Contribute to Rally
       - Request New Features
       - Project Info
[TODO] add 80 symbols margin check similar to what
       Performance Documentation has

Change-Id: I3dc17027a2bfa75214b960573ec8b036b1bc7bb0
This commit is contained in:
Dina Belova
2016-11-16 14:37:45 -08:00
parent ec1f9e4e01
commit 9963be7324
13 changed files with 1960 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
doc/source/index.rst
View File

@@ -38,13 +38,12 @@ Contents
overview/index
install_and_upgrade/index
tutorial
quick_start/index
cli/cli_reference
reports
plugins
plugin/plugin_reference
contribute
gates
feature_requests
project_info
release_notes

192
doc/source/quick_start/gates.rst Normal file
View File

@@ -0,0 +1,192 @@
..
Copyright 2015 Mirantis Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _gates:
Rally OpenStack Gates
=====================
Gate jobs
---------
The **OpenStack CI system** uses the so-called **"Gate jobs"** to control
merges of patches submitted for review on Gerrit. These **Gate jobs** usually
just launch a set of tests -- unit, functional, integration, style -- that
check that the proposed patch does not break the software and can be merged
into the target branch, thus providing additional guarantees for the stability
of the software.
Create a custom Rally Gate job
------------------------------
You can create a **Rally Gate job** for your project to run Rally benchmarks
against the patchsets proposed to be merged into your project.
To create a rally-gate job, you should create a **rally-jobs/** directory at
the root of your project.
As a rule, this directory contains only **{projectname}.yaml**, but more
scenarios and jobs can be added as well. This yaml file is in fact an input
Rally task file specifying benchmark scenarios that should be run in your gate
job.
To make *{projectname}.yaml* run in gates, you need to add *"rally-jobs"* to
the "jobs" section of *projects.yaml* in *openstack-infra/project-config*.
Example: Rally Gate job for Glance
----------------------------------
Let's take a look at an example for the `Glance`_ project:
Edit *jenkins/jobs/projects.yaml:*
.. parsed-literal::
- project:
name: glance
node: 'bare-precise || bare-trusty'
tarball-site: tarballs.openstack.org
doc-publisher-site: docs.openstack.org
jobs:
- python-jobs
- python-icehouse-bitrot-jobs
- python-juno-bitrot-jobs
- openstack-publish-jobs
- translation-jobs
**- rally-jobs**
Also add *gate-rally-dsvm-{projectname}* to *zuul/layout.yaml*:
.. parsed-literal::
- name: openstack/glance
template:
- name: merge-check
- name: python26-jobs
- name: python-jobs
- name: openstack-server-publish-jobs
- name: openstack-server-release-jobs
- name: periodic-icehouse
- name: periodic-juno
- name: check-requirements
- name: integrated-gate
- name: translation-jobs
- name: large-ops
- name: experimental-tripleo-jobs
check:
- check-devstack-dsvm-cells
**- gate-rally-dsvm-glance**
gate:
- gate-devstack-dsvm-cells
experimental:
- gate-grenade-dsvm-forward
To add one more scenario and job, you need to add *{scenarioname}.yaml* file
here, and *gate-rally-dsvm-{scenarioname}* to *projects.yaml*.
For example, you can add *myscenario.yaml* to *rally-jobs* directory in your
project and then edit *jenkins/jobs/projects.yaml* in this way:
.. parsed-literal::
- project:
name: glance
github-org: openstack
node: bare-precise
tarball-site: tarballs.openstack.org
doc-publisher-site: docs.openstack.org
jobs:
- python-jobs
- python-havana-bitrot-jobs
- openstack-publish-jobs
- translation-jobs
- rally-jobs
**- 'gate-rally-dsvm-{name}':
name: myscenario**
Finally, add *gate-rally-dsvm-myscenario* to *zuul/layout.yaml*:
.. parsed-literal::
- name: openstack/glance
template:
- name: python-jobs
- name: openstack-server-publish-jobs
- name: periodic-havana
- name: check-requirements
- name: integrated-gate
check:
- check-devstack-dsvm-cells
- check-tempest-dsvm-postgres-full
- gate-tempest-dsvm-large-ops
- gate-tempest-dsvm-neutron-large-ops
**- gate-rally-dsvm-myscenario**
It is also possible to arrange your input task files as templates based on
``Jinja2``. Say, you want to set the image names used throughout the
*myscenario.yaml* task file as a variable parameter. Then, replace concrete
image names in this file with a variable:
.. code-block:: yaml
...
NovaServers.boot_and_delete_server:
-
args:
image:
name: {{image_name}}
...
NovaServers.boot_and_list_server:
-
args:
image:
name: {{image_name}}
...
and create a file named *myscenario_args.yaml* that will define the parameter
values:
.. code-block:: yaml
---
image_name: "^cirros.*uec$"
this file will be automatically used by Rally to substitute the variables in
*myscenario.yaml*.
Plugins & Extras in Rally Gate jobs
-----------------------------------
Along with scenario configs in yaml, the **rally-jobs** directory can also
contain two subdirectories:
- **plugins**: :ref:`Plugins <plugins>` needed for your gate job;
- **extra**: auxiliary files like bash scripts or images.
Both subdirectories will be copied to *~/.rally/* before the job gets started.
.. references:
.. _Glance: https://wiki.openstack.org/wiki/Glance

30
doc/source/quick_start/index.rst Normal file
View File

@@ -0,0 +1,30 @@
..
Copyright 2015 Mirantis Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
===========
Quick start
===========
This section will guide you through all steps of using Rally - from
installation to its advanced usage in different use cases (including running
Rally in OpenStack CI system gates to control merges of patches submitted for
review on Gerrit code review system).
.. toctree::
:glob:
:maxdepth: 2
tutorial
gates

40
doc/source/quick_start/tutorial.rst Normal file
View File

@@ -0,0 +1,40 @@
..
Copyright 2015 Mirantis Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _tutorial:
Rally step-by-step
==================
In the following tutorial, we will guide you step-by-step through different use
cases that might occur in Rally, starting with the easy ones and moving towards
more complicated cases.
.. toctree::
:glob:
:maxdepth: 1
tutorial/step_0_installation
tutorial/step_1_setting_up_env_and_running_benchmark_from_samples
tutorial/step_2_input_task_format
tutorial/step_3_benchmarking_with_existing_users
tutorial/step_4_adding_success_criteria_for_benchmarks
tutorial/step_5_task_templates
tutorial/step_6_aborting_load_generation_on_sla_failure
tutorial/step_7_working_with_multple_openstack_clouds
tutorial/step_8_discovering_more_plugins
tutorial/step_9_deploying_openstack
tutorial/step_10_verifying_cloud_via_tempest

42
doc/source/quick_start/tutorial/step_0_installation.rst Normal file
View File

@@ -0,0 +1,42 @@
..
Copyright 2015 Mirantis Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _tutorial_step_0_installation:
Step 0. Installation
====================
The easiest way to install Rally is by running its `installation script`_:
.. code-block:: bash
wget -q -O- https://raw.githubusercontent.com/openstack/rally/master/install_rally.sh | bash
# or using curl:
curl https://raw.githubusercontent.com/openstack/rally/master/install_rally.sh | bash
If you execute the script as regular user, Rally will create a new
virtual environment in ``~/rally/`` and install in it Rally, and will
use `sqlite` as database backend. If you execute the script as root,
Rally will be installed system wide. For more installation options,
please refer to the :ref:`installation <install>` page.
**Note:** Rally requires Python version 2.7 or 3.4.
Now that you have Rally installed, you are ready to start
:ref:`benchmarking OpenStack with it <tutorial_step_1_setting_up_env_and_running_benchmark_from_samples>`!
.. references:
.. _installation script: https://raw.githubusercontent.com/openstack/rally/master/install_rally.sh

294
doc/source/quick_start/tutorial/step_1_setting_up_env_and_running_benchmark_from_samples.rst Normal file
View File

@@ -0,0 +1,294 @@
..
Copyright 2015 Mirantis Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _tutorial_step_1_setting_up_env_and_running_benchmark_from_samples:
Step 1. Setting up the environment and running a benchmark from samples
=======================================================================
.. contents::
:local:
In this demo, we will show how to perform some basic operations in Rally, such
as registering an OpenStack cloud, benchmarking it and generating benchmark
reports.
We assume that you have gone through :ref:`tutorial_step_0_installation` and
have an already existing OpenStack deployment with Keystone available at
*<KEYSTONE_AUTH_URL>*.
Registering an OpenStack deployment in Rally
--------------------------------------------
First, you have to provide Rally with an OpenStack deployment it is going to
benchmark. This should be done either through `OpenRC files`_ or through
deployment `configuration files`_. In case you already have an *OpenRC*, it is
extremely simple to register a deployment with the *deployment create* command:
.. code-block:: console
$ . openrc admin admin
$ rally deployment create --fromenv --name=existing
+--------------------------------------+----------------------------+------------+------------------+--------+
| uuid | created_at | name | status | active |
+--------------------------------------+----------------------------+------------+------------------+--------+
| 28f90d74-d940-4874-a8ee-04fda59576da | 2015-01-18 00:11:38.059983 | existing | deploy->finished | |
+--------------------------------------+----------------------------+------------+------------------+--------+
Using deployment : <Deployment UUID>
...
Alternatively, you can put the information about your cloud credentials into a
JSON configuration file (let's call it `existing.json`_). The *deployment
create* command has a slightly different syntax in this case:
.. code-block:: console
$ rally deployment create --file=existing.json --name=existing
+--------------------------------------+----------------------------+------------+------------------+--------+
| uuid | created_at | name | status | active |
+--------------------------------------+----------------------------+------------+------------------+--------+
| 28f90d74-d940-4874-a8ee-04fda59576da | 2015-01-18 00:11:38.059983 | existing | deploy->finished | |
+--------------------------------------+----------------------------+------------+------------------+--------+
Using deployment : <Deployment UUID>
...
Note the last line in the output. It says that the just created deployment is
now used by Rally; that means that all the benchmarking operations from now on
are going to be performed on this deployment. Later we will show how to switch
between different deployments.
Finally, the *deployment check* command enables you to verify that your current
deployment is healthy and ready to be benchmarked:
.. code-block:: console
$ rally deployment check
keystone endpoints are valid and following services are available:
+----------+----------------+-----------+
| services | type | status |
+----------+----------------+-----------+
| cinder | volume | Available |
| cinderv2 | volumev2 | Available |
| ec2 | ec2 | Available |
| glance | image | Available |
| heat | orchestration | Available |
| heat-cfn | cloudformation | Available |
| keystone | identity | Available |
| nova | compute | Available |
| novav21 | computev21 | Available |
| s3 | s3 | Available |
+----------+----------------+-----------+
Benchmarking
------------
Now that we have a working and registered deployment, we can start benchmarking
it. The sequence of benchmarks to be launched by Rally should be specified in a
*benchmark task configuration file* (either in *JSON* or in *YAML* format).
Let's try one of the sample benchmark tasks available in
`samples/tasks/scenarios`_, say, the one that boots and deletes multiple
servers (*samples/tasks/scenarios/nova/boot-and-delete.json*):
.. code-block:: json
{
"NovaServers.boot_and_delete_server": [
{
"args": {
"flavor": {
"name": "m1.tiny"
},
"image": {
"name": "^cirros.*uec$"
},
"force_delete": false
},
"runner": {
"type": "constant",
"times": 10,
"concurrency": 2
},
"context": {
"users": {
"tenants": 3,
"users_per_tenant": 2
}
}
}
]
}
To start a benchmark task, run the ``task start`` command (you can also add the
*-v* option to print more logging information):
.. code-block:: console
$ rally task start samples/tasks/scenarios/nova/boot-and-delete.json
--------------------------------------------------------------------------------
Preparing input task
--------------------------------------------------------------------------------
Input task is:
<Your task config here>
--------------------------------------------------------------------------------
Task 6fd9a19f-5cf8-4f76-ab72-2e34bb1d4996: started
--------------------------------------------------------------------------------
Benchmarking... This can take a while...
To track task status use:
rally task status
or
rally task detailed
--------------------------------------------------------------------------------
Task 6fd9a19f-5cf8-4f76-ab72-2e34bb1d4996: finished
--------------------------------------------------------------------------------
test scenario NovaServers.boot_and_delete_server
args position 0
args values:
{u'args': {u'flavor': {u'name': u'm1.tiny'},
u'force_delete': False,
u'image': {u'name': u'^cirros.*uec$'}},
u'context': {u'users': {u'project_domain': u'default',
u'resource_management_workers': 30,
u'tenants': 3,
u'user_domain': u'default',
u'users_per_tenant': 2}},
u'runner': {u'concurrency': 2, u'times': 10, u'type': u'constant'}}
+--------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count |
+--------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| nova.boot_server | 7.99 | 9.047 | 11.862 | 9.747 | 10.805 | 100.0% | 10 |
| nova.delete_server | 4.427 | 4.574 | 4.772 | 4.677 | 4.725 | 100.0% | 10 |
| total | 12.556 | 13.621 | 16.37 | 14.252 | 15.311 | 100.0% | 10 |
+--------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
Load duration: 70.1310448647
Full duration: 87.545541048
HINTS:
* To plot HTML graphics with this data, run:
rally task report 6fd9a19f-5cf8-4f76-ab72-2e34bb1d4996 --out output.html
* To get raw JSON output of task results, run:
rally task results 6fd9a19f-5cf8-4f76-ab72-2e34bb1d4996
Using task: 6fd9a19f-5cf8-4f76-ab72-2e34bb1d4996
Note that the Rally input task above uses *regular expressions* to specify the
image and flavor name to be used for server creation, since concrete names
might differ from installation to installation. If this benchmark task fails,
then the reason for that might a non-existing image/flavor specified in the
task. To check what images/flavors are available in the deployment you are
currently benchmarking, you might use the *rally show* command:
.. code-block:: console
$ rally show images
+--------------------------------------+-----------------------+-----------+
| UUID | Name | Size (B) |
+--------------------------------------+-----------------------+-----------+
| 8dfd6098-0c26-4cb5-8e77-1ecb2db0b8ae | CentOS 6.5 (x86_64) | 344457216 |
| 2b8d119e-9461-48fc-885b-1477abe2edc5 | CirrOS 0.3.4 (x86_64) | 13287936 |
+--------------------------------------+-----------------------+-----------+
$ rally show flavors
Flavors for user `admin` in tenant `admin`:
+----+-----------+-------+----------+-----------+-----------+
| ID | Name | vCPUs | RAM (MB) | Swap (MB) | Disk (GB) |
+----+-----------+-------+----------+-----------+-----------+
| 1 | m1.tiny | 1 | 512 | | 1 |
| 2 | m1.small | 1 | 2048 | | 20 |
| 3 | m1.medium | 2 | 4096 | | 40 |
| 4 | m1.large | 4 | 8192 | | 80 |
| 5 | m1.xlarge | 8 | 16384 | | 160 |
+----+-----------+-------+----------+-----------+-----------+
Report generation
-----------------
One of the most beautiful things in Rally is its task report generation
mechanism. It enables you to create illustrative and comprehensive HTML reports
based on the benchmarking data. To create and open at once such a report for
the last task you have launched, call:
.. code-block:: bash
rally task report --out=report1.html --open
This will produce an HTML page with the overview of all the scenarios that
you've included into the last benchmark task completed in Rally (in our case,
this is just one scenario, and we will cover the topic of multiple scenarios in
one task in
:ref:`the next step of our tutorial <tutorial_step_2_input_task_format>`):
.. image:: ../../images/Report-Overview.png
:align: center
This aggregating table shows the duration of the load produced by the
corresponding scenario (*"Load duration"*), the overall benchmark scenario
execution time, including the duration of environment preparation with contexts
(*"Full duration"*), the number of iterations of each scenario
(*"Iterations"*), the type of the load used while running the scenario
(*"Runner"*), the number of failed iterations (*"Errors"*) and finally whether
the scenario has passed certain Success Criteria (*"SLA"*) that were set up by
the user in the input configuration file (we will cover these criteria in
:ref:`one of the next steps <tutorial_step_4_adding_success_criteria_for_benchmarks>`).
By navigating in the left panel, you can switch to the detailed view of the
benchmark results for the only scenario we included into our task, namely
**NovaServers.boot_and_delete_server**:
.. image:: ../../images/Report-Scenario-Overview.png
:align: center
This page, along with the description of the success criteria used to check the
outcome of this scenario, shows more detailed information and statistics about
the duration of its iterations. Now, the *"Total durations"* table splits the
duration of our scenario into the so-called **"atomic actions"**: in our case,
the **"boot_and_delete_server"** scenario consists of two actions -
**"boot_server"** and **"delete_server"**. You can also see how the scenario
duration changed throughout its iterations in the *"Charts for the total
duration"* section. Similar charts, but with atomic actions detailed are on the
*"Details"* tab of this page:
.. image:: ../../images/Report-Scenario-Atomic.png
:align: center
Note that all the charts on the report pages are very dynamic: you can change
their contents by clicking the switches above the graph and see more
information about its single points by hovering the cursor over these points.
Take some time to play around with these graphs
and then move on to :ref:`the next step of our tutorial <tutorial_step_2_input_task_format>`.
.. references:
.. _OpenRC files: http://docs.openstack.org/user-guide/content/cli_openrc.html
.. _configuration files: https://github.com/openstack/rally/tree/master/samples/deployments
.. _existing.json: https://github.com/openstack/rally/blob/master/samples/deployments/existing.json
.. _samples/tasks/scenarios: https://github.com/openstack/rally/tree/master/samples/tasks/scenarios

231
doc/source/quick_start/tutorial/step_2_input_task_format.rst Normal file
View File

@@ -0,0 +1,231 @@
..
Copyright 2015 Mirantis Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _tutorial_step_2_input_task_format:
Step 2. Rally input task format
===============================
.. contents::
:local:
Basic input task syntax
-----------------------
Rally comes with a really great collection of
:ref:`plugins <tutorial_step_8_discovering_more_plugins>` and in most
real-world cases you will use multiple plugins to test your OpenStack cloud.
Rally makes it very easy to run **different test cases defined in a single
task**. To do so, use the following syntax:
.. code-block:: json
{
"<ScenarioName1>": [<benchmark_config>, <benchmark_config2>, ...]
"<ScenarioName2>": [<benchmark_config>, ...]
}
where *<benchmark_config>*, as before, is a dictionary:
.. code-block:: json
{
"args": { <scenario-specific arguments> },
"runner": { <type of the runner and its specific parameters> },
"context": { <contexts needed for this scenario> },
"sla": { <different SLA configs> }
}
Multiple benchmarks in a single task
------------------------------------
As an example, let's edit our configuration file from
:ref:`step 1 <tutorial_step_1_setting_up_env_and_running_benchmark_from_samples>`
so that it prescribes Rally to launch not only the
**NovaServers.boot_and_delete_server** scenario, but also the
**KeystoneBasic.create_delete_user** scenario. All we have to do is to append
the configuration of the second scenario as yet another top-level key of our
JSON file:
*multiple-scenarios.json*
.. code-block:: json
{
"NovaServers.boot_and_delete_server": [
{
"args": {
"flavor": {
"name": "m1.tiny"
},
"image": {
"name": "^cirros.*uec$"
},
"force_delete": false
},
"runner": {
"type": "constant",
"times": 10,
"concurrency": 2
},
"context": {
"users": {
"tenants": 3,
"users_per_tenant": 2
}
}
}
],
"KeystoneBasic.create_delete_user": [
{
"args": {},
"runner": {
"type": "constant",
"times": 10,
"concurrency": 3
}
}
]
}
Now you can start this benchmark task as usually:
.. code-block:: console
$ rally task start multiple-scenarios.json
...
+--------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count |
+--------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| nova.boot_server | 8.06 | 11.354 | 18.594 | 18.54 | 18.567 | 100.0% | 10 |
| nova.delete_server | 4.364 | 5.054 | 6.837 | 6.805 | 6.821 | 100.0% | 10 |
| total | 12.572 | 16.408 | 25.396 | 25.374 | 25.385 | 100.0% | 10 |
+--------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
Load duration: 84.1959171295
Full duration: 102.033041
--------------------------------------------------------------------------------
...
+----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count |
+----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| keystone.create_user | 0.676 | 0.875 | 1.03 | 1.02 | 1.025 | 100.0% | 10 |
| keystone.delete_user | 0.407 | 0.647 | 0.84 | 0.739 | 0.79 | 100.0% | 10 |
| total | 1.082 | 1.522 | 1.757 | 1.724 | 1.741 | 100.0% | 10 |
+----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
Load duration: 5.72119688988
Full duration: 10.0808410645
...
Note that the HTML reports you can generate by typing **rally task report
--out=report_name.html** after your benchmark task has completed will get
richer as your benchmark task configuration file includes more benchmark
scenarios. Let's take a look at the report overview page for a task that covers
all the scenarios available in Rally:
.. code-block:: bash
rally task report --out=report_multiple_scenarios.html --open
.. image:: ../../images/Report-Multiple-Overview.png
:align: center
Multiple configurations of the same scenario
--------------------------------------------
Yet another thing you can do in Rally is to launch **the same benchmark
scenario multiple times with different configurations**. That's why our
configuration file stores a list for the key
*"NovaServers.boot_and_delete_server"*: you can just append a different
configuration of this benchmark scenario to this list to get it. Let's say,
you want to run the **boot_and_delete_server** scenario twice: first using the
*"m1.tiny"* flavor and then using the *"m1.small"* flavor:
*multiple-configurations.json*
.. code-block:: json
{
"NovaServers.boot_and_delete_server": [
{
"args": {
"flavor": {
"name": "m1.tiny"
},
"image": {
"name": "^cirros.*uec$"
},
"force_delete": false
},
"runner": {...},
"context": {...}
},
{
"args": {
"flavor": {
"name": "m1.small"
},
"image": {
"name": "^cirros.*uec$"
},
"force_delete": false
},
"runner": {...},
"context": {...}
}
]
}
That's it! You will get again the results for each configuration separately:
.. code-block:: console
$ rally task start --task=multiple-configurations.json
...
+--------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count |
+--------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| nova.boot_server | 7.896 | 9.433 | 13.14 | 11.329 | 12.234 | 100.0% | 10 |
| nova.delete_server | 4.435 | 4.898 | 6.975 | 5.144 | 6.059 | 100.0% | 10 |
| total | 12.404 | 14.331 | 17.979 | 16.72 | 17.349 | 100.0% | 10 |
+--------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
Load duration: 73.2339417934
Full duration: 91.1692159176
--------------------------------------------------------------------------------
...
+--------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count |
+--------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| nova.boot_server | 8.207 | 8.91 | 9.823 | 9.692 | 9.758 | 100.0% | 10 |
| nova.delete_server | 4.405 | 4.767 | 6.477 | 4.904 | 5.691 | 100.0% | 10 |
| total | 12.735 | 13.677 | 16.301 | 14.596 | 15.449 | 100.0% | 10 |
+--------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+
Load duration: 71.029528141
Full duration: 88.0259010792
...
The HTML report will also look similar to what we have seen before:
.. code-block:: bash
rally task report --out=report_multiple_configuraions.html --open
.. image:: ../../images/Report-Multiple-Configurations-Overview.png
:align: center

180
doc/source/quick_start/tutorial/step_3_benchmarking_with_existing_users.rst Normal file
View File

@@ -0,0 +1,180 @@
..
Copyright 2015 Mirantis Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _tutorial_step_3_benchmarking_with_existing_users:
Step 3. Benchmarking OpenStack with existing users
==================================================
.. contents::
:local:
Motivation
----------
There are two very important reasons from the production world of why it is
preferable to use some already existing users to benchmark your OpenStack
cloud:
1. *Read-only Keystone Backends:* creating temporary users for benchmark
scenarios in Rally is just impossible in case of r/o Keystone backends like
*LDAP* and *AD*.
2. *Safety:* Rally can be run from an isolated group of users, and if something
goes wrong, this won’t affect the rest of the cloud users.
Registering existing users in Rally
-----------------------------------
The information about existing users in your OpenStack cloud should be passed
to Rally at the
:ref:`deployment initialization step <tutorial_step_1_setting_up_env_and_running_benchmark_from_samples>`.
You have to use the **ExistingCloud** deployment plugin that just provides
Rally with credentials of an already existing cloud. The difference from the
deployment configuration we've seen previously is that you should set up the
*"users"* section with the credentials of already existing users. Let's call
this deployment configuration file *existing_users.json*:
.. code-block:: json
{
"type": "ExistingCloud",
"auth_url": "http://example.net:5000/v2.0/",
"region_name": "RegionOne",
"endpoint_type": "public",
"admin": {
"username": "admin",
"password": "pa55word",
"tenant_name": "demo"
},
"users": [
{
"username": "b1",
"password": "1234",
"tenant_name": "testing"
},
{
"username": "b2",
"password": "1234",
"tenant_name": "testing"
}
]
}
This deployment configuration requires some basic information about the
OpenStack cloud like the region name, auth url. admin user credentials, and any
amount of users already existing in the system. Rally will use their
credentials to generate load in against this deployment as soon as we register
it as usual:
.. code-block:: console
$ rally deployment create --file existings_users --name our_cloud
+--------------------------------------+----------------------------+-----------+------------------+--------+
| uuid | created_at | name | status | active |
+--------------------------------------+----------------------------+-----------+------------------+--------+
| 1849a9bf-4b18-4fd5-89f0-ddcc56eae4c9 | 2015-03-28 02:43:27.759702 | our_cloud | deploy->finished | |
+--------------------------------------+----------------------------+-----------+------------------+--------+
Using deployment: 1849a9bf-4b18-4fd5-89f0-ddcc56eae4c9
~/.rally/openrc was updated
After that, the **rally show** command lists the resources for each user
separately:
.. code-block:: console
$ rally show images
Images for user `admin` in tenant `admin`:
+--------------------------------------+---------------------------------+-----------+
| UUID | Name | Size (B) |
+--------------------------------------+---------------------------------+-----------+
| 041cfd70-0e90-4ed6-8c0c-ad9c12a94191 | cirros-0.3.4-x86_64-uec | 25165824 |
| 87710f09-3625-4496-9d18-e20e34906b72 | Fedora-x86_64-20-20140618-sda | 209649664 |
| b0f269be-4859-48e0-a0ca-03fb80d14602 | cirros-0.3.4-x86_64-uec-ramdisk | 3740163 |
| d82eaf7a-ff63-4826-9aa7-5fa105610e01 | cirros-0.3.4-x86_64-uec-kernel | 4979632 |
+--------------------------------------+---------------------------------+-----------+
Images for user `b1` in tenant `testing`:
+--------------------------------------+---------------------------------+-----------+
| UUID | Name | Size (B) |
+--------------------------------------+---------------------------------+-----------+
| 041cfd70-0e90-4ed6-8c0c-ad9c12a94191 | cirros-0.3.4-x86_64-uec | 25165824 |
| 87710f09-3625-4496-9d18-e20e34906b72 | Fedora-x86_64-20-20140618-sda | 209649664 |
| b0f269be-4859-48e0-a0ca-03fb80d14602 | cirros-0.3.4-x86_64-uec-ramdisk | 3740163 |
| d82eaf7a-ff63-4826-9aa7-5fa105610e01 | cirros-0.3.4-x86_64-uec-kernel | 4979632 |
+--------------------------------------+---------------------------------+-----------+
Images for user `b2` in tenant `testing`:
+--------------------------------------+---------------------------------+-----------+
| UUID | Name | Size (B) |
+--------------------------------------+---------------------------------+-----------+
| 041cfd70-0e90-4ed6-8c0c-ad9c12a94191 | cirros-0.3.4-x86_64-uec | 25165824 |
| 87710f09-3625-4496-9d18-e20e34906b72 | Fedora-x86_64-20-20140618-sda | 209649664 |
| b0f269be-4859-48e0-a0ca-03fb80d14602 | cirros-0.3.4-x86_64-uec-ramdisk | 3740163 |
| d82eaf7a-ff63-4826-9aa7-5fa105610e01 | cirros-0.3.4-x86_64-uec-kernel | 4979632 |
+--------------------------------------+---------------------------------+-----------+
With this new deployment being active, Rally will use the already existing
users *"b1"* and *"b2"* instead of creating the temporary ones when launching
benchmark task that do not specify the *"users"* context.
Running benchmark scenarios with existing users
-----------------------------------------------
After you have registered a deployment with existing users, don't forget to
remove the *"users"* context from your benchmark task configuration if you want
to use existing users, like in the following configuration file
(*boot-and-delete.json*):
.. code-block:: json
{
"NovaServers.boot_and_delete_server": [
{
"args": {
"flavor": {
"name": "m1.tiny"
},
"image": {
"name": "^cirros.*uec$"
},
"force_delete": false
},
"runner": {
"type": "constant",
"times": 10,
"concurrency": 2
},
"context": {}
}
]
}
When you start this task, it will use the existing users *"b1"* and *"b2"*
instead of creating the temporary ones:
.. code-block:: bash
rally task start samples/tasks/scenarios/nova/boot-and-delete.json
It goes without saying that support of benchmarking with predefined users
simplifies the usage of Rally for generating loads against production clouds.
(based on: http://boris-42.me/rally-can-generate-load-with-passed-users-now/)

164
doc/source/quick_start/tutorial/step_4_adding_success_criteria_for_benchmarks.rst Normal file
View File

@@ -0,0 +1,164 @@
..
Copyright 2015 Mirantis Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _tutorial_step_4_adding_success_criteria_for_benchmarks:
Step 4. Adding success criteria (SLA) for benchmarks
====================================================
.. contents::
:local:
SLA - Service-Level Agreement (Success Criteria)
------------------------------------------------
Rally allows you to set success criteria (also called *SLA - Service-Level
Agreement*) for every benchmark. Rally will automatically check them for you.
To configure the SLA, add the *"sla"* section to the configuration of the
corresponding benchmark (the check name is a key associated with its target
value). You can combine different success criteria:
.. code-block:: json
{
"NovaServers.boot_and_delete_server": [
{
"args": {
...
},
"runner": {
...
},
"context": {
...
},
"sla": {
"max_seconds_per_iteration": 10,
"failure_rate": {
"max": 25
}
}
}
]
}
Such configuration will mark the **NovaServers.boot_and_delete_server**
benchmark scenario as not successful if either some iteration took more than 10
seconds or more than 25% iterations failed.
Checking SLA
------------
Let us show you how Rally SLA work using a simple example based on **Dummy
benchmark scenarios**. These scenarios actually do not perform any
OpenStack-related stuff but are very useful for testing the behaviors of Rally.
Let us put in a new task, *test-sla.json*, 2 scenarios -- one that does nothing
and another that just throws an exception:
.. code-block:: json
{
"Dummy.dummy": [
{
"args": {},
"runner": {
"type": "constant",
"times": 5,
"concurrency": 2
},
"context": {
"users": {
"tenants": 3,
"users_per_tenant": 2
}
},
"sla": {
"failure_rate": {"max": 0.0}
}
}
],
"Dummy.dummy_exception": [
{
"args": {},
"runner": {
"type": "constant",
"times": 5,
"concurrency": 2
},
"context": {
"users": {
"tenants": 3,
"users_per_tenant": 2
}
},
"sla": {
"failure_rate": {"max": 0.0}
}
}
]
}
Note that both scenarios in these tasks have the **maximum failure rate of 0%**
as their **success criterion**. We expect that the first scenario will pass
this criterion while the second will fail it. Let's start the task:
.. code-block:: bash
rally task start test-sla.json
After the task completes, run *rally task sla_check* to check the results again
the success criteria you defined in the task:
.. code-block:: console
$ rally task sla_check
+-----------------------+-----+--------------+--------+-------------------------------------------------------------------------------------------------------+
| benchmark | pos | criterion | status | detail |
+-----------------------+-----+--------------+--------+-------------------------------------------------------------------------------------------------------+
| Dummy.dummy | 0 | failure_rate | PASS | Maximum failure rate percent 0.0% failures, minimum failure rate percent 0% failures, actually 0.0% |
| Dummy.dummy_exception | 0 | failure_rate | FAIL | Maximum failure rate percent 0.0% failures, minimum failure rate percent 0% failures, actually 100.0% |
+-----------------------+-----+--------------+--------+-------------------------------------------------------------------------------------------------------+
Exactly as expected.
SLA in task report
------------------
SLA checks are nicely visualized in task reports. Generate one:
.. code-block:: bash
rally task report --out=report_sla.html --open
Benchmark scenarios that have passed SLA have a green check on the overview
page:
.. image:: ../../images/Report-SLA-Overview.png
:align: center
Somewhat more detailed information about SLA is displayed on the scenario
pages:
.. image:: ../../images/Report-SLA-Scenario.png
:align: center
Success criteria present a very useful concept that enables not only to analyze
the outcome of your benchmark tasks, but also to control their execution. In
:ref:`one of the next sections <tutorial_step_6_aborting_load_generation_on_sla_failure>`
of our tutorial, we will show how to use SLA to abort the load generation before
your OpenStack goes wrong.

378
doc/source/quick_start/tutorial/step_5_task_templates.rst Normal file
View File

@@ -0,0 +1,378 @@
..
Copyright 2015 Mirantis Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _tutorial_step_5_task_templates:
Step 5. Rally task templates
============================
.. contents::
:local:
Basic template syntax
---------------------
A nice feature of the input task format used in Rally is that it supports the
**template syntax** based on `Jinja2`_. This turns out to be extremely useful
when, say, you have a fixed structure of your task but you want to parameterize
this task in some way. For example, imagine your input task file (*task.yaml*)
runs a set of Nova scenarios:
.. code-block:: yaml
---
NovaServers.boot_and_delete_server:
-
args:
flavor:
name: "m1.tiny"
image:
name: "^cirros.*uec$"
runner:
type: "constant"
times: 2
concurrency: 1
context:
users:
tenants: 1
users_per_tenant: 1
NovaServers.resize_server:
-
args:
flavor:
name: "m1.tiny"
image:
name: "^cirros.*uec$"
to_flavor:
name: "m1.small"
runner:
type: "constant"
times: 3
concurrency: 1
context:
users:
tenants: 1
users_per_tenant: 1
In both scenarios above, the *"^cirros.*uec$"* image is passed to the scenario
as an argument (so that these scenarios use an appropriate image while booting
servers). Let’s say you want to run the same set of scenarios with the same
runner/context/sla, but you want to try another image while booting server to
compare the performance. The most elegant solution is then to turn the image
name into a template variable:
.. code-block:: yaml
---
NovaServers.boot_and_delete_server:
-
args:
flavor:
name: "m1.tiny"
image:
name: {{image_name}}
runner:
type: "constant"
times: 2
concurrency: 1
context:
users:
tenants: 1
users_per_tenant: 1
NovaServers.resize_server:
-
args:
flavor:
name: "m1.tiny"
image:
name: {{image_name}}
to_flavor:
name: "m1.small"
runner:
type: "constant"
times: 3
concurrency: 1
context:
users:
tenants: 1
users_per_tenant: 1
and then pass the argument value for **{{image_name}}** when starting a task
with this configuration file. Rally provides you with different ways to do
that:
1. Pass the argument values directly in the command-line interface (with either
a JSON or YAML dictionary):
.. code-block:: bash
rally task start task.yaml --task-args '{"image_name": "^cirros.*uec$"}'
rally task start task.yaml --task-args 'image_name: "^cirros.*uec$"'
2. Refer to a file that specifies the argument values (JSON/YAML):
.. code-block:: bash
rally task start task.yaml --task-args-file args.json
rally task start task.yaml --task-args-file args.yaml
where the files containing argument values should look as follows:
*args.json*:
.. code-block:: json
{
"image_name": "^cirros.*uec$"
}
*args.yaml*:
.. code-block:: yaml
---
image_name: "^cirros.*uec$"
Passed in either way, these parameter values will be substituted by Rally when
starting a task:
.. code-block:: console
$ rally task start task.yaml --task-args "image_name: "^cirros.*uec$""
--------------------------------------------------------------------------------
Preparing input task
--------------------------------------------------------------------------------
Input task is:
---
NovaServers.boot_and_delete_server:
-
args:
flavor:
name: "m1.tiny"
image:
name: ^cirros.*uec$
runner:
type: "constant"
times: 2
concurrency: 1
context:
users:
tenants: 1
users_per_tenant: 1
NovaServers.resize_server:
-
args:
flavor:
name: "m1.tiny"
image:
name: ^cirros.*uec$
to_flavor:
name: "m1.small"
runner:
type: "constant"
times: 3
concurrency: 1
context:
users:
tenants: 1
users_per_tenant: 1
--------------------------------------------------------------------------------
Task cbf7eb97-0f1d-42d3-a1f1-3cc6f45ce23f: started
--------------------------------------------------------------------------------
Benchmarking... This can take a while...
Using the default values
------------------------
Note that the ``Jinja2`` template syntax allows you to set the default values
for your parameters. With default values set, your task file will work even if
you don't parameterize it explicitly while starting a task. The default values
should be set using the *{% set ... %}* clause (*task.yaml*):
.. code-block:: yaml
{% set image_name = image_name or "^cirros.*uec$" %}
---
NovaServers.boot_and_delete_server:
-
args:
flavor:
name: "m1.tiny"
image:
name: {{image_name}}
runner:
type: "constant"
times: 2
concurrency: 1
context:
users:
tenants: 1
users_per_tenant: 1
...
If you don't pass the value for *{{image_name}}* while starting a task, the
default one will be used:
.. code-block:: console
$ rally task start task.yaml
--------------------------------------------------------------------------------
Preparing input task
--------------------------------------------------------------------------------
Input task is:
---
NovaServers.boot_and_delete_server:
-
args:
flavor:
name: "m1.tiny"
image:
name: ^cirros.*uec$
runner:
type: "constant"
times: 2
concurrency: 1
context:
users:
tenants: 1
users_per_tenant: 1
...
Advanced templates
------------------
Rally makes it possible to use all the power of ``Jinja2`` template syntax,
including the mechanism of **built-in functions**. This enables you to
construct elegant task files capable of generating complex load on your cloud.
As an example, let us make up a task file that will create new users with
increasing concurrency. The input task file (*task.yaml*) below uses the
``Jinja2`` **for-endfor** construct to accomplish that:
.. code-block:: yaml
---
KeystoneBasic.create_user:
{% for i in range(2, 11, 2) %}
-
args: {}
runner:
type: "constant"
times: 10
concurrency: {{i}}
sla:
failure_rate:
max: 0
{% endfor %}
In this case, you don’t need to pass any arguments via
*--task-args/--task-args-file*, but as soon as you start this task, Rally will
automatically unfold the for-loop for you:
.. code-block:: console
$ rally task start task.yaml
--------------------------------------------------------------------------------
Preparing input task
--------------------------------------------------------------------------------
Input task is:
---
KeystoneBasic.create_user:
-
args: {}
runner:
type: "constant"
times: 10
concurrency: 2
sla:
failure_rate:
max: 0
-
args: {}
runner:
type: "constant"
times: 10
concurrency: 4
sla:
failure_rate:
max: 0
-
args: {}
runner:
type: "constant"
times: 10
concurrency: 6
sla:
failure_rate:
max: 0
-
args: {}
runner:
type: "constant"
times: 10
concurrency: 8
sla:
failure_rate:
max: 0
-
args: {}
runner:
type: "constant"
times: 10
concurrency: 10
sla:
failure_rate:
max: 0
--------------------------------------------------------------------------------
Task ea7e97e3-dd98-4a81-868a-5bb5b42b8610: started
--------------------------------------------------------------------------------
Benchmarking... This can take a while...
As you can see, the Rally task template syntax is a simple but powerful
mechanism that not only enables you to write elegant task configurations, but
also makes them more readable for other people. When used appropriately, it can
really improve the understanding of your benchmarking procedures in Rally when
shared with others.
.. references:
.. _Jinja2: https://pypi.python.org/pypi/Jinja2

149
doc/source/quick_start/tutorial/step_6_aborting_load_generation_on_sla_failure.rst Normal file
View File

@@ -0,0 +1,149 @@
..
Copyright 2015 Mirantis Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _tutorial_step_6_aborting_load_generation_on_sla_failure:
Step 6. Aborting load generation on success criteria failure
============================================================
Benchmarking pre-production and production OpenStack clouds is not a trivial
task. From the one side it is important to reach the OpenStack cloud’s limits,
from the other side the cloud shouldn't be damaged. Rally aims to make this
task as simple as possible. Since the very beginning Rally was able to generate
enough load for any OpenStack cloud. Generating too big a load was the major
issue for production clouds, because Rally didn't know how to stop the load
until it was too late.
With the **"stop on SLA failure"** feature, however, things are much better.
This feature can be easily tested in real life by running one of the most
important and plain benchmark scenario called *"Authenticate.keystone"*. This
scenario just tries to authenticate from users that were pre-created by Rally.
Rally input task looks as follows (*auth.yaml*):
.. code-block:: yaml
---
Authenticate.keystone:
-
runner:
type: "rps"
times: 6000
rps: 50
context:
users:
tenants: 5
users_per_tenant: 10
sla:
max_avg_duration: 5
In human-readable form this input task means: *Create 5 tenants with 10 users
in each, after that try to authenticate to Keystone 6000 times performing 50
authentications per second (running new authentication request every 20ms).
Each time we are performing authentication from one of the Rally pre-created
user. This task passes only if max average duration of authentication takes
less than 5 seconds.*
**Note that this test is quite dangerous because it can DDoS Keystone**. We are
running more and more simultaneously authentication requests and things may go
wrong if something is not set properly (like on my DevStack deployment in Small
VM on my laptop).
Let’s run Rally task with **an argument that prescribes Rally to stop load on
SLA failure**:
.. code-block:: console
$ rally task start --abort-on-sla-failure auth.yaml
....
+--------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count |
+--------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| total | 0.108 | 8.58 | 65.97 | 19.782 | 26.125 | 100.0% | 2495 |
+--------+-----------+-----------+-----------+---------------+---------------+---------+-------+
On the resulting table there are 2 interesting things:
1. Average duration was 8.58 sec which is more than 5 seconds
2. Rally performed only 2495 (instead of 6000) authentication requests
To understand better what has happened let’s generate HTML report:
.. code-block:: bash
rally task report --out auth_report.html
.. image:: ../../images/Report-Abort-on-SLA-task-1.png
:align: center
On the chart with durations we can observe that the duration of authentication
request reaches 65 seconds at the end of the load generation. **Rally stopped
load at the very last moment just before bad things happened. The reason why it
runs so many attempts to authenticate is because of not enough good success
criteria.** We had to run a lot of iterations to make average duration bigger
than 5 seconds. Let’s chose better success criteria for this task and run it
one more time.
.. code-block:: yaml
---
Authenticate.keystone:
-
runner:
type: "rps"
times: 6000
rps: 50
context:
users:
tenants: 5
users_per_tenant: 10
sla:
max_avg_duration: 5
max_seconds_per_iteration: 10
failure_rate:
max: 0
Now our task is going to be successful if the following three conditions hold:
1. maximum average duration of authentication should be less than 5 seconds
2. maximum duration of any authentication should be less than 10 seconds
3. no failed authentication should appear
Let’s run it!
.. code-block:: console
$ rally task start --abort-on-sla-failure auth.yaml
...
+--------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count |
+--------+-----------+-----------+-----------+---------------+---------------+---------+-------+
| total | 0.082 | 5.411 | 22.081 | 10.848 | 14.595 | 100.0% | 1410 |
+--------+-----------+-----------+-----------+---------------+---------------+---------+-------+
.. image:: ../../images/Report-Abort-on-SLA-task-2.png
:align: center
This time load stopped after 1410 iterations versus 2495 which is much better.
The interesting thing on this chart is that first occurrence of “> 10 second”
authentication happened on 950 iteration. The reasonable question: “Why does
Rally run 500 more authentication requests then?”. This appears from the math:
During the execution of **bad** authentication (10 seconds) Rally performed
about 50 request/sec * 10 sec = 500 new requests as a result we run 1400
iterations instead of 950.
(based on: http://boris-42.me/rally-tricks-stop-load-before-your-openstack-goes-wrong/)

144
doc/source/quick_start/tutorial/step_7_working_with_multple_openstack_clouds.rst Normal file
View File

@@ -0,0 +1,144 @@
..
Copyright 2015 Mirantis Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _tutorial_step_7_working_with_multple_openstack_clouds:
Step 7. Working with multiple OpenStack clouds
==============================================
Rally is an awesome tool that allows you to work with multiple clouds and can
itself deploy them. We already know how to work with
:ref:`a single cloud <tutorial_step_1_setting_up_env_and_running_benchmark_from_samples>`.
Let us now register 2 clouds in Rally: the one that we have access to and the
other that we know is registered with wrong credentials.
.. code-block:: console
$ . openrc admin admin # openrc with correct credentials
$ rally deployment create --fromenv --name=cloud-1
+--------------------------------------+----------------------------+------------+------------------+--------+
| uuid | created_at | name | status | active |
+--------------------------------------+----------------------------+------------+------------------+--------+
| 4251b491-73b2-422a-aecb-695a94165b5e | 2015-01-18 00:11:14.757203 | cloud-1 | deploy->finished | |
+--------------------------------------+----------------------------+------------+------------------+--------+
Using deployment: 4251b491-73b2-422a-aecb-695a94165b5e
~/.rally/openrc was updated
...
$ . bad_openrc admin admin # openrc with wrong credentials
$ rally deployment create --fromenv --name=cloud-2
+--------------------------------------+----------------------------+------------+------------------+--------+
| uuid | created_at | name | status | active |
+--------------------------------------+----------------------------+------------+------------------+--------+
| 658b9bae-1f9c-4036-9400-9e71e88864fc | 2015-01-18 00:38:26.127171 | cloud-2 | deploy->finished | |
+--------------------------------------+----------------------------+------------+------------------+--------+
Using deployment: 658b9bae-1f9c-4036-9400-9e71e88864fc
~/.rally/openrc was updated
...
Let us now list the deployments we have created:
.. code-block:: console
$ rally deployment list
+--------------------------------------+----------------------------+------------+------------------+--------+
| uuid | created_at | name | status | active |
+--------------------------------------+----------------------------+------------+------------------+--------+
| 4251b491-73b2-422a-aecb-695a94165b5e | 2015-01-05 00:11:14.757203 | cloud-1 | deploy->finished | |
| 658b9bae-1f9c-4036-9400-9e71e88864fc | 2015-01-05 00:40:58.451435 | cloud-2 | deploy->finished | * |
+--------------------------------------+----------------------------+------------+------------------+--------+
Note that the second is marked as **"active"** because this is the deployment
we have created most recently. This means that it will be automatically (unless
its UUID or name is passed explicitly via the *--deployment* parameter) used by
the commands that need a deployment, like *rally task start ...* or *rally
deployment check*:
.. code-block:: console
$ rally deployment check
Authentication Issues: wrong keystone credentials specified in your endpoint properties. (HTTP 401).
$ rally deployment check --deployment=cloud-1
keystone endpoints are valid and following services are available:
+----------+----------------+-----------+
| services | type | status |
+----------+----------------+-----------+
| cinder | volume | Available |
| cinderv2 | volumev2 | Available |
| ec2 | ec2 | Available |
| glance | image | Available |
| heat | orchestration | Available |
| heat-cfn | cloudformation | Available |
| keystone | identity | Available |
| nova | compute | Available |
| novav21 | computev21 | Available |
| s3 | s3 | Available |
+----------+----------------+-----------+
You can also switch the active deployment using the **rally deployment use**
command:
.. code-block:: console
$ rally deployment use cloud-1
Using deployment: 658b9bae-1f9c-4036-9400-9e71e88864fc
~/.rally/openrc was updated
...
$ rally deployment check
keystone endpoints are valid and following services are available:
+----------+----------------+-----------+
| services | type | status |
+----------+----------------+-----------+
| cinder | volume | Available |
| cinderv2 | volumev2 | Available |
| ec2 | ec2 | Available |
| glance | image | Available |
| heat | orchestration | Available |
| heat-cfn | cloudformation | Available |
| keystone | identity | Available |
| nova | compute | Available |
| novav21 | computev21 | Available |
| s3 | s3 | Available |
+----------+----------------+-----------+
Note the first two lines of the CLI output for the *rally deployment use*
command. They tell you the UUID of the new active deployment and also say that
the *~/.rally/openrc* file was updated -- this is the place where the "active"
UUID is actually stored by Rally.
One last detail about managing different deployments in Rally is that the
*rally task list* command outputs only those tasks that were run against the
currently active deployment, and you have to provide the *--all-deployments*
parameter to list all the tasks:
.. code-block:: console
$ rally task list
+--------------------------------------+-----------------+----------------------------+----------------+----------+--------+-----+
| uuid | deployment_name | created_at | duration | status | failed | tag |
+--------------------------------------+-----------------+----------------------------+----------------+----------+--------+-----+
| c21a6ecb-57b2-43d6-bbbb-d7a827f1b420 | cloud-1 | 2015-01-05 01:00:42.099596 | 0:00:13.419226 | finished | False | |
| f6dad6ab-1a6d-450d-8981-f77062c6ef4f | cloud-1 | 2015-01-05 01:05:57.653253 | 0:00:14.160493 | finished | False | |
+--------------------------------------+-----------------+----------------------------+----------------+----------+--------+-----+
$ rally task list --all-deployment
+--------------------------------------+-----------------+----------------------------+----------------+----------+--------+-----+
| uuid | deployment_name | created_at | duration | status | failed | tag |
+--------------------------------------+-----------------+----------------------------+----------------+----------+--------+-----+
| c21a6ecb-57b2-43d6-bbbb-d7a827f1b420 | cloud-1 | 2015-01-05 01:00:42.099596 | 0:00:13.419226 | finished | False | |
| f6dad6ab-1a6d-450d-8981-f77062c6ef4f | cloud-1 | 2015-01-05 01:05:57.653253 | 0:00:14.160493 | finished | False | |
| 6fd9a19f-5cf8-4f76-ab72-2e34bb1d4996 | cloud-2 | 2015-01-05 01:14:51.428958 | 0:00:15.042265 | finished | False | |
+--------------------------------------+-----------------+----------------------------+----------------+----------+--------+-----+

115
doc/source/quick_start/tutorial/step_8_discovering_more_plugins.rst Normal file
View File

@@ -0,0 +1,115 @@
..
Copyright 2015 Mirantis Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _tutorial_step_8_discovering_more_plugins:
Step 8. Discovering more plugins in Rally
=========================================
.. contents::
:local:
Plugins in the Rally repository
---------------------------------
Rally currently comes with a great collection of plugins that use the API of
different OpenStack projects like **Keystone**, **Nova**, **Cinder**,
**Glance** and so on. The good news is that you can combine multiple plugins
in one task to test your cloud in a comprehensive way.
First, let's see what plugins are available in Rally. One of the ways to
discover these plugins is just to inspect their `source code`_.
another is to use build-in rally plugin command.
CLI: rally plugin show
----------------------
Rally plugin CLI command is much more convenient way to learn about different
plugins in Rally. This command allows to list plugins and show detailed
information about them:
.. code-block:: console
$ rally plugin show create_meter_and_get_stats
NAME
CeilometerStats.create_meter_and_get_stats
NAMESPACE
default
MODULE
rally.plugins.openstack.scenarios.ceilometer.stats
DESCRIPTION
Meter is first created and then statistics is fetched for the same
using GET /v2/meters/(meter_name)/statistics.
PARAMETERS
+--------+------------------------------------------------+
| name | description |
+--------+------------------------------------------------+
| kwargs | contains optional arguments to create a meter |
| | |
+--------+------------------------------------------------+
In case if multiple found benchmarks found command list all matches elements:
.. code-block:: console
$ rally plugin show NovaKeypair
Multiple plugins found:
+-------------------------------------------------+-----------+-------------------------------------------------------+
| name | namespace | title |
+-------------------------------------------------+-----------+-------------------------------------------------------+
| NovaKeypair.boot_and_delete_server_with_keypair | default | Boot and delete server with keypair. |
| NovaKeypair.create_and_delete_keypair | default | Create a keypair with random name and delete keypair. |
| NovaKeypair.create_and_list_keypairs | default | Create a keypair with random name and list keypairs. |
+-------------------------------------------------+-----------+-------------------------------------------------------+
CLI: rally plugin list
----------------------
This command can be used to list filtered by name list of plugins.
.. code-block:: console
$ rally plugin list --name Keystone
+--------------------------------------------------+-----------+-----------------------------------------------------------------+
| name | namespace | title |
+--------------------------------------------------+-----------+-----------------------------------------------------------------+
| Authenticate.keystone | default | Check Keystone Client. |
| KeystoneBasic.add_and_remove_user_role | default | Create a user role add to a user and disassociate. |
| KeystoneBasic.create_add_and_list_user_roles | default | Create user role, add it and list user roles for given user. |
| KeystoneBasic.create_and_delete_ec2credential | default | Create and delete keystone ec2-credential. |
| KeystoneBasic.create_and_delete_role | default | Create a user role and delete it. |
| KeystoneBasic.create_and_delete_service | default | Create and delete service. |
| KeystoneBasic.create_and_list_ec2credentials | default | Create and List all keystone ec2-credentials. |
| KeystoneBasic.create_and_list_services | default | Create and list services. |
| KeystoneBasic.create_and_list_tenants | default | Create a keystone tenant with random name and list all tenants. |
| KeystoneBasic.create_and_list_users | default | Create a keystone user with random name and list all users. |
| KeystoneBasic.create_delete_user | default | Create a keystone user with random name and then delete it. |
| KeystoneBasic.create_tenant | default | Create a keystone tenant with random name. |
| KeystoneBasic.create_tenant_with_users | default | Create a keystone tenant and several users belonging to it. |
| KeystoneBasic.create_update_and_delete_tenant | default | Create, update and delete tenant. |
| KeystoneBasic.create_user | default | Create a keystone user with random name. |
| KeystoneBasic.create_user_set_enabled_and_delete | default | Create a keystone user, enable or disable it, and delete it. |
| KeystoneBasic.create_user_update_password | default | Create user and update password for that user. |
| KeystoneBasic.get_entities | default | Get instance of a tenant, user, role and service by id's. |
+--------------------------------------------------+-----------+-----------------------------------------------------------------+
.. references:
.. _source code: https://github.com/openstack/rally/tree/master/rally/plugins/