openstack/rally
Code Issues Proposed changes

Merge "[Docs][Verify] Adding page for rally verify [part 1]"

Browse Source
This commit is contained in:
Jenkins 2016-05-23 15:02:55 +00:00 committed by Gerrit Code Review
commit 6b9576c899
3 changed files with 827 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
doc/source/images/Report-verify-xfail.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 62 KiB

12
doc/source/tutorial.rst
View File

@ -25,4 +25,14 @@ In the following tutorial, we will guide you step-by-step through different use
:glob:
:maxdepth: 1
tutorial/**
tutorial/step_0_installation
tutorial/step_1_setting_up_env_and_running_benchmark_from_samples
tutorial/step_2_input_task_format
tutorial/step_3_benchmarking_with_existing_users
tutorial/step_4_adding_success_criteria_for_benchmarks
tutorial/step_5_task_templates
tutorial/step_6_aborting_load_generation_on_sla_failure
tutorial/step_7_working_with_multple_openstack_clouds
tutorial/step_8_discovering_more_plugins
tutorial/step_9_deploying_openstack
tutorial/step_10_verifying_cloud_via_tempest

816
doc/source/tutorial/step_10_verifying_cloud_via_tempest.rst Normal file
View File

@ -0,0 +1,816 @@
..
Copyright 2016 Mirantis Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
.. _tutorial_step_10_verifying_cloud_via_tempest:
Step 10. Verifying cloud via Tempest
====================================
.. contents::
:local:
In this guide, we are going to show how to use Tempest and Rally together.
We assume that you have a :ref:`Rally installation <tutorial_step_0_installation>`
and an already :ref:`registered OpenStack deployment <tutorial_step_1_setting_up_env_and_running_benchmark_from_samples>`
in Rally. So let's get started!
Tempest installation (rally verify install/uninstall/reinstall)
---------------------------------------------------------------
In order to install Tempest execute the following command:
.. code-block:: console
$ rally verify install
2016-05-09 13:23:51.897 21850 INFO rally.verification.tempest.tempest [-] Tempest is not installed for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:23:51.897 21850 INFO rally.verification.tempest.tempest [-] Installing Tempest for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:23:51.908 21850 INFO rally.verification.tempest.tempest [-] Please, wait while Tempest is being cloned.
Cloning into '/home/ubuntu/.rally/tempest/base/tempest_base-ljZwwS'...
remote: Counting objects: 70000, done.
remote: Compressing objects: 100% (37320/37320), done.
remote: Total 70000 (delta 53645), reused 47857 (delta 32525)
Receiving objects: 100% (70000/70000), 9.92 MiB | 1.03 MiB/s, done.
Resolving deltas: 100% (53645/53645), done.
Checking connectivity... done.
2016-05-09 13:24:11.180 21850 INFO rally.verification.tempest.tempest [-] Installing the virtual environment for Tempest.
2016-05-09 13:24:25.596 21850 INFO rally.verification.tempest.tempest [-] Tempest has been successfully installed!
By default, the command clones Tempest from the **https://git.openstack.org/openstack/tempest**
repository and installs it for the current deployment. But it is possible to
install Tempest for any registered deployment in Rally, using the **--deployment**
argument.
.. code-block:: console
$ rally verify install --deployment <UUID or name of a deployment>
Also, Rally allows users to specify a source to clone Tempest from, using the
**--source** argument. The source can be both the path to a local Tempest
repository and a URL, e.g. to some GitHub repository.
.. code-block:: console
$ rally verify install --source /home/ubuntu/tempest/
2016-05-09 13:29:05.004 22382 INFO rally.verification.tempest.tempest [-] Tempest is not installed for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:29:05.004 22382 INFO rally.verification.tempest.tempest [-] Installing Tempest for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:29:05.013 22382 INFO rally.verification.tempest.tempest [-] Please, wait while Tempest is being cloned.
Cloning into '/home/ubuntu/.rally/tempest/base/tempest_base-pscTA7'...
done.
2016-05-09 13:29:05.902 22382 INFO rally.verification.tempest.tempest [-] Installing the virtual environment for Tempest.
2016-05-09 13:29:18.052 22382 INFO rally.verification.tempest.tempest [-] Tempest has been successfully installed!
.. code-block:: console
$ rally verify install --source https://github.com/openstack/tempest.git
2016-05-09 13:30:15.804 22541 INFO rally.verification.tempest.tempest [-] Tempest is not installed for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:30:15.804 22541 INFO rally.verification.tempest.tempest [-] Installing Tempest for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:30:15.814 22541 INFO rally.verification.tempest.tempest [-] Please, wait while Tempest is being cloned.
Cloning into '/home/ubuntu/.rally/tempest/base/tempest_base-BLUv1E'...
remote: Counting objects: 70000, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 70000 (delta 0), reused 0 (delta 0), pack-reused 69993
Receiving objects: 100% (70000/70000), 20.66 MiB | 2.67 MiB/s, done.
Resolving deltas: 100% (52246/52246), done.
Checking connectivity... done.
2016-05-09 13:30:37.602 22541 INFO rally.verification.tempest.tempest [-] Installing the virtual environment for Tempest.
2016-05-09 13:30:49.432 22541 INFO rally.verification.tempest.tempest [-] Tempest has been successfully installed!
Moreover, it is possible to specify a Tempest commit ID or tag, using the
**--version** argument, to install a certain version of Tempest.
.. code-block:: console
$ rally verify install --source /home/ubuntu/tempest/ --version 198e5b4b871c3d09c20afb56dca9637a8cf86ac8
2016-05-09 13:45:55.764 23259 INFO rally.verification.tempest.tempest [-] Tempest is not installed for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:45:55.764 23259 INFO rally.verification.tempest.tempest [-] Installing Tempest for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:45:55.773 23259 INFO rally.verification.tempest.tempest [-] Please, wait while Tempest is being cloned.
Cloning into '/home/ubuntu/.rally/tempest/base/tempest_base-TcKvAX'...
done.
T doc/source/HACKING.rst
T doc/source/REVIEWING.rst
T doc/source/field_guide/api.rst
T doc/source/field_guide/index.rst
T doc/source/field_guide/scenario.rst
T doc/source/field_guide/stress.rst
T doc/source/field_guide/unit_tests.rst
T doc/source/overview.rst
Note: checking out '198e5b4b871c3d09c20afb56dca9637a8cf86ac8'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b new_branch_name
HEAD is now at 198e5b4... Merge "Pass server to RemoteClient in API tests"
2016-05-09 13:45:56.061 23259 INFO rally.verification.tempest.tempest [-] Installing the virtual environment for Tempest.
2016-05-09 13:46:15.278 23259 INFO rally.verification.tempest.tempest [-] Tempest has been successfully installed!
.. code-block:: console
$ rally verify install --source /home/ubuntu/tempest/ --version 10.0.0
2016-05-09 13:50:42.559 23870 INFO rally.verification.tempest.tempest [-] Tempest is not installed for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:50:42.559 23870 INFO rally.verification.tempest.tempest [-] Installing Tempest for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:50:42.568 23870 INFO rally.verification.tempest.tempest [-] Please, wait while Tempest is being cloned.
Cloning into '/home/ubuntu/.rally/tempest/base/tempest_base-cUe5p8'...
done.
T doc/source/HACKING.rst
T doc/source/REVIEWING.rst
T doc/source/field_guide/api.rst
T doc/source/field_guide/index.rst
T doc/source/field_guide/scenario.rst
T doc/source/field_guide/stress.rst
T doc/source/field_guide/unit_tests.rst
T doc/source/overview.rst
Note: checking out '10.0.0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b new_branch_name
HEAD is now at 09a6015... Merge "Make data_processing/baremetal use rest_client"
2016-05-09 13:50:42.903 23870 INFO rally.verification.tempest.tempest [-] Installing the virtual environment for Tempest.
2016-05-09 13:50:55.827 23870 INFO rally.verification.tempest.tempest [-] Tempest has been successfully installed!
Finally, users can specify the **--system-wide** argument that will tell Rally
not to create a virtual environment for Tempest. In this case, it is assumed
that all Tempest requirements are already installed in the local environment.
.. code-block:: console
$ rally verify install --source /home/ubuntu/tempest/ --version 10.0.0 --system-wide
2016-05-09 13:52:34.085 24216 INFO rally.verification.tempest.tempest [-] Tempest is not installed for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:52:34.085 24216 INFO rally.verification.tempest.tempest [-] Installing Tempest for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 13:52:34.094 24216 INFO rally.verification.tempest.tempest [-] Please, wait while Tempest is being cloned.
Cloning into '/home/ubuntu/.rally/tempest/base/tempest_base-8jFGJU'...
done.
T doc/source/HACKING.rst
T doc/source/REVIEWING.rst
T doc/source/field_guide/api.rst
T doc/source/field_guide/index.rst
T doc/source/field_guide/scenario.rst
T doc/source/field_guide/stress.rst
T doc/source/field_guide/unit_tests.rst
T doc/source/overview.rst
Note: checking out '10.0.0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b new_branch_name
HEAD is now at 09a6015... Merge "Make data_processing/baremetal use rest_client"
2016-05-09 13:52:34.519 24216 INFO rally.verification.tempest.tempest [-] Tempest has been successfully installed!
To remove a local Tempest installation for the current deployment execute the
following command:
.. code-block:: console
$ rally verify uninstall
Also, it is possible to remove the local Tempest installation for any
registered deployment in Rally, using the **--deployment** argument.
.. code-block:: console
$ rally verify uninstall --deployment <UUID or name of a deployment>
In addition, there is the
.. code-block:: console
$ rally verify reinstall
command that combines two commands:
.. code-block:: console
$ rally verify uninstall
$ rally verify install
Arguments for **rally verify reinstall** are the same like for **rally verify install**.
Tempest config generation (rally verify genconfig/showconfig)
-------------------------------------------------------------
In order to generate a Tempest config file execute the following command:
.. code-block:: console
$ rally verify genconfig
2016-05-09 14:31:48.050 25906 INFO rally.verification.tempest.tempest [-] Tempest is not configured for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 14:31:48.050 25906 INFO rally.verification.tempest.tempest [-] Creating Tempest configuration file for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 14:31:56.738 25906 INFO rally.verification.tempest.tempest [-] Tempest configuration file has been successfully created!
By default, the command generates the config file for the current deployment,
but it is possible to generate the config file for any registered deployment
in Rally, using the **--deployment** argument.
.. code-block:: console
$ rally verify genconfig --deployment <UUID or name of a deployment>
Also, Rally allows users to specify any local path to the future config file,
e.g. */home/ubuntu/tempest.conf*, and in this case, Rally will generate the
*tempest.conf* file in the */home/ubuntu/* directory.
.. code-block:: console
$ rally verify genconfig --tempest-config /home/ubuntu/tempest.conf
2016-05-09 14:34:07.619 26204 INFO rally.verification.tempest.tempest [-] Tempest is not configured for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 14:34:07.619 26204 INFO rally.verification.tempest.tempest [-] Creating Tempest configuration file for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 14:34:09.449 26204 INFO rally.verification.tempest.tempest [-] Tempest configuration file has been successfully created!
Moreover, it is possible to override the existing Tempest config file by
providing the **--override** argument in the **rally verify genconfig**
command:
.. code-block:: console
$ rally verify genconfig --override
2016-05-09 14:35:11.608 26270 INFO rally.verification.tempest.tempest [-] Creating Tempest configuration file for deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 14:35:13.395 26270 INFO rally.verification.tempest.tempest [-] Tempest configuration file has been successfully created!
In order to see the generated config file execute the following command:
.. code-block:: console
$ rally verify showconfig
Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
[DEFAULT]
debug = True
log_file = tempest.log
use_stderr = False
[auth]
use_dynamic_credentials = True
...
To see the generated config file for a certain deployment specify the
**--deployment** argument.
.. code-block:: console
$ rally verify showconfig --deployment <UUID or name of a deployment>
Start a verification (rally verify start)
-----------------------------------------
In order to start a verification execute the following command:
.. code-block:: console
$ rally verify start
2016-05-09 14:54:07.446 27377 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 14:54:07.529 27377 INFO rally.verification.tempest.tempest [-] Verification de083a94-8b42-46fe-9cdd-2b6066f9c13c | Starting: Run verification.
2016-05-09 14:54:07.613 27377 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpcbg8BK
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpJEOWsG
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpD8Hsxu
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmp2UQC55
{1} setUpClass (tempest.api.baremetal.admin.test_ports_negative.TestPortsNegative) ... SKIPPED: TestPortsNegative skipped as Ironic is not available
{2} setUpClass (tempest.api.baremetal.admin.test_api_discovery.TestApiDiscovery) ... SKIPPED: TestApiDiscovery skipped as Ironic is not available
{2} setUpClass (tempest.api.baremetal.admin.test_chassis.TestChassis) ... SKIPPED: TestChassis skipped as Ironic is not available
{2} setUpClass (tempest.api.baremetal.admin.test_drivers.TestDrivers) ... SKIPPED: TestDrivers skipped as Ironic is not available
{3} setUpClass (tempest.api.baremetal.admin.test_nodes.TestNodes) ... SKIPPED: TestNodes skipped as Ironic is not available
{3} setUpClass (tempest.api.baremetal.admin.test_ports.TestPorts) ... SKIPPED: TestPorts skipped as Ironic is not available
{0} setUpClass (tempest.api.baremetal.admin.test_nodestates.TestNodeStates) ... SKIPPED: TestNodeStates skipped as Ironic is not available
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent [0.712663s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent [0.502782s] ... ok
{3} tempest.api.compute.admin.test_flavors_access_negative.FlavorsAccessNegativeTestJSON.test_add_flavor_access_duplicate [1.011901s] ... ok
...
By default, the command runs the full suite of Tempest tests for the current
deployment, but it is possible to run the tests for any registered deployment
in Rally, using the **--deployment** argument.
.. code-block:: console
$ rally verify start --deployment <UUID or name of a deployment>
Also, Rally allows users to specify a certain Tempest config file location to
use a certain Tempest config file for running the tests.
.. code-block:: console
$ rally verify start --tempest-config /home/ubuntu/tempest.conf
2016-05-09 15:24:02.474 29197 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 15:24:02.558 29197 INFO rally.verification.tempest.tempest [-] Verification 85b90b77-ee32-4e56-83ed-aabf306cb509 | Starting: Run verification.
2016-05-09 15:24:02.641 29197 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpqJcBEn
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmplKu5tZ
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpww2PLm
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmp6ip_UK
{0} setUpClass (tempest.api.baremetal.admin.test_api_discovery.TestApiDiscovery) ... SKIPPED: TestApiDiscovery skipped as Ironic is not available
{0} setUpClass (tempest.api.baremetal.admin.test_ports.TestPorts) ... SKIPPED: TestPorts skipped as Ironic is not available
{0} setUpClass (tempest.api.baremetal.admin.test_ports_negative.TestPortsNegative) ... SKIPPED: TestPortsNegative skipped as Ironic is not available
{3} setUpClass (tempest.api.baremetal.admin.test_nodestates.TestNodeStates) ... SKIPPED: TestNodeStates skipped as Ironic is not available
{3} setUpClass (tempest.api.compute.admin.test_fixed_ips.FixedIPsTestJson) ... SKIPPED: FixedIPsTestJson skipped as neutron is available
{1} setUpClass (tempest.api.baremetal.admin.test_chassis.TestChassis) ... SKIPPED: TestChassis skipped as Ironic is not available
{1} setUpClass (tempest.api.baremetal.admin.test_drivers.TestDrivers) ... SKIPPED: TestDrivers skipped as Ironic is not available
{1} setUpClass (tempest.api.baremetal.admin.test_nodes.TestNodes) ... SKIPPED: TestNodes skipped as Ironic is not available
{3} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram [0.642174s] ... ok
{0} tempest.api.compute.admin.test_aggregates_negative.AggregatesAdminNegativeTestJSON.test_aggregate_add_existent_host [1.069448s] ... ok
Also, there is a possibility to run a certain suite of Tempest tests, using
the **--set** argument.
.. code-block:: console
$ rally verify start --set compute
2016-05-09 14:56:45.258 27685 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 14:56:45.342 27685 INFO rally.verification.tempest.tempest [-] Verification ab0acb96-f664-438a-8323-198fe68d8a96 | Starting: Run verification.
2016-05-09 14:56:45.425 27685 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpm1QuaD
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpxmGWlN
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpsaG1BU
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpbZzU2y
{2} tempest.api.compute.admin.test_aggregates_negative.AggregatesAdminNegativeTestJSON.test_aggregate_add_existent_host [1.623109s] ... ok
{3} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az [1.125569s] ... FAILED
{2} tempest.api.compute.admin.test_aggregates_negative.AggregatesAdminNegativeTestJSON.test_aggregate_add_host_as_user [2.267328s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent [2.507743s] ... ok
{0} tempest.api.compute.admin.test_availability_zone.AZAdminV2TestJSON.test_get_availability_zone_list [1.132218s] ... ok
{0} tempest.api.compute.admin.test_availability_zone.AZAdminV2TestJSON.test_get_availability_zone_list_detail [0.518452s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent [0.796207s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents [0.735133s] ... ok
{2} tempest.api.compute.admin.test_aggregates_negative.AggregatesAdminNegativeTestJSON.test_aggregate_add_non_exist_host [1.941015s] ... ok
{2} tempest.api.compute.admin.test_aggregates_negative.AggregatesAdminNegativeTestJSON.test_aggregate_create_aggregate_name_length_exceeds_255 [0.183736s] ... ok
...
For now, available sets are **full**, **scenario**, **smoke**, **baremetal**,
**compute**, **database**, **data_processing**, **identity**, **image**,
**messaging**, **network**, **object_storage**, **orchestration**,
**telemetry**, **volume**.
Moreover, users can run a certain set of tests, using the **--regex** argument
and specifying a regular expression.
.. code-block:: console
$ rally verify start --regex tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON
2016-05-09 15:04:50.089 28117 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 15:04:50.173 28117 INFO rally.verification.tempest.tempest [-] Verification 32348bcc-edf1-4434-a10b-9449e2370a16 | Starting: Run verification.
2016-05-09 15:04:50.257 28117 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmp3QMRkn
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram [0.574063s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_verify_entry_in_list_details [0.539422s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_with_int_id [0.542389s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_with_none_id [0.525429s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_with_uuid_id [0.539657s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_list_flavor_without_extra_data [0.782256s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_server_with_non_public_flavor [0.536828s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_is_public_string_variations [1.931141s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_list_non_public_flavor [0.691936s] ... ok
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_list_public_flavor_with_other_user [0.569325s] ... ok
======
Totals
======
Ran: 10 tests in 18.0000 sec.
- Passed: 10
- Skipped: 0
- Expected Fail: 0
- Unexpected Success: 0
- Failed: 0
Sum of execute time for each test: 7.2324 sec.
==============
Worker Balance
==============
- Worker 0 (10 tests) => 0:00:07.236862
2016-05-09 15:05:10.473 28117 INFO rally.verification.tempest.tempest [-] Verification 32348bcc-edf1-4434-a10b-9449e2370a16 | Completed: Run verification.
2016-05-09 15:05:10.474 28117 INFO rally.verification.tempest.tempest [-] Verification 32348bcc-edf1-4434-a10b-9449e2370a16 | Starting: Saving verification results.
2016-05-09 15:05:10.677 28117 INFO rally.verification.tempest.tempest [-] Verification 32348bcc-edf1-4434-a10b-9449e2370a16 | Completed: Saving verification results.
Verification UUID: 32348bcc-edf1-4434-a10b-9449e2370a16
In such a way it is possible to run tests from a certain directory or class
and even run a single test.
.. code-block:: console
$ rally verify start --regex tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram
2016-05-09 15:06:18.088 28217 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 15:06:18.170 28217 INFO rally.verification.tempest.tempest [-] Verification dbd4bc2d-2b76-42b7-b737-fce86a92fbfa | Starting: Run verification.
2016-05-09 15:06:18.254 28217 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpoEkv6Q
{0} tempest.api.compute.admin.test_flavors.FlavorsAdminTestJSON.test_create_flavor_using_string_ram [0.547252s] ... ok
======
Totals
======
Ran: 1 tests in 10.0000 sec.
- Passed: 1
- Skipped: 0
- Expected Fail: 0
- Unexpected Success: 0
- Failed: 0
Sum of execute time for each test: 0.5473 sec.
==============
Worker Balance
==============
- Worker 0 (1 tests) => 0:00:00.547252
2016-05-09 15:06:31.207 28217 INFO rally.verification.tempest.tempest [-] Verification dbd4bc2d-2b76-42b7-b737-fce86a92fbfa | Completed: Run verification.
2016-05-09 15:06:31.207 28217 INFO rally.verification.tempest.tempest [-] Verification dbd4bc2d-2b76-42b7-b737-fce86a92fbfa | Starting: Saving verification results.
2016-05-09 15:06:31.750 28217 INFO rally.verification.tempest.tempest [-] Verification dbd4bc2d-2b76-42b7-b737-fce86a92fbfa | Completed: Saving verification results.
Verification UUID: dbd4bc2d-2b76-42b7-b737-fce86a92fbfa
Also, there is a possibility to run Tempest tests from a file. Users can
specify a list of tests in the file and run them, using the **--tests-file**
argument:
.. code-block:: console
$ cat some-file.txt
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent[id-1fc6bdc8-0b6d-4cc7-9f30-9b04fabe5b90]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent[id-470e0b89-386f-407b-91fd-819737d0b335]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents[id-6a326c69-654b-438a-80a3-34bcc454e138]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter[id-eabadde4-3cd7-4ec4-a4b5-5a936d2d4408]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent[id-dc9ffd51-1c50-4f0e-a820-ae6d2a568a9e]
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_get_details[id-eeef473c-7c52-494d-9f09-2ed7fc8fc036]
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_list[id-7f6a1cc5-2446-4cdb-9baa-b6ae0a919b72]
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_remove_host[id-c8e85064-e79b-4906-9931-c11c24294d02]
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_delete[id-0d148aa3-d54c-4317-aa8d-42040a475e20]
.. code-block:: console
$ rally verify start --tests-file some-file.txt
2016-05-09 15:09:10.864 28456 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 15:09:10.948 28456 INFO rally.verification.tempest.tempest [-] Verification 526b0c54-3805-48eb-8a04-4fec0aad3fe5 | Starting: Run verification.
2016-05-09 15:09:11.033 28456 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpjHUGip
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmp358n_n
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent [0.601839s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent [0.501781s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents [0.375056s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_get_details [1.036974s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter [0.640392s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_list [0.850647s] ... ok
{1} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent [0.371227s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_remove_host [0.803282s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_delete [0.635170s] ... ok
======
Totals
======
Ran: 9 tests in 11.0000 sec.
- Passed: 9
- Skipped: 0
- Expected Fail: 0
- Unexpected Success: 0
- Failed: 0
Sum of execute time for each test: 5.8164 sec.
==============
Worker Balance
==============
- Worker 0 (4 tests) => 0:00:03.328229
- Worker 1 (5 tests) => 0:00:02.492475
2016-05-09 15:09:24.668 28456 INFO rally.verification.tempest.tempest [-] Verification 526b0c54-3805-48eb-8a04-4fec0aad3fe5 | Completed: Run verification.
2016-05-09 15:09:24.669 28456 INFO rally.verification.tempest.tempest [-] Verification 526b0c54-3805-48eb-8a04-4fec0aad3fe5 | Starting: Saving verification results.
2016-05-09 15:09:24.872 28456 INFO rally.verification.tempest.tempest [-] Verification 526b0c54-3805-48eb-8a04-4fec0aad3fe5 | Completed: Saving verification results.
Verification UUID: 526b0c54-3805-48eb-8a04-4fec0aad3fe5
Sometimes users may want to use the specific concurrency for running tests
based on their deployments and available resources. In this case, they can use
the **--concurrency** argument to specify how many processes to use to run
Tempest tests. The default value (0) auto-detects CPU count.
.. code-block:: console
$ rally verify start --tests-file some-file.txt --concurrency 1
2016-05-09 15:10:39.050 28744 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 15:10:39.132 28744 INFO rally.verification.tempest.tempest [-] Verification 95fef399-0cfa-4843-ad50-b5ed974928dc | Starting: Run verification.
2016-05-09 15:10:39.216 28744 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpl_FWjP
{0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent [0.586906s] ... ok
{0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent [0.499466s] ... ok
{0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents [0.370536s] ... ok
{0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter [0.620824s] ... ok
{0} tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent [0.365948s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_get_details [0.942561s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_list [0.897054s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_remove_host [0.743319s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_delete [0.629131s] ... ok
======
Totals
======
Ran: 9 tests in 16.0000 sec.
- Passed: 9
- Skipped: 0
- Expected Fail: 0
- Unexpected Success: 0
- Failed: 0
Sum of execute time for each test: 5.6557 sec.
==============
Worker Balance
==============
- Worker 0 (9 tests) => 0:00:09.701447
2016-05-09 15:10:57.861 28744 INFO rally.verification.tempest.tempest [-] Verification 95fef399-0cfa-4843-ad50-b5ed974928dc | Completed: Run verification.
2016-05-09 15:10:57.861 28744 INFO rally.verification.tempest.tempest [-] Verification 95fef399-0cfa-4843-ad50-b5ed974928dc | Starting: Saving verification results.
2016-05-09 15:10:58.173 28744 INFO rally.verification.tempest.tempest [-] Verification 95fef399-0cfa-4843-ad50-b5ed974928dc | Completed: Saving verification results.
Verification UUID: 95fef399-0cfa-4843-ad50-b5ed974928dc
Sometimes users may want to re-run only those tests that failed in the last
verification. In order to re-run failed tests in the last verification execute
the following command:
.. code-block:: console
$ rally verify start --failing
For example, we have one failed test:
.. code-block:: console
$ rally verify start --regex tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON
2016-05-09 15:32:39.666 29727 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 15:32:39.751 29727 INFO rally.verification.tempest.tempest [-] Verification 1a71f82c-e59d-4b8f-9abd-8a98f53c2531 | Starting: Run verification.
2016-05-09 15:32:39.836 29727 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpFQO_SW
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az [0.572658s] ... FAILED
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_get_details [0.877286s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_list [0.938150s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_remove_host [0.902238s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_delete [0.633860s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_delete_with_az [0.654307s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_update_metadata_get_details [0.792414s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_update_with_az [0.823757s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_verify_entry_in_list [0.505302s] ... ok
==============================
Failed 1 tests - output below:
==============================
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az[id-96be03c7-570d-409c-90f8-e4db3c646996]
--------------------------------------------------------------------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "tempest/api/compute/admin/test_aggregates.py", line 214, in test_aggregate_add_host_create_server_with_az
self.client.add_host(aggregate['id'], host=self.host)
File "tempest/lib/services/compute/aggregates_client.py", line 92, in add_host
post_body)
File "tempest/lib/common/rest_client.py", line 259, in post
return self.request('POST', url, extra_headers, headers, body)
File "tempest/lib/services/compute/base_compute_client.py", line 53, in request
method, url, extra_headers, headers, body)
File "tempest/lib/common/rest_client.py", line 641, in request
resp, resp_body)
File "tempest/lib/common/rest_client.py", line 709, in _error_checker
raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: An object with that identifier already exists
Details: {u'message': u'Cannot add host node-2.domain.tld in aggregate 422: host exists', u'code': 409}
...
Now let's re-run it.
.. code-block:: console
$ rally verify start --failing
2016-05-09 15:36:17.389 30104 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 15:36:17.474 30104 INFO rally.verification.tempest.tempest [-] Verification f4e857a7-f032-452c-9ffb-dc42f0d2e124 | Starting: Run verification.
2016-05-09 15:36:17.559 30104 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmpiYREcb
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az [0.665381s] ... FAILED
==============================
Failed 1 tests - output below:
==============================
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az[id-96be03c7-570d-409c-90f8-e4db3c646996]
--------------------------------------------------------------------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "tempest/api/compute/admin/test_aggregates.py", line 214, in test_aggregate_add_host_create_server_with_az
self.client.add_host(aggregate['id'], host=self.host)
File "tempest/lib/services/compute/aggregates_client.py", line 92, in add_host
post_body)
File "tempest/lib/common/rest_client.py", line 259, in post
return self.request('POST', url, extra_headers, headers, body)
File "tempest/lib/services/compute/base_compute_client.py", line 53, in request
method, url, extra_headers, headers, body)
File "tempest/lib/common/rest_client.py", line 641, in request
resp, resp_body)
File "tempest/lib/common/rest_client.py", line 709, in _error_checker
raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: An object with that identifier already exists
Details: {u'message': u'Cannot add host node-2.domain.tld in aggregate 431: host exists', u'code': 409}
...
Also, it is possible to specify the path to a YAML file with a list of Tempest
tests that are expected to fail. In this case, the specified test will have the
**xfail** status instead of **fail** in the verification report. How to build
a verification report we tell you bellow.
.. code-block:: console
$ cat xfails-file.yaml
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az[id-96be03c7-570d-409c-90f8-e4db3c646996]: Some reason why the test fails
.. code-block:: console
$ rally verify start --regex tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON --xfails-file xfails-file.yaml
2016-05-09 16:31:36.236 772 INFO rally.api [-] Starting verification of deployment: 452f3c6b-119a-4054-a6aa-e4e3347824de
2016-05-09 16:31:36.320 772 INFO rally.verification.tempest.tempest [-] Verification 76d41e5d-bf24-4e16-a9ae-5a722f8fad05 | Starting: Run verification.
2016-05-09 16:31:36.402 772 INFO rally.verification.tempest.tempest [-] Using Tempest config file: /home/ubuntu/.rally/tempest/for-deployment-452f3c6b-119a-4054-a6aa-e4e3347824de/tempest.conf
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
OS_TEST_LOCK_PATH=${OS_TEST_LOCK_PATH:-${TMPDIR:-'/tmp'}} \
${PYTHON:-python} -m subunit.run discover -t ${OS_TOP_LEVEL:-./} ${OS_TEST_PATH:-./tempest/test_discover} --load-list /tmp/tmp9sB5u5
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az [0.625294s] ... FAILED
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_get_details [0.897577s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_list [0.865686s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_remove_host [0.710349s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_delete [0.620124s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_delete_with_az [0.642956s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_update_metadata_get_details [0.766061s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_update_with_az [0.795929s] ... ok
{0} tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_create_verify_entry_in_list [0.495695s] ... ok
==============================
Failed 1 tests - output below:
==============================
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az[id-96be03c7-570d-409c-90f8-e4db3c646996]
--------------------------------------------------------------------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "tempest/api/compute/admin/test_aggregates.py", line 214, in test_aggregate_add_host_create_server_with_az
self.client.add_host(aggregate['id'], host=self.host)
File "tempest/lib/services/compute/aggregates_client.py", line 92, in add_host
post_body)
File "tempest/lib/common/rest_client.py", line 259, in post
return self.request('POST', url, extra_headers, headers, body)
File "tempest/lib/services/compute/base_compute_client.py", line 53, in request
method, url, extra_headers, headers, body)
File "tempest/lib/common/rest_client.py", line 641, in request
resp, resp_body)
File "tempest/lib/common/rest_client.py", line 709, in _error_checker
raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: An object with that identifier already exists
Details: {u'message': u'Cannot add host node-2.domain.tld in aggregate 450: host exists', u'code': 409}
...
.. image:: ../images/Report-verify-xfail.png
:align: center
Finally, users can specify the **--system-wide** argument that will tell Rally
not to use the Tempest virtual environment for tests. In this case, it is
assumed that all Tempest requirements are already installed in the local
environment. This argument is useful when users don't have an Internet
connection to install requirements, but they have pre-installed ones in the
local environment.
.. code-block:: console
$ rally verify start --system-wide
...