Merge "Refactor run tests part of Tobiko quick start guide"
This commit is contained in:
Zuul
Gerrit Code Review
commit
c4749e94cb
|
|
@ -0,0 +1,14 @@
|
|||
Disruptive (or faults) test cases are used for testing that after inducing some critical
|
||||
disruption to the operation of the cloud, the services can get back to the expected state
|
||||
after a while. To execute them you can type::
|
||||
|
||||
tox -e faults
|
||||
|
||||
The faults induced by these test cases could be cloud nodes reboot,
|
||||
OpenStack services restart, virtual machines migrations, etc.
|
||||
|
||||
Please note that while scenario test cases are being executed in parallel (to
|
||||
speed up test case execution), disruptive test cases are only executed sequentially.
|
||||
This is because the operations executed by such cases could break some functionality
|
||||
for a short time and alter the regular state of the system which may be assumed by other
|
||||
test cases to be executed.
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
Scenario test cases are used to create workloads that simulate real-world use
|
||||
of OpenStack. They create networks, virtual machines, ports, routers, etc.
|
||||
They also validate that these workloads are functioning.
|
||||
|
||||
Running Tobiko scenario test cases using Tox (may take some minutes to complete)::
|
||||
|
||||
tox -e scenario
|
||||
|
||||
Scenario test cases are also used to check that previously created resources are
|
||||
still up and working as expected. To ensure test cases will not create those
|
||||
resources again we can set `TOBIKO_PREVENT_CREATE` environment variable before
|
||||
re-running test cases::
|
||||
|
||||
TOBIKO_PREVENT_CREATE=yes tox -e scenario
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
Scenario and disruptive test cases, which are being executed in a specific
|
||||
sequence, could be used to uncover more issues with the cloud than disruptive
|
||||
test cases alone.
|
||||
|
||||
- First ensure there are workloads properly running by running scenario test cases::
|
||||
|
||||
tox -e scenario
|
||||
|
||||
.. sidebar:: Note
|
||||
|
||||
As second step we may, instead, update or upgrade OpenStack nodes.
|
||||
|
||||
- Next we could execute disruptive test cases to "stress" the cloud::
|
||||
|
||||
tox -e faults
|
||||
|
||||
- Finally we might re-run scenario test cases to check that everything is still running
|
||||
as expected::
|
||||
|
||||
TOBIKO_PREVENT_CREATE=yes tox -e scenario
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
After executing test cases we can view the results in greater detail via a small
|
||||
set of files:
|
||||
|
||||
- **test_results.html**:
|
||||
A user-browsable HTML view of test case results.
|
||||
- **test_results.log**:
|
||||
A log file with logging traces collected from every individual test case.
|
||||
- **test_results.subunit**:
|
||||
The original subunit binary file generated by test runner.
|
||||
- **test_results.xml**:
|
||||
An XML Junit file to be used, for example, to show test cases result by
|
||||
Jenkins CI server.
|
||||
|
||||
The names of the above files can be changed from the default value (*test_results*)
|
||||
to a custom one by setting the *TOX_REPORT_NAME* environment variable.
|
||||
|
||||
.. sidebar:: Legend
|
||||
|
||||
*{toxinidir}* stands for the Tobiko source files directory.
|
||||
|
||||
*{envname}* is the name of the Tox enviroment to be executed (IE scenario,
|
||||
faults, etc.)
|
||||
|
||||
The above files are saved into a folder that can be specified with
|
||||
*TOX_REPORT_DIR* environment variable.
|
||||
|
||||
By default the full path of the report directory is made from the below::
|
||||
|
||||
{toxinidir}/report/{envname}
|
||||
|
|
@ -41,130 +41,39 @@ Create a virtual environment with tox
|
|||
|
||||
.. include:: _conf_venv_with_tox.rst
|
||||
|
||||
Running Test Cases
|
||||
------------------
|
||||
Run Test Cases
|
||||
--------------
|
||||
|
||||
Running Scenario Test Cases
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
The next section is a quick guide about running some test cases.
|
||||
For more information, please see our
|
||||
:ref:`Tobiko Test Cases Execution Guide<tobiko-test-case-execution-guide>`
|
||||
|
||||
Scenario test cases are used to create workloads that simulate real-world use
|
||||
of OpenStack. They create networks, virtual machines, ports, routers, etc.
|
||||
They also test validate that these workloads functioning.
|
||||
Before running test cases, make sure you
|
||||
:ref:`configure tobiko logging <conf_logging>` according to
|
||||
your needs.
|
||||
|
||||
Running Tobiko scenario test cases using Tox (may take some time to complete (minutes))::
|
||||
.. note::
|
||||
Unlike other testing frameworks, **Tobiko does not delete its**
|
||||
**resources after test cases finish their execution**.
|
||||
You may clean up tobiko workloads after the execution manually, for example
|
||||
heat stacks and glance images.
|
||||
|
||||
tox -e scenario
|
||||
Run Scenario Test Cases
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To list Heat stacks and Glance images created by test cases::
|
||||
.. include:: _run_scenario.rst
|
||||
|
||||
tox -e venv -- openstack image list
|
||||
tox -e venv -- openstack stack list
|
||||
Run Disruptive Test Cases
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Scenario test cases are also used to check that previously created resources are
|
||||
still up and working as expected. To ensure test cases will not create those
|
||||
resources again we can set `TOBIKO_PREVENT_CREATE` environment variable before
|
||||
re-running test cases::
|
||||
.. include:: _run_faults.rst
|
||||
|
||||
TOBIKO_PREVENT_CREATE=yes tox -e scenario
|
||||
|
||||
|
||||
Cleaning Up Tobiko Workloads
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Once Tobiko test cases have been executed, we may want to clean up all
|
||||
workloads remaining on the cloud so that we restore it to its original state.
|
||||
|
||||
|
||||
Cleaning Up Heat Stacks
|
||||
++++++++++++++++++++++++
|
||||
|
||||
Because Tobiko is using Heat stacks for orchestrating the creation of most of
|
||||
the resources, deleting all stacks created with Tobiko will clean up
|
||||
almost all resources::
|
||||
|
||||
tox -e venv -- bash -c 'openstack stack list -f value -c ID | xargs openstack stack delete'
|
||||
|
||||
|
||||
Cleaning Up Glance Images
|
||||
++++++++++++++++++++++++++
|
||||
|
||||
Because Heat doesn't support creation of Glance images, Tobiko implements some
|
||||
specific fixtures to download images from the Web and upload them to the Glance
|
||||
service::
|
||||
|
||||
tox -e venv -- bash -c 'openstack image list -f value -c ID | xargs openstack image delete'
|
||||
|
||||
|
||||
Running Disruptive Test Cases
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Disruptive test cases are used for testing that after inducing some critical
|
||||
disruption to the operation of the cloud, the services return working as expected after
|
||||
a while. To execute them you can type::
|
||||
|
||||
tox -e faults
|
||||
|
||||
The faults induced by these test cases could be cloud nodes reboot,
|
||||
OpenStack services restart, virtual machines migrations, etc.
|
||||
|
||||
Please note that while scenario test cases are being executed in parallel (to
|
||||
speed up test case execution), disruptive test case are only executed sequentially.
|
||||
This is because the operations executed by such cases could break some functionality
|
||||
for a short time and alter the regular state of the system which may be assumed by other
|
||||
test cases to be executed.
|
||||
|
||||
|
||||
Running the Tobiko Workflow
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Scenario and disruptive test cases, being executed in a specific sequence could
|
||||
be used to uncover more issues with the cloud than disruptive test cases alone.
|
||||
|
||||
- First ensure there are workloads properly running by running scenario test cases::
|
||||
|
||||
tox -e scenario
|
||||
|
||||
.. sidebar:: Note
|
||||
|
||||
As second step we may, instead, update or upgrade OpenStack nodes.
|
||||
|
||||
- Next we could execute disruptive test cases to "stress" the cloud::
|
||||
|
||||
tox -e faults
|
||||
|
||||
- Finally we might re-run scenario test cases to check that everything is still running
|
||||
as expected::
|
||||
|
||||
TOBIKO_PREVENT_CREATE=yes tox -e scenario
|
||||
Run the Tobiko Workflow
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: _run_workflow.rst
|
||||
|
||||
Test Cases Report Files
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
After executing test cases we can view the results in greater detail via a small
|
||||
set of files:
|
||||
|
||||
- **test_results.html**:
|
||||
A user-browseable HTML view of test case results
|
||||
- **test_results.log**:
|
||||
a log file with logging traces collected from every individual test case
|
||||
- **test_results.xml**:
|
||||
an XML Junit file to be used, for example, to show test cases result by
|
||||
Jenkins CI server
|
||||
|
||||
The names of the above files can be changed from the default value (*test_results*)
|
||||
to a custom one by setting the *TOX_REPORT_NAME* environment variable.
|
||||
|
||||
.. sidebar:: Legend
|
||||
|
||||
*{toxinidir}* stand for the Tobiko source files directory.
|
||||
|
||||
*{envname}* is the name of the Tox enviroment to be executed (IE scenario,
|
||||
faults, etc.)
|
||||
|
||||
The above files are saved into a folder that can be specified with
|
||||
*TOX_REPORT_DIR* environment variable.
|
||||
|
||||
By default the full path of the report directory is made from the below::
|
||||
|
||||
{toxinidir}/report/{envname}
|
||||
.. include:: _test_cases_report_files.rst
|
||||
|
|
|
|||
Loading…
Reference in New Issue