Refactor run tests part of Tobiko quick start guide
So far, the Tobiko docs were duplicated in some places, different
(although duplicated), and some fragments were not relevant (or
outdated).
This patch refactors the run test cases part of Tobiko quick start
guide by modularizing the fragments ("spilling" fragments instead of
coping them), correcting typos, and deleting/updating irrelevant
information.
Change-Id: I685182cde8573e80ecaabe8216530b903df759e1
This commit is contained in:
Omer
parent
5cd15560e4
commit
3d04906a9f
14
doc/source/user/_run_faults.rst
Normal file
14
doc/source/user/_run_faults.rst
Normal file
@ -0,0 +1,14 @@
|
||||
Disruptive (or faults) test cases are used for testing that after inducing some critical
|
||||
disruption to the operation of the cloud, the services can get back to the expected state
|
||||
after a while. To execute them you can type::
|
||||
|
||||
tox -e faults
|
||||
|
||||
The faults induced by these test cases could be cloud nodes reboot,
|
||||
OpenStack services restart, virtual machines migrations, etc.
|
||||
|
||||
Please note that while scenario test cases are being executed in parallel (to
|
||||
speed up test case execution), disruptive test cases are only executed sequentially.
|
||||
This is because the operations executed by such cases could break some functionality
|
||||
for a short time and alter the regular state of the system which may be assumed by other
|
||||
test cases to be executed.
|
||||
14
doc/source/user/_run_scenario.rst
Normal file
14
doc/source/user/_run_scenario.rst
Normal file
@ -0,0 +1,14 @@
|
||||
Scenario test cases are used to create workloads that simulate real-world use
|
||||
of OpenStack. They create networks, virtual machines, ports, routers, etc.
|
||||
They also validate that these workloads are functioning.
|
||||
|
||||
Running Tobiko scenario test cases using Tox (may take some minutes to complete)::
|
||||
|
||||
tox -e scenario
|
||||
|
||||
Scenario test cases are also used to check that previously created resources are
|
||||
still up and working as expected. To ensure test cases will not create those
|
||||
resources again we can set `TOBIKO_PREVENT_CREATE` environment variable before
|
||||
re-running test cases::
|
||||
|
||||
TOBIKO_PREVENT_CREATE=yes tox -e scenario
|
||||
20
doc/source/user/_run_workflow.rst
Normal file
20
doc/source/user/_run_workflow.rst
Normal file
@ -0,0 +1,20 @@
|
||||
Scenario and disruptive test cases, which are being executed in a specific
|
||||
sequence, could be used to uncover more issues with the cloud than disruptive
|
||||
test cases alone.
|
||||
|
||||
- First ensure there are workloads properly running by running scenario test cases::
|
||||
|
||||
tox -e scenario
|
||||
|
||||
.. sidebar:: Note
|
||||
|
||||
As second step we may, instead, update or upgrade OpenStack nodes.
|
||||
|
||||
- Next we could execute disruptive test cases to "stress" the cloud::
|
||||
|
||||
tox -e faults
|
||||
|
||||
- Finally we might re-run scenario test cases to check that everything is still running
|
||||
as expected::
|
||||
|
||||
TOBIKO_PREVENT_CREATE=yes tox -e scenario
|
||||
29
doc/source/user/_test_cases_report_files.rst
Normal file
29
doc/source/user/_test_cases_report_files.rst
Normal file
@ -0,0 +1,29 @@
|
||||
After executing test cases we can view the results in greater detail via a small
|
||||
set of files:
|
||||
|
||||
- **test_results.html**:
|
||||
A user-browsable HTML view of test case results.
|
||||
- **test_results.log**:
|
||||
A log file with logging traces collected from every individual test case.
|
||||
- **test_results.subunit**:
|
||||
The original subunit binary file generated by test runner.
|
||||
- **test_results.xml**:
|
||||
An XML Junit file to be used, for example, to show test cases result by
|
||||
Jenkins CI server.
|
||||
|
||||
The names of the above files can be changed from the default value (*test_results*)
|
||||
to a custom one by setting the *TOX_REPORT_NAME* environment variable.
|
||||
|
||||
.. sidebar:: Legend
|
||||
|
||||
*{toxinidir}* stands for the Tobiko source files directory.
|
||||
|
||||
*{envname}* is the name of the Tox enviroment to be executed (IE scenario,
|
||||
faults, etc.)
|
||||
|
||||
The above files are saved into a folder that can be specified with
|
||||
*TOX_REPORT_DIR* environment variable.
|
||||
|
||||
By default the full path of the report directory is made from the below::
|
||||
|
||||
{toxinidir}/report/{envname}
|
||||
@ -41,130 +41,39 @@ Create a virtual environment with tox
|
||||
|
||||
.. include:: _conf_venv_with_tox.rst
|
||||
|
||||
Running Test Cases
|
||||
------------------
|
||||
Run Test Cases
|
||||
--------------
|
||||
|
||||
Running Scenario Test Cases
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
The next section is a quick guide about running some test cases.
|
||||
For more information, please see our
|
||||
:ref:`Tobiko Test Cases Execution Guide<tobiko-test-case-execution-guide>`
|
||||
|
||||
Scenario test cases are used to create workloads that simulate real-world use
|
||||
of OpenStack. They create networks, virtual machines, ports, routers, etc.
|
||||
They also test validate that these workloads functioning.
|
||||
Before running test cases, make sure you
|
||||
:ref:`configure tobiko logging <conf_logging>` according to
|
||||
your needs.
|
||||
|
||||
Running Tobiko scenario test cases using Tox (may take some time to complete (minutes))::
|
||||
.. note::
|
||||
Unlike other testing frameworks, **Tobiko does not delete its**
|
||||
**resources after test cases finish their execution**.
|
||||
You may clean up tobiko workloads after the execution manually, for example
|
||||
heat stacks and glance images.
|
||||
|
||||
tox -e scenario
|
||||
Run Scenario Test Cases
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To list Heat stacks and Glance images created by test cases::
|
||||
.. include:: _run_scenario.rst
|
||||
|
||||
tox -e venv -- openstack image list
|
||||
tox -e venv -- openstack stack list
|
||||
Run Disruptive Test Cases
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Scenario test cases are also used to check that previously created resources are
|
||||
still up and working as expected. To ensure test cases will not create those
|
||||
resources again we can set `TOBIKO_PREVENT_CREATE` environment variable before
|
||||
re-running test cases::
|
||||
.. include:: _run_faults.rst
|
||||
|
||||
TOBIKO_PREVENT_CREATE=yes tox -e scenario
|
||||
|
||||
|
||||
Cleaning Up Tobiko Workloads
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Once Tobiko test cases have been executed, we may want to clean up all
|
||||
workloads remaining on the cloud so that we restore it to its original state.
|
||||
|
||||
|
||||
Cleaning Up Heat Stacks
|
||||
++++++++++++++++++++++++
|
||||
|
||||
Because Tobiko is using Heat stacks for orchestrating the creation of most of
|
||||
the resources, deleting all stacks created with Tobiko will clean up
|
||||
almost all resources::
|
||||
|
||||
tox -e venv -- bash -c 'openstack stack list -f value -c ID | xargs openstack stack delete'
|
||||
|
||||
|
||||
Cleaning Up Glance Images
|
||||
++++++++++++++++++++++++++
|
||||
|
||||
Because Heat doesn't support creation of Glance images, Tobiko implements some
|
||||
specific fixtures to download images from the Web and upload them to the Glance
|
||||
service::
|
||||
|
||||
tox -e venv -- bash -c 'openstack image list -f value -c ID | xargs openstack image delete'
|
||||
|
||||
|
||||
Running Disruptive Test Cases
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Disruptive test cases are used for testing that after inducing some critical
|
||||
disruption to the operation of the cloud, the services return working as expected after
|
||||
a while. To execute them you can type::
|
||||
|
||||
tox -e faults
|
||||
|
||||
The faults induced by these test cases could be cloud nodes reboot,
|
||||
OpenStack services restart, virtual machines migrations, etc.
|
||||
|
||||
Please note that while scenario test cases are being executed in parallel (to
|
||||
speed up test case execution), disruptive test case are only executed sequentially.
|
||||
This is because the operations executed by such cases could break some functionality
|
||||
for a short time and alter the regular state of the system which may be assumed by other
|
||||
test cases to be executed.
|
||||
|
||||
|
||||
Running the Tobiko Workflow
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Scenario and disruptive test cases, being executed in a specific sequence could
|
||||
be used to uncover more issues with the cloud than disruptive test cases alone.
|
||||
|
||||
- First ensure there are workloads properly running by running scenario test cases::
|
||||
|
||||
tox -e scenario
|
||||
|
||||
.. sidebar:: Note
|
||||
|
||||
As second step we may, instead, update or upgrade OpenStack nodes.
|
||||
|
||||
- Next we could execute disruptive test cases to "stress" the cloud::
|
||||
|
||||
tox -e faults
|
||||
|
||||
- Finally we might re-run scenario test cases to check that everything is still running
|
||||
as expected::
|
||||
|
||||
TOBIKO_PREVENT_CREATE=yes tox -e scenario
|
||||
Run the Tobiko Workflow
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. include:: _run_workflow.rst
|
||||
|
||||
Test Cases Report Files
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
After executing test cases we can view the results in greater detail via a small
|
||||
set of files:
|
||||
|
||||
- **test_results.html**:
|
||||
A user-browseable HTML view of test case results
|
||||
- **test_results.log**:
|
||||
a log file with logging traces collected from every individual test case
|
||||
- **test_results.xml**:
|
||||
an XML Junit file to be used, for example, to show test cases result by
|
||||
Jenkins CI server
|
||||
|
||||
The names of the above files can be changed from the default value (*test_results*)
|
||||
to a custom one by setting the *TOX_REPORT_NAME* environment variable.
|
||||
|
||||
.. sidebar:: Legend
|
||||
|
||||
*{toxinidir}* stand for the Tobiko source files directory.
|
||||
|
||||
*{envname}* is the name of the Tox enviroment to be executed (IE scenario,
|
||||
faults, etc.)
|
||||
|
||||
The above files are saved into a folder that can be specified with
|
||||
*TOX_REPORT_DIR* environment variable.
|
||||
|
||||
By default the full path of the report directory is made from the below::
|
||||
|
||||
{toxinidir}/report/{envname}
|
||||
.. include:: _test_cases_report_files.rst
|
||||
|
||||
Loading…
Reference in New Issue
Block a user