diff --git a/doc/source/user/_run_faults.rst b/doc/source/user/_run_faults.rst new file mode 100644 index 000000000..f4346eec7 --- /dev/null +++ b/doc/source/user/_run_faults.rst @@ -0,0 +1,14 @@ +Disruptive (or faults) test cases are used for testing that after inducing some critical +disruption to the operation of the cloud, the services can get back to the expected state +after a while. To execute them you can type:: + + tox -e faults + +The faults induced by these test cases could be cloud nodes reboot, +OpenStack services restart, virtual machines migrations, etc. + +Please note that while scenario test cases are being executed in parallel (to +speed up test case execution), disruptive test cases are only executed sequentially. +This is because the operations executed by such cases could break some functionality +for a short time and alter the regular state of the system which may be assumed by other +test cases to be executed. diff --git a/doc/source/user/_run_scenario.rst b/doc/source/user/_run_scenario.rst new file mode 100644 index 000000000..0b612cf95 --- /dev/null +++ b/doc/source/user/_run_scenario.rst @@ -0,0 +1,14 @@ +Scenario test cases are used to create workloads that simulate real-world use +of OpenStack. They create networks, virtual machines, ports, routers, etc. +They also validate that these workloads are functioning. + +Running Tobiko scenario test cases using Tox (may take some minutes to complete):: + + tox -e scenario + +Scenario test cases are also used to check that previously created resources are +still up and working as expected. To ensure test cases will not create those +resources again we can set `TOBIKO_PREVENT_CREATE` environment variable before +re-running test cases:: + + TOBIKO_PREVENT_CREATE=yes tox -e scenario diff --git a/doc/source/user/_run_workflow.rst b/doc/source/user/_run_workflow.rst new file mode 100644 index 000000000..7a60adec1 --- /dev/null +++ b/doc/source/user/_run_workflow.rst @@ -0,0 +1,20 @@ +Scenario and disruptive test cases, which are being executed in a specific +sequence, could be used to uncover more issues with the cloud than disruptive +test cases alone. + +- First ensure there are workloads properly running by running scenario test cases:: + + tox -e scenario + +.. sidebar:: Note + + As second step we may, instead, update or upgrade OpenStack nodes. + +- Next we could execute disruptive test cases to "stress" the cloud:: + + tox -e faults + +- Finally we might re-run scenario test cases to check that everything is still running + as expected:: + + TOBIKO_PREVENT_CREATE=yes tox -e scenario diff --git a/doc/source/user/_test_cases_report_files.rst b/doc/source/user/_test_cases_report_files.rst new file mode 100644 index 000000000..e5aa48fd7 --- /dev/null +++ b/doc/source/user/_test_cases_report_files.rst @@ -0,0 +1,29 @@ +After executing test cases we can view the results in greater detail via a small +set of files: + + - **test_results.html**: + A user-browsable HTML view of test case results. + - **test_results.log**: + A log file with logging traces collected from every individual test case. + - **test_results.subunit**: + The original subunit binary file generated by test runner. + - **test_results.xml**: + An XML Junit file to be used, for example, to show test cases result by + Jenkins CI server. + +The names of the above files can be changed from the default value (*test_results*) +to a custom one by setting the *TOX_REPORT_NAME* environment variable. + +.. sidebar:: Legend + + *{toxinidir}* stands for the Tobiko source files directory. + + *{envname}* is the name of the Tox enviroment to be executed (IE scenario, + faults, etc.) + +The above files are saved into a folder that can be specified with +*TOX_REPORT_DIR* environment variable. + +By default the full path of the report directory is made from the below:: + + {toxinidir}/report/{envname} diff --git a/doc/source/user/quick-start.rst b/doc/source/user/quick-start.rst index 1dd01c272..0d80d5b5e 100644 --- a/doc/source/user/quick-start.rst +++ b/doc/source/user/quick-start.rst @@ -41,130 +41,39 @@ Create a virtual environment with tox .. include:: _conf_venv_with_tox.rst -Running Test Cases ------------------- +Run Test Cases +-------------- -Running Scenario Test Cases -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The next section is a quick guide about running some test cases. +For more information, please see our +:ref:`Tobiko Test Cases Execution Guide` -Scenario test cases are used to create workloads that simulate real-world use -of OpenStack. They create networks, virtual machines, ports, routers, etc. -They also test validate that these workloads functioning. +Before running test cases, make sure you +:ref:`configure tobiko logging ` according to +your needs. -Running Tobiko scenario test cases using Tox (may take some time to complete (minutes)):: +.. note:: + Unlike other testing frameworks, **Tobiko does not delete its** + **resources after test cases finish their execution**. + You may clean up tobiko workloads after the execution manually, for example + heat stacks and glance images. - tox -e scenario +Run Scenario Test Cases +~~~~~~~~~~~~~~~~~~~~~~~ -To list Heat stacks and Glance images created by test cases:: +.. include:: _run_scenario.rst - tox -e venv -- openstack image list - tox -e venv -- openstack stack list +Run Disruptive Test Cases +~~~~~~~~~~~~~~~~~~~~~~~~~ -Scenario test cases are also used to check that previously created resources are -still up and working as expected. To ensure test cases will not create those -resources again we can set `TOBIKO_PREVENT_CREATE` environment variable before -re-running test cases:: +.. include:: _run_faults.rst - TOBIKO_PREVENT_CREATE=yes tox -e scenario - - -Cleaning Up Tobiko Workloads -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Once Tobiko test cases have been executed, we may want to clean up all -workloads remaining on the cloud so that we restore it to its original state. - - -Cleaning Up Heat Stacks -++++++++++++++++++++++++ - -Because Tobiko is using Heat stacks for orchestrating the creation of most of -the resources, deleting all stacks created with Tobiko will clean up -almost all resources:: - - tox -e venv -- bash -c 'openstack stack list -f value -c ID | xargs openstack stack delete' - - -Cleaning Up Glance Images -++++++++++++++++++++++++++ - -Because Heat doesn't support creation of Glance images, Tobiko implements some -specific fixtures to download images from the Web and upload them to the Glance -service:: - - tox -e venv -- bash -c 'openstack image list -f value -c ID | xargs openstack image delete' - - -Running Disruptive Test Cases -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Disruptive test cases are used for testing that after inducing some critical -disruption to the operation of the cloud, the services return working as expected after -a while. To execute them you can type:: - - tox -e faults - -The faults induced by these test cases could be cloud nodes reboot, -OpenStack services restart, virtual machines migrations, etc. - -Please note that while scenario test cases are being executed in parallel (to -speed up test case execution), disruptive test case are only executed sequentially. -This is because the operations executed by such cases could break some functionality -for a short time and alter the regular state of the system which may be assumed by other -test cases to be executed. - - -Running the Tobiko Workflow -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Scenario and disruptive test cases, being executed in a specific sequence could -be used to uncover more issues with the cloud than disruptive test cases alone. - -- First ensure there are workloads properly running by running scenario test cases:: - - tox -e scenario - -.. sidebar:: Note - - As second step we may, instead, update or upgrade OpenStack nodes. - -- Next we could execute disruptive test cases to "stress" the cloud:: - - tox -e faults - -- Finally we might re-run scenario test cases to check that everything is still running - as expected:: - - TOBIKO_PREVENT_CREATE=yes tox -e scenario +Run the Tobiko Workflow +~~~~~~~~~~~~~~~~~~~~~~~ +.. include:: _run_workflow.rst Test Cases Report Files ~~~~~~~~~~~~~~~~~~~~~~~ -After executing test cases we can view the results in greater detail via a small -set of files: - - - **test_results.html**: - A user-browseable HTML view of test case results - - **test_results.log**: - a log file with logging traces collected from every individual test case - - **test_results.xml**: - an XML Junit file to be used, for example, to show test cases result by - Jenkins CI server - -The names of the above files can be changed from the default value (*test_results*) -to a custom one by setting the *TOX_REPORT_NAME* environment variable. - -.. sidebar:: Legend - - *{toxinidir}* stand for the Tobiko source files directory. - - *{envname}* is the name of the Tox enviroment to be executed (IE scenario, - faults, etc.) - -The above files are saved into a folder that can be specified with -*TOX_REPORT_DIR* environment variable. - -By default the full path of the report directory is made from the below:: - - {toxinidir}/report/{envname} +.. include:: _test_cases_report_files.rst