4911a1cbc2
Also reorganized a few things in a way I think makes them a bit clearer, feel free to leave feedback in reviews Change-Id: I879cb957ef35f9b6e1176a4528d8f9f5f33a4c90
361 lines
10 KiB
ReStructuredText
361 lines
10 KiB
ReStructuredText
.. _tobiko-quick-start-guide:
|
|
|
|
========================
|
|
Tobiko Quick Start Guide
|
|
========================
|
|
|
|
|
|
Document Overview
|
|
-----------------
|
|
|
|
This document describes how to install execute Tobiko test cases
|
|
using `Tox <https://tox.readthedocs.io/en/latest/>`__.
|
|
|
|
.. sidebar:: See also
|
|
|
|
To install Tobiko inside a virutalenv please read
|
|
:ref:`tobiko-installation-guide`.
|
|
|
|
To configure Tobiko please read :ref:`tobiko-configuration-guide`.
|
|
|
|
To run Tobiko scenario test cases please look at
|
|
:ref:`tobiko-test-case-execution-guide`.
|
|
|
|
|
|
Install Dependencies
|
|
--------------------
|
|
|
|
Install Basic Python Packages
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Make sure Git and Python 3 are installed on your system.
|
|
|
|
For instance on RedHat Linux / Fedora::
|
|
|
|
sudo dnf install -y git python3 which
|
|
|
|
Check your Python 3 version is 3.6 or greater::
|
|
|
|
python3 --version
|
|
|
|
Make sure pip is installed and up-to date::
|
|
|
|
curl https://bootstrap.pypa.io/get-pip.py | sudo python3
|
|
|
|
Check installed Pip version::
|
|
|
|
python3 -m pip --version
|
|
|
|
Make sure basic Python packages are installed and up-to-date::
|
|
|
|
sudo python3 -m pip install --upgrade setuptools wheel virtualenv tox six
|
|
|
|
Check installed Tox version::
|
|
|
|
tox --version
|
|
|
|
|
|
Clone the Tobiko repository
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Clone the Tobiko repository using Git::
|
|
|
|
git clone https://opendev.org/x/tobiko.git
|
|
cd tobiko
|
|
|
|
|
|
Install Missing Binary Packages
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Install required binary packages::
|
|
|
|
tools/install-bindeps.sh
|
|
|
|
|
|
Configure Logging Options
|
|
-------------------------
|
|
|
|
Test cases load most of the configuration parameters from an INI configuration file,
|
|
typically found at one of the following locations:
|
|
|
|
- ./tobiko.conf (Tobiko source files directory)
|
|
- ~/.tobiko/tobiko.conf
|
|
- /etc/tobiko/tobiko.conf
|
|
|
|
Create it in the Tobiko source directory with the following (or as your preferences). Example::
|
|
|
|
[DEFAULT]
|
|
debug = True
|
|
log_file = tobiko.log
|
|
|
|
The file 'tobiko.log' is the default file where test cases and the Python framework
|
|
are going to write their logging messages. By setting debug as 'true' you
|
|
ensure that messages with the lowest logging level are written there (DEBUG level).
|
|
The log_file location specified above is relative to the tobiko.conf file
|
|
location. In this example it is the Tobiko source files' directory itself.
|
|
|
|
|
|
Configure Tobiko Credentials
|
|
----------------------------
|
|
|
|
In order to run the OpenStack test cases you'll need to set up Keystone
|
|
credentials. You can do it in one of following ways:
|
|
|
|
- :ref:`credentials-from-clouds-file`
|
|
- :ref:`credentials-from-env`
|
|
- :ref:`credentials-from-config`
|
|
|
|
|
|
.. _credentials-from-clouds-file:
|
|
|
|
|
|
Set Tobiko Credentials from clouds.yaml file
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Make sure that in any one of below locations there is a valid
|
|
`OpenStack clouds file <https://docs.openstack.org/python-openstackclient/pike/configuration/index.html#clouds-yaml>`__
|
|
containing valid Keystone credentials:
|
|
|
|
- Tobiko source files directory
|
|
- ~/.config/openstack
|
|
- /etc/openstack
|
|
|
|
|
|
Finally, you will need to specify which credentials Tobiko should pick up via
|
|
'OS_CLOUD' environment variable or by specifying the cloud_name in tobiko.conf file
|
|
(section 'keystone', option 'cloud_name').
|
|
|
|
|
|
Specify 'OS_CLOUD' environment variable
|
|
+++++++++++++++++++++++++++++++++++++++
|
|
|
|
Ensure *OS_CLOUD* environment variable is defined before executing Tobiko test
|
|
cases::
|
|
|
|
export OS_CLOUD=<cloud_name>
|
|
|
|
|
|
Please choose a valid cloud_name from your clouds.yaml file.
|
|
|
|
|
|
Specify cloud_name in tobiko.conf file
|
|
++++++++++++++++++++++++++++++++++++++
|
|
|
|
|
|
Create file `tobiko.conf` in Tobiko sources folder adding a section like below::
|
|
|
|
[keystone]
|
|
cloud_name = <cloud_name>
|
|
|
|
|
|
Please choose a valid cloud_name from your clouds.yaml file.
|
|
|
|
|
|
.. _credentials-from-env:
|
|
|
|
|
|
Set Tobiko Credentials Via Environment Variables
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. sidebar:: See also
|
|
|
|
For more details about supported environment variables please read
|
|
:ref:`authentication-environment-variables` section.
|
|
|
|
You can use an existing shell RC file that is valid for
|
|
`Python OpenStack client <https://docs.openstack.org/python-openstackclient/latest/cli/man/openstack.html#environment-variables>`__
|
|
::
|
|
|
|
source openstackrc
|
|
|
|
An example of 'openstackrc' file could looks like below::
|
|
|
|
export OS_IDENTITY_API_VERSION=3
|
|
export OS_AUTH_URL=https://my_cloud:13000/v3
|
|
export OS_USERNAME=admin
|
|
export OS_PASSWORD=secret
|
|
export OS_PROJECT_NAME=admin
|
|
export OS_USER_DOMAIN_NAME=Default
|
|
export OS_PROJECT_DOMAIN_NAME=Default
|
|
|
|
|
|
.. _credentials-from-config:
|
|
|
|
Set Tobiko Credentials Via :ref:`tobiko-conf` File
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. sidebar:: See also
|
|
|
|
For more details about supported configuration options please read
|
|
:ref:`authentication-configuration` section.
|
|
|
|
Create a file at `~/.tobiko/tobiko.conf` and add a section as in the
|
|
example below (Or add to your existing file)::
|
|
|
|
[keystone]
|
|
api_version = 3
|
|
auth_url = http://my_cloud:13000/v3
|
|
username = admin
|
|
password = secret
|
|
project_name = admin
|
|
user_domain_name = Default
|
|
project_domain_name = Default
|
|
|
|
|
|
Setup Required Resources
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
A public Neutron network is required to be able to execute Tobiko scenario test
|
|
cases by creating a floating IP port on it.
|
|
|
|
To execute commands from a virtualenv created by Tox you can type as below::
|
|
|
|
tox -e venv -- <your-commands>
|
|
|
|
You need to make sure :ref:`authentication-environment-variables` are properly
|
|
set so you can list available public networks::
|
|
|
|
tox -e venv -- openstack network list
|
|
|
|
If there isn't any valid public network, you need to create one before running
|
|
Tobiko OpenStack test cases. Please refer to the `Neutron documentation <https://docs.openstack.org/neutron/latest/>`__
|
|
for additional information.
|
|
|
|
If there is a valid public network for creating floating-IP ports on,
|
|
Tobiko tests cases will automatically use it. To explicitly select a network,
|
|
please add a reference to the network in
|
|
:ref:`tobiko-conf` file::
|
|
|
|
[neutron]
|
|
floating_network = public
|
|
|
|
Running Test Cases
|
|
------------------
|
|
|
|
Running Scenario Test Cases
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Scenario test cases are used to create workloads that simulate real-world use
|
|
of OpenStack. They create networks, virtual machines, ports, routers, etc.
|
|
They also test validate that these workloads functioning.
|
|
|
|
Running Tobiko scenario test cases using Tox (may take some time to complete (minutes))::
|
|
|
|
tox -e scenario
|
|
|
|
To list Heat stacks and Glance images created by test cases::
|
|
|
|
tox -e venv -- openstack image list
|
|
tox -e venv -- openstack stack list
|
|
|
|
Scenario test cases are also used to check that previously created resources are
|
|
still up and working as expected. To ensure test cases will not create those
|
|
resources again we can set `TOBIKO_PREVENT_CREATE` environment variable before
|
|
re-running test cases::
|
|
|
|
TOBIKO_PREVENT_CREATE=yes tox -e scenario
|
|
|
|
|
|
Cleaning Up Tobiko Workloads
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Once Tobiko test cases have been executed, we may want to clean up all
|
|
workloads remaining on the cloud so that we restore it to its original state.
|
|
|
|
|
|
Cleaning Up Heat Stacks
|
|
++++++++++++++++++++++++
|
|
|
|
Because Tobiko is using Heat stacks for orchestrating the creation of most of
|
|
the resources, deleting all stacks created with Tobiko will clean up
|
|
almost all resources::
|
|
|
|
tox -e venv -- bash -c 'openstack stack list -f value -c ID | xargs openstack stack delete'
|
|
|
|
|
|
Cleaning Up Glance Images
|
|
++++++++++++++++++++++++++
|
|
|
|
Because Heat doesn't support creation of Glance images, Tobiko implements some
|
|
specific fixtures to download images from the Web and upload them to the Glance
|
|
service::
|
|
|
|
tox -e venv -- bash -c 'openstack image list -f value -c ID | xargs openstack image delete'
|
|
|
|
|
|
Running Disruptive Test Cases
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Disruptive test cases are used for testing that after inducing some critical
|
|
disruption to the operation of the cloud, the services return working as expected after
|
|
a while. To execute them you can type::
|
|
|
|
tox -e faults
|
|
|
|
The faults induced by these test cases could be cloud nodes reboot,
|
|
OpenStack services restart, virtual machines migrations, etc.
|
|
|
|
Please note that while scenario test cases are being executed in parallel (to
|
|
speed up test case execution), disruptive test case are only executed sequentially.
|
|
This is because the operations executed by such cases could break some functionality
|
|
for a short time and alter the regular state of the system which may be assumed by other
|
|
test cases to be executed.
|
|
|
|
|
|
Running the Tobiko Workflow
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Scenario and disruptive test cases, being executed in a specific sequence could
|
|
be used to uncover more issues with the cloud than disruptive test cases alone.
|
|
|
|
- First ensure there are workloads properly running by running scenario test cases::
|
|
|
|
tox -e scenario
|
|
|
|
.. sidebar:: Note
|
|
|
|
As second step we may, instead, update or upgrade OpenStack nodes.
|
|
|
|
- Next we could execute disruptive test cases to "stress" the cloud::
|
|
|
|
tox -e faults
|
|
|
|
- Finally we might re-run scenario test cases to check that everything is still running
|
|
as expected::
|
|
|
|
TOBIKO_PREVENT_CREATE=yes tox -e scenario
|
|
|
|
|
|
Test Cases Report Files
|
|
~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
After executing test cases we can view the results in greater detail via a small
|
|
set of files:
|
|
|
|
- **test_results.html**:
|
|
A user-browseable HTML view of test case results
|
|
- **test_results.log**:
|
|
a log file with logging traces collected from every individual test case
|
|
- **test_results.subunit**:
|
|
the original subunit binary file generated by test runner
|
|
- **test_results.xml**:
|
|
an XML Junit file to be used, for example, to show test cases result by
|
|
Jenkins CI server
|
|
|
|
The names of the above files can be changed from the default value (*test_results*)
|
|
to a custom one by setting the *TOX_REPORT_NAME* environment variable.
|
|
|
|
.. sidebar:: Legend
|
|
|
|
*{toxinidir}* stand for the Tobiko source files directory.
|
|
|
|
*{envname}* is the name of the Tox enviroment to be executed (IE scenario,
|
|
faults, etc.)
|
|
|
|
The above files are saved into a folder that can be specified with
|
|
*TOX_REPORT_DIR* environment variable.
|
|
|
|
By default the full path of the report directory is made from the below::
|
|
|
|
{toxinidir}/report/{envname}
|