x/tobiko
Code Issues Proposed changes

Document how to configure Tobiko and run scenario test cases

Browse Source 
This also improve other documentation pages and add many
cross section references.

This guide has been tested on CentOS 7 (Python 2.7) and
RHEL8 (Python 3.6).

Change-Id: I827dcf7237792ab94286c960968df405c7e548b9
This commit is contained in:
Federico Ressi 2019-04-29 13:25:19 +02:00
parent 0112246649
commit 144f50fe01
9 changed files with 514 additions and 126 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

46
README.rst
View File

@ -2,23 +2,35 @@
Tobiko
======
Tobiko is an OpenStack testing framework focusing on areas
complementary to Tempest:
- Minor updates and major upgrades. Tobiko tests populate the
cloud with workloads such as instances, allows the CI workflow
to perform an operation such as an update or upgrade, and then runs
tests that validate that the cloud workloads are still functional.
- Fault injection, like restarting nodes, OpenStack services and
dependencies such as OVS, RabbitMQ or the DB.
- White box testing; Specifically the ability to run commands on
nodes.
Test Big Cloud Operations
-------------------------
Links
~~~~~
Tobiko is an OpenStack testing framework focusing on areas mostly
complementary to `Tempest <https://docs.openstack.org/tempest/latest/>`__.
While tempest main focus has been testing OpenStack rest APIs, the main Tobiko
focus would be to test OpenStack system operations while "simulating"
the use of the cloud as the final user would.
* Documentation: `Tobiko Documentation <https://docs.openstack.org/tobiko/latest/>`__
* Usage: `Tobiko User Guide <https://docs.openstack.org/tobiko/latest/user/>`__
* Bugs: `Tobiko StoryBoard <https://storyboard.openstack.org/#!/project/x/tobiko>`__
* Source Code: https://opendev.org/x/tobiko.git
* License: `Apache License v2.0 <https://opendev.org/x/tobiko/src/branch/master/LICENSE>`__
Tobiko's test cases populate the cloud with workloads such as instances, allows
the CI workflow to perform an operation such as an update or upgrade, and then
runs test cases to validate that the cloud workloads are still functional.
Main Project Goals
~~~~~~~~~~~~~~~~~~
- Provide a Python framework to write system scenario test cases;
- Provide CLI tools to implement a workflow designed to test potentially
destructive operations (like cloud nodes reboot, update, upgrade...).
- Provide tools to monitor and recollect the healthy status of the cloud
References
----------
* Free software: Apache License, Version 2.0
* Documentation: https://docs.openstack.org/tobiko/latest/
* Release notes: https://docs.openstack.org/releasenotes/tobiko/
* Source: https://opendev.org/x/tobiko
* Bugs: https://storyboard.openstack.org/#!/project/x/tobiko

13
doc/source/contents.rst Normal file
View File

@ -0,0 +1,13 @@
Tobiko Documentation Contents
-----------------------------
.. toctree::
:maxdepth: 2
user/index
contributor/index
.. toctree::
:maxdepth: 1
reference/index

23
doc/source/index.rst
View File

@ -1,6 +1,3 @@
Tobiko
======
.. include:: ../../README.rst
Document Overview
@ -9,24 +6,14 @@ Document Overview
This document describes the tools for final user and contributors of the
project, and assumes that you are already familiar with OpenStack from an
end-user perspective. If not, hop over to the
`OpenStack doc site <https://docs.openstack.org/>`__
`OpenStack doc site <https://docs.openstack.org/>`__.
You can look for additional documentation also in the
`OpenStack wiki <http://wiki.openstack.org/>`__
`OpenStack wiki <http://wiki.openstack.org/>`__.
This documentation is generated by the Sphinx toolkit and lives in the source
tree.
This documentation is generated by the Sphinx toolkit and lives in the `source
tree <https://opendev.org/x/tobiko/doc/source>`__.
Enjoy!
.. toctree::
:maxdepth: 2
install/index
user/index
contributor/index
.. toctree::
:maxdepth: 1
reference/index
.. include:: contents.rst

50
doc/source/install/index.rst
View File

@ -1,50 +0,0 @@
.. _tobiko-install-guide:
=========================
Tobiko Installation Guide
=========================
Document Overview
-----------------
This document describes how to install Tobiko inside a Python virtualenv. For
a quick and simpler start you can jump to the :ref:`quick-start-guide`.
Install Tobiko Using Virtualenv
-------------------------------
Make sure Gcc, Git and base Python packages are installed on your system.
For instance on RHEL Linux you could type::
sudo yum install -y gcc git python python-devel
Make sure pip and setuptools are installed and up-to date::
wget https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
sudo pip install --upgrade setuptools
Make sure virtualenv and wheel are installed and up to date::
sudo pip install --upgrade tox virtualenv wheel
Get Tobiko source code using Git::
git clone https://opendev.org/x/tobiko.git
cd tobiko
To install Tobiko and its dependencies is safer to create a clean Virtualenv
where to install it. Create a Virtualenv and activate it::
virtualenv .tobiko-env
source .tobiko-env/bin/activate
Install Tobiko and its requirements::
pip install \
-c https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt \
-r requirements.txt \
-r extra-requirements.txt
pip install .

273
doc/source/user/config.rst Normal file
View File

@ -0,0 +1,273 @@
.. _tobiko-configuration-guide:
==========================
Tobiko Configuration Guide
==========================
Document Overview
-----------------
This document describes how to configure Tobiko.
.. sidebar:: See also
For a quick and simpler start you can jump to the
:ref:`tobiko-quick-start-guide`.
To install Tobiko inside a virutalenv please read
:ref:`tobiko-installation-guide`.
To run Tobiko scenario test cases please look at
:ref:`tobiko-test-case-execution-guide`.
Configure Tobiko Framework
--------------------------
In order to make sure Tobiko tools can connect to OpenStack services via Rest
API configuration parameters can be passed either via environment variables or
via a ini configuration file (referred here as :ref:`tobiko-conf`). Please look
at :ref:`authentication-methods` for more details.
To be able to execute scenario test cases there some OpenStack resources that
has to be created before running test cases. Please look at
:ref:`setup-required-resources` for more details.
.. _tobiko-conf:
tobiko.conf
~~~~~~~~~~~
Tobiko tries to load :ref:`tobiko-conf` file from one of below locations:
* current directory::
./tobiko.conf
* user home directory::
~/.tobiko/tobiko.conf
* system directory::
/etc/tobiko/tobiko.conf
Configure Logging
~~~~~~~~~~~~~~~~~
Tobiko can configure logging system to write messages to a log file. You can
edit below options in :ref:`tobiko-conf` to enable it as below::
[DEFAULT]
# Whenever to allow debugging messages to be written out or not
debug = true
# Name of the file where log messages will be appended.
log_file = tobiko.log
# The base directory used for relative log_file paths.
log_dir = .
.. _authentication-methods:
Authentication Methods
~~~~~~~~~~~~~~~~~~~~~~
Tobiko uses
`OpenStack client <https://docs.openstack.org/python-openstackclient/latest/>`__
to connect to OpenStack services.
.. _authentication-environment-variables:
Authentication Environment Variables
++++++++++++++++++++++++++++++++++++
To configure how Tobiko can connect to
services you can use the same
`environment variables <https://docs.openstack.org/python-openstackclient/latest/cli/man/openstack.html#environment-variables>`__
you would use for OpenStack Python client CLI.
Currently supported variables are::
# Identity API version
export OS_IDENTITY_API_VERSION=3
# URL to be used to connect to OpenStack Irentity Rest API service
export OS_AUTH_URL=http://10.0.0.109:5000/v3
# Authentication username (name or ID)
export OS_USERNAME=admin
export OS_USER_ID=...
# Authentication password
export OS_PASSWORD=...
# Project-level authentication scope (name or ID)
export OS_PROJECT_NAME=admin
export OS_TENANT_NAME=admin
export OS_PROJECT_ID=...
export OS_TENANT_ID=...
# Domain-level authorization scope (name or ID)
export OS_DOMAIN_NAME=default
export OS_DOMAIN_ID=...
# Domain name or ID containing user
export OS_USER_DOMAIN_NAME=default
export OS_USER_DOMAIN_ID=...
# Domain name or ID containing project
export OS_PROJECT_DOMAIN_NAME=default
export OS_PROJECT_DOMAIN_ID=...
# ID of the trust to use as a trustee user
export OS_TRUST_ID=...
.. _authentication-configuration:
Autentication Configuration
+++++++++++++++++++++++++++
You can also configure the same authentication parameters by editing 'keystone'
section in :ref:`tobiko-conf` file. For example::
[keystone]
# Identity API version
api_version = 3
# URL to be used to connect to OpenStack Irentity Rest API service
auth_url=http://10.0.0.109:5000/v3
# Authentication username (name or ID)
username = admin
# Authentication password
password = ...
# Project-level authentication scope (name or ID)
project_name = admin
# Domain-level authorization scope (name or ID)
domain = default
# Domain name or ID containing user
user_domain_name = default
# Domain name or ID containing prject
project_domain_name = default
# ID of the trust to use as a trustee user
trust_id = ...
.. _proxy-server-configuration:
Proxy Server Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~
The first thing to make sure is Tobiko can reach OpenStack services. In case
OpenStack is not directly accessible from where test cases or Tobiko CLI are
executed it is possible to use an HTTP proxy server running on a network that
is able to reach all OpenStack Rest API service. This can be performed
by using below standard environment variables::
export http_proxy=http://<proxy-host>:<proxy-port>/
export https_proxy=http://<proxy-host>:<proxy-port>/
export no_proxy=127.0.0.1,...
For convenience it is also possible to specify the same parameters via
:ref:`tobiko-conf`::
[http]
http_proxy = http://<proxy-host>:<proxy-port>/
https_proxy = http://<proxy-host>:<proxy-port>/
no_proxy = 127.0.0.1,...
Because Tobiko test cases could execute local commands (like for example ping)
to reach network services we have to specify in tobiko.conf file a shell
(like OpenSSH client) to be used instead of the default local one
('/bin/sh')::
[shell]
command = /usr/bin/ssh <proxy-host>
Please make sure it is possible to execute commands on local system without
having to pass a password::
/usr/bin/ssh <proxy-host> echo 'Yes it works!'
To archive it please follow one of the
`many guides available on Internet
<https://www.google.com/search?q=passwordless+ssh&oq=passwordless+&aqs=chrome.0.0j69i57j0l4.4775j0j7&sourceid=chrome&ie=UTF-8>`__
.
.. _setup-required-resources:
Setup Required Resources
~~~~~~~~~~~~~~~~~~~~~~~~
To be able to execute Tobiko scenario test cases there some OpenStack
resources that has to be created before running test cases.
Install required Python OpenStack clients::
pip install --upgrade \
-c https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt \
python-openstackclient \
python-glanceclient \
python-novaclient \
python-neutronclient
You need to make sure ref:`authentication-environment-variables` are properly
set::
source openstackrc
openstack image list
openstack flavor list
openstack network list
`Get an image <https://docs.openstack.org/image-guide/obtain-images.html>`__
for Nova instances created by Tobiko::
wget http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
openstack image create cirros \
--file cirros-0.4.0-x86_64-disk.img \
--disk-format qcow2 \
--container-format bare \
--public
Create a flavor to be used with above image::
openstack flavor create --vcpus 1 --ram 64 --disk 1 m1.tiny
Create an SSH key file to be used to ssh to Nova server instances::
ssh-keygen -f ~/.ssh/id_rsa -P ''
Add reference to above resources into your :ref:`tobiko-conf` file::
[nova]
image = cirros
flavor = m1.tiny
key_file=~/.ssh/id_rsa
Add reference to the network where Tobiko should create floating IP instances
in :ref:`tobiko-conf` file::
[neutron]
floating_network = public
What's Next
-----------
To know how to run Tobiko scenario test cases you can look at
:ref:`tobiko-test-case-execution-guide`

5
doc/source/user/index.rst
View File

@ -1,3 +1,5 @@
.. _tobiko-user-guide:
=================
Tobiko User Guide
=================
@ -6,3 +8,6 @@ Tobiko User Guide
:maxdepth: 2
quick-start
install
config
run-test-cases

69
doc/source/user/install.rst Normal file
View File

@ -0,0 +1,69 @@
.. _tobiko-installation-guide:
=========================
Tobiko Installation Guide
=========================
Document Overview
-----------------
This document describes how to install Tobiko inside a
`Python Virtualenv <https://virtualenv.pypa.io/en/latest/>`__.
.. sidebar:: See also
For a quick and simpler start you can jump to the
:ref:`tobiko-quick-start-guide`.
To configure Tobiko please read :ref:`tobiko-configuration-guide`.
To run Tobiko scenario test cases please look at
:ref:`tobiko-test-case-execution-guide`.
Install Tobiko Using Virtualenv
-------------------------------
Make sure Gcc, Git and base Python packages are installed on your system.
For instance on RHEL Linux 7.6 or CentOS 7 you could type::
sudo yum install -y gcc git python python-devel wget
For instance on RHEL Linux 8 or CentOS 8 you could type::
sudo dnf install -y gcc git python3 python3-devel wget
sudo alternatives --set python /usr/bin/python3
Make sure pip is installed and up-to date::
wget https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
PIP=$(which pip)
Make sure setuptools, virtualenv and wheel are installed and up to date::
sudo $PIP install --upgrade setuptools virtualenv wheel
Get Tobiko source code using Git and enter into Tobiko soruce folder::
git clone https://opendev.org/x/tobiko.git
cd tobiko
To install Tobiko and its dependencies is safer to create a clean Virtualenv
where to install it. Create a Virtualenv and activate it::
virtualenv .tobiko-env
source .tobiko-env/bin/activate
Install Tobiko and its requirements::
pip install \
-c https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt \
.
What's Next
-----------
To know how to configure Tobiko please read :ref:`tobiko-configuration-guide`.

122
doc/source/user/quick-start.rst
View File

@ -1,35 +1,50 @@
.. _quick-start-guide:
.. _tobiko-quick-start-guide:
=================
Quick Start guide
=================
========================
Tobiko Quick Start Guide
========================
Document Overview
-----------------
This document describes how to install execute Tobiko scenarios test cases
using Tox. To install Tobiko inside a virtualenv directory please refers to
:ref:`tobiko-install-guide`.
using `Tox <https://tox.readthedocs.io/en/latest/>`__.
.. sidebar:: See also
To install Tobiko inside a virutalenv please read
:ref:`tobiko-installation-guide`.
To configure Tobiko please read :ref:`tobiko-configuration-guide`.
To run Tobiko scenario test cases please look at
:ref:`tobiko-test-case-execution-guide`.
Install Required Packages
-------------------------
Make sure Gcc, Git and base Python packages are installed on your system.
For instance on RHEL Linux you could type::
sudo yum install -y gcc git python python-devel
For instance on RHEL Linux 8 or CentOS 8 you could type::
sudo dnf install -y gcc git python3 python3-devel wget
sudo alternatives --set python /usr/bin/python3
Make sure pip and setuptools are installed and up-to date::
wget https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
sudo pip install --upgrade setuptools
PIP=$(which pip)
Make sure tox, virtualenv and wheel are installed and up to date::
Make sure setuptools, wheel, virtualenv, and tox are installed and up to date::
sudo pip install --upgrade tox virtualenv wheel
sudo $PIP install --upgrade setuptools wheel virtualenv tox
Get Tobiko
@ -56,73 +71,94 @@ credentials. You can do it in one of below ways:
Set Tobiko Credentials Via Environment Variables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can use an existing shell rc file that is valid for Python OpenStack
client::
.. sidebar:: See also
For more details about supported environment variables please read
:ref:`authentication-environment-variables` section.
You can use an existing shell RC file that is valid for
`Python OpenStack client <https://docs.openstack.org/python-openstackclient/latest/cli/man/openstack.html#environment-variables>`__
::
source openstackrc
An example of 'openstackrc' file could looks like below::
export API_VERSION=2
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_URL=https://my_cloud:13000/v3
export OS_USERNAME=admin
export OS_PASSWORD=secret
export PROJECT_NAME=admin
export OS_PROJECT_NAME=admin
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=admin
export OS_AUTH_URL=https://my_cloud:13000/v3
export OS_PROJECT_DOMAIN_NAME=Default
.. _credentials-from-config:
Set Tobiko Credentials Via 'tobiko.conf' File
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Set Tobiko Credentials Via :ref:`tobiko-conf` File
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Create a file at ~/.tobiko/tobiko.conf adding a section like below::
.. sidebar:: See also
For more details about supported configuration options please read
:ref:`authentication-configuration` section.
Create a file at `~/.tobiko/tobiko.conf` adding a section like below::
[keystone]
api_version = 2
api_version = 3
auth_url = http://my_cloud:13000/v3
username = admin
password = secret
project_name = admin
user_domain_name = admin
project_domain_name = admin
auth_url = http://my_cloud:13000/v3
user_domain_name = Default
project_domain_name = Default
Create Required Resources
~~~~~~~~~~~~~~~~~~~~~~~~~
Setup Required Resources
~~~~~~~~~~~~~~~~~~~~~~~~
Install Python OpenStack client::
To be able to execute Tobiko scenario test cases there some OpenStack
resources that has to be created before running test cases.
source overcloudrc
sudo pip install python-openstackclient \
python-novaclient \
python-glanceclient
To execute commands from a virtualenv created by Tox you can type as below::
Create an image for Nova instances created by Tobiko::
tox -e venv -- <your-commands>
wget http://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img
openstack image create "cirros" \
--file cirros-0.3.5-x86_64-disk.img \
You need to make sure ref:`authentication-environment-variables` are properly
set::
tox -e venv -- openstack image list
tox -e venv -- openstack flavor list
tox -e venv -- openstack network list
`Get an image <https://docs.openstack.org/image-guide/obtain-images.html>`__
for Nova instances created by Tobiko::
wget -c http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
tox -e venv -- openstack image create cirros \
--file cirros-0.4.0-x86_64-disk.img \
--disk-format qcow2 \
--container-format bare \
--public
Create a flavor to be used with above image::
openstack flavor create --id 0 --vcpus 1 --ram 64 --disk 1 m1.tiny
tox -e venv -- openstack flavor create --vcpus 1 --ram 64 --disk 1 m1.tiny
Add reference to above resources into your `tobiko.conf` file::
Create an SSH key file to be used to ssh to Nova server instances::
ssh-keygen -f ~/.ssh/id_rsa -P ''
Add reference to above resources into your :ref:`tobiko-conf` file::
[nova]
image = cirros
flavor = m1.tiny
key_file=~/.ssh/id_rsa
Configure Public Network Name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Add reference to the network where Tobiko should create floating IP instances::
Add reference to the network where Tobiko should create floating IP instances
in :ref:`tobiko-conf` file::
[neutron]
floating_network = public
@ -131,6 +167,10 @@ Add reference to the network where Tobiko should create floating IP instances::
Run Test Cases
--------------
Run Tobiko scenario test cases using Tox::
Finally run Tobiko scenario test cases using Tox::
tox -e scenario
List resources stacks created by test cases::
tox -e venv -- openstack stack list

39
doc/source/user/run-test-cases.rst Normal file
View File

@ -0,0 +1,39 @@
.. _tobiko-test-case-execution-guide:
=================================
Tobiko Test Cases Execution Guide
=================================
This document describes how to execute Tobiko scenario test cases.
.. sidebar:: See also
For a quick and simpler start you can jump to the
:ref:`tobiko-quick-start-guide`.
To install Tobiko inside a virutalenv please read
:ref:`tobiko-installation-guide`.
To configure Tobiko please read :ref:`tobiko-configuration-guide`.
Prepare Your System
~~~~~~~~~~~~~~~~~~~
Before running Tobiko test cases you need to be sure you are doing it from
Tobiko source files folder and that you have actived a Virtualenv where Tobiko
and its requirements are installed. Please refers to
:ref:`tobiko-installation-guide` and :ref:`tobiko-configuration-guide` to know
how to setup your system before running test cases.
Run Scenario Test Cases
~~~~~~~~~~~~~~~~~~~~~~~
To run test cases you need a test runner able to execute Python test cases.
Test cases delivered with Tobiko has been tested using
`stestr <https://stestr.readthedocs.io/en/latest/>`__
From Tobiko source folder you can run scenario test cases using below command::
stestr run --test-path tobiko/tests/scenario/