openstack/neutron
Code Issues Proposed changes

[Docs] Guide about running and debugging tempest tests

Browse Source 
Move out tempest related things from TESTING.rst to tempest.rst.
It also gives some basics on how to debug tempest in zuul and locally.

Change-Id: I46c282c865bfdf5e88e26ea8fc31b26cdd3f6b2e
This commit is contained in:
elajkat 2020-11-17 15:01:07 +01:00
parent 4c886998b1
commit db740decec
3 changed files with 276 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
TESTING.rst
View File

@ -236,6 +236,12 @@ reproducible way to verify code while you're still writing it.
More details can be found in :ref:`FullStack Testing<fullstack_testing>` guide.
Integration - tempest tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tempest is the integration test suit of Openstack, more details can be found in
:ref:`Tempest testing<tempest_testing>`
API Tests
~~~~~~~~~

1
doc/source/contributor/testing/index.rst
View File

@ -37,3 +37,4 @@ Testing
db_transient_failure_injection
ci_scenario_jobs
ovn_devstack
tempest

269
doc/source/contributor/testing/tempest.rst Normal file
View File

@ -0,0 +1,269 @@
..
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
Convention for heading levels in Neutron devref:
======= Heading 0 (reserved for the title in a document)
------- Heading 1
~~~~~~~ Heading 2
+++++++ Heading 3
''''''' Heading 4
(Avoid deeper levels because they do not render well.)
.. _tempest_testing:
Tempest testing
===============
Tempest basics in Networking projects
-------------------------------------
Tempest is the integration test suite of Openstack, for details see
`Tempest Testing Project <https://docs.openstack.org/tempest/latest/>`_.
Tempest makes it possible to add project-specific plugins, and for networking
this is `neutron-tempest-plugin <https://opendev.org/openstack/neutron-tempest-plugin>`_.
neutron-tempest-plugin covers API and scenario tests not just for core Neutron
functionality, but for stadium projects as well.
For reference please read `Testing Neutron\'s related sections <testing.html#api-tests>`_
API Tests
~~~~~~~~~
API tests (neutron-tempest-plugin/neutron_tempest_plugin/api/) are
intended to ensure the function
and stability of the Neutron API. As much as possible, changes to
this path should not be made at the same time as changes to the code
to limit the potential for introducing backwards-incompatible changes,
although the same patch that introduces a new API should include an API
test.
Since API tests target a deployed Neutron daemon that is not test-managed,
they should not depend on controlling the runtime configuration
of the target daemon. API tests should be black-box - no assumptions should
be made about implementation. Only the contract defined by Neutron's REST API
should be validated, and all interaction with the daemon should be via
a REST client.
The neutron-tempest-plugin/neutron_tempest_plugin directory was copied from the
Tempest project around the Kilo timeframe. At the time, there was an overlap of tests
between the Tempest and Neutron repositories. This overlap was then eliminated by carving
out a subset of resources that belong to Tempest, with the rest in Neutron.
API tests that belong to Tempest deal with a subset of Neutron's resources:
* Port
* Network
* Subnet
* Security Group
* Router
* Floating IP
These resources were chosen for their ubiquity. They are found in most
Neutron deployments regardless of plugin, and are directly involved in the
networking and security of an instance. Together, they form the bare minimum
needed by Neutron.
This is excluding extensions to these resources (For example: Extra DHCP
options to subnets, or snat_gateway mode to routers) that are not mandatory
in the majority of cases.
Tests for other resources should be contributed to the neutron-tempest-plugin
repository. Scenario tests should be similarly split up between Tempest and
Neutron according to the API they're targeting.
To create an API test, the testing class must at least inherit from
neutron_tempest_plugin.api.base.BaseNetworkTest base class. As some of tests
may require certain extensions to be enabled, the base class provides
``required_extensions`` class attribute which can be used by subclasses to
define a list of required extensions for a particular test class.
Scenario Tests
~~~~~~~~~~~~~~
Scenario tests (neutron-tempest-plugin/neutron_tempest_plugin/scenario), like API tests,
use the Tempest test infrastructure and have the same requirements. Guidelines for
writing a good scenario test may be found at the Tempest developer guide:
https://docs.openstack.org/tempest/latest/field_guide/scenario.html
Scenario tests, like API tests, are split between the Tempest and Neutron
repositories according to the Neutron API the test is targeting.
Some scenario tests require advanced ``Glance`` images (for example, ``Ubuntu``
or ``CentOS``) in order to pass. Those tests are skipped by default. To enable
them, include the following in ``tempest.conf``:
.. code-block:: ini
[compute]
image_ref = <uuid of advanced image>
[neutron_plugin_options]
default_image_is_advanced = True
To use the ``advanced image`` only for the tests that really need it and
``cirros`` for the rest to keep test execution as fast as possible:
.. code-block:: ini
[compute]
image_ref = <uuid of cirros image>
[neutron_plugin_options]
advanced_image_ref = <uuid of advanced image>
advanced_image_flavor_ref = <suitable flavor for the advance image>
advanced_image_ssh_user = <username for the advanced image>
Specific test requirements for advanced images are:
#. ``test_trunk`` requires ``802.11q`` kernel module loaded.
#. ``test_metadata`` requires capabilty to run ``curl`` for IPv6 addresses.
#. ``test_multicast`` needs to execute python script on the VM to open
multicast sockets.
#. ``test_mtu`` requires ping to be able to send packets with specific ``mtu``.
Zuul basics & job structure
---------------------------
Zuul is the gating system behind Openstack, for details see:
`Zuul - A Project Gating System <https://zuul-ci.org/docs/zuul/>`_.
Zuul job definitions are in yaml, ansible in the depths. The job definitions can be
inherited. Networking projects job definitions parents are coming from
`devstack zuul job config <https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml>`_
and from `tempest <https://opendev.org/openstack/tempest/src/branch/master/tempest>`_
and defined in `neutron-tempest-plugin zuul.d folder <https://opendev.org/openstack/neutron-tempest-plugin/src/branch/master/zuul.d>`_
and in `neutron zuul.d folder <https://opendev.org/openstack/neutron/src/branch/master/zuul.d>`_ .
Where to look
-------------
Debugging zuul results
~~~~~~~~~~~~~~~~~~~~~~
Tempest executed with different configurations,
for details check this page
:ref:`Tempest jobs running in Neutron CI<ci_jobs>`
When zuul reports back job results to a review it gives links to the results
as well.
The logs can be checked online if you select ``Logs`` tab on the logs page.
* ``job-output.txt`` is the full log which contains not just test execution
logs, but devstack console output.
* ``test_results.html`` is the clickable html test report.
* ``controller`` and ``compute`` (in case of multinode job) are a dictionary
tree containing the relevant files (configuration files, logs etc)
created in the job. For example under controller/logs/etc/neutron/ you can
check how Neutron services were configured, or in the file
controller/logs/tempest_conf.txt you can check tempest configuration file.
* services' log files are the in files ``controller/logs/screen-`*`.txt``,
so for example neutron l2 agent logs are in the file
controller/logs/screen-q-agt.txt.
Downloading logs
++++++++++++++++
There is a possibility to download all logs related to a job.
If you choose this on the zuul logs page select ``Artifacts`` tab on the
logs page and click on ``Download all logs``. This will download a script
``download-logs.sh``, which when executed downloads all the logs for the job
under ``/tmp/``:
.. code-block:: shell
$ chmod +x download-logs.sh
$ ./download-logs.sh
2020-12-07T18:12:09+01:00 | Querying https://zuul.opendev.org/api/tenant/openstack/build/8caed05f5ba441b4be2b061d1d421e4e for manifest
2020-12-07T18:12:11+01:00 | Saving logs to /tmp/zuul-logs.c8ZhLM
2020-12-07T18:12:11+01:00 | Getting logs from https://3612101d6c142bf9c77a-c96c299047b55dcdeaefef8e344ceab6.ssl.cf1.rackcdn.com/694539/11/check/tempest-slow-py3/8caed05/
2020-12-07T18:12:11+01:00 | compute1/logs/apache/access_log.txt [ 0001/0337 ]
...
$ ls /tmp/zuul-logs.c8ZhLM/
compute1
controller
Executing tempest locally
~~~~~~~~~~~~~~~~~~~~~~~~~
For executing tempest locally you need a working devstack, to make it worse
if you have to debug a test executed in a multinode job you need a multinode
setup as well.
For devstack documentation please refer to this page:
`DevStack <https://docs.openstack.org/devstack/latest/>`_
To have tempest installed and have a proper configuration file for it in your
local.conf file enable tempest as service:
.. code-block:: ini
ENABLED_SERVICES+=tempest
or
.. code-block:: ini
enable_service tempest
To use specific config options for tempest you can add those as well to
local.conf:
.. code-block:: ini
[[test-config|/opt/stack/tempest/etc/tempest.conf]]
[network-feature-enabled]
qos_placement_physnet=physnet1
To make devstack setup neutron and neutron-tempest-plugin as well enable their
devstack plugin:
.. code-block:: ini
enable_plugin neutron https://opendev.org/openstack/neutron
enable_plugin neutron-tempest-plugin https://opendev.org/openstack/neutron-tempest-plugin
If you need a special image for the tests you can set that too in local.conf:
.. code-block:: ini
IMAGE_URLS="http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-i386-disk.img,https://cloud-images.ubuntu.com/releases/bionic/release/ubuntu-18.04-server-cloudimg-amd64.img"
ADVANCED_IMAGE_NAME=ubuntu-18.04-server-cloudimg-amd64
ADVANCED_INSTANCE_TYPE=ds512M
ADVANCED_INSTANCE_USER=ubuntu
If devstack succeeds you can find tempest and neutron-tempest-plugin under
``/opt/stack/`` directory (with all other project folders which are set to be
installed from git).
Tempest's configuration file is under ``/opt/stack/tempest/etc/`` folder, you
can check there if everything is as expected.
You can check if neutron-tempest-plugin is known as a tempest plugin by
tempest:
.. code-block:: shell
$ tempest list-plugins
+---------------------------------+------------------------------------------------------------+
| Name | EntryPoint |
+---------------------------------+------------------------------------------------------------+
| neutron_tests | neutron_tempest_plugin.plugin:NeutronTempestPlugin |
+---------------------------------+------------------------------------------------------------+
To execute a given test or group of tests you can use a regex, or you can use
the idempotent id of a test or the tag associated with the test:
.. code-block:: shell
tempest run --config etc/tempest.conf --regex tempest.scenario
tempest run --config etc/tempest.conf --regex neutron_tempest_plugin.scenario
tempest run --config etc/tempest.conf smoke
tempest run --config etc/tempest.conf ab40fc48-ca8d-41a0-b2a3-f6679c847bfe