Browse Source
Move out tempest related things from TESTING.rst to tempest.rst. It also gives some basics on how to debug tempest in zuul and locally. Change-Id: I46c282c865bfdf5e88e26ea8fc31b26cdd3f6b2echanges/58/765958/8
3 changed files with 276 additions and 0 deletions
@ -0,0 +1,269 @@
|
||||
.. |
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may |
||||
not use this file except in compliance with the License. You may obtain |
||||
a copy of the License at |
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0 |
||||
|
||||
Unless required by applicable law or agreed to in writing, software |
||||
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT |
||||
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the |
||||
License for the specific language governing permissions and limitations |
||||
under the License. |
||||
|
||||
|
||||
Convention for heading levels in Neutron devref: |
||||
======= Heading 0 (reserved for the title in a document) |
||||
------- Heading 1 |
||||
~~~~~~~ Heading 2 |
||||
+++++++ Heading 3 |
||||
''''''' Heading 4 |
||||
(Avoid deeper levels because they do not render well.) |
||||
|
||||
.. _tempest_testing: |
||||
|
||||
Tempest testing |
||||
=============== |
||||
|
||||
Tempest basics in Networking projects |
||||
------------------------------------- |
||||
Tempest is the integration test suite of Openstack, for details see |
||||
`Tempest Testing Project <https://docs.openstack.org/tempest/latest/>`_. |
||||
|
||||
Tempest makes it possible to add project-specific plugins, and for networking |
||||
this is `neutron-tempest-plugin <https://opendev.org/openstack/neutron-tempest-plugin>`_. |
||||
|
||||
neutron-tempest-plugin covers API and scenario tests not just for core Neutron |
||||
functionality, but for stadium projects as well. |
||||
For reference please read `Testing Neutron\'s related sections <testing.html#api-tests>`_ |
||||
|
||||
API Tests |
||||
~~~~~~~~~ |
||||
|
||||
API tests (neutron-tempest-plugin/neutron_tempest_plugin/api/) are |
||||
intended to ensure the function |
||||
and stability of the Neutron API. As much as possible, changes to |
||||
this path should not be made at the same time as changes to the code |
||||
to limit the potential for introducing backwards-incompatible changes, |
||||
although the same patch that introduces a new API should include an API |
||||
test. |
||||
|
||||
Since API tests target a deployed Neutron daemon that is not test-managed, |
||||
they should not depend on controlling the runtime configuration |
||||
of the target daemon. API tests should be black-box - no assumptions should |
||||
be made about implementation. Only the contract defined by Neutron's REST API |
||||
should be validated, and all interaction with the daemon should be via |
||||
a REST client. |
||||
|
||||
The neutron-tempest-plugin/neutron_tempest_plugin directory was copied from the |
||||
Tempest project around the Kilo timeframe. At the time, there was an overlap of tests |
||||
between the Tempest and Neutron repositories. This overlap was then eliminated by carving |
||||
out a subset of resources that belong to Tempest, with the rest in Neutron. |
||||
|
||||
API tests that belong to Tempest deal with a subset of Neutron's resources: |
||||
|
||||
* Port |
||||
* Network |
||||
* Subnet |
||||
* Security Group |
||||
* Router |
||||
* Floating IP |
||||
|
||||
These resources were chosen for their ubiquity. They are found in most |
||||
Neutron deployments regardless of plugin, and are directly involved in the |
||||
networking and security of an instance. Together, they form the bare minimum |
||||
needed by Neutron. |
||||
|
||||
This is excluding extensions to these resources (For example: Extra DHCP |
||||
options to subnets, or snat_gateway mode to routers) that are not mandatory |
||||
in the majority of cases. |
||||
|
||||
Tests for other resources should be contributed to the neutron-tempest-plugin |
||||
repository. Scenario tests should be similarly split up between Tempest and |
||||
Neutron according to the API they're targeting. |
||||
|
||||
To create an API test, the testing class must at least inherit from |
||||
neutron_tempest_plugin.api.base.BaseNetworkTest base class. As some of tests |
||||
may require certain extensions to be enabled, the base class provides |
||||
``required_extensions`` class attribute which can be used by subclasses to |
||||
define a list of required extensions for a particular test class. |
||||
|
||||
Scenario Tests |
||||
~~~~~~~~~~~~~~ |
||||
|
||||
Scenario tests (neutron-tempest-plugin/neutron_tempest_plugin/scenario), like API tests, |
||||
use the Tempest test infrastructure and have the same requirements. Guidelines for |
||||
writing a good scenario test may be found at the Tempest developer guide: |
||||
https://docs.openstack.org/tempest/latest/field_guide/scenario.html |
||||
|
||||
Scenario tests, like API tests, are split between the Tempest and Neutron |
||||
repositories according to the Neutron API the test is targeting. |
||||
|
||||
Some scenario tests require advanced ``Glance`` images (for example, ``Ubuntu`` |
||||
or ``CentOS``) in order to pass. Those tests are skipped by default. To enable |
||||
them, include the following in ``tempest.conf``: |
||||
|
||||
.. code-block:: ini |
||||
|
||||
[compute] |
||||
image_ref = <uuid of advanced image> |
||||
[neutron_plugin_options] |
||||
default_image_is_advanced = True |
||||
|
||||
To use the ``advanced image`` only for the tests that really need it and |
||||
``cirros`` for the rest to keep test execution as fast as possible: |
||||
|
||||
.. code-block:: ini |
||||
|
||||
[compute] |
||||
image_ref = <uuid of cirros image> |
||||
[neutron_plugin_options] |
||||
advanced_image_ref = <uuid of advanced image> |
||||
advanced_image_flavor_ref = <suitable flavor for the advance image> |
||||
advanced_image_ssh_user = <username for the advanced image> |
||||
|
||||
Specific test requirements for advanced images are: |
||||
|
||||
#. ``test_trunk`` requires ``802.11q`` kernel module loaded. |
||||
#. ``test_metadata`` requires capabilty to run ``curl`` for IPv6 addresses. |
||||
#. ``test_multicast`` needs to execute python script on the VM to open |
||||
multicast sockets. |
||||
#. ``test_mtu`` requires ping to be able to send packets with specific ``mtu``. |
||||
|
||||
Zuul basics & job structure |
||||
--------------------------- |
||||
Zuul is the gating system behind Openstack, for details see: |
||||
`Zuul - A Project Gating System <https://zuul-ci.org/docs/zuul/>`_. |
||||
|
||||
Zuul job definitions are in yaml, ansible in the depths. The job definitions can be |
||||
inherited. Networking projects job definitions parents are coming from |
||||
`devstack zuul job config <https://opendev.org/openstack/devstack/src/branch/master/.zuul.yaml>`_ |
||||
and from `tempest <https://opendev.org/openstack/tempest/src/branch/master/tempest>`_ |
||||
and defined in `neutron-tempest-plugin zuul.d folder <https://opendev.org/openstack/neutron-tempest-plugin/src/branch/master/zuul.d>`_ |
||||
and in `neutron zuul.d folder <https://opendev.org/openstack/neutron/src/branch/master/zuul.d>`_ . |
||||
|
||||
Where to look |
||||
------------- |
||||
|
||||
Debugging zuul results |
||||
~~~~~~~~~~~~~~~~~~~~~~ |
||||
Tempest executed with different configurations, |
||||
for details check this page |
||||
:ref:`Tempest jobs running in Neutron CI<ci_jobs>` |
||||
|
||||
When zuul reports back job results to a review it gives links to the results |
||||
as well. |
||||
|
||||
The logs can be checked online if you select ``Logs`` tab on the logs page. |
||||
|
||||
* ``job-output.txt`` is the full log which contains not just test execution |
||||
logs, but devstack console output. |
||||
* ``test_results.html`` is the clickable html test report. |
||||
* ``controller`` and ``compute`` (in case of multinode job) are a dictionary |
||||
tree containing the relevant files (configuration files, logs etc) |
||||
created in the job. For example under controller/logs/etc/neutron/ you can |
||||
check how Neutron services were configured, or in the file |
||||
controller/logs/tempest_conf.txt you can check tempest configuration file. |
||||
* services' log files are the in files ``controller/logs/screen-`*`.txt``, |
||||
so for example neutron l2 agent logs are in the file |
||||
controller/logs/screen-q-agt.txt. |
||||
|
||||
Downloading logs |
||||
++++++++++++++++ |
||||
There is a possibility to download all logs related to a job. |
||||
|
||||
If you choose this on the zuul logs page select ``Artifacts`` tab on the |
||||
logs page and click on ``Download all logs``. This will download a script |
||||
``download-logs.sh``, which when executed downloads all the logs for the job |
||||
under ``/tmp/``: |
||||
|
||||
.. code-block:: shell |
||||
|
||||
$ chmod +x download-logs.sh |
||||
$ ./download-logs.sh |
||||
2020-12-07T18:12:09+01:00 | Querying https://zuul.opendev.org/api/tenant/openstack/build/8caed05f5ba441b4be2b061d1d421e4e for manifest |
||||
2020-12-07T18:12:11+01:00 | Saving logs to /tmp/zuul-logs.c8ZhLM |
||||
2020-12-07T18:12:11+01:00 | Getting logs from https://3612101d6c142bf9c77a-c96c299047b55dcdeaefef8e344ceab6.ssl.cf1.rackcdn.com/694539/11/check/tempest-slow-py3/8caed05/ |
||||
2020-12-07T18:12:11+01:00 | compute1/logs/apache/access_log.txt [ 0001/0337 ] |
||||
... |
||||
|
||||
$ ls /tmp/zuul-logs.c8ZhLM/ |
||||
compute1 |
||||
controller |
||||
|
||||
Executing tempest locally |
||||
~~~~~~~~~~~~~~~~~~~~~~~~~ |
||||
For executing tempest locally you need a working devstack, to make it worse |
||||
if you have to debug a test executed in a multinode job you need a multinode |
||||
setup as well. |
||||
|
||||
For devstack documentation please refer to this page: |
||||
`DevStack <https://docs.openstack.org/devstack/latest/>`_ |
||||
|
||||
To have tempest installed and have a proper configuration file for it in your |
||||
local.conf file enable tempest as service: |
||||
|
||||
.. code-block:: ini |
||||
|
||||
ENABLED_SERVICES+=tempest |
||||
|
||||
or |
||||
|
||||
.. code-block:: ini |
||||
|
||||
enable_service tempest |
||||
|
||||
To use specific config options for tempest you can add those as well to |
||||
local.conf: |
||||
|
||||
.. code-block:: ini |
||||
|
||||
[[test-config|/opt/stack/tempest/etc/tempest.conf]] |
||||
[network-feature-enabled] |
||||
qos_placement_physnet=physnet1 |
||||
|
||||
To make devstack setup neutron and neutron-tempest-plugin as well enable their |
||||
devstack plugin: |
||||
|
||||
.. code-block:: ini |
||||
|
||||
enable_plugin neutron https://opendev.org/openstack/neutron |
||||
enable_plugin neutron-tempest-plugin https://opendev.org/openstack/neutron-tempest-plugin |
||||
|
||||
If you need a special image for the tests you can set that too in local.conf: |
||||
|
||||
.. code-block:: ini |
||||
|
||||
IMAGE_URLS="http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-i386-disk.img,https://cloud-images.ubuntu.com/releases/bionic/release/ubuntu-18.04-server-cloudimg-amd64.img" |
||||
ADVANCED_IMAGE_NAME=ubuntu-18.04-server-cloudimg-amd64 |
||||
ADVANCED_INSTANCE_TYPE=ds512M |
||||
ADVANCED_INSTANCE_USER=ubuntu |
||||
|
||||
If devstack succeeds you can find tempest and neutron-tempest-plugin under |
||||
``/opt/stack/`` directory (with all other project folders which are set to be |
||||
installed from git). |
||||
|
||||
Tempest's configuration file is under ``/opt/stack/tempest/etc/`` folder, you |
||||
can check there if everything is as expected. |
||||
|
||||
You can check if neutron-tempest-plugin is known as a tempest plugin by |
||||
tempest: |
||||
|
||||
.. code-block:: shell |
||||
|
||||
$ tempest list-plugins |
||||
+---------------------------------+------------------------------------------------------------+ |
||||
| Name | EntryPoint | |
||||
+---------------------------------+------------------------------------------------------------+ |
||||
| neutron_tests | neutron_tempest_plugin.plugin:NeutronTempestPlugin | |
||||
+---------------------------------+------------------------------------------------------------+ |
||||
|
||||
To execute a given test or group of tests you can use a regex, or you can use |
||||
the idempotent id of a test or the tag associated with the test: |
||||
|
||||
.. code-block:: shell |
||||
|
||||
tempest run --config etc/tempest.conf --regex tempest.scenario |
||||
tempest run --config etc/tempest.conf --regex neutron_tempest_plugin.scenario |
||||
tempest run --config etc/tempest.conf smoke |
||||
tempest run --config etc/tempest.conf ab40fc48-ca8d-41a0-b2a3-f6679c847bfe |
Loading…
Reference in new issue