openstack/openstack-tempest-skiplist
Code Issues Proposed changes

Retire Tripleo: remove repo content

Browse Source 
TripleO project is retiring
- https://review.opendev.org/c/openstack/governance/+/905145

this commit remove the content of this project repo

Depends-On: https://review.opendev.org/c/openstack/project-config/+/910075

Change-Id: I05ba454853950b40942b5b4ff044f6070aa2117f
This commit is contained in:
Ghanshyam Mann
2024-02-24 11:32:24 -08:00
committed by Ghanshyam
parent ec349ebef6
commit 934fe57620
62 changed files with 8 additions and 5434 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.coveragerc
-6
View File
@@ -1,6 +0,0 @@
[run]
branch = True
source = tempest_skip
[report]
ignore_errors = True
.gitignore
-46
View File
@@ -1,46 +0,0 @@
*.py[cod]
*.pyc
# Packages
*.egg*
*.egg-info
dist
build
eggs
parts
bin
var
sdist
develop-eggs
.installed.cfg
lib
lib64
# Installer logs
pip-log.txt
# Unit test / coverage reports
cover/
.coverage*
!.coveragerc
.tox
.venv
# Mr Developer
.project
.pydevproject
# Sphinx
doc/build
# pbr generates these
AUTHORS
ChangeLog
# Editors
*~
.*.swp
.*sw?
# Files created by releasenotes build
releasenotes/build
.zuul.yaml
-32
View File
@@ -1,32 +0,0 @@
- project:
templates:
- publish-openstack-docs-pti
- openstack-python3-zed-jobs
- release-notes-jobs-python3
check:
jobs:
- openstack-tempest-skiplist-validate
gate:
jobs:
- openstack-tempest-skiplist-validate
- job:
name: openstack-tempest-skiplist-base
parent: unittests
description: Base job for tempest-skiplist
required-projects:
- openstack/openstack-tempest-skiplist
roles:
- zuul: openstack/openstack-tempest-skiplist
irrelevant-files:
- ^doc/.*$
- ^releasenotes/.*$
- ^.*\.rst$
- job:
name: openstack-tempest-skiplist-validate
parent: openstack-tempest-skiplist-base
description: Tempest skiplist to validate yaml file
pre-run:
- playbooks/tempest_skip_install.yaml
run: playbooks/tempest_skip_validate.yaml
README.rst
+8 -78
View File
@@ -1,80 +1,10 @@
openstack-tempest-skiplist
==========================
This project is no longer maintained.
Overview
--------
The contents of this repository are still available in the Git
source code management system. To see the contents of this
repository before it reached its end of life, please check out the
previous commit with "git checkout HEAD^1".
openstack-tempest-skiplist will generate a skip list to be executed by tempest
- Free software: Apache license
- Documentation: https://docs.openstack.org/openstack-tempest-skiplist/latest/
- Source: https://opendev.org/openstack/openstack-tempest-skiplist
- Bugs: https://storyboard.openstack.org/#!/project/1173
- Release notes: https://docs.openstack.org/releasenotes/openstack-tempest-skiplist
Quickstart
==========
- edit tempest_skip.yml
- add the required fields, including the launchpad
- execute tox tests
- submit :)
Installation
------------
Installing from git and virtualenv::
$ git clone https://opendev.org/openstack/openstack-tempest-skiplist
$ cd openstack-tempest-skiplist
$ virtualenv .venv
$ source .venv/bin/activate
$ pip install .
Validation
----------
After edit your file, you can check if it is valid with the following command::
$ tempest-skip validate --file file.yaml
Examples
--------
A simple example of the structure of the yaml file expected by
tempest-skiplist.
For more information about each field, visit the `documentation <https://docs.openstack.org/openstack-tempest-skiplist/latest/yaml/formatting.html>` website:
.. code-block:: yaml
known_failures:
- test: 'full.tempest.test'
bz: 'https://bugzilla.redhat.com/1'
lp: 'https://launchpad.net/bugs/1'
deployment:
- 'undercloud'
- 'overcloud'
jobs:
- job1
- job2
reason: 'default reason'
releases:
- name: master
lp: 'https://launchpad.net/bugs/2'
reason: 'Some reason'
- name: train
bz: 'https://bugzilla.redhat.com/train1'
installers:
- 'osp'
- name: ussuri
installers:
- 'tripleo'
bz: 'https://bugzilla.redhat.com/ussuri1'
In the above example, the test *full.tempest.test* will be executed by tempest
in both deployments, undercloud, and overcloud (depending on the job).
It will also be skipped in the releases master, train and ussuri, specifically
in jobs job1 and job2. It will not be skipped in any other job, no matter what
the release is.
Removing the list of jobs, means it will be skipped everywhere.
For any further questions, please email
openstack-discuss@lists.openstack.org or join #openstack-dev on
OFTC.
doc/requirements.txt
-6
View File
@@ -1,6 +0,0 @@
openstackdocstheme>=2.2.1 # Apache-2.0
reno>=3.1.0 # Apache-2.0
sphinx>=2.0.0,!=2.1.0 # BSD
sphinx-argparse>=0.2.2 # MIT
sphinxcontrib-svg2pdfconverter>=0.1.0 # BSD
validators==0.18.2
doc/source/addtest/addtest.rst
-86
View File
@@ -1,86 +0,0 @@
========================
Adding tests to skiplist
========================
Adding tests
------------
Of course it is possible to directly edit the yaml file and add the test
itself, but that may lead to failures, as for example duplicated entries,
identation issues, having a namespace test instead of full path test name.
In order to avoid that, you can use the `addtest` command that will identify if
the test is already on the skiplist, avoiding duplication. It will also
generate the yaml file properly as well as avoid the use of test namespace. For
example, if a user tries to add the tempest.scenario, it will skip all the
tests under tempest.scenario, which is not the desirable behavior. However, it
is a lot of work to add each entry under tempest.scenario, since we must repeat
all the reasons, bugzilla, releases, etc.
The addtest command solves all these problems for you. First of all, when you
try to add a test passing a test namespace, addtest will give you a list of
tests under that particular namespace, where you can choose from the list which
ones you would like to add. Select the ones you want and you are done!
Examples
--------
The command below add the test tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_connectivity_between_vms_on_different_networks::
$ tempest-skip addtest \
--file roles/validate-tempest/vars/tempest_skip.yml \
--release master \
--test tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_connectivity_between_vms_on_different_networks \
--reason 'Failing on network' \
--lp https://launchpad.net/bug/12345
In this example, we are adding the full path test, and so, the command will not
prompt you a list of tests that you want to choose. If for example, only the
namespace be parsed, you will be prompted with a list of tests under that
namespace::
$ tempest-skip addtest \
--file roles/validate-tempest/vars/tempest_skip.yml \
--release master \
--test tempest.scenario.test_network_basic_ops \
--reason 'Failing on network' \
--lp https://launchpad.net/bug/12345
And this is the output:
.. code-block::
[?] These are the tests available on the namespace, choose which ones you want to add. Press space to select:
> o tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_connectivity_between_vms_on_different_networks
o tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic
o tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_mtu_sized_frames
o tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
o tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_port_security_macspoofing_port
o tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_preserve_preexisting_port
o tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_router_rescheduling
o tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details
o tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_instance_port_admin_state
o tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state
You can use the arrow keys on your keyboard to navigate through the list, and
space to select. Once you are done, just press enter, the command will go
through each test, check if the test exists or not. If it exists, it will also
check the release exists, and properly add the test.
Once you are done, you can validate if the yaml file was generated properly
with the command::
$ tempest-skip validate --file roles/validate-tempest/vars/tempest_skip.yml
There are some arguments you can pass to the addtest, the required ones are:
* --file
* --lp or --bz
* --reason
All the other arguments have default values, for example, `--release` default
value is master and `--deployment` default value is overcloud
For more information, use::
$ tempest-skip addtest --help
doc/source/addtest/index.rst
-5
View File
@@ -1,5 +0,0 @@
.. toctree::
:maxdepth: 2
:includehidden:
addtest
doc/source/conf.py
-86
View File
@@ -1,86 +0,0 @@
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# -- General configuration ----------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = [
'openstackdocstheme',
#'sphinx.ext.intersphinx',
'sphinx.ext.autodoc',
'sphinxcontrib.rsvgconverter',
'sphinxarg.ext'
]
html_theme = 'openstackdocs'
# openstackdocstheme options
openstackdocs_repo_name = 'openstack/openstack-tempest-skiplist'
openstackdocs_pdf_link = True
openstackdocs_use_storyboard = True
# autodoc generation is a bit aggressive and a nuisance when doing heavy
# text edit cycles.
# execute "export SPHINX_DEBUG=1" in your terminal to disable
# The suffix of source filenames.
source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = 'openstack-tempest-skiplist'
copyright = u'2016, Red Hat'
# If true, '()' will be appended to :func: etc. cross-reference text.
add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
add_module_names = True
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'native'
# -- Options for HTML output --------------------------------------------------
# The theme to use for HTML and HTML Help pages. Major themes that come with
# Sphinx are currently 'default' and 'sphinxdoc'.
# html_theme_path = ["."]
# html_theme = '_theme'
# html_static_path = ['static']
# Output file base name for HTML help builder.
htmlhelp_basename = '%sdoc' % project
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass
# [howto/manual]).
latex_documents = [
('index',
'doc-%s.tex' % project,
u'%s Documentation' % project,
u'Red Hat, Inc.', 'manual'),
]
latex_use_xindy = False
latex_elements = {
'extraclassoptions': 'openany,oneside',
}
# Example configuration for intersphinx: refer to the Python standard library.
#intersphinx_mapping = {'http://docs.python.org/': None}
doc/source/index.rst
-21
View File
@@ -1,21 +0,0 @@
Welcome to openstack-tempest-skiplist documentation!
========================================================
openstack-tempest-skiplist is a tool for generate tempest skiplist based on
different criteria such as release, installer type, and job name
Content:
--------
.. toctree::
:maxdepth: 2
overview
install/index
yaml/index
validate/index
listyaml/index
listallowed/index
addtest/index
* :ref:`search`
doc/source/install/index.rst
-5
View File
@@ -1,5 +0,0 @@
.. toctree::
:maxdepth: 2
:includehidden:
installation
doc/source/install/installation.rst
-34
View File
@@ -1,34 +0,0 @@
============
Installation
============
Git
---
1. Clone and change to the directory::
$ git clone https://opendev.org/openstack/openstack-tempest-skiplist
$ cd openstack-tempest-skiplist
2. Create a virtual environment using :command:`virtualenv`::
$ virtualenv .venv
$ source .venv/bin/activate
3. Install requirements in the newly created virtual environment::
(.venv) $ pip install .
4. *(optional)* Instead of manual installation described in steps 2 and 3
above, tox can be used for installing the requirements as well.
To create python 3.6 environment run following::
$ tox -epy36
$ source .tox/py36/bin/activate
Pip installation
----------------
Install ``openstack-tempest-skiplist`` via pip as follows::
$ pip install openstack-tempest-skiplist
doc/source/listallowed/index.rst
-5
View File
@@ -1,5 +0,0 @@
.. toctree::
:maxdepth: 2
:includehidden:
listallowed
doc/source/listallowed/listallowed.rst
-68
View File
@@ -1,68 +0,0 @@
==================
List allowed tests
==================
List allowed tests
------------------
You can use :command:`tempest-skip list-allowed` command to list the tests to
be executed with two positional parameters which are in the
expected format::
1. ``--file`` is the positional parameter - list all the tests in the file
2. ``--group`` or ``--job`` - filter the tests for a specific job, or a
specific group.
Job filter
----------
The job filter, is as the name indicate, it checks the yaml file for a job that
matches (must be full match, not partial), and list the tests related to that
specific job::
$ tempets-skip list-allowed --file tempest_allow.yml --job job1
Group filter
------------
The group filter, which have precedence on the ``--job``, will list the tests
for a particular group. This is good when you have several jobs, that run a
specific set of tests. In this case, you don't need to repeat the same set of
tests for several different jobs::
$ tempest-skip list-allowed --file tempest_allow.yml --group default_group
Release filter
--------------
The release filter, which is default to master, filter based on group or job
for an specific release.
Multiple Groups with same name
------------------------------------------
Multiple groups can have same name with different tests and releases. This
behaviour allows us to classify the tests on the basis of releases. Here, we
cannot define a release more than once in such groups i.e groups having same
name should mandatorily have different releases::
- name: featureset062 # standalone-jobs
tests:
- 'octavia_tempest_plugin.tests.scenario.v2.test_healthmonitor'
- 'octavia_tempest_plugin.tests.scenario.v2.test_listener'
releases:
- master
- wallaby
- name: featureset062 # standalone-jobs
tests:
- 'octavia_tempest_plugin.tests.scenario.v2.test_healthmonitor'
- 'octavia_tempest_plugin.tests.scenario.v2.test_l7policy'
releases:
- train
Wildcard filter for releases
----------------------------
If in the releases list in the yaml file, the release ``all`` is set, that
means, it will not matter which release is passed to ``tempest-skip`` command,
it will be included in the final list.
doc/source/listyaml/index.rst
-5
View File
@@ -1,5 +0,0 @@
.. toctree::
:maxdepth: 2
:includehidden:
listyaml
doc/source/listyaml/listyaml.rst
-32
View File
@@ -1,32 +0,0 @@
======================
List skipped tests
======================
List skipped tests
-------------------
You can use :command:`tempest-skip list-skipped` command to list tests in the yaml
file with one positional and two optional parameters which is in the expected
format::
1. ``--file`` is the positional parameter - lists all the tests in the file::
$ tempest-skip list-skipped yaml --file tempest_skip.yml
2. ``--release``, ``--deployment``, ``--installer`` and ``--job`` are the
optional parameters - list all the tests within a specific release,
deployment or a specific job::
$ tempest-skip list-skipped --file tempest_skip.yml --release train
$ tempest-skip list-skipped --file tempest_skip.yml --job job1
$ tempest-skip list-skipped --file tempest_skip.yml --release train --job job1
$ tempest-skip list-skipped --file tempest_skip.yml --deployment undercloud
$ tempest-skip list-skipped --file tempest_skip.yml --installer tripleo
This will return any tests that match the job, as well as tests that doesn't
have any job configured. This is required when you configure your zuul jobs to
always parse the ``--job`` option. In this scenario, if a job2 is parsed, and
there is no test with job2, it would return zero tests to be skipped, which is
not the intent. The test with no job defined, means, skip everywhere, if you
define the job in the test yaml file, it means, skip all the tests that doesn't
have a job defined, plus this test.
doc/source/overview.rst
-13
View File
@@ -1,13 +0,0 @@
========
Overview
========
openstack-tempest-skiplist will automatically generate the list of tests to
be skipped by tempest based on the criterias passed to the tool
- Free software: Apache license
- Documentation: https://docs.openstack.org/openstack-tempest-skiplist/latest/
- Storyboard: https://storyboard.openstack.org/#!/project/1173
- Source: https://opendev.org/openstack/openstack-tempest-skiplist
- Bugs: https://storyboard.openstack.org/#!/project/1173
- Release notes: https://docs.openstack.org/releasenotes/openstack-tempest-skiplist
doc/source/validate/index.rst
-5
View File
@@ -1,5 +0,0 @@
.. toctree::
:maxdepth: 2
:includehidden:
validate
doc/source/validate/validate.rst
-20
View File
@@ -1,20 +0,0 @@
======================
Validate the yaml file
======================
Validation
----------
You can use :command:`tempest-skip validate` command to validate if the yaml
file is in the expected format for the allowed list::
$ tempest-skip validate --allowed --file good_file.yaml
or for the skipped list:
$ tempest-skip validate --skipped --file good_file.yaml
This will return nothing if the file is valid, or an error otherwise::
$ tempest-skip validate --file bad_file.yaml
required key not provided @ data['known_failures'][0]['releases'][2]['reason']
doc/source/yaml/formatting.rst
-201
View File
@@ -1,201 +0,0 @@
==========
Formatting
==========
YAML File
---------
The YAML file used by `openstack-tempest-skiplist` use the following pattern:
.. code-block:: yaml
known_failures:
- test: 'full.tempest.test'
bz: 'https://bugzilla.redhat.com/1'
lp: 'https://launchpad.net/bugs/1'
deployment:
- 'undercloud'
- 'overcloud'
jobs:
- job1
- job2
reason: 'default reason'
releases:
- name: master
lp: 'https://launchpad.net/bugs/2'
reason: 'Some reason'
- name: train
bz: 'https://bugzilla.redhat.com/train1'
- name: ussuri
bz: 'https://bugzilla.redhat.com/ussuri1'
YAML values
-----------
BZ and LP
+++++++++
One of them is required, not both, but it's crucial that people are able to
trackdown the reason the test is being added in the skiplist.
Here there are two levels of lp and bz: the first one, is the global lp and bz
and the second is based on the release. This is required because it's possible
to have the test failing in two different releases but with different reasons.
If the lp or bz is set on test level, you don't need to add it per release, it
will use the test level.
The value of both lp and bz are their respective URL
Deployment
++++++++++
This is right now TripleO only configuration, since it deploys two different
openstack instances, undercloud and overcloud. Default to overcloud.
Installers
++++++++++
This is a list of installers per release where the test will be
skipped. The options for installers are 'tripleo' and 'osp',
where 'tripleo' is the upstream installer and 'osp' is the
downstream installer.
The default (if this option is not set) is 'tripleo' and 'osp'.
Jobs
++++
This is a list of jobs where the test should be skipped. This is not a
required field, however, once set, the tool will return the test only if the
job matches.
Reason
++++++
This is a description about why this test is being skipped. It exists in two
levels, the first in the test level, that will be the default reason, and in
release level, where reason can be different.
Releases
++++++++
Releases contain a list of releases that the test will be skipped. It's very
common that in a release the test is passing, but in another don't, so we can
manage it here.
Here, it's also required a reason, and a lp or a bz
Examples
--------
Below are some examples of valid yaml files that can be used:
No releases
+++++++++++
Since there's no releases, this will be valid for all releases:
.. code-block:: yaml
known_failures:
- test: 'tempest_skip.tests.test_validate'
bz: 'https://bugzilla.redhat.com/1'
lp: 'https://launchpad.net/bugs/1'
deployment:
- 'undercloud'
reason: 'This test will be skipped in any release'
With releases
+++++++++++++
As release is set, the test will be skipped only on the matching releases
.. code-block:: yaml
known_failures:
- test: 'tempest_skip.tests.test_validate'
bz: 'https://bugzilla.redhat.com/1'
lp: 'https://launchpad.net/bugs/1'
deployment:
- 'undercloud'
reason: 'This test will be skipped in any release'
releases:
- name: rocky
reason: 'Test failing in rock because of network'
lp: 'https://launchpad.net/bugs/1'
- name: ussuri
reason: 'Test is failing in ussuri because of storage bug'
bz: 'https://bugzilla.redhat.com/1'
With jobs
+++++++++
If a list of jobs is set, the test will be skipped only in the matching jobs
.. code-block:: yaml
known_failures:
- test: 'tempest_skip.tests.test_validate'
bz: 'https://bugzilla.redhat.com/1'
lp: 'https://launchpad.net/bugs/1'
deployment:
- 'undercloud'
reason: 'This test will be skipped in any release'
jobs:
- tempest-test-job-skip1
- tempest-test-job-skip2
With jobs and releases
++++++++++++++++++++++
This test will be skipped only when it matches both, job and release
.. code-block:: yaml
known_failures:
- test: 'tempest_skip.tests.test_validate'
bz: 'https://bugzilla.redhat.com/1'
lp: 'https://launchpad.net/bugs/1'
deployment:
- 'undercloud'
reason: 'This test will be skipped in all releases'
releases:
- name: rocky
reason: 'Test failing in rock because of network'
lp: 'https://launchpad.net/bugs/1'
- name: ussuri
reason: 'Test is failing in ussuri because of storage bug'
bz: 'https://bugzilla.redhat.com/1'
jobs:
- tempest-test-job-skip1
- tempest-test-job-skip2
With releases and installers
++++++++++++++++++++++++++++
.. code-block:: yaml
known_failures:
- test: 'tempest_skip.tests.test_validate'
bz: 'https://bugzilla.redhat.com/1'
lp: 'https://launchpad.net/bugs/1'
deployment:
- 'overcloud'
reason: 'This test will be skipped in any release'
releases:
- name: train
reason: 'Test failing in train because of network'
installers:
- 'tripleo'
lp: 'https://launchpad.net/bugs/1'
- name: wallaby
reason: 'Test is failing in /osp-17 because of storage bug'
installers:
- 'osp'
bz: 'https://bugzilla.redhat.com/1'
doc/source/yaml/index.rst
-5
View File
@@ -1,5 +0,0 @@
.. toctree::
:maxdepth: 2
:includehidden:
formatting
openstack-operators/tempest_allow.yml
-36
View File
@@ -1,36 +0,0 @@
groups:
- name: openstack-operator
tests:
- tempest.api.identity.*.v3
releases:
- all
- name: cinder-operator
tests:
- tempest.api.volume
releases:
- all
- name: keystone-operator
tests:
- tempest.api.identity.*.v3
releases:
- all
- name: neutron-operator
tests:
- tempest.api.network.*
releases:
- all
- name: nova-operator
tests:
- tempest.api.compute.*
releases:
- all
- name: manila-operator
tests:
- manila_tempest_tests.tests.api
releases:
- all
- name: ironic-operator
tests:
- ironic_tempest_plugin.tests.api.admin.*
releases:
- all
openstack-operators/tempest_skip.yml
-1256
View File
File diff suppressed because it is too large Load Diff
playbooks/tempest_skip_install.yaml
-5
View File
@@ -1,5 +0,0 @@
---
- hosts: all
roles:
- ensure-pip
- install_skiplist
playbooks/tempest_skip_validate.yaml
-5
View File
@@ -1,5 +0,0 @@
---
- hosts: all
roles:
- validate_yaml
- list_skipped
releasenotes/source/_static/.placeholder
View File
releasenotes/source/_templates/.placeholder
View File
releasenotes/source/conf.py
-268
View File
@@ -1,268 +0,0 @@
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Glance Release Notes documentation build configuration file, created by
# sphinx-quickstart on Tue Nov 3 17:40:50 2015.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
# sys.path.insert(0, os.path.abspath('.'))
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'openstackdocstheme',
'reno.sphinxext',
]
# openstackdocstheme options
openstackdocs_repo_name = 'openstack/openstack-tempest-skiplist'
openstackdocs_use_storyboard = True
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
# source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
copyright = u'2019, OpenStack'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
# language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
# today = ''
# Else, today_fmt is used as the format for a strftime call.
# today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = []
# The reST default role (used for this markup: `text`) to use for all
# documents.
# default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
# add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
# add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
# show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
# modindex_common_prefix = []
# If true, keep warnings as "system message" paragraphs in the built documents.
# keep_warnings = False
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'openstackdocs'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
# html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
# html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
# html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
# html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
# html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
# html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
# html_extra_path = []
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
# html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
# html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
# html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
# html_additional_pages = {}
# If false, no module index is generated.
# html_domain_indices = True
# If false, no index is generated.
# html_use_index = True
# If true, the index is split into individual pages for each letter.
# html_split_index = False
# If true, links to the reST sources are added to the pages.
# html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
# html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
# html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
# html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
# html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'skiplistReleaseNotesdoc'
# -- Options for LaTeX output ---------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
# 'preamble': '',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
('index', 'skiplistReleaseNotes.tex',
u'skiplist Release Notes Documentation', u'OpenStack', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
# latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
# latex_use_parts = False
# If true, show page references after internal links.
# latex_show_pagerefs = False
# If true, show URL addresses after external links.
# latex_show_urls = False
# Documents to append as an appendix to all manuals.
# latex_appendices = []
# If false, no module index is generated.
# latex_domain_indices = True
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'skiplistreleasenotes', u'skiplist Release Notes Documentation',
[u'OpenStack'], 1)
]
# If true, show URL addresses after external links.
# man_show_urls = False
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
('index', 'skiplistReleaseNotes', u'skiplist Release Notes Documentation',
u'OpenStack', 'skiplistReleaseNotes',
'A collection of tempest skipped tests shared across OpenStack projects.',
'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
# texinfo_appendices = []
# If false, no module index is generated.
# texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
# texinfo_show_urls = 'footnote'
# If true, do not generate a @detailmenu in the "Top" node's menu.
# texinfo_no_detailmenu = False
# -- Options for Internationalization output ------------------------------
locale_dirs = ['locale/']
releasenotes/source/index.rst
-8
View File
@@ -1,8 +0,0 @@
============================================
openstack-tempest-skiplist Release Notes
============================================
.. toctree::
:maxdepth: 1
unreleased
releasenotes/source/unreleased.rst
-5
View File
@@ -1,5 +0,0 @@
==============================
Current Series Release Notes
==============================
.. release-notes::
requirements.txt
-6
View File
@@ -1,6 +0,0 @@
cliff
inquirer
ruamel.yaml>=0.17.7
tempest
validators
voluptuous
roles/install_skiplist/README.rst
-11
View File
@@ -1,11 +0,0 @@
Install tempest skiplist
========================
This role install tempest-skiplist.
**Role Variables**
.. zuul:rolevar:: tempest_skip_path
:type: string
A path to the openstack-tempest-skiplist directory
roles/install_skiplist/defaults/main.yaml
-7
View File
@@ -1,7 +0,0 @@
---
# Path for the tempest skip
# tempest_skip_path:
_zuul_path: "{{ zuul.projects['opendev.org/openstack/openstack-tempest-skiplist'].src_dir | default('') }}"
_local_path: "{{ local_working_dir | default('') }}/usr/local/share/ansible"
tempest_skip_path: "{{ (zuul is defined) | ternary(_zuul_path, _local_path) }}"
list_skipped_yaml_file: "{{ tempest_skip_path }}/roles/validate-tempest/vars/tempest_skip.yml"
roles/install_skiplist/tasks/main.yaml
-18
View File
@@ -1,18 +0,0 @@
---
- block:
- name: Create tempest-skiplist venv with latest pip, setuptools and pbr
pip:
name: ['setuptools', 'pbr']
state: latest
extra_args: --user
- name: Ensure a recent version of pip is installed
pip:
name: "pip>=19.1.1"
virtualenv_command: "/usr/bin/python3 -m venv"
- name: Install tempest-skiplist
pip:
name: "."
chdir: "{{ tempest_skip_path }}"
extra_args: --user
roles/list_allowed/README.rst
-23
View File
@@ -1,23 +0,0 @@
List allowed tempest tests
==========================
This role executes the tempest-skip tool and returns the list of tests to be
executed.
**Role Variables**
.. zuul:rolevar:: list_allowed_yaml_file
:type: string
Path to the yaml file containing the skipped tests
.. zuul:rolevar:: list_allowed_job
:type: string
Job name to be used in the filter if required
.. zuul:rolevar:: list_allowed_group
:type: string
Group to be used in the filter. It has more precedence than
list_allowed_job.
roles/list_allowed/library/list_allowed.py
-123
View File
@@ -1,123 +0,0 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from ansible.module_utils.basic import AnsibleModule
from tempest_skip.list_allowed import ListAllowedYaml
import sys
ANSIBLE_METADATA = {
'metadata_version': '0.1',
'status': ['preview'],
'supported_by': 'community'
}
DOCUMENTATION = '''
---
module: list_allowed
author:
- "Arx Cruz (@arxcruz)
version_added: '2.9'
short_description:
- Parse filtered tests from tempest
notes: []
requirements:
- tempest-skip
options:
yaml_file:
description:
- Path to a yaml file containing the tests in
openstack-tempest-skiplist format
required: True
type: str
job:
description:
- Name of the job to be used in the filter.
required: False
type: str
group:
description:
- Group name to be used in the filter. It has more precedence than job
required: False
type: str
'''
EXAMPLES = '''
- name: Get list of allowed tests
list_allowed:
yaml_file: /tmp/allowed.yaml
job: tripleo-ci-centos-8-standalone
group: default
'''
RETURN = '''
allowed_tests:
description:
- List of tests filtered
returned: Always
type: list
sample: [
"tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON"
]
'''
def run_module():
module_args = dict(
yaml_file=dict(type='str', required=True),
job=dict(type='str', required=False, default=None),
group=dict(type='str', required=False, default=None),
release=dict(type='str', required=False, default="master")
)
result = dict(
changed=True,
message='',
filtered_tests=[]
)
module = AnsibleModule(
argument_spec=module_args
)
if not module.params['yaml_file']:
module.fail_json(msg="A yaml file must be provided!")
if not module.params['job'] and not module.params['group']:
module.fail_json(msg="You must specify either job or group parameter!")
cmd = ListAllowedYaml(__name__, sys.argv[1:])
parser = cmd.get_parser(__name__)
parser.file = module.params['yaml_file']
parser.group = module.params['group']
parser.release = module.params['release']
parser.job = module.params['job']
tests = cmd.take_action(parser)
allowed_tests = [test[0] for test in tests[1]]
result.update({'allowed_tests': allowed_tests})
module.exit_json(**result)
def main():
run_module()
if __name__ == '__main__':
main()
roles/list_allowed/tasks/main.yaml
-8
View File
@@ -1,8 +0,0 @@
---
- name: List allowed tests
list_allowed:
yaml_file: "{{ list_allowed_yaml_file }}"
group: "{{ list_allowed_group| default(omit) }}"
release: "{{ list_allowed_release | default(omit) }}"
job: "{{ list_allowed_job | default(omit) }}"
register: tempest_allowed_register
roles/list_skipped/README.rst
-22
View File
@@ -1,22 +0,0 @@
List skipped tempest tests
==========================
This role execute the tempest-skip tool and return the list of tests to be
skipped.
**Role Variables**
.. zuul:rolevar:: list_skipped_yaml_file
:type: string
Path to the yaml file containing the skipped tests
.. zuul:rolevar:: list_skipped_release
:type: string
Release name to be used in the filter if required.
.. zuul:rolevar:: list_skipped_job
:type: string
Job name to be used in the filter if required.
roles/list_skipped/defaults/main.yaml
-8
View File
@@ -1,8 +0,0 @@
---
# list_skipped_yaml_file: /home/zuul/src/openstack/openstack-tempest-skiplist/roles/validate-tempest/vars/tempest_skip.yml
# list_skipped_job: tripleo-ci-centos-8-containers-multinode
# list_skipped_release: master
_zuul_path: "{{ zuul.projects['opendev.org/openstack/openstack-tempest-skiplist'].src_dir | default('') }}"
_local_path: "{{ local_working_dir | default('') }}/usr/local/share/ansible"
tempest_skip_path: "{{ (zuul is defined) | ternary(_zuul_path, _local_path) }}"
list_skipped_yaml_file: "{{ tempest_skip_path }}/roles/validate-tempest/vars/tempest_skip.yml"
roles/list_skipped/library/list_skipped.py
-137
View File
@@ -1,137 +0,0 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from ansible.module_utils.basic import AnsibleModule
from tempest_skip.list_yaml import ListSkippedYaml
import sys
ANSIBLE_METADATA = {
'metadata_version': '0.1',
'status': ['preview'],
'supported_by': 'community'
}
DOCUMENTATION = '''
---
module: list_skipped
author:
- "Arx Cruz (@arxcruz)
version_added: '2.9'
short_description:
- Parse skipped tests from tempest
notes: []
requirements:
- tempest-skip
options:
yaml_file:
description:
- Path to a yaml file containing the skipped tests in
openstack-tempest-skiplist format
required: True
type: str
job:
description:
- Name of the job to be used in the filter. Passing the job it will
filter only tests that have the specified job in the jobs list.
required: False
type: str
release:
description:
- Release name to be used in the filter. Default is set to 'master'
required: False
type: str
deployment:
description:
- Type of deployment, right now is undercloud or overcloud. Default is
set to 'overcloud'
required: False
type: str
installer:
description:
- Type of installer. Right now it's just a string, but in the future
might be specific keys
required: False
type: str
'''
EXAMPLES = '''
- name: Get list of skipped tests
list_skipped:
yaml_file: /tmp/skipped.yaml
job: tripleo-ci-centos-8-standalone
release: master
deployment: 'overcloud'
installer: 'tripleo'
'''
RETURN = '''
skipped_tests:
description:
- List of tests filtered
returned: Always
type: list
sample: [
"tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON"
]
'''
def run_module():
module_args = dict(
yaml_file=dict(type='str', required=True),
job=dict(type='str', required=False, default=None),
release=dict(type='str', required=False, default='master'),
deployment=dict(type='str', required=False, default='overcloud'),
installer=dict(type='str', required=False, default='tripleo')
)
result = dict(
changed=True,
message='',
skipped_tests=[]
)
module = AnsibleModule(
argument_spec=module_args
)
if not module.params['yaml_file']:
module.fail_json(msg="A yaml file must be provided!")
cmd = ListSkippedYaml(__name__, sys.argv[1:])
parser = cmd.get_parser(__name__)
parser.file = module.params['yaml_file']
parser.release = module.params['release']
parser.job = module.params['job']
parser.deployment = module.params['deployment']
parser.installer = module.params['installer']
tests = cmd.take_action(parser)
skipped_tests = [test[0] for test in tests[1]]
result.update({'skipped_tests': skipped_tests})
module.exit_json(**result)
def main():
run_module()
if __name__ == '__main__':
main()
roles/list_skipped/tasks/main.yaml
-9
View File
@@ -1,9 +0,0 @@
---
- name: List skipped tests
list_skipped:
yaml_file: "{{ list_skipped_yaml_file }}"
release: "{{ list_skipped_release | default(omit) }}"
job: "{{ list_skipped_job | default(omit) }}"
deployment: "{{ list_skipped_deployment | default(omit) }}"
installer: "{{ list_skipped_installer | default(omit) }}"
register: tempest_skip_register
roles/validate-tempest/vars/tempest_allow.yml
-268
View File
@@ -1,268 +0,0 @@
groups:
- name: default
tests:
- smoke
releases:
- all
- name: featureset001
tests:
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
releases:
- all
- name: featureset001_periodic
tests:
- 'tempest.api'
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
releases:
- all
- name: featureset002
tests:
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
releases:
- all
- name: featureset002_periodic
tests:
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
releases:
- all
- name: featureset010
tests:
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
releases:
- all
- name: featureset020
tests:
- 'tempest.scenario'
- 'tempest.api'
- 'keystone_tempest_plugin'
- 'neutron_tempest_plugin'
- 'cinder_tempest_plugin'
releases:
- all
- name: featureset020_periodic
tests:
- 'tempest.scenario'
- 'keystone_tempest_plugin.scenario'
- 'neutron_tempest_plugin.scenario'
- 'cinder_tempest_plugin.scenario'
releases:
- all
- name: featureset027
tests:
- 'tempest.api.identity'
- 'tempest.api.network.test_networks'
- 'tempest.api.network.test_networks_negative'
- 'tempest.api.network.test_security_groups_negative'
- 'tempest.api.network.test_ports'
- 'tempest.api.network.test_routers_negative'
releases:
- all
- name: featureset030
tests:
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops'
releases:
- all
- name: featureset035
tests:
- 'test_network_v6'
- 'ipv6'
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
releases:
- all
- name: featureset035_periodic
tests:
- 'test_network_v6'
- 'ipv6'
- 'tempest.api'
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
releases:
- all
- name: featureset039
tests:
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
releases:
- all
- name: featureset003
tests:
- 'tempest.api.identity'
- 'tempest.api.image.v2.test_images.BasicOperationsImagesTest'
- 'tempest.api.image.v2.test_images_tags_negative.ImagesTagsNegativeTest'
- 'tempest.api.image.v2.test_images_member_negative.ImagesMemberNegativeTest'
- 'tempest.api.compute.admin.test_flavors'
- 'tempest.api.compute.flavors'
- 'tempest.api.compute.keypairs'
- 'tempest.api.compute.admin.test_quotas'
- 'tempest.api.compute.admin.test_aggregates_negative'
- 'tempest.api.compute.security_groups'
- 'tempest.api.network.test_networks'
- 'tempest.api.network.test_networks_negative'
- 'tempest.api.network.test_security_groups_negative'
- 'tempest.api.network.test_ports'
- 'tempest.api.network.test_routers_negative'
releases:
- all
- name: featureset052
tests:
- 'tempest.scenario.test_minimum_basic.TestMinimumBasicScenario'
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
- 'tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern'
releases:
- all
- name: featureset052_periodic
tests:
- 'smoke'
releases:
- all
- name: featureset056 # standalone-upgrade
tests:
- 'tempest.scenario.test_minimum_basic.TestMinimumBasicScenario'
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
- 'tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern'
releases:
- all
- name: featureset060 # scenario012-standalone
tests:
- 'ironic_tempest_plugin.tests.api.admin.test_chassis.TestChassis'
- 'ironic_tempest_plugin.tests.api.admin.test_api_discovery.TestApiDiscovery'
- 'ironic_tempest_plugin.tests.api.admin.test_conductor.TestConductors'
- 'ironic_tempest_plugin.tests.api.admin.test_deploy_templates.TestDeployTemplates'
- 'ironic_tempest_plugin.tests.api.admin.test_deploy_templates.TestDeployTemplatesNegative'
- 'ironic_tempest_plugin.tests.api.admin.test_deploy_templates.TestDeployTemplatesOldAPI'
releases:
- all
- name: featureset061 # containers-undercloud-minion
tests:
- 'tempest.api.identity'
- 'tempest.api.image.v2.test_images.BasicOperationsImagesTest'
- 'tempest.api.image.v2.test_images_tags_negative.ImagesTagsNegativeTest'
- 'tempest.api.image.v2.test_images_member_negative.ImagesMemberNegativeTest'
- 'tempest.api.compute.admin.test_flavors'
- 'tempest.api.compute.flavors'
- 'tempest.api.compute.keypairs'
- 'tempest.api.compute.admin.test_quotas'
- 'tempest.api.compute.admin.test_aggregates_negative'
- 'tempest.api.compute.security_groups'
- 'tempest.api.network.test_networks'
- 'tempest.api.network.test_networks_negative'
- 'tempest.api.network.test_security_groups_negative'
- 'tempest.api.network.test_ports'
- 'tempest.api.network.test_routers_negative'
releases:
- all
- name: featureset062 # standalone-jobs
tests:
- 'octavia_tempest_plugin.tests.scenario.v2.test_healthmonitor'
- 'octavia_tempest_plugin.tests.scenario.v2.test_listener'
- 'octavia_tempest_plugin.tests.scenario.v2.test_member'
- 'octavia_tempest_plugin.tests.scenario.v2.test_load_balancer'
releases:
- master
- zed
- wallaby
- name: featureset062 # standalone-jobs
tests:
- 'octavia_tempest_plugin.tests.scenario.v2.test_healthmonitor'
- 'octavia_tempest_plugin.tests.scenario.v2.test_l7policy'
- 'octavia_tempest_plugin.tests.scenario.v2.test_load_balancer'
- 'octavia_tempest_plugin.tests.scenario.v2.test_pool'
- 'octavia_tempest_plugin.tests.scenario.v2.test_listener'
- 'octavia_tempest_plugin.tests.scenario.v2.test_l7rule'
- 'octavia_tempest_plugin.tests.scenario.v2.test_member'
releases:
- train
- name: featureset062_periodic
tests:
- 'octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest.test_basic_http_traffic'
- 'octavia_tempest_plugin.tests.scenario.v2.test_pool'
releases:
- all
- name: featureset066
tests:
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
- 'tempest.api.compute.admin.test_flavors'
- 'tempest.api.compute.flavors'
- 'tempest.api.compute.keypairs'
- 'tempest.api.compute.security_groups'
releases:
- wallaby
- name: featureset063
tests:
- 'tempest.api.identity'
- 'tempest.api.image.v2.test_images.BasicOperationsImagesTest'
- 'tempest.api.image.v2.test_images_tags_negative.ImagesTagsNegativeTest'
- 'tempest.api.image.v2.test_images_member_negative.ImagesMemberNegativeTest'
- 'tempest.api.compute.admin.test_flavors'
- 'tempest.api.compute.flavors'
- 'tempest.api.compute.keypairs'
- 'tempest.api.compute.admin.test_quotas'
- 'tempest.api.compute.admin.test_aggregates_negative'
- 'tempest.api.compute.security_groups'
- 'tempest.api.network.test_networks'
- 'tempest.api.network.test_networks_negative'
- 'tempest.api.network.test_security_groups_negative'
- 'tempest.api.network.test_ports'
- 'tempest.api.network.test_routers_negative'
releases:
- all
- name: featureset064
tests:
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
releases:
- all
- name: featureset067
tests:
- 'tempest.scenario.test_minimum_basic.TestMinimumBasicScenario'
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps'
- 'tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern'
releases:
- all
jobs:
- name: tripleo-ci-scenario000-standalone-options
tests:
- 'keystone_tempest_plugin.tests.api.identity.v3'
releases:
- all
- name: tripleo-ci-scenario001-standalone-options
tests:
- 'tempest.api.identity.v3'
- 'tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern'
- 'tempest.scenario.test_snapshot_pattern.TestSnapshotPattern.test_snapshot_pattern'
- 'telemetry_tempest_plugin.scenario.test_telemetry_integration.TestTelemetryIntegration'
releases:
- all
- name: tripleo-ci-scenario002-standalone-options
tests:
- 'tempest.api.object_storage.test_object_services.ObjectTest'
- 'barbican_tempest_plugin.tests.scenario.test_volume_encryption.VolumeEncryptionTest'
- 'barbican_tempest_plugin.tests.scenario.test_image_signing.ImageSigningSnapshotTest'
- 'barbican_tempest_plugin.tests.scenario.test_image_signing.ImageSigningTest.test_signed_image_upload_and_boot'
- 'telemetry_tempest_plugin.scenario.test_telemetry_integration.TestTelemetryIntegration'
releases:
- all
- name: tripleo-ci-scenario003-standalone-options
tests:
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops'
- 'sahara_tempest_plugin.tests.api.test_data_sources.DataSourceTest'
- 'mistral_tempest_tests.tests.api.v2.test_actions.ActionTestsV2'
- 'designate_tempest_plugin.tests.api.v2.test_recordset.RecordsetsTest'
- 'designate_tempest_plugin.tests.api.v2.test_zones.ZonesTest'
releases:
- all
- name: tripleo-ci-scenario004-standalone-options
tests:
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops'
- 'manila_tempest_tests.tests.api.test_shares.SharesNFSTest.test_create_get_delete_share'
- 'tempest.scenario.test_object_storage_basic_ops'
releases:
- all
- name: tripleo-ci-scenario007-standalone-options
tests:
- 'tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops'
- 'tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario'
releases:
- all
- name: tripleo-ci-scenario010-ovn-provider-standalone-options
tests:
- '^octavia_tempest_plugin.tests.scenario.v2.test_load_balancer.LoadBalancerScenarioTest'
releases:
- all
roles/validate-tempest/vars/tempest_skip.yml
-1085
View File
File diff suppressed because it is too large Load Diff
roles/validate_yaml/README.rst
-13
View File
@@ -1,13 +0,0 @@
Validate yaml schema for tempest skip
=====================================
This role execute the validate command from tempest-skiplist to ensure that
the yaml file used to store the tempest tests to be skipped are
in the correct schema format.
**Role Variables**
.. zuul:rolevar:: tempest_skip_path
:type: string
A path to the openstack-tempest-skiplist directory
roles/validate_yaml/defaults/main.yaml
-3
View File
@@ -1,3 +0,0 @@
---
# Path for the tempest skkip
# tempest_skip_path:
roles/validate_yaml/tasks/main.yaml
-21
View File
@@ -1,21 +0,0 @@
---
- block:
- name: Validate tempest-skip skipped yaml file
shell: |
set -ex
export PATH=$PATH:/usr/local/sbin:/usr/sbin:$HOME/.local/bin
tempest-skip validate --skipped --file roles/validate-tempest/vars/tempest_skip.yml
tempest-skip validate --skipped --file openstack-operators/tempest_skip.yml
args:
chdir: "{{ tempest_skip_path }}"
executable: /bin/bash
- name: Validate tempest-skip allowed yaml file
shell: |
set -ex
export PATH=$PATH:/usr/local/sbin:/usr/sbin:$HOME/.local/bin
tempest-skip validate --allowed --file roles/validate-tempest/vars/tempest_allow.yml
tempest-skip validate --allowed --file openstack-operators/tempest_allow.yml
args:
chdir: "{{ tempest_skip_path }}"
executable: /bin/bash
setup.cfg
-53
View File
@@ -1,53 +0,0 @@
[metadata]
name = openstack-tempest-skiplist
summary = Tool and repository to store tempest skip list for different installers
description_file =
README.rst
author = OpenStack
author_email = openstack-discuss@lists.openstack.org
home_page = https://pypi.org/project/openstack-tempest-skiplist
classifier =
Environment :: OpenStack
Intended Audience :: Information Technology
Intended Audience :: System Administrators
License :: OSI Approved :: Apache Software License
Operating System :: POSIX :: Linux
Programming Language :: Python
Programming Language :: Python :: 3
Programming Language :: Python :: 3.6
Programming Language :: Python :: 3.7
Programming Language :: Python :: 3.8
[files]
packages =
tempest_skip
data_files =
usr/local/share/ansible/roles/install_skiplist = roles/install_skiplist/*
usr/local/share/ansible/roles/validate_yaml = roles/validate_yaml/*
usr/local/share/ansible/roles/list_skipped = roles/list_skipped/*
usr/local/share/ansible/roles/list_allowed = roles/list_allowed*
usr/local/share/ansible/roles/validate-tempest = roles/validate-tempest/*
usr/local/share/openstack-operator = openstack-operator/*
[entry_points]
console_scripts =
tempest-skip = tempest_skip.main:main
tempest_skip.cm =
validate = tempest_skip.validate:Validate
list = tempest_skip.list_yaml:ListSkippedYaml
list-skipped = tempest_skip.list_yaml:ListSkippedYaml
addtest = tempest_skip.add_test:AddTest
list-allowed = tempest_skip.list_allowed:ListAllowedYaml
[build_sphinx]
source-dir = doc/source
build-dir = doc/build
all_files = 1
[upload_sphinx]
upload-dir = doc/build/html
[build_releasenotes]
all_files = 1
build-dir = releasenotes/build
source-dir = releasenotes/source
setup.py
-29
View File
@@ -1,29 +0,0 @@
# Copyright 2020 Red Hat, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT
import setuptools
# In python < 2.7.4, a lazy loading of package `pbr` will break
# setuptools if some other modules registered functions in `atexit`.
# solution from: http://bugs.python.org/issue15881#msg170215
try:
import multiprocessing # noqa
except ImportError:
pass
setuptools.setup(
setup_requires=['pbr>=1.8'],
pbr=True)
tempest_skip.yml
-1
View File
@@ -1 +0,0 @@
roles/validate-tempest/vars/tempest_skip.yml
tempest_skip/__init__.py
View File
tempest_skip/add_test.py
-189
View File
@@ -1,189 +0,0 @@
# Copyright 2020 Red Hat, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
import os
import tempfile
from argparse import ArgumentTypeError
import inquirer
import ruamel.yaml as yaml
import validators
from cliff.command import Command
from stestr import config_file
from tempest.cmd.init import TempestInit
from tempest.cmd.run import TempestRun
class AddTest(Command):
"""Command to add a test into the skiplist"""
log = logging.getLogger(__name__)
def take_action(self, parsed_args):
self.log.debug('Running add_test command')
self.abs_path = os.path.abspath(parsed_args.file)
self.yaml_file = yaml.round_trip_load(open(self.abs_path),
preserve_quotes=True)
test_list = self._get_tests_list()
filter_tests = list(filter(lambda x: parsed_args.test in x, test_list))
if len(filter_tests) > 1:
question = [inquirer.Checkbox('list_tests',
'These are the tests available on '
'the namespace, choose which ones '
'you want to add. Press space to '
'select',
choices=filter_tests)]
answers = inquirer.prompt(question)
for test in answers.get('list_tests', []):
self._add_test_in_yaml(test, parsed_args.deployment,
parsed_args.release,
parsed_args.reason, lp=parsed_args.lp,
bz=parsed_args.bz, job=parsed_args.job)
else:
self._add_test_in_yaml(parsed_args.test, parsed_args.deployment,
parsed_args.release,
parsed_args.reason, lp=parsed_args.lp,
bz=parsed_args.bz, job=parsed_args.job)
with open(self.abs_path, 'w') as f:
yaml.round_trip_dump(self.yaml_file, f,
Dumper=yaml.RoundTripDumper,
indent=4, block_seq_indent=2)
def _add_test_in_yaml(self, test_name, deployment, release, reason,
lp=None, bz=None, job=None):
was_inserted = False
release_exist = False
deployment_exist = False
job_exist = False
jobs = []
for test in self.yaml_file.get('known_failures'):
if test.get('test') == test_name:
# Test already exist
for d in test.get('deployment'):
if d == deployment:
deployment_exist = True
for r in test.get('releases'):
if r.get('name') == release:
was_inserted = True
release_exist = True
for j in test.get('jobs'):
if j == job:
job_exist = True
if not release_exist:
entry = {'name': release,
'reason': reason}
if lp:
entry['lp'] = lp
if bz:
entry['bz'] = bz
test.get('releases').append(entry)
was_inserted = True
if not deployment_exist:
test.get('deployment').append(deployment)
if not job_exist:
test.get('jobs').append(job)
if not was_inserted:
release = {'name': release, 'reason': reason}
if lp:
release['lp'] = lp
if bz:
release['bz'] = bz
if job:
jobs = [job]
entry = {'test': test_name, 'deployment': [deployment],
'releases': [release],
'jobs': jobs}
self.yaml_file.get('known_failures').append(entry)
self.log.debug('Test {} added with release {}'.format(
test_name, release))
else:
self.log.warning(
'Test {} already exist for release {}, doing nothing'.format(
test_name, release))
def _get_tests_list(self):
tempest_run = TempestRun(__name__, [])
tempest_init = TempestInit(__name__, [])
path = tempfile.mkdtemp(dir='/tmp')
parser = tempest_init.get_parser(__name__)
parser.workspace_path = None
parser.name = path.split(os.path.sep)[-1]
parser.config_dir = None
parser.dir = path
parser.show_global_dir = False
tempest_init.take_action(parser)
parser = tempest_run.get_parser('tempest')
parser.list_tests = True
parser.config_file = None
parser.workspace = path.split(os.path.sep)[-1]
parser.state = None
parser.smoke = False
parser.regex = ''
parser.whitelist_file = None
parser.blacklist_file = None
parser.black_regex = None
parser.workspace_path = None
tempest_run._create_stestr_conf()
# config = os.path.join(path, '.stestr.conf')
os.chdir(path)
conf = config_file.TestrConf('.stestr.conf')
cmd = conf.get_run_command()
try:
cmd.setUp()
ids = cmd.list_tests()
finally:
cmd.cleanUp()
return [i.split('[')[0] for i in ids]
def _validate_url(self, url):
if not validators.url(url):
raise ArgumentTypeError('{} is not a valid url'.format(url))
return url
def get_parser(self, prog_name):
parser = super(AddTest, self).get_parser(prog_name)
parser.add_argument('--file', dest='file', required=True,
help='Skiplist config file')
parser.add_argument('--release', dest='release', default='master',
help='Release where the test will be added')
parser.add_argument('--job', dest='job',
help='Specify in which job this test will be '
'skipped')
parser.add_argument('--test', dest='test', required=True,
help='Test to be skipped')
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument('--lp', dest='lp',
help='Launchpad bug', type=self._validate_url)
group.add_argument('--bz', dest='bz',
help='Bugzilla bug', type=self._validate_url)
parser.add_argument('--reason', dest='reason', required=True,
help='Reason to test be skipped')
parser.add_argument('--deployment', dest='deployment', required=False,
default='overcloud',
choices=['overcloud', 'undercloud'])
return parser
tempest_skip/list_allowed.py
-87
View File
@@ -1,87 +0,0 @@
#!/usr/bin/python
# Copyright 2020 Red Hat, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
from cliff.lister import Lister
import yaml
DEFAULT_TESTS = ['smoke']
class ListAllowedYaml(Lister):
"""Command to list the tests to be included, using a YAML file, based on
release and/or job"""
log = logging.getLogger(__name__)
def take_action(self, parsed_args):
self.log.debug('Running list_allowed command')
yaml_file = yaml.safe_load(open(parsed_args.file))
tests = self._filter_allowed_tests(parsed_args, yaml_file)
return (('Test name',), ((test,) for
test in tests))
def _filter_allowed_tests(self, parsed_args, yaml_file):
tests = []
if len(yaml_file) > 0:
if parsed_args.group:
everything = yaml_file.get('groups', [])
group_tests = list(filter(
lambda x:
x.get('name', '') == parsed_args.group, everything))
group_tests = list(filter(
lambda x:
any(r in ['all', parsed_args.release] for r in x.get(
'releases', [])),
group_tests))
if len(group_tests) > 0:
tests = group_tests
if parsed_args.job:
everything = yaml_file.get('jobs', [])
job_tests = list(filter(
lambda x: x.get('name', '') == parsed_args.job,
everything))
job_tests = list(filter(
lambda x:
any(r in ['all', parsed_args.release] for r in x.get(
'releases', [])),
job_tests))
if len(job_tests) > 0:
tests = job_tests
if len(tests) > 0:
return tests[0].get('tests')
return []
def get_parser(self, prog_name):
parser = super(ListAllowedYaml, self).get_parser(prog_name)
parser.add_argument('--file', dest='file',
required=True,
help='List the tests to be included in the '
'YAML file'
)
parser.add_argument('--job', dest='job',
help='List the tests to be included in the '
'given job', required=True)
parser.add_argument('--group', dest='group',
help='List the tests to be included in the '
'given group', default='')
parser.add_argument('--release', dest='release',
help='Filter the tests per release',
default='master')
return parser
tempest_skip/list_yaml.py
-91
View File
@@ -1,91 +0,0 @@
#!/usr/bin/python
# Copyright 2020 Red Hat, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
from cliff.lister import Lister
import yaml
DEFAULT_TESTS = ['smoke']
class ListSkippedYaml(Lister):
"""Command to list the tests to be skipped, using a YAML file, based on
release and/or job"""
log = logging.getLogger(__name__)
def take_action(self, parsed_args):
self.log.debug('Running list_yaml command')
yaml_file = yaml.safe_load(open(parsed_args.file))
tests = []
tests = self._filter_skipped_tests(parsed_args, yaml_file)
return (('Test name',), ((test['test'],) for test in tests))
def _filter_skipped_tests(self, parsed_args, yaml_file):
if len(yaml_file) > 0:
tests = yaml_file.get('known_failures', [])
if parsed_args.job:
tests = [test for test in tests
if (not test.get('jobs', []) or (
parsed_args.job in test.get('jobs')))]
if parsed_args.release:
tests = [test for test in tests
if [release for release in test.get('releases', [])
if release['name'] == parsed_args.release]]
if parsed_args.installer:
tests = [test for test in tests
if [release for release in test.get('releases', [])
if parsed_args.installer in
release.get('installers', ['tripleo', 'osp'])]]
if parsed_args.deployment:
tests = [test for test in tests
if (not test.get('deployment', []) or (
parsed_args.deployment in
test.get('deployment')))]
return tests
return []
def get_parser(self, prog_name):
parser = super(ListSkippedYaml, self).get_parser(prog_name)
parser.add_argument('--file', dest='file',
required=True,
help='List the tests to be skipped in the '
'YAML file'
)
parser.add_argument('--job', dest='job',
help='List the tests to be skipped in the '
'given job'
)
parser.add_argument('--release', dest='release',
help='List the tests to be skipped in the '
'given release'
)
parser.add_argument('--deployment', dest='deployment',
help='List the tests to be skipped in the '
'given deployment')
parser.add_argument('--installer', dest='installer',
default='tripleo', help='Tests to be skipped for '
'a particular installer. Use '
'tripleo for upstream, and osp'
' for downstream')
return parser
tempest_skip/main.py
-51
View File
@@ -1,51 +0,0 @@
#!/usr/bin/python
# Copyright 2020 Red Hat, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import sys
from cliff.app import App
from cliff.commandmanager import CommandManager
class TempestSkip(App):
def __init__(self):
super(TempestSkip, self).__init__(
description='Tempest skiplist tool',
version='0.1',
command_manager=CommandManager('tempest_skip.cm'),
deferred_help=True
)
def initialize_app(self, argv):
self.LOG.debug('Initializing tempest skiplist tool')
def prepare_to_run_command(self, cmd):
self.LOG.debug('prepare_to_run_command %s', cmd.__class__.__name__)
def clean_up(self, cmd, result, err):
self.LOG.debug('clean_up %s', cmd.__class__.__name__)
if err:
self.LOG.debug('Error: %s', err)
def main(argv=sys.argv[1:]):
tempest_skip = TempestSkip()
return tempest_skip.run(argv)
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
tempest_skip/tests/__init__.py
View File
tempest_skip/tests/base.py
-32
View File
@@ -1,32 +0,0 @@
# Copyright 2020 Red Hat, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import os
import tempfile
from oslotest import base
class TestCase(base.BaseTestCase):
def write_yaml_file(self, file_content):
fd, path = tempfile.mkstemp()
self.addCleanup(os.remove, path)
yaml_file = os.fdopen(fd, 'w')
yaml_file.write(file_content)
yaml_file.close()
return path
tempest_skip/tests/test_add_test.py
-107
View File
@@ -1,107 +0,0 @@
# Copyright 2020 Red Hat, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import mock
from tempest_skip.add_test import AddTest
from tempest_skip.tests import base
import yaml
class TestAddFile(base.TestCase):
def setUp(self):
super(TestAddFile, self).setUp()
self.list_file = """
known_failures:
- test: 'tempest_skip.tests.test_list_yaml'
deployment:
- 'overcloud'
releases:
- name: 'master'
reason: 'It can be removed when bug for OVN is repaired'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1832166'
jobs: []
- test: 'tempest_skip.tests.test_list_yaml_2'
deployment:
- 'overcloud'
releases:
- name: 'master'
reason: 'This test was enabled recently on ovn'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1832166'
jobs: []
- test: 'tempest_skip.tests.test_list_yaml_3'
deployment:
- 'overcloud'
- 'undercloud'
releases:
- name: 'train'
reason: 'Test failing on train release'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1832166'
jobs:
- 'job1'
"""
self.path = self.write_yaml_file(self.list_file)
self.cmd = AddTest(__name__, [])
self.parser = self.cmd.get_parser(__name__)
self.parser.file = self.path
self.parser.job = None
self.parser.release = 'master'
self.parser.lp = 'https://launchpad.net/bug/12345'
self.parser.bz = None
self.parser.reason = 'A good reason'
self.parser.deployment = 'overcloud'
self.tests_list = ['tempest_skip.tests.test1',
'tempest_skip.tests.test2',
'tempest_skip.tests.test3']
@mock.patch('inquirer.Checkbox')
@mock.patch('inquirer.prompt')
@mock.patch('tempest_skip.add_test.AddTest._get_tests_list')
def test_add_test_new_test(self, tests_list, prompt_mock, checkbox_mock):
self.parser.test = 'tempest_skip.tests'
tests_list.return_value = self.tests_list
prompt_mock.return_value = {'list_tests': ['tempest_skip.tests.test3']}
self.cmd.take_action(self.parser)
checkbox_mock.assert_called_once()
prompt_mock.assert_called_once()
yaml_file = yaml.safe_load(open(self.path))
# There are 3 tests in the skip file, adding one more test, should be 4
self.assertEqual(4, len(yaml_file['known_failures']))
@mock.patch('tempest_skip.add_test.AddTest._get_tests_list')
def test_add_test_already_exist(self, tests_list):
self.parser.test = 'tempest_skip.tests.test_list_yaml'
tests_list.return_value = self.tests_list
self.cmd.take_action(self.parser)
yaml_file = yaml.safe_load(open(self.path))
self.assertEqual(3, len(yaml_file['known_failures']))
@mock.patch('inquirer.Checkbox')
@mock.patch('inquirer.prompt')
@mock.patch('tempest_skip.add_test.AddTest._get_tests_list')
def test_add_test_no_prompt(self, tests_list, prompt_mock, checkbox_mock):
self.parser.test = 'tempest_skip.tests.test1'
tests_list.return_value = self.tests_list
prompt_mock.return_value = {'list_tests': ['tempest_skip.tests.test3']}
self.cmd.take_action(self.parser)
checkbox_mock.assert_not_called()
prompt_mock.assert_not_called()
yaml_file = yaml.safe_load(open(self.path))
# There are 3 tests in the skip file, adding one more test, should be 4
self.assertEqual(4, len(yaml_file['known_failures']))
tempest_skip/tests/test_list_allowed.py
-161
View File
@@ -1,161 +0,0 @@
# Copyright 2020 Red Hat, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from tempest_skip.tests import base
from tempest_skip.list_allowed import ListAllowedYaml
class TestListYamlAllowed(base.TestCase):
def setUp(self):
super(TestListYamlAllowed, self).setUp()
self.list_file = """
groups:
- name: group1
tests:
- test_group_1
- test_group_2
releases:
- master
- name: group2
tests:
- test_group_2_1
- test_group_2_2
releases:
- all
jobs:
- name: job1
tests:
- test1
- test2
- test3
releases:
- master
- another
- name: job2
tests:
- test4
- test5
releases:
- master
- name: job3
tests:
- test6
- test7
releases:
- all
"""
self.path = self.write_yaml_file(self.list_file)
self.cmd = ListAllowedYaml(__name__, [])
self.parser = self.cmd.get_parser(__name__)
self.parser.file = self.path
self.parser.release = None
self.parser.deployment = None
self.parser.job = None
self.parser.group = None
self.parser.release = 'master'
def test_list_allowed_with_job(self):
self.parser.job = 'job1'
cmd_result = self.cmd.take_action(self.parser)
expected = [('test1',), ('test2',), ('test3',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(expected, list_tests)
def test_list_allowed_without_job(self):
cmd_result = self.cmd.take_action(self.parser)
list_tests = [test for test in cmd_result[1]]
self.assertEqual([], list_tests)
def test_list_allowed_with_no_job(self):
self.parser.job = 'no-exist'
cmd_result = self.cmd.take_action(self.parser)
list_tests = [test for test in cmd_result[1]]
self.assertEqual([], list_tests)
def test_list_allowed_with_group(self):
self.parser.group = 'group1'
cmd_result = self.cmd.take_action(self.parser)
expected = [('test_group_1',), ('test_group_2',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(expected, list_tests)
def test_list_allowed_with_existing_group_and_job(self):
self.parser.group = 'group1'
self.parser.job = 'job2'
cmd_result = self.cmd.take_action(self.parser)
expected = [('test4',), ('test5',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(expected, list_tests)
def test_list_allowed_with_existing_group_and_no_existing_job(self):
self.parser.group = 'group1'
self.parser.job = 'jobnoexist'
cmd_result = self.cmd.take_action(self.parser)
expected = [('test_group_1',), ('test_group_2',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(expected, list_tests)
def test_list_allowed_with_no_existing_group_and_no_existing_job(self):
self.parser.group = 'groupnoexist'
self.parser.job = 'jobnoexist'
cmd_result = self.cmd.take_action(self.parser)
list_tests = [test for test in cmd_result[1]]
self.assertEqual([], list_tests)
def test_list_allowed_with_no_existing_group_and_existing_job(self):
self.parser.group = 'groupnoexist'
self.parser.job = 'job2'
cmd_result = self.cmd.take_action(self.parser)
list_tests = [test for test in cmd_result[1]]
expected = [('test4',), ('test5',)]
self.assertEqual(expected, list_tests)
def test_list_allowed_with_no_group(self):
self.parser.group = 'no-exist'
cmd_result = self.cmd.take_action(self.parser)
list_tests = [test for test in cmd_result[1]]
self.assertEqual([], list_tests)
def test_list_allowed_job_with_release_all(self):
self.parser.job = 'job3'
self.parser.release = 'whatever'
cmd_result = self.cmd.take_action(self.parser)
expected = [('test6',), ('test7',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(expected, list_tests)
def test_list_allowed_group_with_release_all(self):
self.parser.group = 'group2'
self.parser.release = 'whatever'
cmd_result = self.cmd.take_action(self.parser)
expected = [('test_group_2_1',), ('test_group_2_2',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(expected, list_tests)
def test_list_allowed_with_no_release(self):
self.parser.release = 'no-exist'
cmd_result = self.cmd.take_action(self.parser)
list_tests = [test for test in cmd_result[1]]
self.assertEqual([], list_tests)
def test_list_allowed_with_no_release_but_job(self):
self.parser.release = 'no-exist'
self.parser.job = 'job2'
cmd_result = self.cmd.take_action(self.parser)
list_tests = [test for test in cmd_result[1]]
self.assertEqual([], list_tests)
tempest_skip/tests/test_list_yaml.py
-205
View File
@@ -1,205 +0,0 @@
# Copyright 2020 Red Hat, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from tempest_skip.tests import base
from tempest_skip.list_yaml import ListSkippedYaml
class TestListSkippedYaml(base.TestCase):
def setUp(self):
super(TestListSkippedYaml, self).setUp()
self.list_file = """
known_failures:
- test: 'tempest_skip.tests.test_list_yaml'
deployment:
- 'overcloud'
releases:
- name: 'master'
reason: 'It can be removed when bug for OVN is repaired'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1832166'
jobs: []
- test: 'tempest_skip.tests.test_list_yaml_2'
deployment:
- 'overcloud'
releases:
- name: 'master'
reason: 'This test was enabled recently on ovn'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1832166'
jobs: []
- test: 'tempest_skip.tests.test_list_yaml_3'
deployment:
- 'overcloud'
- 'undercloud'
releases:
- name: 'train'
reason: 'Test failing on train release'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1832166'
jobs:
- 'job1'
- test: 'tempest_skip.tests.test_list_yaml_4'
deployment:
- 'overcloud'
releases:
- name: 'train'
reason: 'Test failing on train release'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1832166'
installers:
- 'osp'
jobs:
- 'job3'
- test: 'tempest_skip.tests.test_list_yaml_5'
deployment:
- 'overcloud'
releases:
- name: 'train'
reason: 'Test failing on train release'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1832166'
installers:
- 'osp'
- 'tripleo'
- name: 'wallaby'
reason: 'Test failing on train release'
lp: 'https://bugs.launchpad.net/tripleo/+bug/1832166'
jobs: []
"""
self.path = self.write_yaml_file(self.list_file)
self.cmd = ListSkippedYaml(__name__, [])
self.parser = self.cmd.get_parser(__name__)
self.parser.file = self.path
self.parser.release = None
self.parser.deployment = None
self.parser.job = None
self.parser.installer = 'tripleo'
def test_list_yaml(self):
cmd_result = self.cmd.take_action(self.parser)
exptected = [('tempest_skip.tests.test_list_yaml',),
('tempest_skip.tests.test_list_yaml_2',),
('tempest_skip.tests.test_list_yaml_3',),
('tempest_skip.tests.test_list_yaml_5',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(exptected, list_tests)
def test_list_yaml_with_release(self):
self.parser.release = 'train'
self.parser.job = None
cmd_result = self.cmd.take_action(self.parser)
exptected = [('tempest_skip.tests.test_list_yaml_3',),
('tempest_skip.tests.test_list_yaml_5',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(exptected, list_tests)
def test_list_yaml_with_deployment(self):
self.parser.deployment = 'undercloud'
cmd_result = self.cmd.take_action(self.parser)
expected = [('tempest_skip.tests.test_list_yaml_3',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(expected, list_tests)
self.parser.deployment = 'notfound'
cmd_result = self.cmd.take_action(self.parser)
list_tests = [test for test in cmd_result[1]]
self.assertEqual([], list_tests)
self.parser.deployment = 'undercloud'
self.parser.release = 'master'
cmd_result = self.cmd.take_action(self.parser)
list_tests = [test for test in cmd_result[1]]
self.assertEqual([], list_tests)
self.parser.deployment = 'undercloud'
self.parser.release = 'train'
cmd_result = self.cmd.take_action(self.parser)
expected = [('tempest_skip.tests.test_list_yaml_3',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(expected, list_tests)
def test_list_yaml_with_installer(self):
self.parser.installer = 'osp'
self.parser.release = 'master'
cmd_result = self.cmd.take_action(self.parser)
expected = [('tempest_skip.tests.test_list_yaml',),
('tempest_skip.tests.test_list_yaml_2',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(expected, list_tests)
def test_list_yaml_with_installer_and_deployment(self):
self.parser.installer = 'osp'
self.parser.deployment = 'overcloud'
cmd_result = self.cmd.take_action(self.parser)
expected = [('tempest_skip.tests.test_list_yaml',),
('tempest_skip.tests.test_list_yaml_2',),
('tempest_skip.tests.test_list_yaml_3',),
('tempest_skip.tests.test_list_yaml_4',),
('tempest_skip.tests.test_list_yaml_5',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(expected, list_tests)
def test_list_yaml_with_installer_and_invalid_deployment(self):
self.parser.installer = 'osp'
self.parser.deployment = 'underclouds'
cmd_result = self.cmd.take_action(self.parser)
expected = []
list_tests = [test for test in cmd_result[1]]
self.assertEqual(expected, list_tests)
def test_list_yaml_with_release_not_found(self):
self.parser.release = 'not-found'
self.parser.job = None
cmd_result = self.cmd.take_action(self.parser)
exptected = []
list_tests = [test for test in cmd_result[1]]
self.assertEqual(exptected, list_tests)
def test_list_yaml_with_job(self):
self.parser.job = 'job1'
cmd_result = self.cmd.take_action(self.parser)
exptected = [('tempest_skip.tests.test_list_yaml',),
('tempest_skip.tests.test_list_yaml_2',),
('tempest_skip.tests.test_list_yaml_3',),
('tempest_skip.tests.test_list_yaml_5',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(exptected, list_tests)
def test_list_yaml_with_job_not_found(self):
self.parser.job = 'job2'
cmd_result = self.cmd.take_action(self.parser)
exptected = [('tempest_skip.tests.test_list_yaml',),
('tempest_skip.tests.test_list_yaml_2',),
('tempest_skip.tests.test_list_yaml_5',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(exptected, list_tests)
def test_list_yaml_with_release_and_job(self):
self.parser.job = 'job1'
self.parser.release = 'train'
cmd_result = self.cmd.take_action(self.parser)
exptected = [('tempest_skip.tests.test_list_yaml_3',),
('tempest_skip.tests.test_list_yaml_5',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(exptected, list_tests)
def test_list_yaml_with_release_and_job_not_found(self):
self.parser.job = 'job2'
self.parser.release = 'not-found'
cmd_result = self.cmd.take_action(self.parser)
exptected = []
list_tests = [test for test in cmd_result[1]]
self.assertEqual(exptected, list_tests)
tempest_skip/tests/test_validate.py
-165
View File
@@ -1,165 +0,0 @@
# Copyright 2020 Red Hat, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import os
import tempfile
from tempest_skip.tests import base
from tempest_skip.validate import Validate
from voluptuous.error import MultipleInvalid
class TestValidate(base.TestCase):
def setUp(self):
super(TestValidate, self).setUp()
self.cmd = Validate(__name__, [])
self.parser = self.cmd.get_parser(__name__)
self.parser.skipped = False
self.parser.allowed = False
class TestValidateAllowed(TestValidate):
def setUp(self):
super(TestValidateAllowed, self).setUp()
def test_validate_passes(self):
fd, path = tempfile.mkstemp()
self.addCleanup(os.remove, path)
valid_yaml = """
jobs:
- name: 'job1'
tests:
- test1
- test2
releases:
- master
- name: 'job2'
tests:
- another.test
- a.different.test
releases:
- master
"""
yaml_file = os.fdopen(fd, 'w')
yaml_file.write(valid_yaml)
yaml_file.close()
self.parser.allowed = True
self.parser.filename = path
cmd_result = self.cmd.take_action(self.parser)
self.assertEqual(cmd_result, None)
def test_validate_fails(self):
fd, path = tempfile.mkstemp()
self.addCleanup(os.remove, path)
valid_yaml = """
jobs:
- name: 'job1.will.fail'
test:
- test1
- test2
- name: 'job2'
tests:
- another.test
- a.different.test
"""
yaml_file = os.fdopen(fd, 'w')
yaml_file.write(valid_yaml)
yaml_file.close()
self.parser.allowed = True
self.parser.filename = path
self.assertRaises(MultipleInvalid, self.cmd.take_action, self.parser)
class TestValidateSkipped(TestValidate):
def setUp(self):
super(TestValidateSkipped, self).setUp()
def test_validate_passes(self):
fd, path = tempfile.mkstemp()
self.addCleanup(os.remove, path)
valid_yaml = """
known_failures:
- test: 'tempest_skip.tests.test_validate'
bz: 'https://bugzilla.redhat.com/1'
lp: 'https://launchpad.net/bugs/1'
deployment:
- 'undercloud'
jobs:
- openstack-tempest-skip-job1
- openstack-tempest-skip-job2
reason: 'This test is failing'
releases:
- name: 'master'
lp: 'https://launchpad.net/bugs/1'
reason: 'Test with launchpad'
installers:
- 'tripleo'
- 'osp'
- name: 'train'
bz: 'https://bugzilla.redhat.com/1'
reason: 'Test with bugzilla'
- name: 'ussuri'
reason: 'Test without launchpad or bugzilla'
lp: 'https://launchpad.net/bugs/1'
"""
yaml_file = os.fdopen(fd, 'w')
yaml_file.write(valid_yaml)
yaml_file.close()
self.parser.skipped = True
self.parser.filename = path
cmd_result = self.cmd.take_action(self.parser)
self.assertEqual(cmd_result, None)
def test_validate_fails(self):
fd, path = tempfile.mkstemp()
self.addCleanup(os.remove, path)
valid_yaml = """
known_failures:
- test: 'tempest_skip.tests.test_validate'
bz: 'https://bugzilla.redhat.com/1'
lp: 'https://launchpad.net/bugs/1'
deployment:
- 'undercloud'
jobs:
- openstack-tempest-skip-job1:
option: '1'
- openstack-tempest-skip-job2
reason: 'This test is failing'
releases:
- name: 'master'
lp: 'https://launchpad.net/bugs/1'
reason: 'Test with launchpad'
- name: 'train'
bz: 'https://bugzilla.redhat.com/1'
reason: 'Test with bugzilla'
- name: 'ussuri'
reason: 'Test without launchpad or bugzilla'
"""
yaml_file = os.fdopen(fd, 'w')
yaml_file.write(valid_yaml)
yaml_file.close()
self.parser.skipped = True
self.parser.filename = path
self.assertRaises(MultipleInvalid, self.cmd.take_action, self.parser)
tempest_skip/validate.py
-81
View File
@@ -1,81 +0,0 @@
#!/usr/bin/python
# Copyright 2020 Red Hat, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
from cliff.command import Command
import voluptuous as v
import yaml
class Validate(Command):
"Command to validate the yaml file parsed to tempest skiplist"
log = logging.getLogger(__name__)
validate_skipped = v.Schema({
'known_failures': [{
v.Required('test'): str,
v.Optional('bz'): v.Url(),
v.Optional('lp'): v.Url(),
v.Required('deployment'): [v.Any('undercloud', 'overcloud')],
v.Optional('reason'): str,
v.Required('releases'): [
v.Schema({
v.Required('name'): str,
v.Required(v.SomeOf(
validators=[v.Any('lp', 'bz')], min_valid=1)): v.Url(),
v.Required('reason'): str,
v.Optional('installers'): [str]
})
],
v.Optional('jobs'): [str]
}]
})
validate_allowed = v.Schema({
'groups': [{
v.Required('name'): str,
v.Required('tests'): [str],
v.Required('releases'): [str]
}],
'jobs': [{
v.Required('name'): str,
v.Required('tests'): [str],
v.Required('releases'): [str]
}]
})
def take_action(self, parsed_args):
self.log.debug('Running validate command')
yaml_file = yaml.safe_load(open(parsed_args.filename))
if parsed_args.skipped:
self.validate_skipped(yaml_file)
if parsed_args.allowed:
self.validate_allowed(yaml_file)
def get_parser(self, prog_name):
parser = super(Validate, self).get_parser(prog_name)
parser.add_argument('--file', dest='filename',
help='Path to the YAML file to be validate',
required=True)
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument('--skipped', dest='skipped', action='store_true',
default=False, help='Validate skipped schema')
group.add_argument('--allowed', dest='allowed', action='store_true',
default=False, help='Validate allowed schema')
return parser
test-requirements.txt
-8
View File
@@ -1,8 +0,0 @@
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
flake8==3.8.3 # MIT
mock
oslotest>=3.2.0 # Apache-2.0
stestr>=1.1.0 # Apache-2.0
tox.ini
-70
View File
@@ -1,70 +0,0 @@
[tox]
minversion = 3.1.1
envlist = py38,py37,py36,pypy,pep8
skipsdist = True
# this allows tox to infer the base python from the environment name
# and override any basepython configured in this file
ignore_basepython_conflict = true
[testenv]
basepython = python3
usedevelop = True
install_command = pip install -c{env:UPPER_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/master} {opts} {packages}
setenv =
VIRTUAL_ENV={envdir}
PYTHONWARNINGS=default::DeprecationWarning
allowlist_externals = *
deps =
-r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands =
find . -type f -name "*.pyc" -delete
stestr --test-path ./tempest_skip/tests run {posargs}
[testenv:pep8]
commands = flake8 {posargs}
[testenv:venv]
commands = {posargs}
[testenv:cover]
setenv =
{[testenv]setenv}
PYTHON=coverage run --source tempest_skip --parallel-mode
commands =
coverage erase
find . -type f -name "*.pyc" -delete
stestr --test-path ./tempest_skip/tests run {posargs}
coverage combine
coverage html -d cover
coverage xml -o cover/coverage.xml
coverage report
[testenv:docs]
deps = -r{toxinidir}/doc/requirements.txt
commands = sphinx-build -b html doc/source doc/build/html
[testenv:pdf-docs]
deps = {[testenv:docs]deps}
allowlist_externals =
make
commands =
sphinx-build -W -b latex doc/source doc/build/pdf
make -C doc/build/pdf
[testenv:releasenotes]
deps = -r{toxinidir}/doc/requirements.txt
commands =
sphinx-build -a -E -W -d releasenotes/build/doctrees -b html releasenotes/source releasenotes/build/html
[testenv:debug]
commands = oslo_debug_helper {posargs}
[flake8]
# E123, E125 skipped as they are invalid PEP-8.
show-source = True
ignore = E123,E125
enable-extensions = H106,H230,H904
builtins = _
exclude=.venv,.git,.tox,dist,doc,*lib/python*,*egg,build