openstack
/
openstack-tempest-skiplist
Code Issues Proposed changes

Merge "Allowed tests command added"

This commit is contained in:
Zuul 2021-04-15 12:02:33 +00:00 committed by Gerrit Code Review
parent 8b451dfc13 f19474e273
commit f292293b06
18 changed files with 521 additions and 56 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
doc/source/index.rst
View File

@ -15,6 +15,7 @@ Content:
yaml/index
validate/index
listyaml/index
listallowed/index
addtest/index
* :ref:`search`

5
doc/source/listallowed/index.rst Normal file
View File

@ -0,0 +1,5 @@
.. toctree::
:maxdepth: 2
:includehidden:
listallowed

39
doc/source/listallowed/listallowed.rst Normal file
View File

@ -0,0 +1,39 @@
==================
List allowed tests
==================
List allowed tests
------------------
You can use :command:`tempest-skip list-allowed` command to list the tests to
be executed with two positional parameters which are in the
expected format::
1. ``--file`` is the positional parameter - list all the tests in the file
2. ``--group`` or ``--job`` - filter the tests for a specific job, or a
specific group.
Job filter
----------
The job filter, is as the name indicate, it checks the yaml file for a job that
matches (must be full match, not partial), and list the tests related to that
specific job::
$ tempets-skip list-allowed --file tempest_allow.yml --job job1
Group filter
------------
The group filter, which have precedence on the ``--job``, will list the tests
for a particular group. This is good when you have several jobs, that run a
specific set of tests. In this case, you don't need to repeat the same set of
tests for several different jobs::
$ tempest-skip list-allowed --file tempest_allow.yml --group default_group
Release filter
--------------
The release filter, which is default to master, filter based on group or job
for an specific release.

18
doc/source/listyaml/listyaml.rst
View File

@ -1,26 +1,26 @@
======================
List the yaml file
List skipped tests
======================
List Yaml
----------
List skipped tests
-------------------
You can use :command:`tempest-skip` list command to list tests in the yaml
You can use :command:`tempest-skip list-skipped` command to list tests in the yaml
file with one positional and two optional parameters which is in the expected
format::
1. ``--file`` is the positional parameter - lists all the tests in the file::
$ tempest-skip list yaml --file tempest_skip.yml
$ tempest-skip list-skipped yaml --file tempest_skip.yml
2. ``--release``, ``--deployment`` and ``--job`` are the optional
parameters - list all the tests within a specific release, deployment
or a specific job::
$ tempest-skip list --file tempest_skip.yml --release train
$ tempest-skip list --file tempest_skip.yml --job job1
$ tempest-skip list --file tempest_skip.yml --release train --job job1
$ tempest-skip list --file tempest_skip.yml --deployment undercloud
$ tempest-skip list-skipped --file tempest_skip.yml --release train
$ tempest-skip list-skipped --file tempest_skip.yml --job job1
$ tempest-skip list-skipped --file tempest_skip.yml --release train --job job1
$ tempest-skip list-skipped --file tempest_skip.yml --deployment undercloud
This will return any tests that match the job, as well as tests that doesn't
have any job configured. This is required when you configure your zuul jobs to

11
doc/source/validate/validate.rst
View File

@ -5,13 +5,16 @@ Validate the yaml file
Validation
----------
You can use :command:`tempest-skip` validate command to validate if the yaml
file is in the expected format::
You can use :command:`tempest-skip validate` command to validate if the yaml
file is in the expected format for the allowed list::
$ tempest-skip validate --file good_file.yaml
$ tempest-skip validate --allowed --file good_file.yaml
or for the skipped list:
$ tempest-skip validate --skipped --file good_file.yaml
This will return nothing if the file is valid, or an error otherwise::
$ tempest-skip validate --file bad_file.yaml
required key not provided @ data['known_failures'][0]['releases'][2]['reason']
required key not provided @ data['known_failures'][0]['releases'][2]['reason']

23
roles/list_allowed/README.rst Normal file
View File

@ -0,0 +1,23 @@
List allowed tempest tests
==========================
This role executes the tempest-skip tool and returns the list of tests to be
executed.
**Role Variables**
.. zuul:rolevar:: list_allowed_yaml_file
:type: string
Path to the yaml file containing the skipped tests
.. zuul:rolevar:: list_allowed_job
:type: string
Job name to be used in the filter if required
.. zuul:rolevar:: list_allowed_group
:type: string
Group to be used in the filter. It has more precedence than
list_allowed_job.

121
roles/list_allowed/library/list_allowed.py Normal file
View File

@ -0,0 +1,121 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from ansible.module_utils.basic import AnsibleModule
from tempest_skip.list_allowed import ListAllowedYaml
import sys
ANSIBLE_METADATA = {
'metadata_version': '0.1',
'status': ['preview'],
'supported_by': 'community'
}
DOCUMENTATION = '''
---
module: list_allowed
author:
- "Arx Cruz (@arxcruz)
version_added: '2.9'
short_description:
- Parse filtered tests from tempest
notes: []
requirements:
- tempest-skip
options:
yaml_file:
description:
- Path to a yaml file containing the tests in
openstack-tempest-skiplist format
required: True
type: str
job:
description:
- Name of the job to be used in the filter.
required: False
type: str
group:
description:
- Group name to be used in the filter. It has more precedence than job
required: False
type: str
'''
EXAMPLES = '''
- name: Get list of allowed tests
list_allowed:
yaml_file: /tmp/allowed.yaml
job: tripleo-ci-centos-8-standalone
group: default
'''
RETURN = '''
allowed_tests:
description:
- List of tests filtered
returned: Always
type: list
sample: [
"tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON"
]
'''
def run_module():
module_args = dict(
yaml_file=dict(type='str', required=True),
job=dict(type='str', required=False, default=None),
group=dict(type='str', required=False, default=None)
)
result = dict(
changed=True,
message='',
filtered_tests=[]
)
module = AnsibleModule(
argument_spec=module_args
)
if not module.params['yaml_file']:
module.fail_json(msg="A yaml file must be provided!")
if not module.params['job'] and not module.params['group']:
module.fail_json(msg="You must specify either job or group parameter!")
cmd = ListAllowedYaml(__name__, sys.argv[1:])
parser = cmd.get_parser(__name__)
parser.file = module.params['yaml_file']
parser.group = module.params['release']
parser.job = module.params['job']
tests = cmd.take_action(parser)
allowed_tests = [test[0] for test in tests[1]]
result.update({'allowed_tests': allowed_tests})
module.exit_json(**result)
def main():
run_module()
if __name__ == '__main__':
main()

7
roles/list_allowed/tasks/main.yaml Normal file
View File

@ -0,0 +1,7 @@
---
- name: List allowed tests
list_allowed:
yaml_file: "{{ list_skipped_yaml_file }}"
group: "{{ list_skipped_release | default(omit) }}"
job: "{{ list_skipped_job | default(omit) }}"
register: tempest_allowed_register

4
roles/list_skipped/library/list_skipped.py
View File

@ -14,7 +14,7 @@
# limitations under the License.
from ansible.module_utils.basic import AnsibleModule
from tempest_skip.list_yaml import ListYaml
from tempest_skip.list_yaml import ListSkippedYaml
import sys
@ -106,7 +106,7 @@ def run_module():
if not module.params['yaml_file']:
module.fail_json(msg="A yaml file must be provided!")
cmd = ListYaml(__name__, sys.argv[1:])
cmd = ListSkippedYaml(__name__, sys.argv[1:])
parser = cmd.get_parser(__name__)
parser.file = module.params['yaml_file']
parser.release = module.params['release']

6
roles/validate-tempest/vars/tempest_allow.yml Normal file
View File

@ -0,0 +1,6 @@
groups:
- name: default
tests:
- smoke
releases:
- master

26
roles/validate_yaml/tasks/main.yaml
View File

@ -1,23 +1,19 @@
---
- block:
# TODO(arxcruz) Right now there are yaml files that are not
# a valid tempest-skiplist file, so we are using just one for now
# The task isn't useful until we get rid of the old yaml files
# (see next task).
- name: Find all yaml files to be tested
find:
paths: "{{ tempest_skip_path }}/roles/validate-tempest/vars"
patterns: '*.yml,*.yaml'
register: yaml_files
# TODO(arxcruz) Once we get rid of the old yaml files,
# use yaml_files.files in a loop
- name: Validate tempest-skip yaml files
- name: Validate tempest-skip skipped yaml file
shell: |
set -ex
export PATH=$PATH:/usr/local/sbin:/usr/sbin:$HOME/.local/bin
tempest-skip validate --file roles/validate-tempest/vars/tempest_skip.yml
tempest-skip validate --skipped --file roles/validate-tempest/vars/tempest_skip.yml
args:
chdir: "{{ tempest_skip_path }}"
executable: /bin/bash
- name: Validate tempest-skip allowed yaml file
shell: |
set -ex
export PATH=$PATH:/usr/local/sbin:/usr/sbin:$HOME/.local/bin
tempest-skip validate --allowed --file roles/validate-tempest/vars/tempest_allow.yml
args:
chdir: "{{ tempest_skip_path }}"
executable: /bin/bash
register: validate_output

5
setup.cfg
View File

@ -25,6 +25,7 @@ data_files =
usr/local/share/ansible/roles/install_skiplist = roles/install_skiplist/*
usr/local/share/ansible/roles/validate_yaml = roles/validate_yaml/*
usr/local/share/ansible/roles/list_skipped = roles/list_skipped/*
usr/local/share/ansible/roles/list_allowed = roles/list_allowed*
usr/local/share/ansible/roles/validate-tempest = roles/validate-tempest/*
[entry_points]
@ -32,8 +33,10 @@ console_scripts =
tempest-skip = tempest_skip.main:main
tempest_skip.cm =
validate = tempest_skip.validate:Validate
list = tempest_skip.list_yaml:ListYaml
list = tempest_skip.list_yaml:ListSkippedYaml
list-skipped = tempest_skip.list_yaml:ListSkippedYaml
addtest = tempest_skip.add_test:AddTest
list-allowed = tempest_skip.list_allowed:ListAllowedYaml
[build_sphinx]
source-dir = doc/source

83
tempest_skip/list_allowed.py Normal file
View File

@ -0,0 +1,83 @@
#!/usr/bin/python
# Copyright 2020 Red Hat, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
from cliff.lister import Lister
import yaml
DEFAULT_TESTS = ['smoke']
class ListAllowedYaml(Lister):
"""Command to list the tests to be included, using a YAML file, based on
release and/or job"""
log = logging.getLogger(__name__)
def take_action(self, parsed_args):
self.log.debug('Running list_allowed command')
yaml_file = yaml.safe_load(open(parsed_args.file))
tests = self._filter_allowed_tests(parsed_args, yaml_file)
return (('Test name',), ((test,) for
test in tests))
def _filter_allowed_tests(self, parsed_args, yaml_file):
tests = []
if len(yaml_file) > 0:
if parsed_args.group:
everything = yaml_file.get('groups', [])
tests = list(filter(
lambda x:
x.get('name', '') == parsed_args.group, everything))
tests = list(filter(
lambda x:
parsed_args.release in x.get('releases', []), tests))
if parsed_args.job:
everything = yaml_file.get('jobs', [])
tests = list(filter(
lambda x: x.get('name', '') == parsed_args.job,
everything))
tests = list(filter(
lambda x:
parsed_args.release in x.get('releases', []), tests))
if len(tests) > 0:
return tests[0].get('tests')
return []
def get_parser(self, prog_name):
parser = super(ListAllowedYaml, self).get_parser(prog_name)
parser.add_argument('--file', dest='file',
required=True,
help='List the tests to be included in the '
'YAML file'
)
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument('--job', dest='job',
help='List the tests to be included in the '
'given job'
)
group.add_argument('--group', dest='group',
help='List the tests to be included in the '
'given group'
)
parser.add_argument('--release', dest='release',
help='Filter the tests per release',
default='master')
return parser

16
tempest_skip/list_yaml.py
View File

@ -20,7 +20,10 @@ from cliff.lister import Lister
import yaml
class ListYaml(Lister):
DEFAULT_TESTS = ['smoke']
class ListSkippedYaml(Lister):
"""Command to list the tests to be skipped, using a YAML file, based on
release and/or job"""
@ -30,6 +33,11 @@ class ListYaml(Lister):
self.log.debug('Running list_yaml command')
yaml_file = yaml.safe_load(open(parsed_args.file))
tests = []
tests = self._filter_skipped_tests(parsed_args, yaml_file)
return (('Test name',), ((test['test'],) for test in tests))
def _filter_skipped_tests(self, parsed_args, yaml_file):
if len(yaml_file) > 0:
tests = yaml_file.get('known_failures', [])
@ -48,8 +56,8 @@ class ListYaml(Lister):
if parsed_args.deployment:
self.parsed_deployment = parsed_args.deployment
tests = list(filter(self._filter_deployment, tests))
return (('Test name',), ((test['test'],) for test in tests))
return tests
return []
def _filter_deployment(self, test):
if not test.get('deployment', []):
@ -66,7 +74,7 @@ class ListYaml(Lister):
return False
def get_parser(self, prog_name):
parser = super(ListYaml, self).get_parser(prog_name)
parser = super(ListSkippedYaml, self).get_parser(prog_name)
parser.add_argument('--file', dest='file',
required=True,
help='List the tests to be skipped in the '

102
tempest_skip/tests/test_list_allowed.py Normal file
View File

@ -0,0 +1,102 @@
# Copyright 2020 Red Hat, Inc.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from tempest_skip.tests import base
from tempest_skip.list_allowed import ListAllowedYaml
class TestListYamlAllowed(base.TestCase):
def setUp(self):
super(TestListYamlAllowed, self).setUp()
self.list_file = """
groups:
- name: group1
tests:
- test_group_1
- test_group_2
releases:
- master
jobs:
- name: job1
tests:
- test1
- test2
- test3
releases:
- master
- another
- name: job2
tests:
- test4
- test5
releases:
- master
"""
self.path = self.write_yaml_file(self.list_file)
self.cmd = ListAllowedYaml(__name__, [])
self.parser = self.cmd.get_parser(__name__)
self.parser.file = self.path
self.parser.release = None
self.parser.deployment = None
self.parser.job = None
self.parser.group = None
self.parser.release = 'master'
def test_list_allowed_with_job(self):
self.parser.job = 'job1'
cmd_result = self.cmd.take_action(self.parser)
expected = [('test1',), ('test2',), ('test3',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(expected, list_tests)
def test_list_allowed_without_job(self):
cmd_result = self.cmd.take_action(self.parser)
list_tests = [test for test in cmd_result[1]]
self.assertEqual([], list_tests)
def test_list_allowed_with_no_job(self):
self.parser.job = 'no-exist'
cmd_result = self.cmd.take_action(self.parser)
list_tests = [test for test in cmd_result[1]]
self.assertEqual([], list_tests)
def test_list_allowed_with_group(self):
self.parser.group = 'group1'
cmd_result = self.cmd.take_action(self.parser)
expected = [('test_group_1',), ('test_group_2',)]
list_tests = [test for test in cmd_result[1]]
self.assertEqual(expected, list_tests)
def test_list_allowed_with_no_group(self):
self.parser.group = 'no-exist'
cmd_result = self.cmd.take_action(self.parser)
list_tests = [test for test in cmd_result[1]]
self.assertEqual([], list_tests)
def test_list_allowed_with_no_release(self):
self.parser.release = 'no-exist'
cmd_result = self.cmd.take_action(self.parser)
list_tests = [test for test in cmd_result[1]]
self.assertEqual([], list_tests)
def test_list_allowed_with_no_release_but_job(self):
self.parser.release = 'no-exist'
self.parser.job = 'job2'
cmd_result = self.cmd.take_action(self.parser)
list_tests = [test for test in cmd_result[1]]
self.assertEqual([], list_tests)

21
tempest_skip/tests/test_list_yaml.py
View File

@ -13,26 +13,13 @@
# License for the specific language governing permissions and limitations
# under the License.
from cliff.lister import Lister
from tempest_skip.tests import base
from tempest_skip.list_yaml import ListYaml
from tempest_skip.list_yaml import ListSkippedYaml
class TestListYamlRelease(Lister):
def get_parser(self, prog_name):
parser = super(TestListYamlRelease, self).get_parser(prog_name)
return parser
def take_action(self, parsed_args):
tests = ['tempest_skip.tests.test_list_yaml',
'tempest_skip.tests.test_list_yaml_2']
return (('Test name', ), ((test,) for test in tests))
class TestListYaml(base.TestCase):
class TestListSkippedYaml(base.TestCase):
def setUp(self):
super(TestListYaml, self).setUp()
super(TestListSkippedYaml, self).setUp()
self.list_file = """
known_failures:
- test: 'tempest_skip.tests.test_list_yaml'
@ -64,7 +51,7 @@ class TestListYaml(base.TestCase):
"""
self.path = self.write_yaml_file(self.list_file)
self.cmd = ListYaml(__name__, [])
self.cmd = ListSkippedYaml(__name__, [])
self.parser = self.cmd.get_parser(__name__)
self.parser.file = self.path
self.parser.release = None

64
tempest_skip/tests/test_validate.py
View File

@ -33,6 +33,64 @@ class TestValidate(base.TestCase):
self.assertEqual(p.returncode, expected, msg)
return out, err
class TestValidateAllowed(TestValidate):
def setUp(self):
super(TestValidateAllowed, self).setUp()
def test_validate_passes(self):
fd, path = tempfile.mkstemp()
self.addCleanup(os.remove, path)
valid_yaml = """
jobs:
- name: 'job1'
tests:
- test1
- test2
releases:
- master
- name: 'job2'
tests:
- another.test
- a.different.test
releases:
- master
"""
yaml_file = os.fdopen(fd, 'w')
yaml_file.write(valid_yaml)
yaml_file.close()
self.assertRunExit(['tempest-skip', 'validate',
'--allowed', '--file', path], 0)
def test_validate_fails(self):
fd, path = tempfile.mkstemp()
self.addCleanup(os.remove, path)
valid_yaml = """
jobs:
- name: 'job1.will.fail'
test:
- test1
- test2
- name: 'job2'
tests:
- another.test
- a.different.test
"""
yaml_file = os.fdopen(fd, 'w')
yaml_file.write(valid_yaml)
yaml_file.close()
self.assertRunExit(['tempest-skip', 'validate',
'--allowed', '--file', path], 1)
class TestValidateSkipped(TestValidate):
def setUp(self):
super(TestValidateSkipped, self).setUp()
def test_validate_passes(self):
fd, path = tempfile.mkstemp()
self.addCleanup(os.remove, path)
@ -63,7 +121,8 @@ class TestValidate(base.TestCase):
yaml_file.write(valid_yaml)
yaml_file.close()
self.assertRunExit(['tempest-skip', 'validate', '--file', path], 0)
self.assertRunExit(['tempest-skip', 'validate',
'--skipped', '--file', path], 0)
def test_validate_fails(self):
fd, path = tempfile.mkstemp()
@ -95,4 +154,5 @@ class TestValidate(base.TestCase):
yaml_file.write(valid_yaml)
yaml_file.close()
self.assertRunExit(['tempest-skip', 'validate', '--file', path], 1)
self.assertRunExit(['tempest-skip', 'validate',
'--skipped', '--file', path], 1)

25
tempest_skip/validate.py
View File

@ -26,7 +26,7 @@ class Validate(Command):
log = logging.getLogger(__name__)
validate = v.Schema({
validate_skipped = v.Schema({
'known_failures': [{
v.Required('test'): str,
v.Optional('bz'): v.Url(),
@ -45,10 +45,26 @@ class Validate(Command):
}]
})
validate_allowed = v.Schema({
'groups': [{
v.Required('name'): str,
v.Required('tests'): [str],
v.Required('releases'): [str]
}],
'jobs': [{
v.Required('name'): str,
v.Required('tests'): [str],
v.Required('releases'): [str]
}]
})
def take_action(self, parsed_args):
self.log.debug('Running validate command')
yaml_file = yaml.safe_load(open(parsed_args.filename))
self.validate(yaml_file)
if parsed_args.skipped:
self.validate_skipped(yaml_file)
if parsed_args.allowed:
self.validate_allowed(yaml_file)
def get_parser(self, prog_name):
parser = super(Validate, self).get_parser(prog_name)
@ -56,4 +72,9 @@ class Validate(Command):
help='Path to the YAML file to be validate',
required=True)
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument('--skipped', dest='skipped', action='store_true',
default=False, help='Validate skipped schema')
group.add_argument('--allowed', dest='allowed', action='store_true',
default=False, help='Validate allowed schema')
return parser