openstack/tripleo-quickstart-extras
Code Issues Proposed changes
9e4495770c
tripleo-quickstart-extras/roles/validate-tempest
History
John Trowbridge 9e4495770c Allow support for subunit2junitxml < 1.2 
Versions of subunit2junitxml less than 1.2 do not support the
syntax we have in the validate tempest role. Namely:

  subunit2junitxml foo bar

Instead they need the destination file specified with '--output-to'

  subunit2junitxml foo --output-to=bar

As far as I can tell, the newer versions of subunit still support
the '--output-to' syntax. As such, we should use the syntax that
supports all versions.

Closes-Bug: 1720859

Change-Id: I3b52db71bb1b1b1f0faf241038348e6dedca943f
2017-11-20 15:24:18 -05:00
..
defaults Allow removing of options from tempest conf 2017-10-08 12:04:21 +00:00
files Tempest notifications for Storage components 2017-10-20 14:55:29 +02:00
meta Integrate undercloud_user var into oooq-extras 2017-01-18 09:32:16 -05:00
tasks Allow support for subunit2junitxml < 1.2 2017-11-20 15:24:18 -05:00
templates Changing python-tempestconf url 2017-11-13 10:30:40 +01:00
vars Add test_mtu_sized_frames to the skip list during investigation 2017-11-19 22:00:39 +00:00
README.md Fix role name to validate-tempest in role files 2017-11-12 13:51:36 +00:00

README.md

validate-tempest

Run tempest tests on undercloud or overcloud.

Requirements

This Ansible role allows to run tempest tests against installed undercloud or overcloud. It can use both upstream sources with virtualenv and RDO packages for tempest.

Role Variables

  • tempest_undercloud: false/true - whether to run tempest on installed undercloud
  • tempest_overcloud: false/true - whether to run tempest on installed overcloud
  • tempest_format: venv/packages - Which tempest installation to use - either install python virtual environment with installed there python modules from requirements file, or to use installed with RDO RPM packages
  • tempest_log_file - name of log file for tempest run
  • test_regex - tests regular expression for testr run, i.e. smoke or tempest.api.object_storage|keystone_tempest_plugin.
  • run_tempest: false/true - to run tempest or not
  • tempest_config: false/true - whether to prepare the script which configures and runs tempest or not
  • tempest_whitelist: list - list of tests you want to be executed. set skip_file_src to empty if you don't want to run with blacklist_file option. Default value is an empty list
  • skip_file_src: path to skip tests file, set it empty if running without skip list: skip_file_src=''
  • tempest_workers: int - how many parallel workers to run (default is number of cores)
  • tempest_until_failure: false/true - default is false, repeat the run again and again until failure occurs
  • tempest_exit_on_failure: true/false - whether to exit from role with tempest exit code (default: true)
  • tempestmail_config: config.yaml - name of config file for tempestmail script
  • tempest_track_resources: true/false - whether to save the state of resources after tempest run (default: true)
  • check_tempest_bugs: true/false - Will check every bugzilla and launchpad bug in the yaml skip file
  • tempest_plugins: list - List of openstack services tempest plugins to be installed
  • tempest_extra_config: dict - A dict of tempest configuration which needs to be overridden in tempest.conf, It should be like section.key: value.
  • tempest_conf_removal: dict - A dict of tempest configuration which will be removed from tempest.conf file. Format: section.key: value
  • public_physical_network: The name of the border physical network (default: datacentre).

Skip tests file

Tempest test can be skipped if they are in a skip file, using the whole test name or a regular expression. Since the intention is to use the list of skip tests in other parts of quickstart-extras, it's easier to have this list of skip tests in yaml format where you can set the skip test, reason and launchpad or bugzilla. This make easier later to check whether the test still valid or not to be skipped. Skip files are under roles/validate-tempest/vars/tempest_skip_VERSION.yaml. In these files you can add a regex, together with a reason and a bugzilla or launchpad bug related to that particular test. Launchpad and/or bugzilla re optional, however you must specify a reason why you are skipping the test.

Example of skip file:

---
known_failures:
  - test: 'tempest.scenario.test_volume_boot_pattern'
    reason: 'rdo-manager tempest smoke test failing on "floating ip pool not found"'
    bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1272289'
  - test: 'tempest.api.volume.test_volumes_get'
    reason: 'rdo-manager tempest smoke test failing on "floating ip pool not found"'
  - test: 'tempest.api.identity.*v3'
    reason: 'Tempest Identify tests failed on V3 api on OSP-D puddle'
    bz: 'https://bugzilla.redhat.com/show_bug.cgi?id=1266947'
  - test: 'tempest.api.image.v2.test_images_metadefs_namespace_properties.MetadataNamespacePropertiesTest.test_basic_meta_def_namespace_property'
    reason: 'glance is not calling glance-manage db_load_metadefs'
    lp: 'https://bugs.launchpad.net/tripleo/+bug/1664995'

Real time bug check

If check_tempest_bugs is set to true, a script will be called and will check in real time, all tests who has a bugzilla or a launchpad bug. This will generate a new skip file, removing all the bugs that were already closed but wasn't updated in the yaml skip file yet.

Dependencies

No dependencies

Example Playbook

---
- name:  Run tempest
  hosts: undercloud
  gather_facts: no
  roles:
    - validate-tempest

License

Apache 2.0

Author Information

RDO-CI Team