Convert tempest-scenario-all job to tempest-slow

As per discussion on ML[1], we do not have any job which runs only
slow tests.

This commit provide a separate job which run only slow tests which can
be API or scenario tests. Further this job can be used cross projects
like nova, cinder, neutron.

Why we need separate job for slow tests:
 - slow tests create gate issues on job timeout which we observed\
   during Rocky cycle[2]. Not to effect gate let's start them executing
   on separate job.
Why to convert the tempest-scenario-all job
 - tempest-scenario-all job was created to run the slow tests along with
   other scenario tests. But slow tests can be API tests[3] and mixing them
   with other scenario or API tests is not good idea.
 - All scenario tests excluding slow tests are run as part of tempest-full
   job so let's not run scenario tests twice on gate.

[1] http://lists.openstack.org/pipermail/openstack-dev/2018-July/132527.html
[2] https://bugs.launchpad.net/tempest/+bug/1783405
[3] https://ethercalc.openstack.org/dorupfz6s9qt

Change-Id: I5b1d3041c0140bea11705ad287f1472e74aacb5c
This commit is contained in:
ghanshyam 2018-08-01 08:53:00 +00:00 committed by Ghanshyam Mann
parent 1969a69441
commit 10326c7dc2
2 changed files with 17 additions and 10 deletions

View File

@ -143,25 +143,24 @@
voting: false
- job:
name: tempest-scenario-all
name: tempest-slow
parent: tempest-multinode-full
branches:
- master
description: |
This multinode integration job will run all scenario tests including slow
tests with lvm multibackend setup. This job will not run any API tests.
This multinode integration job will run all the tests tagged as slow.
It enables the lvm multibackend setup to cover few scenario tests.
This job will run only slow tests(API or Scenario) serially.
Former names for this job were:
* legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend
* tempest-scenario-multinode-lvm-multibackend
timeout: 10800
vars:
# 'all' is used for applying the custom regex below.
tox_envlist: all
tox_envlist: slow-serial
devstack_localrc:
CINDER_ENABLED_BACKENDS: lvm:lvmdriver-1,lvm:lvmdriver-2
tempest_concurrency: 2
tempest_test_regex: (^tempest\.(scenario))
- job:
name: tempest-full-queens
@ -353,7 +352,7 @@
- ^tempest/hacking/.*$
- ^tempest/tests/.*$
- tempest-tox-plugin-sanity-check
- tempest-scenario-all:
- tempest-slow:
irrelevant-files:
- ^(test-|)requirements.txt$
- ^.*\.rst$
@ -363,7 +362,6 @@
- ^setup.cfg$
- ^tempest/hacking/.*$
- ^tempest/tests/.*$
- ^tempest/api/.*$
- nova-cells-v1:
irrelevant-files:
- ^(test-|)requirements.txt$
@ -397,7 +395,7 @@
- ^setup.cfg$
- ^tempest/hacking/.*$
- ^tempest/tests/.*$
- tempest-scenario-all:
- tempest-slow:
irrelevant-files:
- ^(test-|)requirements.txt$
- ^.*\.rst$
@ -407,7 +405,6 @@
- ^setup.cfg$
- ^tempest/hacking/.*$
- ^tempest/tests/.*$
- ^tempest/api/.*$
experimental:
jobs:
- tempest-cinder-v2-api:

10
tox.ini
View File

@ -135,6 +135,16 @@ commands =
find . -type f -name "*.pyc" -delete
tempest run --serial --regex '\[.*\bsmoke\b.*\]' {posargs}
[testenv:slow-serial]
envdir = .tox/tempest
sitepackages = {[tempestenv]sitepackages}
setenv = {[tempestenv]setenv}
deps = {[tempestenv]deps}
# The regex below is used to select the slow tagged tests to run serially:
commands =
find . -type f -name "*.pyc" -delete
tempest run --serial --regex '\[.*\bslow\b.*\]' {posargs}
[testenv:venv]
deps =
-c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt}