Convert tempest-scenario-all job to tempest-slow

As per discussion on ML[1], we do not have any job which runs only
slow tests.

This commit provide a separate job which run only slow tests which can
be API or scenario tests. Further this job can be used cross projects
like nova, cinder, neutron.

Why we need separate job for slow tests:
 - slow tests create gate issues on job timeout which we observed\
   during Rocky cycle[2]. Not to effect gate let's start them executing
   on separate job.
Why to convert the tempest-scenario-all job
 - tempest-scenario-all job was created to run the slow tests along with
   other scenario tests. But slow tests can be API tests[3] and mixing them
   with other scenario or API tests is not good idea.
 - All scenario tests excluding slow tests are run as part of tempest-full
   job so let's not run scenario tests twice on gate.

[1] http://lists.openstack.org/pipermail/openstack-dev/2018-July/132527.html
[2] https://bugs.launchpad.net/tempest/+bug/1783405
[3] https://ethercalc.openstack.org/dorupfz6s9qt

Change-Id: I5b1d3041c0140bea11705ad287f1472e74aacb5c
This commit is contained in:
ghanshyam 2018-08-01 08:53:00 +00:00 committed by Ghanshyam Mann
parent 1969a69441
commit 10326c7dc2
2 changed files with 17 additions and 10 deletions

View File

@ -143,25 +143,24 @@
voting: false voting: false
- job: - job:
name: tempest-scenario-all name: tempest-slow
parent: tempest-multinode-full parent: tempest-multinode-full
branches: branches:
- master - master
description: | description: |
This multinode integration job will run all scenario tests including slow This multinode integration job will run all the tests tagged as slow.
tests with lvm multibackend setup. This job will not run any API tests. It enables the lvm multibackend setup to cover few scenario tests.
This job will run only slow tests(API or Scenario) serially.
Former names for this job were: Former names for this job were:
* legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend * legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend
* tempest-scenario-multinode-lvm-multibackend * tempest-scenario-multinode-lvm-multibackend
timeout: 10800 timeout: 10800
vars: vars:
# 'all' is used for applying the custom regex below. tox_envlist: slow-serial
tox_envlist: all
devstack_localrc: devstack_localrc:
CINDER_ENABLED_BACKENDS: lvm:lvmdriver-1,lvm:lvmdriver-2 CINDER_ENABLED_BACKENDS: lvm:lvmdriver-1,lvm:lvmdriver-2
tempest_concurrency: 2 tempest_concurrency: 2
tempest_test_regex: (^tempest\.(scenario))
- job: - job:
name: tempest-full-queens name: tempest-full-queens
@ -353,7 +352,7 @@
- ^tempest/hacking/.*$ - ^tempest/hacking/.*$
- ^tempest/tests/.*$ - ^tempest/tests/.*$
- tempest-tox-plugin-sanity-check - tempest-tox-plugin-sanity-check
- tempest-scenario-all: - tempest-slow:
irrelevant-files: irrelevant-files:
- ^(test-|)requirements.txt$ - ^(test-|)requirements.txt$
- ^.*\.rst$ - ^.*\.rst$
@ -363,7 +362,6 @@
- ^setup.cfg$ - ^setup.cfg$
- ^tempest/hacking/.*$ - ^tempest/hacking/.*$
- ^tempest/tests/.*$ - ^tempest/tests/.*$
- ^tempest/api/.*$
- nova-cells-v1: - nova-cells-v1:
irrelevant-files: irrelevant-files:
- ^(test-|)requirements.txt$ - ^(test-|)requirements.txt$
@ -397,7 +395,7 @@
- ^setup.cfg$ - ^setup.cfg$
- ^tempest/hacking/.*$ - ^tempest/hacking/.*$
- ^tempest/tests/.*$ - ^tempest/tests/.*$
- tempest-scenario-all: - tempest-slow:
irrelevant-files: irrelevant-files:
- ^(test-|)requirements.txt$ - ^(test-|)requirements.txt$
- ^.*\.rst$ - ^.*\.rst$
@ -407,7 +405,6 @@
- ^setup.cfg$ - ^setup.cfg$
- ^tempest/hacking/.*$ - ^tempest/hacking/.*$
- ^tempest/tests/.*$ - ^tempest/tests/.*$
- ^tempest/api/.*$
experimental: experimental:
jobs: jobs:
- tempest-cinder-v2-api: - tempest-cinder-v2-api:

10
tox.ini
View File

@ -135,6 +135,16 @@ commands =
find . -type f -name "*.pyc" -delete find . -type f -name "*.pyc" -delete
tempest run --serial --regex '\[.*\bsmoke\b.*\]' {posargs} tempest run --serial --regex '\[.*\bsmoke\b.*\]' {posargs}
[testenv:slow-serial]
envdir = .tox/tempest
sitepackages = {[tempestenv]sitepackages}
setenv = {[tempestenv]setenv}
deps = {[tempestenv]deps}
# The regex below is used to select the slow tagged tests to run serially:
commands =
find . -type f -name "*.pyc" -delete
tempest run --serial --regex '\[.*\bslow\b.*\]' {posargs}
[testenv:venv] [testenv:venv]
deps = deps =
-c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt}