26 Commits

Author SHA1 Message Date
98f2393641 Use stable branch upper-constraints till 2023.1
Stable/2023.1 and Unmaintained xena/yoga/zed runs ubuntu-focal
jobs and since with [1] py3.8 pins are dropped from master
upper-constraints, so these branches can no longer use master
constraints with envs running with python3.8.
With this patch making these branches to use stable constraints.

[1] https://review.opendev.org/c/openstack/requirements/+/925201
[2] https://lists.openstack.org/archives/list/openstack-discuss@\
    lists.openstack.org/thread/FOWV4UQZTH4DPDA67QDEROAESYU5Z3LE/

Needed-By: https://review.opendev.org/c/openstack/devstack/+/933704
Related-Bug: #2085886
Change-Id: I8d2ff33384aec93820aa27ad27f1ff9bd7e87098
2024-10-30 11:45:55 -07:00
Ghanshyam Mann
b42fc49058 Fix constraint usage for unmaintained/victoria|wallaby
vistoria and wallaby branches are renamed to unmaintained/
namespace. Fixing run-tempest role to adopt those new name
for those branches so that it can set the correct constraints
while installing the tempest.

Change-Id: I595c3530461173fe3c9ba258aeb1ae90314874bf
2024-04-18 11:59:28 -07:00
Rajesh Tailor
a85bdb47ef [codespell] fix typos
This change fixes typos in tempest code, doc, zuul yaml
and tox.ini file, which are found by codespell.

Change-Id: I8051d979e28f7dbf532181f339cbef9cd7220c76
2024-04-02 12:12:32 +05:30
Ghanshyam Mann
671a931e54 Increase the default concurrency for tempest run
run-tempest role which run tempest test for all tempest
based job limit the concurrency (number of worker used to
run the tests) to number-of-cpu/2 (if number of cpu > 3)
which limit the jobs to use half of the available cpu to
run the tests. We are seeing a lot of timeout now a days
and increasing the test runner worker can possibily solve
the timeout issue.

This commit modifies the logic of default concurrency to
set as number-of-cpu -2 so that we will be utlizing the
available cpu in more efficient way.

Related-Bug: #2004780

Change-Id: I8beeedade3098ad346d3a307b94bc3b254cde90a
2023-07-21 11:59:42 -05:00
Lukáš Piwowarski
bd6481d934 Fix default values for variables in run-tempest role
With the new version of tox the parsing of postargs value has changed.
Previously an empty string was discarded by tox if it appeared in the
postargs value in contrast with the latest version of tox which
leaves the empty string intact.

The behavior of the latest tox version makes some jobs fail as some
variables in the run-tempest role have an empty string as a default
value. This leads in some cases to the execution of an incorrect tempest
command, for example:

tempest run --regex '[.*\bsmoke\b.*]' '' --concurrency=2

The patch changes the default value for blacklist_option,
exclude_list_option and tempest_test_regex variables to None to prevent
an empty string from being passed to postargs variable in tox.

Change-Id: I006e33c5ea0753b68cf1feb8580cb029209af9a8
2023-01-31 14:14:45 +01:00
Ghanshyam Mann
51a5d4f234 Use UPPER_CONSTRAINTS_FILE for stable/wallaby testing
stable/wallaby is in EM state and Tempest is going to be
pinned for stable/wallaby which need to modify run-tempest
role to use the constraints set by the devstack.

Change-Id: I985c222a1a461dff10ff33716fd2263a1a89813a
2023-01-25 14:49:48 -06:00
Ghanshyam Mann
d9b6d7ca1a Use UPPER_CONSTRAINTS_FILE for stable/victoria testing
stable/victoria is in EM state and Tempest is going to be
pinned for stable/victoria which need to modify run-tempest
role to use the constraints set by the devstack or any job.

Because of Tempest 26.1.0 and with stestr 3.0.1 (beacause of upper
constraints of stestr 3.0.1 in stable/ussuri ) which does not have
new args exclude-regex so let's fallback to old arg if new arg is
passed.

Change-Id: I9cad0ccb02ee7bef0f94a1ee40e0fa117d89b5eb
2022-05-30 10:12:12 +00:00
Ghanshyam Mann
7aa3b21e89 Use UPPER_CONSTRAINTS_FILE for stable/ussuri testing
Tempest is going to be pinned for stable/ussuri(depends on) which
need to modify run-tempest role to use the constraints set by
the devstack or any job.

Because of Tempest 26.1.0 and with stestr 2.5.1 or 3.0.1 (beacause of upper
constraints of stestr 2.5.1 in stable/train and 3.0.1 in stable/ussuri )
which does not have new args exclude-regex so let's fallback to old arg if
new arg is passed.

Change-Id: I89ae6bd5137e57cb6ec5bc3c9edecc29489bca94
2022-05-28 23:53:16 +00:00
Ghanshyam Mann
5ff5dabd6e Fix stable/train testing with tempest 26.1.0
Tis issue came up before also during we cap the Tempest
version for stable branch tests(the one moved to EM state).

Tempest use the tox constraints file while creating the tempest
virtual env in devstack/lib/tempest, installing the tempest
plugins and running the tests like in run-tempest ansible roles.
If anywhere tox constraints file is used differently then venv
will be recreated and installed tempest plugins will be gone from
that recreated venv so tempest will not be able to run the plugins
tests and fail job.

Current failure: https://aca43ed1a01d95dea0ee-dcdb6cbb330bdac08ffee1284f86c919.ssl.cf2.rackcdn.com/822225/3/check/octavia-v2-dsvm-scenario/f60e4d3/job-output.txt

Earlier I tried to fix it in a generic way by fetching the devstack
localrc - https://review.opendev.org/c/openstack/tempest/+/774913
but this solution does not seems working as we can see it is skipped
in current stable/train testing with tempest 26.1.0

Adding stable/train explicitly in list for using the stable constraint
in run-tempest role also which will be in sync with what devstack/lib/tempest
use. And later I will debug why generic way is not working.

Because of Tempest 26.1.0 and with stestr 2.5.1 (beacause of upper
constraints of stestr 2.5.1 in stable/train) which does not have
new args exclude-regex so let's fallback to old arg if new arg is passed.

Depends-On: https://review.opendev.org/c/openstack/devstack/+/822380
Depends-On: https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/825708

Closes-Bug: #1955418
Change-Id: I105c839c38adfba1f9e22123ec3a99ffc0b5b72e
2022-01-25 17:03:31 +00:00
Ghanshyam Mann
926f823fbd Fix TEMPEST_VENV_UPPER_CONSTRAINTS default value comparision
TEMPEST_VENV_UPPER_CONSTRAINTS is default to 'master' in devstack
not 'default', fixing this in tempest role.

Change-Id: I5ccc379cc03d8c4d6bd7ebb914bbbc8ffb8ba34d
2021-04-07 16:52:41 -05:00
Ghanshyam Mann
62d11d8479 Use configured upper-constraints for non-master Tempest run
If any job use older Tempest version then we can not need
to use the master upper-constraints as they can be
incompatible most of the time. We need to use the compatible
constraints of what version of Tempest is configured to run.

Below devstack patch adding the new way to configure
the upper-constraints via new variable TEMPEST_VENV_UPPER_CONSTRAINTS
when Tempest master is not used.
- https://review.opendev.org/c/openstack/devstack/+/774711

We need to use the same variable to use the configured constraints
in run-tempest roles also so that when run-tempest recreate the
tempest venv then it honor the same configured constraints instead
if using master and error For example:
- https://zuul.opendev.org/t/openstack/build/854450dc514041a78fd250dde0eb138e/log/job-output.txt#31341

Depends-On: https://review.opendev.org/c/openstack/devstack/+/774716

Change-Id: I32d539b02366c8ae7ea70195674a97ed6a8ae759
2021-02-22 09:49:55 -06:00
Zuul
76f63460b3 Merge "Allow use custom uppper constraints in stable/stein" 2021-02-21 03:52:13 +00:00
Martin Kopec
8a8c85ddfc Fix run-tempest black/exclude var setting
Before [1] we didn't have conditionals for tempest_black_regex
being set or not, therefore it had a default value of empty
string set in the role's defaults. However after [1] we do
have such conditionals which make setting the default value
unnecessary - especially when we're asking if the var is
defined - it will be always defined.
Therefore this patch removes the default setting of
tempest_black_regex and tempest_exclude_regex.

The patch also removes 'Set tempest_exclude_regex' task
introduced by [1] as it appears unnecessary as well. This
task simply copied tempest_black_regex value to
tempest_exclude_regex. However there are 2 tasks below which
do the same logic plus more - they build CLI arguments for
tempest where depending on which ansible var is used the
appropriate CLI arg is chosen - tempest_black_regex is set,
--black-regex will be passed to tempest (for backward
compatibility) and if tempest_exclude_regex is set,
--exclude-regex will passed to tempest.

The patch creates a new temporary ansible var which will be used
only internally. The role will store in it the CLI argument
chosen dynamically which is passed to tempest later. Before we
reused the tempest_exclude_regex variable for that, however that
causes problems whent the var is set on a role level as set_fact
won't override such var.

[1] https://review.opendev.org/c/openstack/tempest/+/768583

Change-Id: I3e45741057e7a95b282b03bab4695f46c62d5271
2021-02-11 17:18:25 +00:00
Riccardo Pittau
bd2acbfbc7 Allow use custom uppper constraints in stable/stein
Openstack Stein still supports Python 2.7 which means it is necessary to
set different upper constraints for that specific release for tempest
testing using that Python version.
The current master upper constraints won't work as they exclusively
consider Python version >= 3.x

Change-Id: Ieafd0ebc242eba2f4cbaf22823dbe94735236189
2021-02-04 11:23:34 +01:00
Martin Kopec
dc84423b72 Inclusive jargon
Following stestr's example where arguments such as --blacklist-file,
--black-regex and --whitelist-file are deprecated since its
3.1.0 release, let's do the change here as well in order to
get tempest consumers some time for the transition.

This change deprecates the following arguments and replaces them
by new ones which are functionally equivavelnt:
* --black-regex is replaced by --exclude-regex
* --blacklist-file is replaced by --exclude-list
* --whitelist-file is replaced by --include-list

For now, Tempest will accept both (new and old) arguments to make
the transition smoother for all consumers.

The patch also bumps min version of tox to 3.18.0 in order to
replace tox's whitelist_externals by allowlist_externals option:
https://github.com/tox-dev/tox/blob/master/docs/changelog.rst#v3180-2020-07-23

Change-Id: I3e09b31f63d2cd7ea41c48e62432bd3bc54fcf44
2021-01-20 08:40:57 +00:00
Zuul
b714ae3895 Merge "run-tempest: save the result of tempest execution" 2020-05-01 00:12:07 +00:00
Andreas Jaeger
99b085bdb9 Set TOX_CONSTRAINTS_FILE
UPPER_CONSTRAINTS_FILE is deprecatedi[1], set the newer
TOX_CONSTRAINTS_FILE as well when interacting with other repos for the
transition.

Update internal use but not the tox.ini file, since users calling it
might use the old variable.

[1] https://zuul-ci.org/docs/zuul-jobs/python-roles.html#rolevar-tox.tox_constraints_file

Change-Id: Iea09402cf918a6978120a341950643185c94ec85
2020-04-22 15:01:21 +02:00
Luigi Toscano
b774666aae run-tempest: save the result of tempest execution
This is useful when a job needs to execute additional tests
after tempest run, so ignoring the tempest run failures,
and combine the results afterwards.

Change-Id: I61fc5ebf69db47a035ae8294cfda4047d9f7e21a
2020-04-17 13:31:00 +02:00
Ghanshyam
56149024a0 Use stable constraint in run-tempest for stable branch testing
Stable branches till stable/rocky is using python
version <py3.6. Tempest test those branch in venv
but Tempest tox use the master upper-constraint[1]
which block installation due to dependencies
require >=py3.6. For exmaple, oslo.concurrency 4.0.0
is not compatible for <py3.6.

As we pin Tempest for EM stable brach, we should be
able to use stable constraint for Tempest installation
as well as while running during run-tempest playbook.

tox.ini is hard coded to use master constraint[1] which force
run-tempest to recreate the tox env and use the master constraint.

Devstack can set stable u-c to use via env var but this is
not enough as Tempest role run-tempest recreate the tox
env with master u-c.
- https://review.opendev.org/#/c/705089/4/lib/tempest

Currently we are fixing run-tempest to use stable constraint
till stable/rocky which use distro having <py3.6.

Ideally, we can change compatible branch constraint in tox.ini
while doing release tag for Tempest. But that can be done
for future tags only.

[1] bc9fe8eca8/tox.ini (L14)

Change-Id: I4268049ff91e5527e4b95fd8a70fa53ba27d56ed
2020-02-06 13:12:07 +00:00
Dmitry Tantsur
5952f85343 Allow overridding OS_TEST_TIMEOUT via the role variables
Change-Id: I7e464ba27704a0b4b13d6e0787b81750deee965e
2019-09-02 14:56:07 +02:00
Sergey Vilgelm
2979bff949
Pass tox_extra_args to Run tempest task
Patrole needs access to global packages, but the 'all-plugin' is
deprecated[0] and 'all' defines a sitepackages option as False[1].
Adding tox_extra_args option makes it possible to configure Zuul and
pass --sitepackages argument to tox.

[0] 9b4c50cf63/tox.ini (L64)
[1] 9b4c50cf63/tox.ini (L7)

Change-Id: Ib3264f54075216471720483c8dd816887c1f413c
2018-11-06 15:33:04 -06:00
Chandan Kumar
a9445043df Added tempest_black_regex to skip tempest tests
* https://review.openstack.org/547106 adds a black_regex flag to
  tempest run. We can use the same in devstack-tempest base job
  to skip/blacklist tests.

Change-Id: I9ed91b01ddb7ec0a3d23305031afbac7c4b16c55
2018-08-08 06:17:37 +00:00
Michel Peterson
ef1a16d4a8 Add support for blacklist file for Zuul jobs
Currently there is tempest_test_regex that can be used to select or
ignore tests, however when the list is complex it can become complex to
maintain and define. For such complex scenarios is that the blacklist
file exist.

This patch adds support for the blacklist file in Zuul jobs.

Change-Id: Ib63955446ea59d81d0d882704666dcce4a625526
2018-02-26 15:17:58 +02:00
Andreas Jaeger
ba2f479dc2 Rename tox_venvlist
For consistency with the tox jobs, use tox_envlist instead of
tox_venvlist. See
https://docs.openstack.org/infra/zuul-jobs/jobs.html#job-tox for
definition of tox job.

All projects using tox_venvlist have been changed to use tox_envlist as
well (see dependencies). Now we can rename tox_venvlist to tox_envlist -
and then the projects can remove tox_venvlist and use only tox_envlist.

Change-Id: I67a20b1faa77ff0166ad96fd95432c11948abc74
Depends-On: I035584aaeaa71b62c1d0c7e0a62dbf053a3488b6
Depends-On: I2fb02ead3b45c2a05d28f59fd28d062a3b40e07e
Depends-On: Id03d4c1351d5a002dc9d73247d0315bef274f627
Depends-On: Iff3093e61c7d2cb684667d5bc5f2a75580569aab
2018-01-12 16:49:47 +01:00
Luigi Toscano
e8a1f53986 run-tempest role: new tempest_test_regex variable
Specify a regular expression to filter the tests.

Change-Id: I44911c6bd6ede68c3376c8a6b42700310dce7a06
2017-11-16 15:22:07 +01:00
Andrea Frittoli (andreaf)
7d5445dae2 Initial skeleton for devstack-tempest base job
Setup the initial folder and play to run tempest.
This simply runs tempest full for now, with not support for config
options.

Change-Id: I5a76dd23900a1b5fb1764fafd837d69baf9ed8b1
Depends-on: Iffe54fbccbccd68db08f79a1b51dd7f76dbff408
2017-11-08 18:04:39 +11:00