19 Commits

Author SHA1 Message Date
Ghanshyam Mann
926f823fbd Fix TEMPEST_VENV_UPPER_CONSTRAINTS default value comparision
TEMPEST_VENV_UPPER_CONSTRAINTS is default to 'master' in devstack
not 'default', fixing this in tempest role.

Change-Id: I5ccc379cc03d8c4d6bd7ebb914bbbc8ffb8ba34d
2021-04-07 16:52:41 -05:00
Ghanshyam Mann
62d11d8479 Use configured upper-constraints for non-master Tempest run
If any job use older Tempest version then we can not need
to use the master upper-constraints as they can be
incompatible most of the time. We need to use the compatible
constraints of what version of Tempest is configured to run.

Below devstack patch adding the new way to configure
the upper-constraints via new variable TEMPEST_VENV_UPPER_CONSTRAINTS
when Tempest master is not used.
- https://review.opendev.org/c/openstack/devstack/+/774711

We need to use the same variable to use the configured constraints
in run-tempest roles also so that when run-tempest recreate the
tempest venv then it honor the same configured constraints instead
if using master and error For example:
- https://zuul.opendev.org/t/openstack/build/854450dc514041a78fd250dde0eb138e/log/job-output.txt#31341

Depends-On: https://review.opendev.org/c/openstack/devstack/+/774716

Change-Id: I32d539b02366c8ae7ea70195674a97ed6a8ae759
2021-02-22 09:49:55 -06:00
Zuul
76f63460b3 Merge "Allow use custom uppper constraints in stable/stein" 2021-02-21 03:52:13 +00:00
Martin Kopec
8a8c85ddfc Fix run-tempest black/exclude var setting
Before [1] we didn't have conditionals for tempest_black_regex
being set or not, therefore it had a default value of empty
string set in the role's defaults. However after [1] we do
have such conditionals which make setting the default value
unnecessary - especially when we're asking if the var is
defined - it will be always defined.
Therefore this patch removes the default setting of
tempest_black_regex and tempest_exclude_regex.

The patch also removes 'Set tempest_exclude_regex' task
introduced by [1] as it appears unnecessary as well. This
task simply copied tempest_black_regex value to
tempest_exclude_regex. However there are 2 tasks below which
do the same logic plus more - they build CLI arguments for
tempest where depending on which ansible var is used the
appropriate CLI arg is chosen - tempest_black_regex is set,
--black-regex will be passed to tempest (for backward
compatibility) and if tempest_exclude_regex is set,
--exclude-regex will passed to tempest.

The patch creates a new temporary ansible var which will be used
only internally. The role will store in it the CLI argument
chosen dynamically which is passed to tempest later. Before we
reused the tempest_exclude_regex variable for that, however that
causes problems whent the var is set on a role level as set_fact
won't override such var.

[1] https://review.opendev.org/c/openstack/tempest/+/768583

Change-Id: I3e45741057e7a95b282b03bab4695f46c62d5271
2021-02-11 17:18:25 +00:00
Riccardo Pittau
bd2acbfbc7 Allow use custom uppper constraints in stable/stein
Openstack Stein still supports Python 2.7 which means it is necessary to
set different upper constraints for that specific release for tempest
testing using that Python version.
The current master upper constraints won't work as they exclusively
consider Python version >= 3.x

Change-Id: Ieafd0ebc242eba2f4cbaf22823dbe94735236189
2021-02-04 11:23:34 +01:00
Martin Kopec
dc84423b72 Inclusive jargon
Following stestr's example where arguments such as --blacklist-file,
--black-regex and --whitelist-file are deprecated since its
3.1.0 release, let's do the change here as well in order to
get tempest consumers some time for the transition.

This change deprecates the following arguments and replaces them
by new ones which are functionally equivavelnt:
* --black-regex is replaced by --exclude-regex
* --blacklist-file is replaced by --exclude-list
* --whitelist-file is replaced by --include-list

For now, Tempest will accept both (new and old) arguments to make
the transition smoother for all consumers.

The patch also bumps min version of tox to 3.18.0 in order to
replace tox's whitelist_externals by allowlist_externals option:
https://github.com/tox-dev/tox/blob/master/docs/changelog.rst#v3180-2020-07-23

Change-Id: I3e09b31f63d2cd7ea41c48e62432bd3bc54fcf44
2021-01-20 08:40:57 +00:00
Zuul
b714ae3895 Merge "run-tempest: save the result of tempest execution" 2020-05-01 00:12:07 +00:00
Andreas Jaeger
99b085bdb9 Set TOX_CONSTRAINTS_FILE
UPPER_CONSTRAINTS_FILE is deprecatedi[1], set the newer
TOX_CONSTRAINTS_FILE as well when interacting with other repos for the
transition.

Update internal use but not the tox.ini file, since users calling it
might use the old variable.

[1] https://zuul-ci.org/docs/zuul-jobs/python-roles.html#rolevar-tox.tox_constraints_file

Change-Id: Iea09402cf918a6978120a341950643185c94ec85
2020-04-22 15:01:21 +02:00
Luigi Toscano
b774666aae run-tempest: save the result of tempest execution
This is useful when a job needs to execute additional tests
after tempest run, so ignoring the tempest run failures,
and combine the results afterwards.

Change-Id: I61fc5ebf69db47a035ae8294cfda4047d9f7e21a
2020-04-17 13:31:00 +02:00
Martin Kopec
4af65407b1 Update README for run-tempest role
Updates README in order to clear confusion about defining
multi-line regexes.

Closes-Bug: 1827234
Change-Id: Ie55ef879b1d6c971b202553b3906d02c29b981d6
2020-02-10 07:25:09 +00:00
Ghanshyam
56149024a0 Use stable constraint in run-tempest for stable branch testing
Stable branches till stable/rocky is using python
version <py3.6. Tempest test those branch in venv
but Tempest tox use the master upper-constraint[1]
which block installation due to dependencies
require >=py3.6. For exmaple, oslo.concurrency 4.0.0
is not compatible for <py3.6.

As we pin Tempest for EM stable brach, we should be
able to use stable constraint for Tempest installation
as well as while running during run-tempest playbook.

tox.ini is hard coded to use master constraint[1] which force
run-tempest to recreate the tox env and use the master constraint.

Devstack can set stable u-c to use via env var but this is
not enough as Tempest role run-tempest recreate the tox
env with master u-c.
- https://review.opendev.org/#/c/705089/4/lib/tempest

Currently we are fixing run-tempest to use stable constraint
till stable/rocky which use distro having <py3.6.

Ideally, we can change compatible branch constraint in tox.ini
while doing release tag for Tempest. But that can be done
for future tags only.

[1] bc9fe8eca8/tox.ini (L14)

Change-Id: I4268049ff91e5527e4b95fd8a70fa53ba27d56ed
2020-02-06 13:12:07 +00:00
Dmitry Tantsur
5952f85343 Allow overridding OS_TEST_TIMEOUT via the role variables
Change-Id: I7e464ba27704a0b4b13d6e0787b81750deee965e
2019-09-02 14:56:07 +02:00
Sergey Vilgelm
2979bff949
Pass tox_extra_args to Run tempest task
Patrole needs access to global packages, but the 'all-plugin' is
deprecated[0] and 'all' defines a sitepackages option as False[1].
Adding tox_extra_args option makes it possible to configure Zuul and
pass --sitepackages argument to tox.

[0] 9b4c50cf63/tox.ini (L64)
[1] 9b4c50cf63/tox.ini (L7)

Change-Id: Ib3264f54075216471720483c8dd816887c1f413c
2018-11-06 15:33:04 -06:00
Chandan Kumar
a9445043df Added tempest_black_regex to skip tempest tests
* https://review.openstack.org/547106 adds a black_regex flag to
  tempest run. We can use the same in devstack-tempest base job
  to skip/blacklist tests.

Change-Id: I9ed91b01ddb7ec0a3d23305031afbac7c4b16c55
2018-08-08 06:17:37 +00:00
Michel Peterson
ef1a16d4a8 Add support for blacklist file for Zuul jobs
Currently there is tempest_test_regex that can be used to select or
ignore tests, however when the list is complex it can become complex to
maintain and define. For such complex scenarios is that the blacklist
file exist.

This patch adds support for the blacklist file in Zuul jobs.

Change-Id: Ib63955446ea59d81d0d882704666dcce4a625526
2018-02-26 15:17:58 +02:00
Andreas Jaeger
ba2f479dc2 Rename tox_venvlist
For consistency with the tox jobs, use tox_envlist instead of
tox_venvlist. See
https://docs.openstack.org/infra/zuul-jobs/jobs.html#job-tox for
definition of tox job.

All projects using tox_venvlist have been changed to use tox_envlist as
well (see dependencies). Now we can rename tox_venvlist to tox_envlist -
and then the projects can remove tox_venvlist and use only tox_envlist.

Change-Id: I67a20b1faa77ff0166ad96fd95432c11948abc74
Depends-On: I035584aaeaa71b62c1d0c7e0a62dbf053a3488b6
Depends-On: I2fb02ead3b45c2a05d28f59fd28d062a3b40e07e
Depends-On: Id03d4c1351d5a002dc9d73247d0315bef274f627
Depends-On: Iff3093e61c7d2cb684667d5bc5f2a75580569aab
2018-01-12 16:49:47 +01:00
Michel Peterson
515edd4035 Document how to write multiline regex with comments
It is possible to add multiline and docummented regex to the
tempest_test_regex but it might not be evident at first, although very
useful for some scenarios.

This patch documents how this should be achieved to make it easier for
people in the future.

Change-Id: Id54679debedb1667cc70004b9a28fba921e50c0b
2017-12-25 14:40:32 +02:00
Luigi Toscano
e8a1f53986 run-tempest role: new tempest_test_regex variable
Specify a regular expression to filter the tests.

Change-Id: I44911c6bd6ede68c3376c8a6b42700310dce7a06
2017-11-16 15:22:07 +01:00
Andrea Frittoli (andreaf)
7d5445dae2 Initial skeleton for devstack-tempest base job
Setup the initial folder and play to run tempest.
This simply runs tempest full for now, with not support for config
options.

Change-Id: I5a76dd23900a1b5fb1764fafd837d69baf9ed8b1
Depends-on: Iffe54fbccbccd68db08f79a1b51dd7f76dbff408
2017-11-08 18:04:39 +11:00