stable/stein uses old Tempest 26.0.0 to run the test
and that is why we use stable/stein upper-constraints
in the testing. But when any jobs with tempest plugins
try to run plugins tests, they fail.
This is because a new role run-tempest-26 (used for testing
with tempest 26.0.0) does not include the stable/stein in the
list of branches for whom we use stable constraints instead of
master constraints. and due to that run-tempest-26 finds constraints
mismatch and recreates the tempest virtual env. On recreation, venv
remove the previously installed tempest pluings and so does not run
their test.
Closes-Bug: #1938227
Change-Id: I00ac642fa602980d14612adc1ce65068d67db8ad
run-tempest is changed recently to add the new variables but
keep supporting the old ones too, for example:
tempest_black_regex, tempest_exclude_regex. and if both
old and new var are used in job definition (parent and child) then
new variables are picked. Because Tempest is branchless, zuul pick
the Tempest master playbooks/roles. That is why job running on stable
branch gate will pick the base job definition from Tempest master.
This way if any stable jobs which were defining the old var and using old
Tempest are broken if any of their parent job define the new var.
This commit pin the older run-tempest role for such stable branches.
Change-Id: If49ab0c31aca5b7837636727096a9bc83f891b1b
from Tempest to DevStack as it tests DevStack side of things and
is useful for projects not using Tempest.
This is part 2 of 2.
The 1st part is DevStack-side, in Depends-On.
The script is left calling out to devstack because legacy (dsvm)
jobs rely on its presence.
Depends-On: Ie166730843f874b9c99e37244e460d7ad33b7eeb
Change-Id: I6fa17ae413f106453303c4882925573bd8e05029
TEMPEST_VENV_UPPER_CONSTRAINTS is default to 'master' in devstack
not 'default', fixing this in tempest role.
Change-Id: I5ccc379cc03d8c4d6bd7ebb914bbbc8ffb8ba34d
If any job use older Tempest version then we can not need
to use the master upper-constraints as they can be
incompatible most of the time. We need to use the compatible
constraints of what version of Tempest is configured to run.
Below devstack patch adding the new way to configure
the upper-constraints via new variable TEMPEST_VENV_UPPER_CONSTRAINTS
when Tempest master is not used.
- https://review.opendev.org/c/openstack/devstack/+/774711
We need to use the same variable to use the configured constraints
in run-tempest roles also so that when run-tempest recreate the
tempest venv then it honor the same configured constraints instead
if using master and error For example:
- https://zuul.opendev.org/t/openstack/build/854450dc514041a78fd250dde0eb138e/log/job-output.txt#31341
Depends-On: https://review.opendev.org/c/openstack/devstack/+/774716
Change-Id: I32d539b02366c8ae7ea70195674a97ed6a8ae759
Before [1] we didn't have conditionals for tempest_black_regex
being set or not, therefore it had a default value of empty
string set in the role's defaults. However after [1] we do
have such conditionals which make setting the default value
unnecessary - especially when we're asking if the var is
defined - it will be always defined.
Therefore this patch removes the default setting of
tempest_black_regex and tempest_exclude_regex.
The patch also removes 'Set tempest_exclude_regex' task
introduced by [1] as it appears unnecessary as well. This
task simply copied tempest_black_regex value to
tempest_exclude_regex. However there are 2 tasks below which
do the same logic plus more - they build CLI arguments for
tempest where depending on which ansible var is used the
appropriate CLI arg is chosen - tempest_black_regex is set,
--black-regex will be passed to tempest (for backward
compatibility) and if tempest_exclude_regex is set,
--exclude-regex will passed to tempest.
The patch creates a new temporary ansible var which will be used
only internally. The role will store in it the CLI argument
chosen dynamically which is passed to tempest later. Before we
reused the tempest_exclude_regex variable for that, however that
causes problems whent the var is set on a role level as set_fact
won't override such var.
[1] https://review.opendev.org/c/openstack/tempest/+/768583
Change-Id: I3e45741057e7a95b282b03bab4695f46c62d5271
Openstack Stein still supports Python 2.7 which means it is necessary to
set different upper constraints for that specific release for tempest
testing using that Python version.
The current master upper constraints won't work as they exclusively
consider Python version >= 3.x
Change-Id: Ieafd0ebc242eba2f4cbaf22823dbe94735236189
Following stestr's example where arguments such as --blacklist-file,
--black-regex and --whitelist-file are deprecated since its
3.1.0 release, let's do the change here as well in order to
get tempest consumers some time for the transition.
This change deprecates the following arguments and replaces them
by new ones which are functionally equivavelnt:
* --black-regex is replaced by --exclude-regex
* --blacklist-file is replaced by --exclude-list
* --whitelist-file is replaced by --include-list
For now, Tempest will accept both (new and old) arguments to make
the transition smoother for all consumers.
The patch also bumps min version of tox to 3.18.0 in order to
replace tox's whitelist_externals by allowlist_externals option:
https://github.com/tox-dev/tox/blob/master/docs/changelog.rst#v3180-2020-07-23
Change-Id: I3e09b31f63d2cd7ea41c48e62432bd3bc54fcf44
The patch implements a new flag which will fail a job when any
resources were leaked - that can be used for verification that
tests are cleaning their resources after they are finished.
Change-Id: I212cdce9713c80491153b0bbdb313e75c1f96f1d
* add additional handler so that INFO messages are printed to stdout
besided tempest.log
* replace print commands by LOG.info calls
* when --debug argument used print DEBUG messages to stdout
* add additional debug messages to improve transparency of the cleanup
execution
* use --debug argument within tempest-cleanup ansible role to see more
details directly in the output
Change-Id: Iee72dc411fdf5cda8a8eed8dd7f349e0866648ab
The patch adds a new ansible role called tempest-cleanup which will
allow us to test tempest cleanup in the gate jobs.
Change-Id: I2cef2da6fee13e622da07b890da88850fe420152
UPPER_CONSTRAINTS_FILE is deprecatedi[1], set the newer
TOX_CONSTRAINTS_FILE as well when interacting with other repos for the
transition.
Update internal use but not the tox.ini file, since users calling it
might use the old variable.
[1] https://zuul-ci.org/docs/zuul-jobs/python-roles.html#rolevar-tox.tox_constraints_file
Change-Id: Iea09402cf918a6978120a341950643185c94ec85
This is useful when a job needs to execute additional tests
after tempest run, so ignoring the tempest run failures,
and combine the results afterwards.
Change-Id: I61fc5ebf69db47a035ae8294cfda4047d9f7e21a
Stable branches till stable/rocky is using python
version <py3.6. Tempest test those branch in venv
but Tempest tox use the master upper-constraint[1]
which block installation due to dependencies
require >=py3.6. For exmaple, oslo.concurrency 4.0.0
is not compatible for <py3.6.
As we pin Tempest for EM stable brach, we should be
able to use stable constraint for Tempest installation
as well as while running during run-tempest playbook.
tox.ini is hard coded to use master constraint[1] which force
run-tempest to recreate the tox env and use the master constraint.
Devstack can set stable u-c to use via env var but this is
not enough as Tempest role run-tempest recreate the tox
env with master u-c.
- https://review.opendev.org/#/c/705089/4/lib/tempest
Currently we are fixing run-tempest to use stable constraint
till stable/rocky which use distro having <py3.6.
Ideally, we can change compatible branch constraint in tox.ini
while doing release tag for Tempest. But that can be done
for future tags only.
[1] bc9fe8eca8/tox.ini (L14)
Change-Id: I4268049ff91e5527e4b95fd8a70fa53ba27d56ed
Devstack IPv6 base job 'devstack-IPv6' setup the IPv6 setting
to deploy the services to listen on IPv6 address.
Tempest 'devstack-tempest-ipv6' job derived from 'devstack-IPv6'
job adds the playbooks to run the tests.
As part of Train community goal 'Support IPv6-Only Deployments',
we will expand the 'devstack-tempest-ipv6' job to do
IPv6-only deployments verification.
This commit define the new roles of 'ipv6-only-deployments-verification'
which will be invoked as part of run phase of 'devstack-tempest-ipv6' job.
This role will do IPv6-only setting and deployments verification via
tools/verify-ipv6-only-deployement.sh
IPv6 verification script can be extended further to perform more checks
and via project specific test case. Those tests will run as part of project
specific child jobs.
The 'devstack-tempest-ipv6' job will be used as parent for project specific
IPv6-only job. Those child job can extend the project specific IPv6 verification
by defining new playebook for post-run. That way the base verification
done in 'devstack-tempest-ipv6' will still run in addition to project specific
verificaiton and tests run.
Verification structure will be:
- 'devstack-IPv6' deploy the service on IPv6
- 'devstack-tempest-ipv6' run will verify the IPv6-only setting and listen address
- Child jobs derived from 'devstack-tempest-ipv6' will run the IPv6 related test case or
any further IPv6 deployment verification.
This commit also adds the new job 'tempest-ipv6-only' which will run smoke
and ipv6 related tests present in Tempest. This job will be used to run
on 6 services (Nova, Neutron, Cinder, Keystone, Glance, Swift) deployed
by devstack.
Story: #2005477
Task: #35923
Change-Id: Ie1e5e5e02ee45ab7a4aae0f35bdb42afc2c3fe2e
Patrole needs access to global packages, but the 'all-plugin' is
deprecated[0] and 'all' defines a sitepackages option as False[1].
Adding tox_extra_args option makes it possible to configure Zuul and
pass --sitepackages argument to tox.
[0] 9b4c50cf63/tox.ini (L64)
[1] 9b4c50cf63/tox.ini (L7)
Change-Id: Ib3264f54075216471720483c8dd816887c1f413c
* https://review.openstack.org/547106 adds a black_regex flag to
tempest run. We can use the same in devstack-tempest base job
to skip/blacklist tests.
Change-Id: I9ed91b01ddb7ec0a3d23305031afbac7c4b16c55
The process-stackviz role only helps with analyzing runs, when it fails
it should not cause the whole job to suffer a POST_FAILURE.
Change-Id: Icfd6b2fc7adb42afb7900ea4458fa60cefc76fdb
Related-Bug: 1758054
Currently there is a zuul-jobs role only used in this tempest job while
other jobs mostly use fetch-subunit-output role.
This changes tries to consolidate towards the most used role, to be able
to clean up zuul-jobs a bit.
Change-Id: I404760ba34767660f8816f48712e13dccad67440
Depends-On: https://review.openstack.org/548406
Currently there is tempest_test_regex that can be used to select or
ignore tests, however when the list is complex it can become complex to
maintain and define. For such complex scenarios is that the blacklist
file exist.
This patch adds support for the blacklist file in Zuul jobs.
Change-Id: Ib63955446ea59d81d0d882704666dcce4a625526
The current default value of stage_dir does not match the valid
default from zuul-jobs, which causes stackviz to be skipped when
stage_dir is not overwritten, which is the case in all CI jobs.
Setting default and docs to "{{ ansible_user_dir }}".
Change-Id: I9fd3f8541c8e3fbc406c5e5c0c5fe8427bf74112
For consistency with the tox jobs, use tox_envlist instead of
tox_venvlist. See
https://docs.openstack.org/infra/zuul-jobs/jobs.html#job-tox for
definition of tox job.
All projects using tox_venvlist have been changed to use tox_envlist as
well (see dependencies). Now we can rename tox_venvlist to tox_envlist -
and then the projects can remove tox_venvlist and use only tox_envlist.
Change-Id: I67a20b1faa77ff0166ad96fd95432c11948abc74
Depends-On: I035584aaeaa71b62c1d0c7e0a62dbf053a3488b6
Depends-On: I2fb02ead3b45c2a05d28f59fd28d062a3b40e07e
Depends-On: Id03d4c1351d5a002dc9d73247d0315bef274f627
Depends-On: Iff3093e61c7d2cb684667d5bc5f2a75580569aab
It is possible to add multiline and docummented regex to the
tempest_test_regex but it might not be evident at first, although very
useful for some scenarios.
This patch documents how this should be achieved to make it easier for
people in the future.
Change-Id: Id54679debedb1667cc70004b9a28fba921e50c0b
Setup the initial folder and play to run tempest.
This simply runs tempest full for now, with not support for config
options.
Change-Id: I5a76dd23900a1b5fb1764fafd837d69baf9ed8b1
Depends-on: Iffe54fbccbccd68db08f79a1b51dd7f76dbff408
Ghanshyam Mann