This enables tempest cleanup with record_resources in the tempest-all job
to create a list of all the resources created during the tempest run.
Then the list will be used to run 'tempest cleanup' to clean only
the resources in the list.
Also, enable the dry cleanup with record_resources in tempest-extra-tests
job.
Change-Id: Ia64a1c09d4b5e9815ff5926161d550b56433ec64
vistoria and wallaby branches are renamed to unmaintained/
namespace. Fixing run-tempest role to adopt those new name
for those branches so that it can set the correct constraints
while installing the tempest.
Change-Id: I595c3530461173fe3c9ba258aeb1ae90314874bf
This change fixes typos in tempest code, doc, zuul yaml
and tox.ini file, which are found by codespell.
Change-Id: I8051d979e28f7dbf532181f339cbef9cd7220c76
A warning in command description of run cleanup first with dry-run
is added. The cleanup behavior is extended to allow users to delete
only resources if their name starts with a certain prefix.
Closes-Bug: #1945082
Change-Id: I65dfe051c891b3679538acec713e8616746c47f6
The role will find out the available hosts and set
compute.migration_source_host and compute.migration_dest_host
options in tempest.conf if tempest_set_src_dest_host is set to
True.
Change-Id: Ic01970b100a7bb30d9c78229204c7de67681108c
run-tempest role which run tempest test for all tempest
based job limit the concurrency (number of worker used to
run the tests) to number-of-cpu/2 (if number of cpu > 3)
which limit the jobs to use half of the available cpu to
run the tests. We are seeing a lot of timeout now a days
and increasing the test runner worker can possibily solve
the timeout issue.
This commit modifies the logic of default concurrency to
set as number-of-cpu -2 so that we will be utlizing the
available cpu in more efficient way.
Related-Bug: #2004780
Change-Id: I8beeedade3098ad346d3a307b94bc3b254cde90a
With the new version of tox the parsing of postargs value has changed.
Previously an empty string was discarded by tox if it appeared in the
postargs value in contrast with the latest version of tox which
leaves the empty string intact.
The behavior of the latest tox version makes some jobs fail as some
variables in the run-tempest role have an empty string as a default
value. This leads in some cases to the execution of an incorrect tempest
command, for example:
tempest run --regex '[.*\bsmoke\b.*]' '' --concurrency=2
The patch changes the default value for blacklist_option,
exclude_list_option and tempest_test_regex variables to None to prevent
an empty string from being passed to postargs variable in tox.
Change-Id: I006e33c5ea0753b68cf1feb8580cb029209af9a8
stable/wallaby is in EM state and Tempest is going to be
pinned for stable/wallaby which need to modify run-tempest
role to use the constraints set by the devstack.
Change-Id: I985c222a1a461dff10ff33716fd2263a1a89813a
The 'all-plugin' tox environment was deprecated by this patch [1].
Instead of the 'all-plugin' it is recommended to use the 'all' tox
environment.
This patch removes any reference to 'all-plugin' tox environment and
updates the documentation so that the installation steps work with
the 'all' tox venv.
[1] https://review.opendev.org/c/openstack/tempest/+/543974
Related-Bug: 1862240
Change-Id: I0a86d9cf33bff6576a6337967d6b663ab7d9f4a6
stable/victoria is in EM state and Tempest is going to be
pinned for stable/victoria which need to modify run-tempest
role to use the constraints set by the devstack or any job.
Because of Tempest 26.1.0 and with stestr 3.0.1 (beacause of upper
constraints of stestr 3.0.1 in stable/ussuri ) which does not have
new args exclude-regex so let's fallback to old arg if new arg is
passed.
Change-Id: I9cad0ccb02ee7bef0f94a1ee40e0fa117d89b5eb
Tempest is going to be pinned for stable/ussuri(depends on) which
need to modify run-tempest role to use the constraints set by
the devstack or any job.
Because of Tempest 26.1.0 and with stestr 2.5.1 or 3.0.1 (beacause of upper
constraints of stestr 2.5.1 in stable/train and 3.0.1 in stable/ussuri )
which does not have new args exclude-regex so let's fallback to old arg if
new arg is passed.
Change-Id: I89ae6bd5137e57cb6ec5bc3c9edecc29489bca94
Tis issue came up before also during we cap the Tempest
version for stable branch tests(the one moved to EM state).
Tempest use the tox constraints file while creating the tempest
virtual env in devstack/lib/tempest, installing the tempest
plugins and running the tests like in run-tempest ansible roles.
If anywhere tox constraints file is used differently then venv
will be recreated and installed tempest plugins will be gone from
that recreated venv so tempest will not be able to run the plugins
tests and fail job.
Current failure: https://aca43ed1a01d95dea0ee-dcdb6cbb330bdac08ffee1284f86c919.ssl.cf2.rackcdn.com/822225/3/check/octavia-v2-dsvm-scenario/f60e4d3/job-output.txt
Earlier I tried to fix it in a generic way by fetching the devstack
localrc - https://review.opendev.org/c/openstack/tempest/+/774913
but this solution does not seems working as we can see it is skipped
in current stable/train testing with tempest 26.1.0
Adding stable/train explicitly in list for using the stable constraint
in run-tempest role also which will be in sync with what devstack/lib/tempest
use. And later I will debug why generic way is not working.
Because of Tempest 26.1.0 and with stestr 2.5.1 (beacause of upper
constraints of stestr 2.5.1 in stable/train) which does not have
new args exclude-regex so let's fallback to old arg if new arg is passed.
Depends-On: https://review.opendev.org/c/openstack/devstack/+/822380
Depends-On: https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/825708
Closes-Bug: #1955418
Change-Id: I105c839c38adfba1f9e22123ec3a99ffc0b5b72e
stable/stein uses old Tempest 26.0.0 to run the test
and that is why we use stable/stein upper-constraints
in the testing. But when any jobs with tempest plugins
try to run plugins tests, they fail.
This is because a new role run-tempest-26 (used for testing
with tempest 26.0.0) does not include the stable/stein in the
list of branches for whom we use stable constraints instead of
master constraints. and due to that run-tempest-26 finds constraints
mismatch and recreates the tempest virtual env. On recreation, venv
remove the previously installed tempest pluings and so does not run
their test.
Closes-Bug: #1938227
Change-Id: I00ac642fa602980d14612adc1ce65068d67db8ad
run-tempest is changed recently to add the new variables but
keep supporting the old ones too, for example:
tempest_black_regex, tempest_exclude_regex. and if both
old and new var are used in job definition (parent and child) then
new variables are picked. Because Tempest is branchless, zuul pick
the Tempest master playbooks/roles. That is why job running on stable
branch gate will pick the base job definition from Tempest master.
This way if any stable jobs which were defining the old var and using old
Tempest are broken if any of their parent job define the new var.
This commit pin the older run-tempest role for such stable branches.
Change-Id: If49ab0c31aca5b7837636727096a9bc83f891b1b
from Tempest to DevStack as it tests DevStack side of things and
is useful for projects not using Tempest.
This is part 2 of 2.
The 1st part is DevStack-side, in Depends-On.
The script is left calling out to devstack because legacy (dsvm)
jobs rely on its presence.
Depends-On: Ie166730843f874b9c99e37244e460d7ad33b7eeb
Change-Id: I6fa17ae413f106453303c4882925573bd8e05029
TEMPEST_VENV_UPPER_CONSTRAINTS is default to 'master' in devstack
not 'default', fixing this in tempest role.
Change-Id: I5ccc379cc03d8c4d6bd7ebb914bbbc8ffb8ba34d
If any job use older Tempest version then we can not need
to use the master upper-constraints as they can be
incompatible most of the time. We need to use the compatible
constraints of what version of Tempest is configured to run.
Below devstack patch adding the new way to configure
the upper-constraints via new variable TEMPEST_VENV_UPPER_CONSTRAINTS
when Tempest master is not used.
- https://review.opendev.org/c/openstack/devstack/+/774711
We need to use the same variable to use the configured constraints
in run-tempest roles also so that when run-tempest recreate the
tempest venv then it honor the same configured constraints instead
if using master and error For example:
- https://zuul.opendev.org/t/openstack/build/854450dc514041a78fd250dde0eb138e/log/job-output.txt#31341
Depends-On: https://review.opendev.org/c/openstack/devstack/+/774716
Change-Id: I32d539b02366c8ae7ea70195674a97ed6a8ae759
Before [1] we didn't have conditionals for tempest_black_regex
being set or not, therefore it had a default value of empty
string set in the role's defaults. However after [1] we do
have such conditionals which make setting the default value
unnecessary - especially when we're asking if the var is
defined - it will be always defined.
Therefore this patch removes the default setting of
tempest_black_regex and tempest_exclude_regex.
The patch also removes 'Set tempest_exclude_regex' task
introduced by [1] as it appears unnecessary as well. This
task simply copied tempest_black_regex value to
tempest_exclude_regex. However there are 2 tasks below which
do the same logic plus more - they build CLI arguments for
tempest where depending on which ansible var is used the
appropriate CLI arg is chosen - tempest_black_regex is set,
--black-regex will be passed to tempest (for backward
compatibility) and if tempest_exclude_regex is set,
--exclude-regex will passed to tempest.
The patch creates a new temporary ansible var which will be used
only internally. The role will store in it the CLI argument
chosen dynamically which is passed to tempest later. Before we
reused the tempest_exclude_regex variable for that, however that
causes problems whent the var is set on a role level as set_fact
won't override such var.
[1] https://review.opendev.org/c/openstack/tempest/+/768583
Change-Id: I3e45741057e7a95b282b03bab4695f46c62d5271
Openstack Stein still supports Python 2.7 which means it is necessary to
set different upper constraints for that specific release for tempest
testing using that Python version.
The current master upper constraints won't work as they exclusively
consider Python version >= 3.x
Change-Id: Ieafd0ebc242eba2f4cbaf22823dbe94735236189
Following stestr's example where arguments such as --blacklist-file,
--black-regex and --whitelist-file are deprecated since its
3.1.0 release, let's do the change here as well in order to
get tempest consumers some time for the transition.
This change deprecates the following arguments and replaces them
by new ones which are functionally equivavelnt:
* --black-regex is replaced by --exclude-regex
* --blacklist-file is replaced by --exclude-list
* --whitelist-file is replaced by --include-list
For now, Tempest will accept both (new and old) arguments to make
the transition smoother for all consumers.
The patch also bumps min version of tox to 3.18.0 in order to
replace tox's whitelist_externals by allowlist_externals option:
https://github.com/tox-dev/tox/blob/master/docs/changelog.rst#v3180-2020-07-23
Change-Id: I3e09b31f63d2cd7ea41c48e62432bd3bc54fcf44
The patch implements a new flag which will fail a job when any
resources were leaked - that can be used for verification that
tests are cleaning their resources after they are finished.
Change-Id: I212cdce9713c80491153b0bbdb313e75c1f96f1d
* add additional handler so that INFO messages are printed to stdout
besided tempest.log
* replace print commands by LOG.info calls
* when --debug argument used print DEBUG messages to stdout
* add additional debug messages to improve transparency of the cleanup
execution
* use --debug argument within tempest-cleanup ansible role to see more
details directly in the output
Change-Id: Iee72dc411fdf5cda8a8eed8dd7f349e0866648ab
The patch adds a new ansible role called tempest-cleanup which will
allow us to test tempest cleanup in the gate jobs.
Change-Id: I2cef2da6fee13e622da07b890da88850fe420152
UPPER_CONSTRAINTS_FILE is deprecatedi[1], set the newer
TOX_CONSTRAINTS_FILE as well when interacting with other repos for the
transition.
Update internal use but not the tox.ini file, since users calling it
might use the old variable.
[1] https://zuul-ci.org/docs/zuul-jobs/python-roles.html#rolevar-tox.tox_constraints_file
Change-Id: Iea09402cf918a6978120a341950643185c94ec85
This is useful when a job needs to execute additional tests
after tempest run, so ignoring the tempest run failures,
and combine the results afterwards.
Change-Id: I61fc5ebf69db47a035ae8294cfda4047d9f7e21a
Stable branches till stable/rocky is using python
version <py3.6. Tempest test those branch in venv
but Tempest tox use the master upper-constraint[1]
which block installation due to dependencies
require >=py3.6. For exmaple, oslo.concurrency 4.0.0
is not compatible for <py3.6.
As we pin Tempest for EM stable brach, we should be
able to use stable constraint for Tempest installation
as well as while running during run-tempest playbook.
tox.ini is hard coded to use master constraint[1] which force
run-tempest to recreate the tox env and use the master constraint.
Devstack can set stable u-c to use via env var but this is
not enough as Tempest role run-tempest recreate the tox
env with master u-c.
- https://review.opendev.org/#/c/705089/4/lib/tempest
Currently we are fixing run-tempest to use stable constraint
till stable/rocky which use distro having <py3.6.
Ideally, we can change compatible branch constraint in tox.ini
while doing release tag for Tempest. But that can be done
for future tags only.
[1] bc9fe8eca8/tox.ini (L14)
Change-Id: I4268049ff91e5527e4b95fd8a70fa53ba27d56ed
Devstack IPv6 base job 'devstack-IPv6' setup the IPv6 setting
to deploy the services to listen on IPv6 address.
Tempest 'devstack-tempest-ipv6' job derived from 'devstack-IPv6'
job adds the playbooks to run the tests.
As part of Train community goal 'Support IPv6-Only Deployments',
we will expand the 'devstack-tempest-ipv6' job to do
IPv6-only deployments verification.
This commit define the new roles of 'ipv6-only-deployments-verification'
which will be invoked as part of run phase of 'devstack-tempest-ipv6' job.
This role will do IPv6-only setting and deployments verification via
tools/verify-ipv6-only-deployement.sh
IPv6 verification script can be extended further to perform more checks
and via project specific test case. Those tests will run as part of project
specific child jobs.
The 'devstack-tempest-ipv6' job will be used as parent for project specific
IPv6-only job. Those child job can extend the project specific IPv6 verification
by defining new playebook for post-run. That way the base verification
done in 'devstack-tempest-ipv6' will still run in addition to project specific
verificaiton and tests run.
Verification structure will be:
- 'devstack-IPv6' deploy the service on IPv6
- 'devstack-tempest-ipv6' run will verify the IPv6-only setting and listen address
- Child jobs derived from 'devstack-tempest-ipv6' will run the IPv6 related test case or
any further IPv6 deployment verification.
This commit also adds the new job 'tempest-ipv6-only' which will run smoke
and ipv6 related tests present in Tempest. This job will be used to run
on 6 services (Nova, Neutron, Cinder, Keystone, Glance, Swift) deployed
by devstack.
Story: #2005477
Task: #35923
Change-Id: Ie1e5e5e02ee45ab7a4aae0f35bdb42afc2c3fe2e
Patrole needs access to global packages, but the 'all-plugin' is
deprecated[0] and 'all' defines a sitepackages option as False[1].
Adding tox_extra_args option makes it possible to configure Zuul and
pass --sitepackages argument to tox.
[0] 9b4c50cf63/tox.ini (L64)
[1] 9b4c50cf63/tox.ini (L7)
Change-Id: Ib3264f54075216471720483c8dd816887c1f413c
* https://review.openstack.org/547106 adds a black_regex flag to
tempest run. We can use the same in devstack-tempest base job
to skip/blacklist tests.
Change-Id: I9ed91b01ddb7ec0a3d23305031afbac7c4b16c55
The process-stackviz role only helps with analyzing runs, when it fails
it should not cause the whole job to suffer a POST_FAILURE.
Change-Id: Icfd6b2fc7adb42afb7900ea4458fa60cefc76fdb
Related-Bug: 1758054
Currently there is a zuul-jobs role only used in this tempest job while
other jobs mostly use fetch-subunit-output role.
This changes tries to consolidate towards the most used role, to be able
to clean up zuul-jobs a bit.
Change-Id: I404760ba34767660f8816f48712e13dccad67440
Depends-On: https://review.openstack.org/548406
Currently there is tempest_test_regex that can be used to select or
ignore tests, however when the list is complex it can become complex to
maintain and define. For such complex scenarios is that the blacklist
file exist.
This patch adds support for the blacklist file in Zuul jobs.
Change-Id: Ib63955446ea59d81d0d882704666dcce4a625526
The current default value of stage_dir does not match the valid
default from zuul-jobs, which causes stackviz to be skipped when
stage_dir is not overwritten, which is the case in all CI jobs.
Setting default and docs to "{{ ansible_user_dir }}".
Change-Id: I9fd3f8541c8e3fbc406c5e5c0c5fe8427bf74112
For consistency with the tox jobs, use tox_envlist instead of
tox_venvlist. See
https://docs.openstack.org/infra/zuul-jobs/jobs.html#job-tox for
definition of tox job.
All projects using tox_venvlist have been changed to use tox_envlist as
well (see dependencies). Now we can rename tox_venvlist to tox_envlist -
and then the projects can remove tox_venvlist and use only tox_envlist.
Change-Id: I67a20b1faa77ff0166ad96fd95432c11948abc74
Depends-On: I035584aaeaa71b62c1d0c7e0a62dbf053a3488b6
Depends-On: I2fb02ead3b45c2a05d28f59fd28d062a3b40e07e
Depends-On: Id03d4c1351d5a002dc9d73247d0315bef274f627
Depends-On: Iff3093e61c7d2cb684667d5bc5f2a75580569aab
It is possible to add multiline and docummented regex to the
tempest_test_regex but it might not be evident at first, although very
useful for some scenarios.
This patch documents how this should be achieved to make it easier for
people in the future.
Change-Id: Id54679debedb1667cc70004b9a28fba921e50c0b