Commit Graph

56 Commits (master)

Author SHA1 Message Date
Ghanshyam Mann a3e09f7000 Add RBAC new defaults and scope job for stable/zed
stable/zed has the new defaults and scope available but
they were disabled by default. Let's test those by enabling
in stable/zed also so that we know if they work fine in
integrated way or not

Change-Id: Id2bd58854f60f38dccf0ef805de284c7e95db2f3
2 months ago
Zuul 32c3a09dca Merge "Add Tempest gate job for stable/2023.1" 3 months ago
Ghanshyam Mann 9eab9073dd Add Tempest gate job for stable/2023.1
Now we have stable/2023.1 branch ready for devstack
and so does for all service projects.

This commit adds the Tempest testing for stable/2023.1
by adding new jobs running on stable/2023.1 version of

Change-Id: I5994c6fb56e01c9697d4d80bcbff2939ef2eff04
3 months ago
Ghanshyam Mann 1062a8ddd4 Add periodic job to experimental queue
Adding a few missing periodic jobs to experimental
queue so that we can run then on demand also.

Change-Id: Ice57829388d379e53811186a2e3c3ebb9feebaa6
3 months ago
Zuul 1632cf5085 Merge "Revert "Mark tempest-multinode-full-py3 as n-v"" 4 months ago
Zuul 5e9e677d8f Merge "Separate the extra tests to run in a separate job" 4 months ago
Zuul f88e635d39 Merge "Prepare tempest-slow-parallel job and run periodically" 4 months ago
Zuul 614b326bde Merge "Fix retry_bad_request() context manager" 4 months ago
Ghanshyam 61194b73de Revert "Mark tempest-multinode-full-py3 as n-v"
This reverts commit 517563fde8.

Reason for revert: we did few improvement in job/tests run, let's see if timeout is solved or less frequent.

Change-Id: Ib29fb6d817b801e1472fc22fe761a2309f473dc0
4 months ago
Ghanshyam Mann 518e426ab4 Separate the extra tests to run in a separate job
Recently we are seeing a lot of job timeout(bug#2004780)
and we see many tests taking time and also number of tests
increasing over time. This is to prepare the list of extra tests
(here extra tests means the tests which are covered by the other
API or scenario tests) which we do not need to run in every
integrated jobs. Instead, we can run them in a separete job(s).

Currently I am adding admin (except keystone) and negative tests
in the 'extra tests' list but we can add more tests here which
we think are covered in some other tests.

As negative tests are important for interop, adding those extra
tests coverage for stable branch job also but running them in
periodic run only.

Related-Bug: #2004780
Change-Id: Id02221df0d6180519751c63e890851bd59fdafa0
4 months ago
Ghanshyam Mann 6bb98c2aa4 Prepare tempest-slow-parallel job and run periodically
tempest-slow-py3 job run all the slow test serially which
takes lot of time and end up job timeout. This preparing
tempest-slow-parallel job which will run slow tests parallelly
in periodic run. Based on the results, later we can make
tempest-slow-py3 job to run tests in parallel.

Also, run tempest-full-parallel in periodic and based on the
result we can run tempest-full-py3 job scenario tests in parallel.

Relavant-Bug: #2004780

Change-Id: I876dacb40daa384cddc8faae3200cd3d39506ddc
4 months ago
Ghanshyam Mann a9bad00512 Move a few jobs to periodic
We have a few jobs running gate even they are non voting
and run in periodic as well. They do not need to run in
every change but running/checking them in periodic run
is enough coverage. Below are the jobs moving to periodic:

* tempest-full-py3-ipv6
We do run tempest-ipv6-only job as voting in gate to
cover the ipv6 run and tempest-full-py3-ipv6 job can
run perdiocially to test the full tempest on ipv6.

we already discussed and agreed in TC also that centos stream
testing is best effort and can be in periodic or non voting.

Checking if tempest can be run without admin in periodic and not
on every change is enough.

* tempest-full-yoga
We do run all supported stable branch jobs periodically and running only
latest and oldest supported in check pipeline should be enough to catch
any breaking change on stable branches.

Relavant-Bug: #2004780
Change-Id: I8a2da7288e3f2264ce3cc39115c1d807b21fff95
4 months ago
Dan Smith 1ae54e331e Fix retry_bad_request() context manager
When I converted this from a decorator to a context manager, I didn't
remove the fn parameter, nor was it even doing the thing I expected
on the second and later iterations because it doesn't fail in my local
environment. Apparently we're not running this test in
the tempest gate, so this adds to experimental so we can at least
run it on command.

Change-Id: Ia72b50f7f7bf64fe0ddd3f1a415b1807ff264b66
4 months ago
Martin Kopec 517563fde8 Mark tempest-multinode-full-py3 as n-v
The commit marks the job temporarily as non-voting to unblock the
CI. There are multiple patches waiting to be merged which address
other bugs (e.g. the timeout issues) and this job fails at ~50%
rate which makes the merging of other patches comlicated.

Related-Bug: #1998916
Change-Id: I4ef3a6e5c4bbef93d355bfa42589fdb60db43663
4 months ago
Zuul b74ee598b0 Merge "End support for stable/wallaby" 6 months ago
Ghanshyam Mann 5844fd9aeb End support for stable/wallaby
This marks the end of support for stable/wallaby
which is EM now [1]. This remove the stable/wallaby
jobs and also adds relesenote to mark the end of
support of Wallaby.



Change-Id: Ie3bbbc80e4b2e7bca8e8b70fce1d31166220c6d3
6 months ago
Ghanshyam Mann 7ab45a9be8 Add new tempest job enable the rbac scope checks and new defaults
We have many services (Nova, Neutron, Glance etc) implemented the
new RBAC (project scope and project personas). For these services,
all tests should pass as projects personas (project reader) does
not impact existing testing/usage.

keystone has system scope adopted in their policy for now which
we need to make it work for project scope also and until then
we will see test failing.

This commit adds a new tempest full job which enable the scope
and new defaults of RBAC for applicable services.


Change-Id: Ib8f2f0e25205edba332fb9bd2a73012016d45061
6 months ago
Ghanshyam Mann 24a3a3667a Migrate tempest jobs to Ubuntu Jammy (22.04)
As per 2023.1 cycle testing runtime and community goal[1] we need to migrate
upstream CI/CD to Ubuntu Jammy(22.04). Most of the Tempest jobs will be migrate
automatically once devstack base job start running on Jammy(Depends-On).
As we need to run at least one job on Ubuntu Focal(20.04), I am adding
a new job tempest-full-ubuntu-focal.

Few job like tempest-full-multinode-py3 and plugin sanity job needs to override
the nodeset. multinode job needs to run on Focal till stable/zed and on Jammy
from master(2023.1) onwards. Stable branch jobs testing stable branch needs to
keep running on their supported distro version which is Focal.



Change-Id: I2e37603b098880b778100613800496b01167b22f
7 months ago
Zuul 461cab98c0 Merge "Add tempest-multinode-full-py3 job in gate pipeline" 8 months ago
Zuul b0313aa43e Merge "Add nova-multi-cell job in experimental pipeline" 8 months ago
Martin Kopec f1d0e395e9 Add Tempest gate job for stable/zed
Now we have stable/zed branch ready for devstack and so does
for all service projects.

This commit adds the Tempest testing for stable/zed by adding
new jobs running on stable/zed version of openstack.

Change-Id: I6a331a6f2466aa4e4d9f607660b87fa60a63bf4b
8 months ago
ghanshyam mann 3e1d5df6a5 Add tempest-multinode-full-py3 job in gate pipeline
tempest-multinode-full-py3 is voting job running
in check pipeline only, we should run it in gate
pipeline also.

Change-Id: Ifdd7e48b31ef29c54b2bfa805e1eb3197589f925
9 months ago
ghanshyam mann 9a9d5883c5 Add nova-multi-cell job in experimental pipeline
Nova does not support few operation for exmaple
migration, unshelve etc cross cell and we need to
skip such operation testing for cross cell job.
nova-multi-cell job is multi cell job which will be
helpful to check the new tests in tempest.

Adding it in experimental queue so that we can run it
on demand basis.

Change-Id: Ib59f00f78921afd00a7dce26cfd453288eea462c
9 months ago
Zuul 4e994020d0 Merge "Test fips job on centos-9" 10 months ago
Balazs Gibizer 5bab06a031 Make nova-live-migration voting
We merged a broken test case in Icdb3398061d0274fec8b4e10d955a4b00edf4524
as nova-live-migration job was non voting and we missed that the job
failed on the patch.

Let's make the job voting to avoid this in the future.

Change-Id: Ibe8c439da6c1ab6657b6f2618c3a72a074e24305
12 months ago
Ade Lee 0687b9cf39 Test fips job on centos-9
Added FIPS experimental jobs for centos-8 and centos-9.

Change-Id: I42208f82b995f67f45c733e76d6d285a1faf9f95
12 months ago
Ghanshyam Mann 78f5439b73 Add py310 job in gate
As 3.10 is next python version we will be testing in
next cycle (after Zed), let's add it in our testing in advance.

Change-Id: I1d60c21473c7b657a3f4ad57525c4561f2cfcfd5
1 year ago
Ghanshyam Mann 87661fcef7 Drop py3.6 and py3.7 from Tempest
In zed cycle, OpenStack projects and oslo lib
has dropped the py3.6 and py3.7 support


With oslo not supporting the py3.6 and py3.7, Tempest master is
failing and cannot be run on py3.6|7 env.


We discussed about how to solve it in ML[1] and QA meeting[2] and
decided to drop it from Tempest too. Stable branches testing from
stable/victoria to stable/yoga (currently supported branch by Tempest
master) can be tested on py3.8 or if cloud run those releases on py3.6
then Tempest can be run in virtual env with py38.

Closes-Bug: #1975036


Change-Id: Ie844887a798a006794729e733748ceaa5e06d2db
1 year ago
Ghanshyam Mann 263825a17a End support for stable/victoria
This marks the end of support for stable/victoria
which is EM now[1]. This remove the stable/victoria
jobs and also adds relesenote to mark the end of
support of Victoria.


Change-Id: Ia63cdc682a32a2bcfd748f924368db6028b64d8c
1 year ago
Ghanshyam Mann 2c3a6381cb Make tempest plugin sanity job voting again
murano-tempest-plugin fix is proposed (depends on) so
making plugin sanity job voting again.

Change-Id: Iefd62a74b201968ca1987d182e086b9f4de6fb51
1 year ago
Ghanshyam Mann b1ea432710 Unblock gate to temporary remove the py3.6|7 job
py3.6 and py3.7 jobs are failing due to oslo dropped the
py3.6|7 support. We will add them back once bug#1975036
is resolved

Tempest plugin sanity job is broken due to failure in
murano-tempest-plugin. To make this patch merge which
will unblock the gate, I am making plugin sanity job as
non voting. We will revert that once below fix is merged

tempest-full-centos-9-stream job is also failing which fix
is also up and waiting for this patch to merge first, so making
them non voting.

devstack-plugin-ceph-tempest-py3 job is also mostly failing, let's
make it temporarily non voting in order to avoid endless rechecks


Related-Bug: #1975036
Change-Id: Ibcaf1bce80521e05d76199ddebccb33488360644
1 year ago
Ghanshyam Mann e0c0fbe4c4 Add tempest-slow stable branch jobs on Tempest master gate
To cover the stable branch testing on branchless Tempest
master gate, we test tempest-full version of stable branches.
Let's test the tempest-slow job also for stables branches.

All supported stable branches jobs are added in periodic
stable pipeline. As slow jobs are slow, it will make check pipelines
too heavy if we add all the stable branches slow jobs so
I am adding slow jobs only for the oldest supported stable job
which should cover the tempest master working on most
of the stable.

This will cover the Tempest slow tests stability for stable branches.

Change-Id: Idb8f178fa4fb4b669f3a163d5c5a9517b9d43c9a
1 year ago
Ghanshyam Mann 3ed52100fd Drop centos 8 stream testing
In Zed cycle testing runtime, we are targetting the centos 9 stream

With dropping the python 3.6 support, project started adding python 3.8
as minimum, example nova:
- 56b5aed08c/setup.cfg (L13)

with that, centos 8 stream job is failing 100%

This commit drops centos-8-stream testing and move their job definition
to stable/jobs.yaml file as they are being used on stable branch testing.

Change-Id: Ie72eaaf30ee15710f9fec266e93a6c8ad4eb5ddc
1 year ago
Ghanshyam Mann 6ffb4fa7fb End support for stable/ussuri
This marks the end of support for stable/ussuri
which is EM now[1]. This remove the stable/ussuri
jobs which are failing due to bug#1968798 and also
adds relesenote to mark the end of support of Ussuri.


Change-Id: Ieeb4da8859d44470c5cb67a62f2dcfd7786daf3a
1 year ago
Martin Kopec e74b7e2cf5 Add Tempest gate job for stable/yoga
Now we have stable/yoga branch ready for devstack and so does
for all service projects.

This commit adds the Tempest testing for stable/yoga by adding
new jobs running on stable/yoga version of openstack.

Change-Id: Id731c26a0f134428094e8d830fb615695dffe10b
1 year ago
Ghanshyam Mann 7304e3ac89 Move ServerStableDeviceRescueTest to wait for SSH-able server
ServerStableDeviceRescueTest also perform the attach_volume
on rescue server and in cleanup detach_volume. As described in
the bug#1960346 we need to wait for server readiness before
detach volume called.

Also making centos stream 9 job as voting.

Closes-Bug: #1960346
Change-Id: Ia213297b13f42d39213dea9a3b2cfee561cdcf28
1 year ago
Ghanshyam Mann 952fe9c76d Add tempest-full-centos-9-stream non voting job
centos 9 stream is testing runtime for Yoga so
let's add it in tempest gate. This job is failing
due to bug#1960346 and it will be helpful to know the
job status when we add the fixes with below series:

Related-Bug: #1960346
Change-Id: Ib91f67fb9a592e91649e06bb2821f452b36b49f0
1 year ago
Ghanshyam Mann eb1afd539a Make stable/ussuri job voting
stable/ussuri job was made non voting due to


As bug is fixed now, let's revert it to voting again.

Change-Id: I834300ee8d9ee8952284b55f53d97456a1ca6032
1 year ago
yatinkarel 115065a239 Fix TypeError in QoSBandwidthAndPacketRateTests
[1] Added random int to flavor name but that resulted into
TypeError: can only concatenate str (not "int") to str.

Concatenate using % operator to fix it.

Also make tempest-full-ussuri-py3 job non-voting temporary
to unblock gate, can be reverted once [2] is fixed.


Related-Bug: #1959467
Change-Id: I76e15b6b52bb44fec550ead510186c644c4a1c7f
1 year ago
Ghanshyam Mann e6b7056d84 Make broken tempest-full-py3-opensuse15 job only until stable/xena
tempest-full-py3-opensuse15 is failing all the time and
opensuse is not tested/supported distro in testing runtime
or devstack anymore. These job is removed from master gate
but it is in use until stable/xena.


Modifying it for until stable/xena only.

Change-Id: Ibe9166091011c08644cba783b7db77124cb17417
1 year ago
Ade Lee 6ded070b51 Add support for ecdsa keys
In FIPS mode, using RSA keys for ssh is fine as long as SHA-1 is
not used for the signature algorithm.  Unfortunately, the version
of cirros used in OpenStack CI does not have a version of dropbear
that supports SHA-2 signatures.  So, any connections from a FIPS
enabled machine will fail as the cirros instance will only support
ssh-rsa (SHA-1 signatures).

To get around this, we add a new option to specify the key type
(validation.ssh_key_type).  This will allow the addition of other
key types in future if needed.

Tempest now supports 'rsa' and 'ecdsa' key types.

We also add a fips job to the experimental queue to test the usage
of the new key type.

Change-Id: Ib59eb8432fa1a2813b3047955157d1b3d24a55f8
1 year ago
Ghanshyam Mann 59f6aa6a90 Remove stable/train jobs from master gate
stable/train is in EM state and tempest master
does not support stable/train. depends on patch
also cap the tempest on stable/train testing.


Change-Id: Ia298e20d1db77f307c850f90741d5fb46cb87ae0
1 year ago
Zuul 79da6961ed Merge "Test all supported python version for stable branch" 2 years ago
Lee Yarwood 0dba41c17a zuul: Add a CentOS 8 stream job to integrated-gate-compute
As set out in the governance repo for Xena CentOS 8 stream with py36 is
a supported platform and runtime for the release:

As a result the Nova team want to run CentOS 8 stream based jobs within
the integrated-gate-compute template.

An additional tempest-full-py3-centos-8-stream job is added to Tempest's
check and gate queues to ensure coverage here.

Both jobs are given additional swap to workaround bug #1949606, a
behaviour change in QEMU >= 5.0.0 when using [libvirt]virt_type=qemu
that causes additional memory to be consumed by each running instance.

This workaround of additional swap will be removed in the future once
Nova is able to workaround this itself through a new libvirt domain

Change-Id: I4b074dca58960b2622dde15ad7d9a91b3ba0c096
2 years ago
Ghanshyam Mann 1aac05c7b2 Test all supported python version for stable branch
As Tempest is branchless and needs to support older
supported stable branches, let's test it with all the
python version supported in those supported stable branches.
For that, we need to explicitly dd the py jobs instead of using
the template. This way, we will explicitly know that we are testing
all supported python versions when the new release template bumps the
min python version in the template.

In Yoga testing, we are making py3.9 job to run as voting, so
adding py3.9 job as voting.

Change-Id: I422da8da06509e529f4091e21cdc3d392d3cf910
2 years ago
Ghanshyam Mann eb8bd6ac81 Add Tempest gate job for stable/xena
Now we have stable/xena branch ready for devstack
and so does for all service projects.

This commit adds the Tempest testing for stable/xena
by adding new jobs running on stable/xena version of

Change-Id: If6f188aebe48f17d8a343c958cdeb5507421a40b
2 years ago
Ghanshyam Mann ba28d78c30 Add experimental job to run test with 'latest' microversion
API microversions are inhertied in nature from features points
of view, means higher microversion will have all the features/
changes done in lower microversion.

In Tempest we write the microversion tests by capping
the min and max microversion so that they can request
the API with correct microversion.

But for non microversion tests we do not test if they are
run-able for all the configured microversion in Tempest config
file. To test it at some extend this commit adds a experimental
job to run the API tests (compute and volume) with 'latest'
microversion to know if that run successfully or need modification.

This job is experimental as now as I expect lot of test failure
with the 'latest' microversion and as we keep fixing those and all
the tests pass then we can move it to voting job in check/gate pipeline.

Change-Id: Ie840dd0997519d7e11a661853f9376f43c6e7905
2 years ago
Ghanshyam Mann acf37cf89b Avoid running tempest-ipv6-only for unrelated tools files
Currently tempest-ipv6-only job run on all the files change
present in /tools dir which is not required. For example
when we update the tempest plugin sanity scripts present in
/tools dir then this job run which is unneccessary.

Optimizing the tools dir file as per related job required to run.

Change-Id: Ia1d8a8a6dd22bec02e9f93598ddfd3d110b6c009
2 years ago
Rodolfo Alonso Hernandez 354c57df5d Make explicit the network backend used in the CI jobs
All Neutron CI jobs (except for unit, functional and fullstack jobs),
have explicitly defined the network backend used:
- linuxbridge
- ovs
- ovn

That was discussed and approved during the Neutron CI meetings [1].


Change-Id: I4d18ef3e3adf24cf646e0546e7c38145f5a4aa92
2 years ago
Ghanshyam Mann 2cd8f00457 Remove non existing puppet-openstack-integration from check pipeline
puppet-openstack-integration* jobs are not defined anymore in
puppet-openstack-integration repo

and they does not run in check pipeline even raise config error on
modifying the project.yaml


Let's remove those and if new puppet jobs are required to run
in Tempest gate then we can add the correct one.

Change-Id: Icbc5670320682a7071e957119919247497733c9b
2 years ago