tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-object-storage'
gate template for swift gate which will run only Swift, Cinder and glance tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-object-storage template on swift gate.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: I70851d41174462b83cc8c525cecdb1b3d65e4ec7
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-compute'
gate template for nova gate which will run only nova, neutron, cinder
and glance tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-compute template on nova gate.
It will also help to disable the non-nova related serviecs and tests which
cannot be disabled from tempest-full job.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: I807d4cac6214395df292a6955686ea44677c3481
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-placement'
gate template for placement gate which will run only nova and neutron tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-placement template on placement gate.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: I200197e2fb57879aaeb382266d6309f0d1137b0c
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-storage'
gate template for cinder gate which will run Cinder, Glance, Swift and
Nova tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-storage template on cinder and glance gate.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: Iae7d26325228aad0edb94f7e191c8fd0d60e1362
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-networking'
gate template for neutron gate which will run only nova and neutron tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-networking template on neutron gate.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: Ifca8172f2f929105af4bad8ab72b05c2c45a5f58
This commit removes x/networking-ansible from the tempest plugins list.
The original patch[0] was abandoned because networking-ansible took a
different approach[1] for the testing. So, it's not a tempest plugin
anymore.
[0] https://review.opendev.org/#/c/634647/
[1] https://review.opendev.org/#/c/634918/
Change-Id: I554c438f27a7032f31e56eac2aac110e08bb3d7d
This commit changes to publish non-openstack namespace tempest plugins
to the tempest registry doc page. We missed some tempest plugins such as
airship's tempest-plugin because of the recent namespace change. And
also, this commit adds two projects[0] to .zuul.yaml because it had
been added already.
This commit republish them on the doc page, and add two projects to the
blacklist to pass the sanity tests.
Change-Id: Ieecb8989ff8959f1b2c0e5f2f6b8ea2c3ebbcaa9
The nova-cells-v1 job was removed from nova in the Train
release with change I2031bf2cc914d134e5ed61156d2cfc621f006e0b
so we can remove it from tempest's experimental queue.
Change-Id: Ie759315bf7a4440eaae91b5ee4b672234692dd25
This commit adds opendev.org/openstack/cyborg-tempest-plugin to
required-projects to pass the plugin-sanity-check job. It's a new
tempest plugin. It doesn't work properly on master yet. So, I put it on
the BLACKLIST.
And this commit also removes octavia because octavia tempest tests were
already migrated to octavia-tempest-plugin, and there is no tempest test
in the octavia repo. The removal patch[0] in octavia was already merged.
[0] https://review.opendev.org/#/c/659516/
Change-Id: Ied13dbf774472e1c36dca4ccca157104a81fbf9d
currently we do not run the gate jobs if any change is for
test|requirements.txt only which can break the things.
Example- https://review.openstack.org/#/c/649804/
Because Tempest does not have functional testing which might
be enough for teting the requirement change, we start running
the all gate jobs for such change.
Change-Id: I54384f5c664673ef32937cd1d7c4a46c10c18f9b
Now we have stable/stein branch ready for devstack
and so does for all service projects.
This commit adds the Tempest testing for stable/stein
by adding new jobs running on stable/stein version of
openstack.
Depends-On: https://review.openstack.org/#/c/647867/
Change-Id: Ia22988821b5c786a0b25a980cb4ca329dac54bac
Nova has migrated the legacy-tempest-dsvm-nova-v20-api
job to zuulv3 native - "nova-tempest-v2-api"
This commit use the new job in tempest gate.
Change-Id: I81a07d8fbb529b0ac55ce40c50a6e5348e1f43df
Before change Icb971831c8d4fe5f940d9e7993d53f1c3765e30f in devstack
in Rocky, the ENABLE_VOLUME_MULTIATTACH variable is used to control
whether or not the UCA is enabled on the node and defaults to False.
The tempest-slow job is multi-node and runs on stable branches since
change Ie5492603a12ca4ae56baa744bf650ae4d8036256 and as a result
is configuring the UCA on the subnode but not the controller, so
nova-compute on each node is running with different versions of QEMU
which can cause live migration failures if trying to live migrate
from a newer version of QEMU to an older version.
This fixes the issue by configuring the subnode with the correct
localrc value for ENABLE_VOLUME_MULTIATTACH.
Change-Id: Idea94dde6973805995c1678fcfb6b0e93025492a
Closes-Bug: #1819910
Below nodesets defined in Tempest are removed now:
- openstack-bionic-node This is unused after I5a24c697103090f6712d6e9168abe6c86f7c380c
- openstack-opensuse150-node is duplicate of devstack-single-node-opensuse-150
Depends-On: https://review.openstack.org/#/c/641746/
Change-Id: I39c03bfa52191a0b1acf6a73fa31b3c071ee60c1
This commit makes tempest-tox-plugin-sanity-check job
to run on Bionic. This job only run on master Tempest gate
so no need to worry about its xenial version.
Change-Id: I6d8c63a62416b35e9d203cbeabc9868aa6a95f44
tempest-slow has been added for stable branches also
- I5cb829beebcc59de682cba927f5b01793ec70d00
let's make this job definition to allow to run on stable branches also.
Change-Id: Ie5492603a12ca4ae56baa744bf650ae4d8036256
tempest-multinode-full should be available to run on
all stable branch not just master. All the devstack
changes are backported already.
tempest-multinode-full will use Bionic node from
stein onwards and xenial for all other stable branches.
Change-Id: I94f66028df58a69aa7c26e8a03428bf48596d25c
* To use a fresh venv, use `--clear` option when creating a virtualenv
* Removes the project as the script fails to uninstall the dependencies.
* Added missing tempest plugins
* Log the stdout of each plugin in a file
* Adds some plugins to BLACKLIST to avoid errors
Change-Id: I5068d96105a22cbfd33f5900f4bdc21b47a65c06
The volume multiattach tests originally required special
devstack configuration in Queens for the Ubuntu Cloud Archive,
but that is no longer necessary. This change enables the volume
multiattach tests in the tempest-full(-py3) and tempest-slow jobs
so we can drop the nova-multiattach job, which is mostly redundant
coverage of the tempest.api.compute.* tests, and reduce the total
number of jobs we run against nova/cinder/tempest changes.
Due to intermittent bug 1807723 when running the
test_volume_swap_with_multiattach test with two compute services,
that test is conditionally skipped if there is more than one compute.
This is probably no great loss in test coverage for now given
swapping multiattach volumes is likely rarely used (see bug 1775418).
Related ML thread:
http://lists.openstack.org/pipermail/openstack-dev/2018-October/135299.html
Depends-On: https://review.openstack.org/634977
Change-Id: I522a15ba3dbfee5d8ef417e43288a12319abf6ff
'interop-tempest-consistency' job
has been added to check the test renaming
compatibility with interop usage.
This job was added as n-v to check the stability
and it is running fine since it was added so
making this as voting job in check pipeline.
Change-Id: I088f64d87b04b5db2319ccd3b3d02821a06f35e9
This job is based on tempest-slow job but it runs
on Python 3.
This patch also removes tempest-slow job from gate pipeline.
Running only tempest-slow-py3 job there is sufficient.
To keep py2 coverage tempest-slow job will be still run
in check queue together with tempest-slow-py3.
Change-Id: I9fad35c6327fac30a25e940f7fec8515995c68c8
'integrated-gate-py35' template is going to be
renamed to 'integrated-gate-py3' in https://review.openstack.org/#/c/626078/
Integrated jobs are running on Bionic now where python 3.6 is available.
Which means gate jobs in 'integrated-gate-py35' template are
running on python 3.6 not on 3.5 which makes this template name confusing.
depends on commit rename the 'integrated-gate-py35' to 'integrated-gate-py3'
so that it can convey that template will use available python 3 version
in used distro. For example: 3.5 in xenial and 3.6 in bionic.
This commit starts using the new template name so that old
template name can be removed.
Depends-On: https://review.openstack.org/#/c/626078/
Change-Id: Ic5264b3b79d0a4caaa7f466f8686d38f54fe1ffe
Adding irrelevant-files for new job grenade-py3 to avoid
running this integrated testing job on doc or unit test changes.
Change-Id: I2918b6563a65729e70323ca5986fc7a66e2bcc94
Nova is moving the nova-cells-v1 job to its experimental
queue so we should do the same in Tempest.
Depends-On: https://review.openstack.org/623538
Change-Id: I277f3a4bea69501ea4c07118c552343d5ea38f6a
Related-Bug: #1807407
All the base jobs provided by Tempest are migrated to run in bionic
from stein onwards. As Tempest is branchless, we need to keep running or
available those base job running on xenial for stein before which is
stable/rocky, stable/queens and stable/pike.
The selection of the correct nodeset for this happens mostly
automatically when we avoid selecting nodesets explicitly because we
inherit them from the correct branches of devstack which have the
matching job variants defined.
There are only two places left where we need to explicitly override the
default nodesets:
1. tempest-multinode-full
Here we need to select a nodeset with two nodes. Since this job runs
only on master, we simply select two Bionic nodes instead of two Xenial
nodes now.
2. Stable branch testing jobs like tempest-full-rocky
These do not use branch selectors, but "override-checkout" instead, so
they inherit Bionic nodesets from devstack master branch. We need to
force them to run on Xenial for backwards compatibility.
Drop the tempest-full-py36 job because it will be identical to the
tempest-full-py3 job now.
Depends-On: https://review.openstack.org/610977
Depends-On: https://review.openstack.org/611594
Change-Id: I5a24c697103090f6712d6e9168abe6c86f7c380c
Currently, the sanity plugin check job is failing because
of errors like the following:
ERROR! /home/zuul/src/git.openstack.org/openstack/cinder-tempest-plugin not found
In Zuul v3 all repositories used need to be declared
in the 'required-projects' parameter on the job.
To fix this issue, add:
openstack/cinder-tempest-plugin [0]
So this adds missing projects like tempest plugin repos to get
the job passing.
Included in this patch set:
* Add missing repos to required-projects
* Blacklist networking-plumgrid since it is failing because
it hasn't been supported in years [1]
* Remove trio2o from required-projects because it is already
blacklisted
* Add irrelevant-files for sanity check job
[0] http://logs.openstack.org/96/615396/4/check/tempest-tox-plugin-sanity-check/a18e9a9/ara-report/
[1] https://github.com/openstack/networking-plumgrid
Change-Id: Ifee972d87ae83759000db925449764a60a98f7ba