This reverts commit f9d31f2c46.
Reason for revert: Tempest 26.0.0 is released so we need to switch to master constraints now.
Change-Id: I78780ef88c5f8394ec11b144f148f30b060ed657
We are going to release Tempest new tag 26.0.0 to declare
the end of support for stable/stein. So that new tag use
stable/victoria constraint in the tox env as victoria constraints
for sure will work with tag. It did not use wallaby constraints
as wallaby dev cycle is in progress and so does their constraints.
Further information:
https://docs.openstack.org/tempest/latest/requirement_upper_constraint_for_tempest.html
Change-Id: I31433c15984e38e5651b2f3d10c8769f59e30cc1
We need to move 'gabbi_tempest_path' var to gabbi-tempest plugin jobs side
which is the only place it is being used. This var is specific to gabbi-tempest
not used for any other devstack-tempest jobs so defining it in devstack-tempest
base job is not the correct way. We might face issue like bug#1821072 again in
future which can end up blocking all gate jobs.
Below patch is moving this var to gabbi-tempest plugin.
- https://review.opendev.org/#/c/705339/
Change-Id: Id96e9b3f1271fce50aae982fdf12871693107e1d
Closes-Bug: #1861592
The docs requirements migrated to doc/requirements.txt
we need not install things from requirements.txt.
Change-Id: I90212bdb5f2663851b02c58538c5b54e5c9bf4a9
We are going to release Tempest new tag to declare the start of support
for stable/victoria. So that new tag use stable/victoria constraint in
the tox env.
For example, if anyone use Tempest 25.0.0 in future, let's say 1 year
later then tox env also should use the victoria constraint for
compatibility instead of master.
These need to be move back to master constraint, once release is done.
By using master constraint in tox for all the old tag is problem and we
faced the issue of constraint incompatibility due to that and we have to
workaround it all over the devstack, tempest role, grenade to set the
stable constraint via env var.
Further information:
https://docs.openstack.org/tempest/latest/requirement_upper_constraint_for_tempest.html
Change-Id: I1f8c788327b4a1a7f804cf01f8c22609e6646dd9
Generate documentation for each test from its docstring.
The sphinx-apidoc tool is used to generate the .rst
files necessary for the generation of the documentation.
When the documentation is being build only functions that
start with "test" are passed to the resulting documentation.
Change-Id: I1878f67e10df9bb1f8289beda8e59a56b5057fe4
WHile releasing the new tag for Tempest, we modified the
constraint in tox to be used of stable/ussuri
- https://review.opendev.org/#/c/720578/
Now new tag is released and we can bring back the master
constraint in tox.ini so that Tempest master us master constraint.
Change-Id: Ie17d4830cac1902371123c8acb4d3106ba2fd300
We are going to releease Tempest new tag to declare the start of
support for stable/ussuri. So that new tag use stable/ussuri
constraint in the tox env.
For example: if anyone use Tempest 24.0.0 in future say 1 year later then
tox env also should use the ussuri contstraint for compatibility instead
master.
These need to be move back to master constraint once release is done.
By using master constraint in tox for all the old tag is problem and
we faced the issue of constraint incompatibility due to that and we
have to work around it all over the devstack, tempest role, grenade to
set the stable constraint cia env var.
- I4268049ff91e5527e4b95fd8a70fa53ba27d56ed
Change-Id: I39e3e23a85a9abc1eda6848a6f4af77540cb81b6
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.
Fix problems found.
Update local hacking checks for new flake8, remove vi check, it's now in
hacking as H106.
The tempest plugin is used on older branches as well.
We really only need hacking on this repo anyways,
where we no longer support python 2, so here we
make the requirement specific to python 3.
Change-Id: I3f4216f66606fbc450a46c93de306399b7f3cd65
If3ff71dbe636ef98072b15f6599e6e641c222f30 had switched this to python3.6
but this breaks on distros where 3.6 is no longer provided by default,
such as Fedora.
Closes-Bug: #1861308
Change-Id: I0e0b9e88eea2499cb1b9727a7787c050c869c819
While OpenStack is in transition to drop py2, many
lib or dependency of Tempest or its plugins cannot
be installed on python 2.7.
To install Tempest on py3 evnv, let's change the basepython
as python 3 for tox env.
Defining basepython as python3.6 for all venv using
.tox/tempest.
Depends-On: https://review.opendev.org/#/c/703476/
Depends-On: https://review.opendev.org/#/c/703679/
Closes-Bug: 1860033
Change-Id: If3ff71dbe636ef98072b15f6599e6e641c222f30
This commit add PDF building tox.ini environment and options for LaTeX
output. overview.rst is copied from README.rst. And, the original
README.rst file is shrunk because we don't need such a long information
in the README file. People can see the same contents in overview.rst
now.
Change-Id: Id654c814988e78704726d2ba8bea9a03ce8596f8
This commit makes two bashate rules E005 and E042 treat as error and
ignores E006. It is difficult to keep E006 rule because bash scripts
tend to be long naturally. Both E005 and E042 are good to be errors
since it should be safer.
These additions can be seen in devstack[0] too.
E005: file does not begin with #! or have a .sh prefix
E006: check for lines longer than 79 columns
E042: local declaration hides errors
[0] 0fc6b2c5a8/tox.ini (L35)
Change-Id: I56c017569aa8660648764502cce5adfe2e946a58
This commit introduces bashate[0] which is a code style checker for bash
scripts. We already have some bash scripts in Tempest repo. So, it might
be good to check the style automatically.
Some scripts under the tools directory are also fixed due to the bashate
violation.
Change-Id: I07820f10387552f93a9d8891b58a8c7fcdb83046
Devstack IPv6 base job 'devstack-IPv6' setup the IPv6 setting
to deploy the services to listen on IPv6 address.
Tempest 'devstack-tempest-ipv6' job derived from 'devstack-IPv6'
job adds the playbooks to run the tests.
As part of Train community goal 'Support IPv6-Only Deployments',
we will expand the 'devstack-tempest-ipv6' job to do
IPv6-only deployments verification.
This commit define the new roles of 'ipv6-only-deployments-verification'
which will be invoked as part of run phase of 'devstack-tempest-ipv6' job.
This role will do IPv6-only setting and deployments verification via
tools/verify-ipv6-only-deployement.sh
IPv6 verification script can be extended further to perform more checks
and via project specific test case. Those tests will run as part of project
specific child jobs.
The 'devstack-tempest-ipv6' job will be used as parent for project specific
IPv6-only job. Those child job can extend the project specific IPv6 verification
by defining new playebook for post-run. That way the base verification
done in 'devstack-tempest-ipv6' will still run in addition to project specific
verificaiton and tests run.
Verification structure will be:
- 'devstack-IPv6' deploy the service on IPv6
- 'devstack-tempest-ipv6' run will verify the IPv6-only setting and listen address
- Child jobs derived from 'devstack-tempest-ipv6' will run the IPv6 related test case or
any further IPv6 deployment verification.
This commit also adds the new job 'tempest-ipv6-only' which will run smoke
and ipv6 related tests present in Tempest. This job will be used to run
on 6 services (Nova, Neutron, Cinder, Keystone, Glance, Swift) deployed
by devstack.
Story: #2005477
Task: #35923
Change-Id: Ie1e5e5e02ee45ab7a4aae0f35bdb42afc2c3fe2e
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-object-storage'
gate template for swift gate which will run only Swift, Cinder and glance tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-object-storage template on swift gate.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: I70851d41174462b83cc8c525cecdb1b3d65e4ec7
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-compute'
gate template for nova gate which will run only nova, neutron, cinder
and glance tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-compute template on nova gate.
It will also help to disable the non-nova related serviecs and tests which
cannot be disabled from tempest-full job.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: I807d4cac6214395df292a6955686ea44677c3481
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-placement'
gate template for placement gate which will run only nova and neutron tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-placement template on placement gate.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: I200197e2fb57879aaeb382266d6309f0d1137b0c
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-storage'
gate template for cinder gate which will run Cinder, Glance, Swift and
Nova tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-storage template on cinder and glance gate.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: Iae7d26325228aad0edb94f7e191c8fd0d60e1362
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-networking'
gate template for neutron gate which will run only nova and neutron tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-networking template on neutron gate.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: Ifca8172f2f929105af4bad8ab72b05c2c45a5f58
Thorough replacement of git.openstack.org and review.openstack.org URLs
with their opendev.org counterparts.
Change-Id: I88e894db7b854d32593c770f5aa9b8a91fad7866
This commit sets a default value '-' for the tox tasks. Users can run
the tox tasks all, all-plugin and all-sitepackages without specifying
regexes. Without this patch, users face such an error when users run
the tox tasks without specifying regexes.
Change-Id: Ia523895e8f66f25e9a4103cb9be1f1c3fc3f1b62
This commit removes airship-tempest-plugin entry from the BLACKLIST
since the patch was already merged. And airship/tempest-plugin has a
gate job to verify it, too. There is no reason to store it in the
BLACKLIST anymore.
And this commit also make plugin-sanity-check's basepython python3
because we should use it as a default.
Change-Id: I9c50d327df65fecf8510f6f54f06e9f42da9bea0
autopep8 is an automated code formatting tool for python,
it does not know everything, also not super fast,
but it is faster than I can manually reformat the code.
tools/format.sh will call the formatting,
the tox will check did you called it.
Also adding a tox -eautopep8 way to use it.
autopep8 formats the lines in one way,
where tempest used a another way it was changed.
Change-Id: I6d51b14a5a5b87761071d0927fca23ba1651aa41
This PS will add debugger in tox.ini
So it will be easy to check the tests in debug mode
for failure and success too.
Change-Id: I604515a7a5fd369f15c215c75db4a0e91c93ca69
The gabbi-tempest plugin uses an environment variable,
GABBI_TEMPEST_PATH, to identify directories in which to
find the gabbi [1] tests that will be run. This will be
used by a forthcoming zuul job [2] (hosted by the plugin)
that will automate gabbi-based service testing as described
in https://anticdent.org/gabbi-in-the-gate.html .
By setting the environment in the devstack-tempest playbook
we avoid needing to duplicate the playbook: we can use
it directly and have less risk of plays diverging. The
calling job (which doesn't allow the "environment" key)
sets a "var" which then sets the environment variable.
[1] https://gabbi.readthedocs.io
[2] https://review.openstack.org/611678
Needed-By: https://review.openstack.org/611678
Change-Id: I5aa890baa489f434b95d21e4e618f5202ac550e4
The all-plugin tox env uses sitepackages so that plugins
installed outsite of Tempest virtual environment can be discovered.
After the implementation during the Queens release cycle of the
goal of moving Tempest plugins in dedicated git repos, this
environment should not be used anymore. "all" should be used
instead with the appropriate regex filtering.
The deprecation is introduced only in the form of a comment in
tox.ini and a WARNING echoed before the test run.
Patrole needs an environment with site-packages on, so create a
new one identical to the old all-plugin, but with a less confusing
name, for patrole to use while avoiding deprecation warnings.
Change-Id: Ib97fb3d0ce64b6faac6a190afc0006b624ce0a62
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.
We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.
We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.
Change-Id: I7cbf62cee124db959b1ac2698b6989c3e59a55ab
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
This commit bumps up the python version to 3.6. This change encourages
developers to use our latest python version on unit tests.
Change-Id: I45d79c90939aa0f743750fec4b4e286f3679563b
As per discussion on ML[1], we do not have any job which runs only
slow tests.
This commit provide a separate job which run only slow tests which can
be API or scenario tests. Further this job can be used cross projects
like nova, cinder, neutron.
Why we need separate job for slow tests:
- slow tests create gate issues on job timeout which we observed\
during Rocky cycle[2]. Not to effect gate let's start them executing
on separate job.
Why to convert the tempest-scenario-all job
- tempest-scenario-all job was created to run the slow tests along with
other scenario tests. But slow tests can be API tests[3] and mixing them
with other scenario or API tests is not good idea.
- All scenario tests excluding slow tests are run as part of tempest-full
job so let's not run scenario tests twice on gate.
[1] http://lists.openstack.org/pipermail/openstack-dev/2018-July/132527.html
[2] https://bugs.launchpad.net/tempest/+bug/1783405
[3] https://ethercalc.openstack.org/dorupfz6s9qt
Change-Id: I5b1d3041c0140bea11705ad287f1472e74aacb5c
hacking 1.0+ adds a couple of additional checks that are helpful as part
of the transition to Python 3. However, hacking 1.1.0 incorporated
flake8 2.6.x, which transitioned from pep8 to pycodestyle. Tempest
provides some hacking extensions and these use pep8 imports. As things
stand, projects that wish to use these hacking extensions must either
constrain themselves to the same older hacking versions or install pep8
manually. Neither of these approaches is particularly attractive so
instead transition to hacking 1.1.0 ourselves. This has the benefit of
giving us the additional checks for Python 3.
Change-Id: I09e3d2461366bb439bcb817149ce807a232acfbb
This commit switches to stestr in README.rst and the method name. We
already switched to use stestr instead of testr/ostestr. However,
a few are still remaining in README.rst and the unit test name. This
patch leaves some `testr` things because they are for the 'legacy run
method'. So, we don't need to remove them completely.
Change-Id: I0e8e19e2150d6e0ce9a299a10bdd51652d86497e
We stopped running all scenarios in parallel,
which leads to longer job times and less concurrent test run.
The full parallel run had too high failure rate,
this job is supposed to prove it is possible to return back to
the old way. If the issue(s) are still exists it will
produce usable artifacts for further debugging.
Change-Id: I776ebb7761d90d2490cf49f9df9b322c2b34da59
This commit fixes an oversight in the tox job definitions for unit
tests. Because for better or worse we still use oslotest as the base
test class for tempest's unit tests the stdout, stderr, and logging
capture fixtures are only enabled by an env var. We were never setting
that, so these things weren't captured. This commit fixes the oversight
to enable the capture of these once again.
Change-Id: I81e4077dfda9e9a8a8001f52955ab0164f39d31d
Ghanshyam