This commit removes barbican-tempest-plugin from the blacklist. The
patch[1] got already merged. The plugin should work for the sanity now.
[1] https://review.opendev.org/#/c/634631/
Change-Id: I34d119c58afa6b3c728d1a540b4ea1df27d9c4d0
This commit adds x/group-based-policy plugin to the blacklist. It looks
this plugin doesn't support python3 yet.
Change-Id: I2f97f2245434d21b83c790efd37a7092ef1bfeb2
This commit retries http requests to generate a plugins list. We often
see internal server errors(500) in the gate. It makes sanity jobs fail.
So, it would be better to retry internally instead of 'recheck'. To
accomplish it, this uses urllib3 module.
NOTE: The default total number of retries[0] is 10. It should be enough
for us.
[0] https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.retry.Retry
Change-Id: Id3da3e2895fc2324d5a2db91f82b12ec5bb610f8
This commit makes two bashate rules E005 and E042 treat as error and
ignores E006. It is difficult to keep E006 rule because bash scripts
tend to be long naturally. Both E005 and E042 are good to be errors
since it should be safer.
These additions can be seen in devstack[0] too.
E005: file does not begin with #! or have a .sh prefix
E006: check for lines longer than 79 columns
E042: local declaration hides errors
[0] 0fc6b2c5a8/tox.ini (L35)
Change-Id: I56c017569aa8660648764502cce5adfe2e946a58
This commit removes x/ranger-tempest-plugin from BLACKLIST. The patch[0]
is trying to support the tempest-sanity job. So, when the patch is
merged, we can remove it from BLACKLIST.
[0] https://review.opendev.org/#/c/672585/
Change-Id: Ia7f0bdad9f854c0f6724537e1ca0234f2952aca8
Depends-On: https://review.opendev.org/672585/
This commit introduces bashate[0] which is a code style checker for bash
scripts. We already have some bash scripts in Tempest repo. So, it might
be good to check the style automatically.
Some scripts under the tools directory are also fixed due to the bashate
violation.
Change-Id: I07820f10387552f93a9d8891b58a8c7fcdb83046
Devstack IPv6 base job 'devstack-IPv6' setup the IPv6 setting
to deploy the services to listen on IPv6 address.
Tempest 'devstack-tempest-ipv6' job derived from 'devstack-IPv6'
job adds the playbooks to run the tests.
As part of Train community goal 'Support IPv6-Only Deployments',
we will expand the 'devstack-tempest-ipv6' job to do
IPv6-only deployments verification.
This commit define the new roles of 'ipv6-only-deployments-verification'
which will be invoked as part of run phase of 'devstack-tempest-ipv6' job.
This role will do IPv6-only setting and deployments verification via
tools/verify-ipv6-only-deployement.sh
IPv6 verification script can be extended further to perform more checks
and via project specific test case. Those tests will run as part of project
specific child jobs.
The 'devstack-tempest-ipv6' job will be used as parent for project specific
IPv6-only job. Those child job can extend the project specific IPv6 verification
by defining new playebook for post-run. That way the base verification
done in 'devstack-tempest-ipv6' will still run in addition to project specific
verificaiton and tests run.
Verification structure will be:
- 'devstack-IPv6' deploy the service on IPv6
- 'devstack-tempest-ipv6' run will verify the IPv6-only setting and listen address
- Child jobs derived from 'devstack-tempest-ipv6' will run the IPv6 related test case or
any further IPv6 deployment verification.
This commit also adds the new job 'tempest-ipv6-only' which will run smoke
and ipv6 related tests present in Tempest. This job will be used to run
on 6 services (Nova, Neutron, Cinder, Keystone, Glance, Swift) deployed
by devstack.
Story: #2005477
Task: #35923
Change-Id: Ie1e5e5e02ee45ab7a4aae0f35bdb42afc2c3fe2e
This commit adds ranger-tempest-plugin to the blacklist. The plugin
looks like very new and it looks it doesn't work correctly yet. So, it
might be better to skip to sanity for the plugin now.
Change-Id: I6d5d647d92916e5b62e8c67094d04f028e25ee67
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-object-storage'
gate template for swift gate which will run only Swift, Cinder and glance tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-object-storage template on swift gate.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: I70851d41174462b83cc8c525cecdb1b3d65e4ec7
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-compute'
gate template for nova gate which will run only nova, neutron, cinder
and glance tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-compute template on nova gate.
It will also help to disable the non-nova related serviecs and tests which
cannot be disabled from tempest-full job.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: I807d4cac6214395df292a6955686ea44677c3481
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-placement'
gate template for placement gate which will run only nova and neutron tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-placement template on placement gate.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: I200197e2fb57879aaeb382266d6309f0d1137b0c
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-storage'
gate template for cinder gate which will run Cinder, Glance, Swift and
Nova tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-storage template on cinder and glance gate.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: Iae7d26325228aad0edb94f7e191c8fd0d60e1362
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.
But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.
This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.
One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.
This commit propose the idea of defining the 'Integrated-gate-networking'
gate template for neutron gate which will run only nova and neutron tests.
ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html
Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-networking template on neutron gate.
[1] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: Ifca8172f2f929105af4bad8ab72b05c2c45a5f58
This commit removes the nova-lxd plugin from the blacklist. The proposed
patch was already merged, so this plugin should pass the sanity job now.
Change-Id: I72eb62af993bc978127185a1aed0763af18ce9f3
This commit removes x/networking-ansible from the tempest plugins list.
The original patch[0] was abandoned because networking-ansible took a
different approach[1] for the testing. So, it's not a tempest plugin
anymore.
[0] https://review.opendev.org/#/c/634647/
[1] https://review.opendev.org/#/c/634918/
Change-Id: I554c438f27a7032f31e56eac2aac110e08bb3d7d
This commit removes the cyborg-tempest-plugin entry from the BLACKLIST
since the patch was already merged and this plugin should pass the
sanity job now.
Change-Id: Ifec2e45761961bc4dfc3b5c11709c0ac9e81855f
This commit makes the tempest plugin function raise non 404 errors
when it catches them. Without doing that, we just ignore the errors.
However we shouldn't do that because it just causes another error when
the variable is used.
Change-Id: Ia82ef2aa9cb0959ccee1600feff297abb451176a
This commit fixes to add prefixes to the blacklist. The blacklist was
moved to generate-tempest-plugins-list.py by the recent patch[0].
However, the blacklist was updated by the other patch[1] before that.
So, it should have been updated correctly when the conflict was
occurred.
[0] I707aba930037e86f5f94dac73da957f5afacf935
[1] Ieecb8989ff8959f1b2c0e5f2f6b8ea2c3ebbcaa9
Change-Id: I090dd235dd7911aa9bb0eb13d8e65b501dc206fe
This commit makes the BLACKLIST to be published on the Tempest plugin
registry page which we discussed at the PTG[0].
[0] https://etherpad.openstack.org/p/qa-train-ptg
Change-Id: I707aba930037e86f5f94dac73da957f5afacf935
This commit changes to publish non-openstack namespace tempest plugins
to the tempest registry doc page. We missed some tempest plugins such as
airship's tempest-plugin because of the recent namespace change. And
also, this commit adds two projects[0] to .zuul.yaml because it had
been added already.
This commit republish them on the doc page, and add two projects to the
blacklist to pass the sanity tests.
Change-Id: Ieecb8989ff8959f1b2c0e5f2f6b8ea2c3ebbcaa9
This commit make passed and failed plugins be printed at the end of the
tempest-plugin-sanity execution. This is useful to ensure what plugins
were tested.
Change-Id: Iffd9ff6afd71c1d61322e5202b9861ec218cb574
This commit removes vitrage-tempest-plugin from the BLACKLIST since the
patch[0] was already merged.
NOTE: The patch on the comment[1] was duplicated of [0]. So it was
abandoned.
[0] https://review.opendev.org/#/c/638931/
Change-Id: I5539159bad10b3359981fab88b3be5e4e997fcb9
Thorough replacement of git.openstack.org and review.openstack.org URLs
with their opendev.org counterparts.
Change-Id: I88e894db7b854d32593c770f5aa9b8a91fad7866
This commit adds opendev.org/openstack/cyborg-tempest-plugin to
required-projects to pass the plugin-sanity-check job. It's a new
tempest plugin. It doesn't work properly on master yet. So, I put it on
the BLACKLIST.
And this commit also removes octavia because octavia tempest tests were
already migrated to octavia-tempest-plugin, and there is no tempest test
in the octavia repo. The removal patch[0] in octavia was already merged.
[0] https://review.opendev.org/#/c/659516/
Change-Id: Ied13dbf774472e1c36dca4ccca157104a81fbf9d
This commit removes airship-tempest-plugin entry from the BLACKLIST
since the patch was already merged. And airship/tempest-plugin has a
gate job to verify it, too. There is no reason to store it in the
BLACKLIST anymore.
And this commit also make plugin-sanity-check's basepython python3
because we should use it as a default.
Change-Id: I9c50d327df65fecf8510f6f54f06e9f42da9bea0
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.
This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.
This update should result in no functional change.
For more information see the thread at
http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html
Change-Id: Iff8f93426925a3480cedfb899216e443b12cd826
autopep8 is an automated code formatting tool for python,
it does not know everything, also not super fast,
but it is faster than I can manually reformat the code.
tools/format.sh will call the formatting,
the tox will check did you called it.
Also adding a tox -eautopep8 way to use it.
autopep8 formats the lines in one way,
where tempest used a another way it was changed.
Change-Id: I6d51b14a5a5b87761071d0927fca23ba1651aa41
* To use a fresh venv, use `--clear` option when creating a virtualenv
* Removes the project as the script fails to uninstall the dependencies.
* Added missing tempest plugins
* Log the stdout of each plugin in a file
* Adds some plugins to BLACKLIST to avoid errors
Change-Id: I5068d96105a22cbfd33f5900f4bdc21b47a65c06
Currently, the sanity plugin check job is failing because
of errors like the following:
ERROR! /home/zuul/src/git.openstack.org/openstack/cinder-tempest-plugin not found
In Zuul v3 all repositories used need to be declared
in the 'required-projects' parameter on the job.
To fix this issue, add:
openstack/cinder-tempest-plugin [0]
So this adds missing projects like tempest plugin repos to get
the job passing.
Included in this patch set:
* Add missing repos to required-projects
* Blacklist networking-plumgrid since it is failing because
it hasn't been supported in years [1]
* Remove trio2o from required-projects because it is already
blacklisted
* Add irrelevant-files for sanity check job
[0] http://logs.openstack.org/96/615396/4/check/tempest-tox-plugin-sanity-check/a18e9a9/ara-report/
[1] https://github.com/openstack/networking-plumgrid
Change-Id: Ifee972d87ae83759000db925449764a60a98f7ba
There might be chance that few plugin name and url
does not fit into hard coded column width and give error.
This commit count the column width as per plugins name.
Also display the number of plugins in new SR column.
Change-Id: Ia5736d2581ee3b6f851b8d08f855c2a19c157c47