283 Commits

Author SHA1 Message Date
Masayuki Igawa
6fca957b09
Remove barbican-tempest-plugin from blacklist
This commit removes barbican-tempest-plugin from the blacklist. The
patch[1] got already merged. The plugin should work for the sanity now.

[1] https://review.opendev.org/#/c/634631/

Change-Id: I34d119c58afa6b3c728d1a540b4ea1df27d9c4d0
2019-09-24 16:53:32 +09:00
Masayuki Igawa
8913b1db63
Add group-based-policy to blacklist
This commit adds x/group-based-policy plugin to the blacklist. It looks
this plugin doesn't support python3 yet.

Change-Id: I2f97f2245434d21b83c790efd37a7092ef1bfeb2
2019-08-29 11:21:28 +09:00
Zuul
ce682d0312 Merge "Fix Tempest Plugin Registry page for plugins url" 2019-08-16 19:12:06 +00:00
Zuul
22258f28e0 Merge "Remove x/ranger-tempest-plugin from BLACKLIST" 2019-08-15 22:40:17 +00:00
Zuul
b48cf5dea5 Merge "Retry http requests to generate plugins list" 2019-08-15 09:23:03 +00:00
saswati
61f03b7fed Fix Tempest Plugin Registry page for plugins url
There is extra 'openstack/' is added in each plugin url, for example:
- https://opendev.org/openstack/openstack/barbican-tempest-plugin
- https://opendev.org/openstack/x/trio2o

https://docs.openstack.org/tempest/latest/plugin-registry.html

Change-Id: Idc19bedeb171a91079008e79267304158504d932
2019-08-15 08:48:38 +08:00
Masayuki Igawa
411f4bbc5c
Retry http requests to generate plugins list
This commit retries http requests to generate a plugins list. We often
see internal server errors(500) in the gate. It makes sanity jobs fail.
So, it would be better to retry internally instead of 'recheck'. To
accomplish it, this uses urllib3 module.
NOTE: The default total number of retries[0] is 10. It should be enough
for us.

[0] https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.retry.Retry

Change-Id: Id3da3e2895fc2324d5a2db91f82b12ec5bb610f8
2019-08-08 18:24:40 +09:00
Masayuki Igawa
fdd964494a
Make bashate E005,E042 error and ignore E006
This commit makes two bashate rules E005 and E042 treat as error and
ignores E006. It is difficult to keep E006 rule because bash scripts
tend to be long naturally. Both E005 and E042 are good to be errors
since it should be safer.

These additions can be seen in devstack[0] too.

E005: file does not begin with #! or have a .sh prefix
E006: check for lines longer than 79 columns
E042: local declaration hides errors

[0] 0fc6b2c5a8/tox.ini (L35)

Change-Id: I56c017569aa8660648764502cce5adfe2e946a58
2019-08-02 14:29:45 +09:00
Zuul
236a14bf3c Merge "Introduce bashate" 2019-08-02 02:53:28 +00:00
Masayuki Igawa
be05dfab76 Remove x/ranger-tempest-plugin from BLACKLIST
This commit removes x/ranger-tempest-plugin from BLACKLIST. The patch[0]
is trying to support the tempest-sanity job. So, when the patch is
merged, we can remove it from BLACKLIST.

[0] https://review.opendev.org/#/c/672585/

Change-Id: Ia7f0bdad9f854c0f6724537e1ca0234f2952aca8
Depends-On: https://review.opendev.org/672585/
2019-07-31 22:14:35 +00:00
Masayuki Igawa
de1153b361
Introduce bashate
This commit introduces bashate[0] which is a code style checker for bash
scripts. We already have some bash scripts in Tempest repo. So, it might
be good to check the style automatically.

Some scripts under the tools directory are also fixed due to the bashate
violation.

Change-Id: I07820f10387552f93a9d8891b58a8c7fcdb83046
2019-07-30 18:02:07 +09:00
Ghanshyam
a0b8ca43e4 Setup framework for IPv6-only deployment verification
Devstack IPv6 base job 'devstack-IPv6' setup the IPv6 setting
to deploy the services to listen on IPv6 address.

Tempest 'devstack-tempest-ipv6' job derived from 'devstack-IPv6'
job adds the playbooks to run the tests.

As part of Train community goal 'Support IPv6-Only Deployments',
we will expand the 'devstack-tempest-ipv6' job to do
IPv6-only deployments verification.

This commit define the new roles of 'ipv6-only-deployments-verification'
which will be invoked as part of run phase of 'devstack-tempest-ipv6' job.
This role will do IPv6-only setting and deployments verification via
tools/verify-ipv6-only-deployement.sh

IPv6 verification script can be extended further to perform more checks
and via project specific test case. Those tests will run as part of project
specific child jobs.

The 'devstack-tempest-ipv6' job will be used as parent for project specific
IPv6-only job. Those child job can extend the project specific IPv6 verification
by defining new playebook for post-run. That way the base verification
done in 'devstack-tempest-ipv6' will still run in addition to project specific
verificaiton and tests run.

Verification structure will be:
- 'devstack-IPv6' deploy the service on IPv6
- 'devstack-tempest-ipv6' run will verify the IPv6-only setting and listen address
- Child jobs derived from 'devstack-tempest-ipv6' will run the IPv6 related test case or
  any further IPv6 deployment verification.

This commit also adds the new job 'tempest-ipv6-only' which will run smoke
and ipv6 related tests present in Tempest. This job will be used to run
on 6 services (Nova, Neutron, Cinder, Keystone, Glance, Swift) deployed
by devstack.

Story: #2005477
Task: #35923

Change-Id: Ie1e5e5e02ee45ab7a4aae0f35bdb42afc2c3fe2e
2019-07-29 07:37:33 +00:00
Zuul
e53d6b9800 Merge "Add ranger-tempest-plugin to blacklist" 2019-07-19 03:59:18 +00:00
Masayuki Igawa
4f87f6bd2e
Add ranger-tempest-plugin to blacklist
This commit adds ranger-tempest-plugin to the blacklist. The plugin
looks like very new and it looks it doesn't work correctly yet. So, it
might be better to skip to sanity for the plugin now.

Change-Id: I6d5d647d92916e5b62e8c67094d04f028e25ee67
2019-07-17 11:58:24 +09:00
Ghanshyam
95358523e4 Define the Integrated-gate-object-storage gate template
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.

But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.

This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.

One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.

This commit propose the idea of defining the 'Integrated-gate-object-storage'
gate template for swift gate which will run only Swift, Cinder and glance tests.

ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html

Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-object-storage template on swift gate.

[1] https://etherpad.openstack.org/p/qa-train-ptg

Change-Id: I70851d41174462b83cc8c525cecdb1b3d65e4ec7
2019-07-16 04:37:49 +00:00
Ghanshyam
ee7fd1f7e9 Define the Integrated-gate-compute gate template
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.

But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.

This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.

One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.

This commit propose the idea of defining the 'Integrated-gate-compute'
gate template for nova gate which will run only nova, neutron, cinder
and glance tests.

ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html

Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-compute template on nova gate.

It will also help to disable the non-nova related serviecs and tests which
cannot be disabled from tempest-full job.

[1] https://etherpad.openstack.org/p/qa-train-ptg

Change-Id: I807d4cac6214395df292a6955686ea44677c3481
2019-07-16 04:37:34 +00:00
Ghanshyam
31609dc1da Define the Integrated-gate-placement gate template
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.

But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.

This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.

One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.

This commit propose the idea of defining the 'Integrated-gate-placement'
gate template for placement gate which will run only nova and neutron tests.

ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html

Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-placement template on placement gate.

[1] https://etherpad.openstack.org/p/qa-train-ptg

Change-Id: I200197e2fb57879aaeb382266d6309f0d1137b0c
2019-07-16 04:37:16 +00:00
Zuul
aa30a65c9e Merge "Define the Integrated-gate-storage gate template" 2019-07-12 04:14:04 +00:00
Zuul
b3f8338289 Merge "Remove networking-ansible from tempest plugins list" 2019-07-09 16:12:54 +00:00
Zuul
a25ba46c64 Merge "Remove nova-lxd plugin from blacklist" 2019-07-09 16:11:36 +00:00
Ghanshyam Mann
a699445d28 Define the Integrated-gate-storage gate template
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.

But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.

This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.

One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.

This commit propose the idea of defining the 'Integrated-gate-storage'
gate template for cinder gate which will run Cinder, Glance, Swift and
Nova tests.

ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html

Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-storage template on cinder and glance gate.

[1] https://etherpad.openstack.org/p/qa-train-ptg

Change-Id: Iae7d26325228aad0edb94f7e191c8fd0d60e1362
2019-07-09 13:55:04 +00:00
Zuul
36ddb51d82 Merge "Remove cyborg-tempest-plugin from BLACKLIST" 2019-07-09 09:19:50 +00:00
Ghanshyam
5072a24080 Define the Integrated-gate-networking gate template
tempest-full job run all services API and scenario tests (non slow)
and this job is part of Integrated-gate gate template which run on
multiple services (nova, neutron, cinder, glance, placement) check
and gate pipeline.

But there are many unrelated tests run on each project gate. For example,
neutron gate run cinder only tests and fail due to cinder bugs. There is
no point of breaking the neutron gate due to cinder only test failure and
slow down the neutron development phase.

This is what we dicussed in Train PTG in denver to improve the integrated
gate[1] and came up with idea of split the integated-gate template into
much logical way so that only dependent services tests run on their gate.

One way was to move the service only tests to service tempest plugin side
but Tempest is used by interop as well as on production cloud testing so
moving the test is not good idea.

This commit propose the idea of defining the 'Integrated-gate-networking'
gate template for neutron gate which will run only nova and neutron tests.

ML discussion: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html

Next step will be to replace the Integrated-gate-py3 template with new
Integrated-gate-networking template on neutron gate.

[1] https://etherpad.openstack.org/p/qa-train-ptg

Change-Id: Ifca8172f2f929105af4bad8ab72b05c2c45a5f58
2019-07-05 08:04:23 +00:00
Masayuki Igawa
0187a6647a
Remove nova-lxd plugin from blacklist
This commit removes the nova-lxd plugin from the blacklist. The proposed
patch was already merged, so this plugin should pass the sanity job now.

Change-Id: I72eb62af993bc978127185a1aed0763af18ce9f3
2019-07-04 16:51:56 +09:00
Masayuki Igawa
012dc98fb2
Remove networking-ansible from tempest plugins list
This commit removes x/networking-ansible from the tempest plugins list.
The original patch[0] was abandoned because networking-ansible took a
different approach[1] for the testing. So, it's not a tempest plugin
anymore.

[0] https://review.opendev.org/#/c/634647/
[1] https://review.opendev.org/#/c/634918/

Change-Id: I554c438f27a7032f31e56eac2aac110e08bb3d7d
2019-07-04 16:20:15 +09:00
Masayuki Igawa
3f5748906b
Remove cyborg-tempest-plugin from BLACKLIST
This commit removes the cyborg-tempest-plugin entry from the BLACKLIST
since the patch was already merged and this plugin should pass the
sanity job now.

Change-Id: Ifec2e45761961bc4dfc3b5c11709c0ac9e81855f
2019-07-04 16:14:40 +09:00
Zuul
2df6114e44 Merge "Raise non 404 Error in generate-tempest-plugin-list.py" 2019-07-03 03:13:04 +00:00
Masayuki Igawa
2957b4020a
Raise non 404 Error in generate-tempest-plugin-list.py
This commit makes the tempest plugin function raise non 404 errors
when it catches them. Without doing that, we just ignore the errors.
However we shouldn't do that because it just causes another error when
the variable is used.

Change-Id: Ia82ef2aa9cb0959ccee1600feff297abb451176a
2019-06-24 15:22:13 +09:00
Masayuki Igawa
6617b834ff
Add project prefix to blacklist
This commit fixes to add prefixes to the blacklist. The blacklist was
moved to generate-tempest-plugins-list.py by the recent patch[0].
However, the blacklist was updated by the other patch[1] before that.
So, it should have been updated correctly when the conflict was
occurred.

[0] I707aba930037e86f5f94dac73da957f5afacf935
[1] Ieecb8989ff8959f1b2c0e5f2f6b8ea2c3ebbcaa9

Change-Id: I090dd235dd7911aa9bb0eb13d8e65b501dc206fe
2019-06-20 16:23:43 +09:00
Masayuki Igawa
ac8ae64d5b
Publish BLACKLIST on Tempest plugin registry page
This commit makes the BLACKLIST to be published on the Tempest plugin
registry page which we discussed at the PTG[0].

[0] https://etherpad.openstack.org/p/qa-train-ptg

Change-Id: I707aba930037e86f5f94dac73da957f5afacf935
2019-06-13 11:57:09 +09:00
Zuul
cd77312a7a Merge "Publish non-openstack namespace tempest plugins" 2019-06-12 17:02:29 +00:00
Masayuki Igawa
e36fe67e86
Publish non-openstack namespace tempest plugins
This commit changes to publish non-openstack namespace tempest plugins
to the tempest registry doc page. We missed some tempest plugins such as
airship's tempest-plugin because of the recent namespace change. And
also, this commit adds two projects[0] to .zuul.yaml because it had
been added already.

This commit republish them on the doc page, and add two projects to the
blacklist to pass the sanity tests.

Change-Id: Ieecb8989ff8959f1b2c0e5f2f6b8ea2c3ebbcaa9
2019-06-10 11:08:06 +09:00
Masayuki Igawa
c6b0f140bc
Print passed and failed plugins at the end
This commit make passed and failed plugins be printed at the end of the
tempest-plugin-sanity execution. This is useful to ensure what plugins
were tested.

Change-Id: Iffd9ff6afd71c1d61322e5202b9861ec218cb574
2019-05-30 18:38:51 +09:00
Masayuki Igawa
32a4b11f15
Remove vitrage-tempest-plugin from blacklist
This commit removes vitrage-tempest-plugin from the BLACKLIST since the
patch[0] was already merged.

NOTE: The patch on the comment[1] was duplicated of [0]. So it was
abandoned.

[0] https://review.opendev.org/#/c/638931/

Change-Id: I5539159bad10b3359981fab88b3be5e4e997fcb9
2019-05-25 17:01:42 +02:00
caoyuan
349ba75410 Replace git.openstack.org URLs with opendev.org URLs
Thorough replacement of git.openstack.org and review.openstack.org URLs
with their opendev.org counterparts.

Change-Id: I88e894db7b854d32593c770f5aa9b8a91fad7866
2019-05-23 10:19:31 +00:00
Masayuki Igawa
7f5dd855c6
Add cyborg-tempest-plugin to required-projects and remove octavia
This commit adds opendev.org/openstack/cyborg-tempest-plugin to
required-projects to pass the plugin-sanity-check job. It's a new
tempest plugin. It doesn't work properly on master yet. So, I put it on
the BLACKLIST.

And this commit also removes octavia because octavia tempest tests were
already migrated to octavia-tempest-plugin, and there is no tempest test
in the octavia repo. The removal patch[0] in octavia was already merged.

[0] https://review.opendev.org/#/c/659516/

Change-Id: Ied13dbf774472e1c36dca4ccca157104a81fbf9d
2019-05-17 08:24:15 +09:00
Masayuki Igawa
8460cb1844
Remove airship-tempest-plugin from blacklist
This commit removes airship-tempest-plugin entry from the BLACKLIST
since the patch was already merged. And airship/tempest-plugin has a
gate job to verify it, too. There is no reason to store it in the
BLACKLIST anymore.

And this commit also make plugin-sanity-check's basepython python3
because we should use it as a default.

Change-Id: I9c50d327df65fecf8510f6f54f06e9f42da9bea0
2019-05-16 18:48:27 +09:00
Attila Fazekas
c07edb5f80 Mention autopep8 in Hacking.rst
The scripts in the tools now is able to distinguish,
the autopep8 error codes better.

Change-Id: I9ea19009118ed418e3666fad45d2e0eb676204c8
2019-03-29 20:50:23 +00:00
Ian Wienand
ff3851b593 Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: Iff8f93426925a3480cedfb899216e443b12cd826
2019-03-24 20:36:15 +00:00
afazekas
40fcb9ba28 Add autopep8 to tox.ini
autopep8 is an automated code formatting tool for python,
it does not know everything, also not super fast,
but it is faster than I can manually reformat the code.

tools/format.sh will call the formatting,
the tox will check did you called it.

Also adding a tox -eautopep8 way to use it.

autopep8 formats the lines in one way,
where tempest used a another way it was changed.

Change-Id: I6d51b14a5a5b87761071d0927fca23ba1651aa41
2019-03-08 16:04:05 +01:00
Chandan Kumar
97b1ad78ac
Improved tempest plugin sanity check script
* To use a fresh venv, use `--clear` option when creating a virtualenv
* Removes the project as the script fails to uninstall the dependencies.
* Added missing tempest plugins
* Log the stdout of each plugin in a file
* Adds some plugins to BLACKLIST to avoid errors

Change-Id: I5068d96105a22cbfd33f5900f4bdc21b47a65c06
2019-02-25 13:09:08 +09:00
Zuul
f99db4f558 Merge "Fix required projects for sanity plugin check job" 2018-11-21 21:13:38 +00:00
Felipe Monteiro
68273ac78a Fix required projects for sanity plugin check job
Currently, the sanity plugin check job is failing because
of errors like the following:

  ERROR! /home/zuul/src/git.openstack.org/openstack/cinder-tempest-plugin not found
  In Zuul v3 all repositories used need to be declared
  in the 'required-projects' parameter on the job.
  To fix this issue, add:

    openstack/cinder-tempest-plugin [0]

So this adds missing projects like tempest plugin repos to get
the job passing.

Included in this patch set:

* Add missing repos to required-projects
* Blacklist networking-plumgrid since it is failing because
  it hasn't been supported in years [1]
* Remove trio2o from required-projects because it is already
  blacklisted
* Add irrelevant-files for sanity check job

[0] http://logs.openstack.org/96/615396/4/check/tempest-tox-plugin-sanity-check/a18e9a9/ara-report/
[1] https://github.com/openstack/networking-plumgrid

Change-Id: Ifee972d87ae83759000db925449764a60a98f7ba
2018-11-07 19:24:11 -05:00
Felipe Monteiro
ec4c668764 docs: Fix plugin registry generation failures
This patch set fixes plugin registry generation script which
is resulting in the documentation page ending up empty [0].

This is because the script `tools/generate-tempest-plugins-list.sh`
is using an old flag (`-A`).

This drops the `declare -A plugins` command because `plugins` isn't
used anywhere and also uses `decode('utf-8')` on the initial response
lookup for all the projects which is done already in the script
elsewhere [2].

[0] https://docs.openstack.org/tempest/latest/plugin-registry.html
[1] https://bash.cyberciti.biz/bash-reference-manual/Bash-Builtins.html#index-declare-132
[2] da8aec9aed/tools/generate-tempest-plugins-list.py (L67)

Change-Id: I70b59c869df06f35c0f7ca6d20e3584502cda8ef
2018-11-07 03:00:52 +00:00
akhiljain23
a2260de550 Adding links to git reference in docs
Created hyperlinks for git urls in doc [0].
[0] https://docs.openstack.org/tempest/latest/plugin-registry.html

Change-Id: Ied124b1829cc328bfe2dd150f819c7daf480d0f9
2018-08-06 16:11:39 +05:30
ghanshyam
df037310ba Dynamic width count for plugin registry table column
There might be chance that few plugin name and url
does not fit into hard coded column width and give error.

This commit count the column width as per plugins name.

Also display the number of plugins in new SR column.
Change-Id: Ia5736d2581ee3b6f851b8d08f855c2a19c157c47
2018-07-31 17:09:37 +05:30
Stephen Finucane
7f4a6211bf Mark raw strings as such
This is raising warnings now and will be an error in future versions of
Python 3.

Change-Id: I8827cb36ef2ef85e8f245a2a181fb23ee75bfd16
2018-07-06 14:01:28 +01:00
Zuul
11aa0360cd Merge "Skip puppet-* projects for plugin search" 2018-04-19 05:11:18 +00:00
ghanshyam
95b68befc4 Skip puppet-* projects for plugin search
openstack/puppet-* projects can be skipped to
search for tempest plugin.

Change-Id: I8ca7bd944f250b168077ffc8c82b9bddb338eddb
2018-04-12 02:30:29 +00:00
chenpengzi
2b4bf4b4b3 Delete meaningless and unused parameters
Change-Id: I63af79051b6848cf7ed0148bf26b3374f7771fbf
2018-01-29 11:39:06 +08:00