Tempest and its plugin are very tightly coupled for testing
Upstream or Production Cloud. Due to its branchless model,
one of the challenges we faced/facing is to provide their
compatible version set.
Since Queens release and with operator request, we started
doing a coordinated release of plugins along with Tempest
which will be marked as the First compatible version for
that particular release.
For example https://releases.openstack.org/train/index.html#tempest-plugins
We test our stable branch with master version of Tempest &
plugins until it is moved to Extended Maintainance state.
One stable release is in EM state, Tempest officially declares
the end of support tag so that we can use that as the last compatible
version for that EM stable testing. But we do not have the correct
last compatible version from plugins which can be used for testing
the EM branch. It should be with hash from the master at the time
stable release is moved to EM not the hash when the stable release
was done otherwise it will be incompatible with Tempest.
We have faced and fixed many issue due to that during py2-drop work
where we have to pick and try all different versions from plugin to
make it work against EM testing.
Change-Id: I99a6df0a2c02e19efaf484fcd97b875887f08d8e
This is useful when a job needs to execute additional tests
after tempest run, so ignoring the tempest run failures,
and combine the results afterwards.
Change-Id: I61fc5ebf69db47a035ae8294cfda4047d9f7e21a
When Tempest is used in customer site, often we are required to
provide a testcase list including testcase names and descriptions.
Now no this kind of doc is available, so we can add descriptions
with the format of doc string for every testcase, so later we
can generata such a testcase description list.
There are hundreds of testcases missing descriptions, so we can
add them gradually, and limit the modified files in one patch
for the convenience of reviewing.
BTW: some minor code refactor is done along with this patch.
Change-Id: I072bd2c86a7f50559e9e452dda3e1669fad3be00
partially-implements: blueprint testcase-description
When Tempest is used in customer site, often we are required to
provide a testcase list including testcase names and descriptions.
Now no this kind of doc is available, so we can add descriptions
with the format of doc string for every testcase, so later we
can generata such a testcase description list.
There are hundreds of testcases missing descriptions, so we can
add them gradually, and limit the modified files in one patch
for the convenience of reviewing.
Change-Id: I9840ea976d290a8fd22d8c3bae17db9ed42f629a
partially-implements: blueprint testcase-description
While loading data form plugin like tests,
config options etc we do not have enough logs
to debug if any plugins is missed.
It is difficult to debug in such cases where we do not
find config options registered by plugins,
For example: https://review.opendev.org/#/c/706785
Adding more logs to know what all data we are loading from
all plugins.
Change-Id: I09d2349a0e32c2265e6beb3dad4619ad33782a51
In case "TaggedBootDevicesTest.verify_device_metadata" "json.loads"
method returns an exception, catch it and return "False". This could
happen if, for example, the string to load is empty.
Because this function is called in an active wait loop, the content
string will be retrieved again and the check retried.
Change-Id: Id3a5b32f8ab197bcb1c59b5f6364dde4d8648eab
Closes-Bug: #1867904
When Tempest is used in customer site, often we are required to
provide a testcase list including testcase names and descriptions.
Now no this kind of doc is available, so we can add descriptions
with the format of doc string for every testcase, so later we
can generata such a testcase description list.
There are hundreds of testcases missing descriptions, so we can
add them gradually, and limit the modified files in one patch
for the convenience of reviewing.
Change-Id: I3d1634fab5826928a45bd47c18aba6fc32021075
partially-implements: blueprint testcase-description
Currently we rely on [service_available]ironic which
1) will give incorrect results in a mixed environment
2) doesn't work in grenade jobs
This change only allows ports to be DOWN when they have VNIC type
of baremetal.
Change-Id: If24913b7097c5252a214bb780fc786aa0c3f983b
When Tempest is used in customer site, often we are required to
provide a testcase list including testcase names and descriptions.
Now no this kind of doc is available, however, we can extract info
from tempest code to create one, but, some testcases don't have
descriptions about what the testcases are testing, so, we can add
descrption for each testcase.
There are hundreds of testcases missing descriptions, so we can
add them gradually, and limit the modified files in one patch
for the convenience of reviewing.
Change-Id: Id90d6fedccc84aa9548fba62b5926a75050ab443
partially-implements: blueprint testcase-description
This commit adds x/glare and x/tap-as-a-service to the BLACKLIST to
avoid sanity-job failures like following.
Failed Plugins: x/glare, x/tap-as-a-service,
+ [[ -n x/glare, x/tap-as-a-service, ]]
+ exit 1
It looks the root cause is related to pg_config installation but not
sure. So, this commit just blacklists them as a workaround.
Error: pg_config executable not found.
Change-Id: I03c6f01593b8e414b5e111d0e0c971ae156da715
This change adds a simple set of tests to exercise the stable device
rescue feature. A new [compute-feature-enabled]/stable_rescue
configurable is introduced to control when these tests are executed,
defaulting to False.
Change-Id: I0680fe4a6590aa3c4887d338a02b04fabfcdbb11
Microversion v2.73 was introduced in Nova by I46edd595e7417c58410648
changing the request responses of ``GET servers/{server_id}``, ``GET
/servers/detail``, ``POST /servers/{server_id}/action`` where the
action is rebuild and ``PUT servers/{server_id}``.
Change-Id: I8c800f6e280ec6fdf204d344124d994a25d0af19
- Fix noVNC failed TCs tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc:
* Current noVNC request miss the / prefix in GET header in the websocket upgrade request.
* Current GET header GET ?token=xxx HTTP/1 can cause "HTTP/1.1 400 Bad Request" error.
* Fix the issue by adding / prefix and
# updated request format is: GET /?token=xxx HTTP/1.1 or GET /websockify HTTP/1.1
Change-Id: I38a91b8e293e39625d08073e2b898451b6126ecb
With the introduction of expiring group memberships, there is a
new attribute `membership_expires_at` when listing user groups.
This patch updates the test to check the attribute and then
ignore it for group dict comparison.
Change-Id: I4294a879071dde07e5eb1da4df133de8032e1059
Partial-Bug: 1809116
The introduction of Icd7fcb87a09c35a13e4e14235feb30a289d22778 blocks
attempting to update or swap a volume that is currently attached to
multiple instances. This change adds a simple test to assert that a
BadRequest is raised when this is attempted directly against the compute
API.
Future work will also cover this failure when part of a wider volume
retype or migration flow.
Change-Id: I931a810901cf7847a8259e44a30b0150aad859f2
Previously, create_test_server() in the base compute test class did
not provide a way to create a server with admin privileges. This can
be necessary when using APIs that are admin-only by default, for
example the `host` parameter when creating a server in Nova
microversion 2.74. This patch adds a clients kwarg to
create_test_server(). This kwarg is then passed to compute.common's
create_test_server() instead of the cls.os_primary clients that was
harcoded previously.
Change-Id: Ic47e6a4ced54739de9325bb14e332ba15cdb210f
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.
Fix problems found.
Update local hacking checks for new flake8, remove vi check, it's now in
hacking as H106.
The tempest plugin is used on older branches as well.
We really only need hacking on this repo anyways,
where we no longer support python 2, so here we
make the requirement specific to python 3.
Change-Id: I3f4216f66606fbc450a46c93de306399b7f3cd65
Let's check no specific group present instead of list is empty
in test_group_snapshot_create_show_list_delete.
This maybe a reason of failed test due to some inconsistency.
Closes-Bug: #1869852
Change-Id: I41884567ae07b4d6a28667d2a3a792b63e1a6cfa
x/kingbird doens't seem to be compatible with tempest anymore
which is breaking our tempest-tox-plugin-sanity-check gate job.
Change-Id: Icdaff49bbc2ba0e59cd528d55cad6c35014b7ad0
This patch add the option --worker-file on tempest run command.
This will enable users to specify a worker file that will be used
by stestr to manually schedule how the tests will run.
Change-Id: I747f3abe179492da063fcaaf1123ffcf6362f966
Currently race condition can occure in
tempest.api.compute.servers.test_attach_interfaces.
AttachInterfacesUnderV243Test.test_add_remove_fixed_ip
when floating IP added during resource preparation doesn't appear in
the list of original IPs that is created at the beggining of the test,
which then confuses the test
and floating ip is later recognized as fixed IP added in the test.
more details including log:
https://bugzilla.redhat.com/show_bug.cgi?id=1752416
This change ensures possible floating IP added during server
creation is always present in the set of original IPs and also
during every comparasion of IPs.
Closes-Bug: #1866179
Change-Id: Ic3a3e0708714b6d6c9c226e641e1c520e5ebde9d
Signed-off-by: David Sedlák <dsedlak@redhat.com>
Network quotas could be deleted after deletion of project. This
caused the cleanup to fail as network quotas have to be deleted
before the deletion of project.
This fix makes sure that network quotas are deleted before the
deletion of project.
Change-Id: I8576e2c6f1ae6dc90850dd3d8eade7c10942e413
add unittest for create and list resource types association
interface of metadata in glance module
Change-Id: I3f7bff1fcad32450b71f4ddd16e579cd53961fe9
