Nova list the interfaces attached to a server based on list of ports
bound to the server. However during detach interface nova unbounds the
port first and then deallocates the resources used by that port in
placement. The current detach waiter only waits until the interface
disappears from the interface list. This can cause that waiter returns
before the resource allocation is removed from placement cause a race in
the test asserting such allocation.
So this patch changes the waiter to wait for the successful detach
os-instance-action event for the server as that is only recorded after
the port is fully deallocated.
blueprint: qos-minimum-guaranteed-packet-rate
Change-Id: I8740f8e0cc18ffea31a9a068bccee50bf4e6fe28
In bug 1923194, a floating IP is associated to a server. Upon
performing the operation, Neutron sends Nova a network-vif-changed
external event, causing Nova to update its network info cache. Until
Nova does this, the new floating IP is not reflected in `server show`.
Tempest's `server show` is racing with this process, causing
intermittent failures when the new floating IP does not show up in
`server show` in time.
This patch adds a new waiter that waits for a floating IP to either
appear to disappear in the `server show` output, and modifies two
tests to use the new helper.
Closes bug: 1923194
Change-Id: I0f7e1c9096dc1903903fb31c5e854f07800efbfd
This change reworks both the waiters while attaching a volume *and* the
ordering of the waiters when detaching.
This is done to ensure the wait_for_volume_attachment_remove_from_server
waiter is called first as this uses the servers client and is able to
dump the contents of the instance console when we hit a timeout. The
contents of the instance console being incredibly useful to debug issues
within the guestOS when detaching devices as seen in bug #.
The wait_for_volume_attachment_remove_from_server waiter is also
extended to ignore missing Nova volume attachments as this can easily
happen if tests have manually detached volumes ahead of the cleanups
being called.
TODOs are also left to move away from using the volume status to
determine when a given volume has been attached and instead use the
state of volume attachments both in Nova and Cinder.
Related-Bug: #1931702
Change-Id: I8f7986dc6d8689d569b7fba74cca38de4236c6d6
Once Glance finish the import operation(change image status
to active), it move the task to 'success' state but in between of
image become active and task is transitioning to 'success', tempest
try to check the task status and race condition happen.
Adding waiter method in test for task status check.
Closes-Bug: #1926671
Change-Id: I960b80314f1b0926eca33af830bc827f31cbeda6
During a glance import, if we fail, we are not going to change the
state of the image in all cases, and definitely not to 'active'.
Thus waiting for active just means we keep polling a broken image
for a long time, wasting resources. We should also be checking the
list of failed stores, and if something pops in there, abort right
then and there.
This patch makes us use the wait_for_image_imported_to_stores()
waiter, and modifies it to not look at image['stores'] if no
stores are provided. In the case where we don't have multistore
support enabled, we won't be able to know that the import failed
until we time out, but otherwise we will (barring a glance bug for
which I also have a fix). This also makes the waiter not fail on
KeyError if os_glance_failed_import is not present on the image,
as would be the case if used when stores are not enabled.
Note that I'm doing this because sometimes web-download fails in the
gate due to a timeout trying to pull our http_image and we just keep
looping until our own timeout instead of noticing. It also means
we just report "never reached active state" instead of what we know
to be true, which is that import failed.
Change-Id: Ie0569b84dd1c397a79d13af609bf23db52a2dfc8
The test_attach_scsi_disk_with_config_drive test cases is racy at volume
detach. Nova first detach the volume in cinder then later deletes the
BlockDeviceMapping from the nova DB. Test waits for the volume to become
available and then checks the list of volume attachments in the nova API
then it can see that the attachment is still there.
Closes-Bug: #1907084
Change-Id: I814ae3215f39d1e8407c4ca1c7117a314941c80b
This is to fix the "TypeError: format requires a mapping" in
wait_for_image_imported_to_stores and wait_for_image_copied_to_stores.
Change-Id: I8a33bd4f4ea189ce9247dfecec1e53001682b870
Due to issue described in related bug report, it seems that it may
happen sometimes that tempest will start trying to ssh to the instance
before ssh key will be really configured in the instance and in such
case it may happened that there will be AuthenticationFailure error
always there. Even if ssh key will be configured properly later during
the test.
To workaround that issue and avoid failures of tests, this patch adds
check if the vm is really booted and ready to ssh. It is done by
checking console log of the VM and looking for specific string "login:"
which appears at least in case of Cirros and Ubuntu images used in our
CI jobs.
In case when such string will not be found, test will continue to run
and will still try to ssh to the instance. So in worst case it may slow
ssh to the instance a bit but shouldn't really have any bad impact on
test as before this patch it would probably also wait similar amount of
time but on trying to SSH to the instance.
Similar workaround was introduced in the neutron-tempest-plugin with
[1]. After it was merged number of failed jobs due to ssh "Authentication
failure" issues lower down significantly.
[1] https://review.opendev.org/#/c/758968/
Change-Id: I84a7c2fc389d181388ef821848ffad4bc67f1213
Related-Bug: #1892861
The instance may have been deleted when calling 'force_delete_server'.
So this patch ignores exception 'lib_exc.NotFound' that avoid
error in tearing down stage.
Change-Id: I29d6d1a41e3fb1bbbdb1f16ef4e54916df2e1ec7
Closes-Bug: #1896865
Adding tests for below scenarios
1. test for import_to_all_stores(--all-stores true)
2. test for import_to_specified_stores(--stores store_list)
Change-Id: I3aa413c624881065a1cf46c6345f9eda923df8eb
A volume attached to multiple servers has multiple attachment structures
that are distinguished by their "attachment_id" field, not the "id" one.
So when setting up a waiter watching for an attachment being removed,
the "attachment_id" field must be passed. Thus, when attaching a volume,
it is necessary to wait until the attachment shows up in the volume's
data and then use the "attachment_id" field of that record to pass to
the waiter invoked at tear-down time.
Change-Id: I9ab9f786b23061dd3a6a3482ab9739ba504d2bc0
Closes-Bug: 1894724
This change adds a specific waiter for tracking the removal of
attachments from a volume. This should allow callers to track the
detaching of multiattached volumes from instances while the volume
remains in an `in-use` state.
Related-bug: #1858841
Change-Id: I85917f683d2bc81fdf3f6d3351bf56fdb649740e
If reclaim_instance_interval > 0 in nova.conf, some testcases will
fail cause of resource's state is 'in-use'. This patch adds the logic
in `wait_for_server_termination` that if the virtual machine state is
SOFT_DELETED, then call `force_delete`.
Change-Id: I406188ac2aa54419fa4e44f547ae80a31b58b858
Closes-Bug: #1850568
Now we can get error messages like "Resource 595f6527-d84c-
4e14-a86f-c7be186a864c failed to delete and is in ERROR status",
and this has two problems:
1. It doesn't indicate which kind of resource, e.g., volume or server
2. For DeleteErrorException raised in volumes_client, the volume
status is error_deleting, not ERROR.
So this is to pass full and accurate message to DeleteErrorException
to avoid the confusion.
Change-Id: I8cfd9f84b888b3380b6109e967b09f7fa74e4051
When extending volume reaches status error_extending, we should
return from wait_for_volume_resource_status immediately instead
of waiting for the TimeoutException.
Change-Id: I3db0325c23ebb8d42fe77f4c28552fe9cf5ed807
This change introduces a true cinder host to host attached volume
migration test in addition to the existing attached volume retype test.
To enable this two new calls are introduced to the v3 volume client to
allow all volume backends to be listed per project and to also call for
a direct volume migration between backends.
Related-bug: #1803961
Depends-On: I1bdf3431bda2da98380e0dcaa9f952e6768ca3af
Change-Id: I501eb0cd5eb101b4dc553c2cdbc414693dd5b681
The method waits for an interface to be detached from a server.
It will be used in the subsequent patch that tests tagged device
attach and detach (nova API 2.49).
Change-Id: If23bd5947dea345b30a77bc83c5b3dbfa5a4267b
This reverts commit 030dd17f5e.
This masks a behavior change in the compute API when attaching
a volume to a shelved offloaded instance, as indicated by the
need to change Tempest to wait for a volume to be 'reserved'
vs 'in-use' in this scenario with the new Cinder v3 attach
flow.
This is a problem that needs to be handled in Nova to keep
API compatibility with the old volume attachment flow. Nova
can either do that under the covers, or add a new
microversion for callers to opt into this new behavior, but
masking over it in Tempest is wrong.
The related nova change:
Ifc01dbf98545104c998ab96f65ff8623a6db0f28
Change-Id: If8e1f42d853f366a399c9e454a80ba3bf8cd136e
Generic volume groups support was added to Cinder in the
Newton release:
https://blueprints.launchpad.net/cinder/+spec/generic-volume-group
This is the 2nd patch that adds the tempest tests for generic
volume groups APIs in Cinder. It adds group_snapshots clients.
It adds tests for the following APIs:
* create group snapshot
* delete group snapshot
* show group snapshot
* list group snapshot
Change-Id: I21f07f7b3e8f22055e7978c0bf4aa01b80b493d9
The new Cinder attach API treats attaching to a shelved instance
differently than the old code. In the old case we fake things a
bit and mark the volume as attached, even though it's really not,
it's actually just reserved and there's no valid connector
or connection associated with it.
In the new (Cinder V3.27) Attach API's we instead give a more
accurate response, create an attachment record and mark the
volume as 'reserved' until the Instance is unhselved at which
time we complete the connection and move the volume to 'in-use'.
This patch modifes the volume_status waiter slightly so that it
can accept multiple statuses to consider "ok". We also modify the
test_attach_volume_shelved_or_offload_server and
test_detach_volume_shelved_or_offload_server tests to specify that
either 'in-use' or 'reserved' are acceptable responses. This way
we're still compatable with older combinatons of nova/cinder and
we also work with the new combination as well.
Change-Id: Ia84f4325ddb0080521241ace26f89d1161db9dca
This is the 1st patch that adds the tempest tests for generic
volume groups APIs in Cinder. It adds groups and group_types clients.
It tests the following group APIs:
* create group
* delete group
* show group
* list group
Change-Id: Id1779fe5dce50b062ca0ff841a7ede72cac9dd73
This is very minor, but the server formal argument to the
wait_for_interface_status method is actually a server id.
This changes the name of the parameter to be clear on the
type of that object, since it could easily be confused
with a full server response body dict.
Change-Id: Id7cab21ab75092226f0784877ae55f17e880229c
This function waits for a volume resource to reach a given status.
The function is a common function for volume, snapshot and backup
resources.
Change-Id: I070a9f84b8b199df22765391482f7d69a5414db3
In function “wait_for_server_termination”，it is in the waiting
for server to reach termination. So it's more reasonable to throw
a deleting error exception here when server going error.
This patch adds a DeleteErrorException and let function
“wait_for_server_termination” to use it. This makes the exception
message more clear when server(or other resources) entering error
status in the waiting for deletion.
And the DeleteErrorException will be used
in change I6f92f00b8fe4651c27ff4da783b004d29c526dfe
Change-Id: I6646ce732a4f3802338c8f2a478fd81ff5a16a55
The TimeoutException was coming from the wrong module,
which might have just been a rebase mistake while this
stuff was moving around.
Change-Id: Id494aea1527d03148ba9284c48a1c171d3469911
Closes-Bug: #1664633
When waiting for a server to go ACTIVE we log state
transitions, but don't do that on delete. It'd be
useful to see this on delete too, so this patch adds
that.
Change-Id: I1c022cb217f715d584b19493edae76320cba1388
Test scenario with a bootable volume that is migrated via:
retype --on-demand <volume_id>
This will exercise cinder retype and migration as well as
nova swap_volume for a bootable volume.
Requires at least 2 Cinder backends which are configured in
devstack with:
CINDER_ENABLED_BACKENDS=lvm:lvmdriver-1,lvm:lvmdriver-2
Co-Authored-By: Erlon R. Cruz <erlon.cruz@fit-tecnologia.org.br>
Change-Id: I4ed9b1f30d4e4e595c44a0ce243c2463069833d7
ironic project now has a tempest plugin and its tests will be maintained there.
Closes-Bug: #1614516
Depends-On: I0b7e32dfad2ed63f9dd4d7cad130da39bc869a8a
Change-Id: Id518a6d87d0949737cd1c50cb6a83149b85e5f85
This is to move wait_for_interface_status to waiters so it can
be used elsewhere. Besides, wait_for_interface_status returns
the detail of show_interface which is needed in testcases.
Change-Id: I3a8cee9cd344be295cb0cdef5930793403cc6bca
This patch is to fix the issue which described in the bug #1632410
A timeout is being raised on a volume backup test scenario
which could be that because the volume backup is slow or
we tripped a cinder bug. The real issue is that wait loop
for backups in tempest is referencing a non-existent exception class,
"VolumeBackupException".
Change-Id: I2d74afc29cd52da7f3642bf56ea896c92cfccdca
Closes-bug: #1632410
Currently, wait_for_qos_operations method resides on
BaseVolumeAdminTest class. I think we should move it
to common.waiters module as other waiters functions.
Partially implements blueprint consistent-service-method-names
Change-Id: Ia687451e4fddfc1408c2570697068415c1124c38
Currently, wait_for_backup_status function resides on
services/volume/base/base_backups_client.py.
I think we should move it to common.waiters module as
other waiters functions.
Partially implements blueprint consistent-service-method-names
Change-Id: Ie1d416e32c583d8e48140c4046df989cc209b84f
Tempest has dropped XML support and the removed str() becames
meaningles. So this patch removes it.
Change-Id: I02184d725d40417d501f0dc57eb1977bffc77f72
find_test_caller is now moved from misc to test_utils, but there
are still some calling to find_test_caller which will cause "WARNING
tempest.lib.common.utils.misc [-] tempest.lib.common.utils.misc.
find_test_caller is deprecated in favor of tempest.lib.common.
utils.test_utils.find_test_caller".
Change-Id: Icbd4c7a45677b4bb031fc40d6b47dd6a72af6f02
Zuul