Nova list the interfaces attached to a server based on list of ports
bound to the server. However during detach interface nova unbounds the
port first and then deallocates the resources used by that port in
placement. The current detach waiter only waits until the interface
disappears from the interface list. This can cause that waiter returns
before the resource allocation is removed from placement cause a race in
the test asserting such allocation.
So this patch changes the waiter to wait for the successful detach
os-instance-action event for the server as that is only recorded after
the port is fully deallocated.
blueprint: qos-minimum-guaranteed-packet-rate
Change-Id: I8740f8e0cc18ffea31a9a068bccee50bf4e6fe28
In bug 1923194, a floating IP is associated to a server. Upon
performing the operation, Neutron sends Nova a network-vif-changed
external event, causing Nova to update its network info cache. Until
Nova does this, the new floating IP is not reflected in `server show`.
Tempest's `server show` is racing with this process, causing
intermittent failures when the new floating IP does not show up in
`server show` in time.
This patch adds a new waiter that waits for a floating IP to either
appear to disappear in the `server show` output, and modifies two
tests to use the new helper.
Closes bug: 1923194
Change-Id: I0f7e1c9096dc1903903fb31c5e854f07800efbfd
This change reworks both the waiters while attaching a volume *and* the
ordering of the waiters when detaching.
This is done to ensure the wait_for_volume_attachment_remove_from_server
waiter is called first as this uses the servers client and is able to
dump the contents of the instance console when we hit a timeout. The
contents of the instance console being incredibly useful to debug issues
within the guestOS when detaching devices as seen in bug #.
The wait_for_volume_attachment_remove_from_server waiter is also
extended to ignore missing Nova volume attachments as this can easily
happen if tests have manually detached volumes ahead of the cleanups
being called.
TODOs are also left to move away from using the volume status to
determine when a given volume has been attached and instead use the
state of volume attachments both in Nova and Cinder.
Related-Bug: #1931702
Change-Id: I8f7986dc6d8689d569b7fba74cca38de4236c6d6
Once Glance finish the import operation(change image status
to active), it move the task to 'success' state but in between of
image become active and task is transitioning to 'success', tempest
try to check the task status and race condition happen.
Adding waiter method in test for task status check.
Closes-Bug: #1926671
Change-Id: I960b80314f1b0926eca33af830bc827f31cbeda6
During a glance import, if we fail, we are not going to change the
state of the image in all cases, and definitely not to 'active'.
Thus waiting for active just means we keep polling a broken image
for a long time, wasting resources. We should also be checking the
list of failed stores, and if something pops in there, abort right
then and there.
This patch makes us use the wait_for_image_imported_to_stores()
waiter, and modifies it to not look at image['stores'] if no
stores are provided. In the case where we don't have multistore
support enabled, we won't be able to know that the import failed
until we time out, but otherwise we will (barring a glance bug for
which I also have a fix). This also makes the waiter not fail on
KeyError if os_glance_failed_import is not present on the image,
as would be the case if used when stores are not enabled.
Note that I'm doing this because sometimes web-download fails in the
gate due to a timeout trying to pull our http_image and we just keep
looping until our own timeout instead of noticing. It also means
we just report "never reached active state" instead of what we know
to be true, which is that import failed.
Change-Id: Ie0569b84dd1c397a79d13af609bf23db52a2dfc8
The test_attach_scsi_disk_with_config_drive test cases is racy at volume
detach. Nova first detach the volume in cinder then later deletes the
BlockDeviceMapping from the nova DB. Test waits for the volume to become
available and then checks the list of volume attachments in the nova API
then it can see that the attachment is still there.
Closes-Bug: #1907084
Change-Id: I814ae3215f39d1e8407c4ca1c7117a314941c80b
This is to fix the "TypeError: format requires a mapping" in
wait_for_image_imported_to_stores and wait_for_image_copied_to_stores.
Change-Id: I8a33bd4f4ea189ce9247dfecec1e53001682b870
Due to issue described in related bug report, it seems that it may
happen sometimes that tempest will start trying to ssh to the instance
before ssh key will be really configured in the instance and in such
case it may happened that there will be AuthenticationFailure error
always there. Even if ssh key will be configured properly later during
the test.
To workaround that issue and avoid failures of tests, this patch adds
check if the vm is really booted and ready to ssh. It is done by
checking console log of the VM and looking for specific string "login:"
which appears at least in case of Cirros and Ubuntu images used in our
CI jobs.
In case when such string will not be found, test will continue to run
and will still try to ssh to the instance. So in worst case it may slow
ssh to the instance a bit but shouldn't really have any bad impact on
test as before this patch it would probably also wait similar amount of
time but on trying to SSH to the instance.
Similar workaround was introduced in the neutron-tempest-plugin with
[1]. After it was merged number of failed jobs due to ssh "Authentication
failure" issues lower down significantly.
[1] https://review.opendev.org/#/c/758968/
Change-Id: I84a7c2fc389d181388ef821848ffad4bc67f1213
Related-Bug: #1892861
A volume attached to multiple servers has multiple attachment structures
that are distinguished by their "attachment_id" field, not the "id" one.
So when setting up a waiter watching for an attachment being removed,
the "attachment_id" field must be passed. Thus, when attaching a volume,
it is necessary to wait until the attachment shows up in the volume's
data and then use the "attachment_id" field of that record to pass to
the waiter invoked at tear-down time.
Change-Id: I9ab9f786b23061dd3a6a3482ab9739ba504d2bc0
Closes-Bug: 1894724
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: If1de2f48da1fd6ed3f96c98b8dce9eace4f8095d
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This change adds a specific waiter for tracking the removal of
attachments from a volume. This should allow callers to track the
detaching of multiattached volumes from instances while the volume
remains in an `in-use` state.
Related-bug: #1858841
Change-Id: I85917f683d2bc81fdf3f6d3351bf56fdb649740e
When extending volume reaches status error_extending, we should
return from wait_for_volume_resource_status immediately instead
of waiting for the TimeoutException.
Change-Id: I3db0325c23ebb8d42fe77f4c28552fe9cf5ed807
This change introduces a true cinder host to host attached volume
migration test in addition to the existing attached volume retype test.
To enable this two new calls are introduced to the v3 volume client to
allow all volume backends to be listed per project and to also call for
a direct volume migration between backends.
Related-bug: #1803961
Depends-On: I1bdf3431bda2da98380e0dcaa9f952e6768ca3af
Change-Id: I501eb0cd5eb101b4dc553c2cdbc414693dd5b681
TestInterfaceWaiters test case were not verifyinf the main waiters
functionality and sometime they could intermittently fail without
reason like seen in below openstack-tox-py27 Zuul check.
https://review.openstack.org/#/c/559069/33
Solve tests methods failure when time.time() returns two consecutive
floats that rounded by int() function is returning two different integers.
This patch also verify the condition verification method are properly
called and that waiters methods actually returns expected part of
returned body.
Change-Id: I0a2081d823b304d58a5a69a8965858bd4f1303a0
The method waits for an interface to be detached from a server.
It will be used in the subsequent patch that tests tagged device
attach and detach (nova API 2.49).
Change-Id: If23bd5947dea345b30a77bc83c5b3dbfa5a4267b
Generic volume groups support was added to Cinder in the
Newton release:
https://blueprints.launchpad.net/cinder/+spec/generic-volume-group
This is the 2nd patch that adds the tempest tests for generic
volume groups APIs in Cinder. It adds group_snapshots clients.
It adds tests for the following APIs:
* create group snapshot
* delete group snapshot
* show group snapshot
* list group snapshot
Change-Id: I21f07f7b3e8f22055e7978c0bf4aa01b80b493d9
This function waits for a volume resource to reach a given status.
The function is a common function for volume, snapshot and backup
resources.
Change-Id: I070a9f84b8b199df22765391482f7d69a5414db3
Tempest provides a stable library interface and I think we can use it
for volume clients.
Partially implements blueprint consistent-service-method-names
Change-Id: Ia2d87360a6d88e5f54ecd00efbf6730b06484c99
In Tempest we have the waiters.wait_for_image_status function that
is already used in some places. So this commit replaces the use of the
wait_for_image_status method defined in the images_client.py file with
the use of the waiters.wait_for_image_status function.
Change-Id: If42cfbfa1f7dae5af276e80f38f6294ced00ac5b
The recent commit I45d73891f3c0829a2378a032dfc02e5ac0ee34b7
deduplicated the base test class methods from the tempest.lib
reintegration. However the surviving base class was put in the wrong
namespace. The base class is just that a base to build the individual
test subclass off of. It's very confusing for that to not live in the
base tests namespace. While the code in lib is supposed to be self
contained there isn't a reason to do that here for the unit tests,
especially because in this case the import heiarchy becomes more
confusing with the base living in a submodule. This commit moves the
base class to the base tempest.tests namespace where it really
belongs.
Change-Id: I079eeb1135eed3254e5e9dbebac8a52b979303c7
Those 2 files were identical. We should use `tempest.tests.lib.base`
now that we are commited to tempest.lib.
Change-Id: I45d73891f3c0829a2378a032dfc02e5ac0ee34b7
Similar to what Cinder did here [1], this patch mocks time.sleep
to make tests run faster. Some code actually measure the wall clock
to wait for a specific duration before raising a TimeoutError, so
we also need to mock time.time.
This removes ~5sec in unit test and also removes some busy waiting.
[1] https://review.openstack.org/#/c/285658/
Change-Id: I69ba35eff591a5df28049273f3aba15c31f52d00
The tempest/services/volume directory had the following structure:
tempest
|___services
|___volume
|____json
| |____admin
| | |____<client files>
| |
| |____<client files>
|
|____v2
|____json
|____admin
| |____<client files>
|
|____<client files>
The structure was changed to keep consistency with other clients directories.
For example, the tempest/services/identity derectory and the tempest/services/image
directory. So now the structure is the following:
tempest
|___services
|___volume
|____base
| |____<client files>
|
|____v1
| |____json
| |____admin
| | |____<client files>
| |
| |____<client files>
|
|____v2
|____json
|____admin
| |____<client files>
|
|____<client files>
The tempest/services/volume/base directory was created and now it containes base
client files with base client classes for v1 and v2 Cinder clients.
Change-Id: I0ac37e15eddffb12d150dcd5b68e0f933712d22c
Provide the entire response object for all methods of the
VolumeServicesClient
partially implements: blueprint method-return-value-and-move-service-clients-to-lib
Change-Id: I31c7ae14c514d736773db034baa5fe77e3f23370
This simply makes the test_waiters module mimic the source
location for tempest/common/waiters so that goofballs such as
myself don't try to create a new test module to add new tests.
Change-Id: Ic82f05f6c3032f7412dc9a588112815d6df1db22
Zuul