When TOBIKO_PREVENT_CREATE is True, the retry count for
creating a stack were set to 0, causing the retry to
repeat forever until success.
This prevents it to happen by handling TOBIKO_PREVENT_CREATE
variable explicitly and avoid entering in the retry
creation loop in case it is True
This change also adds some additional log messages to
help investigating on stack creation problems.
Change-Id: I911e073c6a1dee926a6b16ea59930bc78c3f8396
Some faults tests are not stable because stack deletion is not completed during
the cleanup of the previous tests
Sometimes the disruption of the next test is executed while the stack
deletion is in progress and due to this the stack remains in state
delete in progress forever
Change-Id: I07f1e21d89174cfeaca77a93bfa8772a757b1afb
This patch avoids the creation of a stack that has not been completely
deleted before by waiting stack disappears after reaching
DELETE_COMPLETE state and before restarting creating it again.
Change-Id: If21ed5e8b99973e42e1beb05bbbe9aa619584ade
THE CHANGE
This also introduce some new features:
- allows to override a stack verification step to be executed
at the end of stack creation to check the stack is in healthy
cond and eventually retry creating it
- restructure fail safe and concurrent creation loop
to make it easier to understand and maintain
- this should fix some different-host of same-host test case
failures tipically seen in downstream jobs by checking
nova scheduler hints are respected and eventually skip neutron
test cases thare pretend to check connectivity between
vms running on different (or the same) hypervisor host.
THE PROBLEM
When schedulers hints are given to nova to chose where to allocate
new VMs, there are no warranties to make sure it actually follows
them. In special it has saw many times that it fails to do it
expecially when many VMs are being created on the same time, for
example when running tobiko from a machine with many CPU core.
On such case test runner decides to spawn many parallel workers
(typically 2 for every core when executing on hyperthreading CPUs).
It has been seen the more test runner workers executing tests
the highest is the probability these hints are unmet by Nova.
THE WORKAROUND
To workaround this problem with this change we are verifying
scheduler hints has met. If not it retries creating them up
to two times before raising a SkipException to skip test
current case.
Change-Id: I8c0cff5ca69680aba3842bd7738f27651a677633
This is a workaround for a concurrency issue detected on Heat
OSP-14, but that could be affecting any version of Heat (no
investigation has been perfomed). It is unknow if this is a bug
or just a limitation of the service.
The issue:
Heat looks like has a concurrency issue when creating new stacks with
the same name on ame time from more parallel processes.
When Tobiko Neutron test cases are executed using tox -e neutron command
test case execution is spawn along some worker processes.
It happens that the same HeatStackFixture, as it is shared between more
test cases, is being created on the same time from more workers.
The stack creation request is therefore submitted to Heat service with
the same parameters and despite what understood from the documentation,
more than one stack starts being created on the same time with the same
name.
Workaround implementation
After these stack creation is started tobiko asks for stack status
using stack name and all worker processes receives information of
the same stack (with the same stack ID).
Therefore those workers that creates a new stack with an ID different
from the one got by the stack name are deleted, leaving on most of the
cases the cloud with only one stack instance for the same stack name.
Change-Id: Ibcdc04d9436664b788d2ed6d68e2a20c74bd6147
Zuul