Previously, the stop_stack message accidentally used the
engine_life_check_timeout (by default, 2s). But unlike other messages sent
using that timeout, stop_stack needs to synchronously kill all running
threads operating on the stack. For a very large stack, this can easily
take much longer than a couple of seconds. This patch increases the timeout
to give a better chance of being able to start the delete.
Change-Id: I4b36ed7f1025b6439aeab63d71041bb2000363a0
Closes-Bug: #1499669
The error messages 'Command Out of Sync' are due to the threads being
stopped in the middle of the database operations. This happens in the
legacy action when delete is requested during a stack create.
We have the thread cancel message but that was not being used in this
case. Thread cancel should provide a more graceful way of ensuring the
stack is in a FAILED state before the delete is attempted.
This changes does the following in the delete_stack service method for
legace engine:
- if the stack is still locked, send thread cancel message
- in a subthread wait for the lock to be released, or until a
timeout based on the 4 minute cancel grace period
- if the stack is still locked, do a thread stop as before
Closes-Bug: #1499669
Closes-Bug: #1546431
Closes-Bug: #1536451
Change-Id: I4cd613681f07d295955c4d8a06505d72d83728a0
Context is added as the first argument to stack_lock_* db functions
but is currently ignored for getting the session used for stack lock
operations.
This is required later for bug 1479723 but is added in its own change
here to ease reviewing load.
Change-Id: Ieb3e4e2ee67150777cbe1e961d0d1806cf1f7e46
Related-Bug: #1479723
Having a stack which is wedged UPDATE_IN_PROGRESS due to another bug,
restarting heat-engine should have resulted in the stack being reset to
UPDATE_FAILED. However the reset was skipped because the lock engine_id
was None.
This change removes the engine_id check since it is possible in practice
for an UPDATE_IN_PROGRESS stack to not be locked.
Since it is possible for the stack to change state in the unlocked
window, a check is added that the stack is still IN_PROGRESS before the
reset is performed.
Change-Id: I1739ccbdf75af35aac5be16b99200975df58b8e2
Closes-Bug: #1514615
This patch splits the stack watch related unit tests into a separate
module. Some mox calls are replaced with mock calls where approriate.
It also moves the existing modules related to engine service tests
into a separate subdirectory. This is to make room for other unit
tests that are coming in.
Change-Id: I7ea61c22e017785c480d78d6c07132164ed0b889
Zane Bitter