masakari/releasenotes/notes/evacuation_in_threads-cc9c79b10acfb5f6.yaml
dineshbhor 25d33d2cb1 Fix race condition between evacuation and its confirmation
Masakari can face a race condition where after evacuation of an
instance to other host user might perform some actions on that
instance which gives wrong instance vm_state to ConfirmEvacuationTask
that results into notification failure.

To fix this issue this patch proposes to lock the instance before
evacuation till its confirmation so that any normal user will not
be able to perform any actions on it. To achieve this the
ConfirmEvacuationTask is completly removed and the confirmation is
done in the EvacuateInstancesTask itself by per instance.
Evacuating an instance and confirming it's evacuation immediately
can reduce the performance so this patch uses the
eventlet.greenpool.GreenPool which executes the complete evacuation
and confirmation of an instance in a separate thread.
To check if the server is already locked or not upgraded the
novaclient's NOVA_API_VERSION from 2.1 to 2.9  as the 'locked'
property is available in nova api_version 2.9 and above.

This patch introduces a new config option
'host_failure_recovery_threads' which will be the number of threads
to be used for evacuating and confirming the instances evacuation.
The default value for this config option is 3.

Closes-Bug: #1693728
Change-Id: Ib5145878633fd424bca5bcbd5cfed13d20362f94
2017-06-23 13:08:03 +05:30

19 lines
642 B
YAML

---
fixes:
- |
Fixes `bug 1693728`_ which will fix the race condition where after
evacuation of an instance to other host user might perform some actions on
that instance which gives wrong instance vm_state to ConfirmEvacuationTask
that results into notification failure.
To fix this issue, following config option is added under ``DEFAULT``
section in 'masakari.conf' file::
[DEFAULT]
host_failure_recovery_threads = 3
This config option decides the number of threads going to be used for
evacuating the instances.
.. _`bug 1693728`: https://bugs.launchpad.net/masakari/+bug/1693728