* The functionality of graceful engine shutdown is now possible
due to correct calculation of the "graceful" flag in the engine
server's stop() method. Unfortunately, the Oslo Service framework
doesn't pass it correctly, it simply ignores it in the call chain.
So the only way to understand if the shutdown is graceful is to
peek at the configuration property "graceful_shutdown_timeout"
provided by Oslo Service. If it's greater than zero then we can
treat it as graceful.
* Oslo Service handles only four OS signals: SIGTERM, SIGINT,
SIGHUP and SIGALRM. Only sending SIGTERM to the process leads
to a graceful shutdown. For example, SIGINT (which is equal to
ctrl + C in a unix shell) interrupts the process immediately.
So the only way to do a graceful shutdown of an engine instance
using a unix shell is to run the "kill <PID>" command. This
needs to be taken into account when using it.
* The patch also changes the order in which the engine server
stops its inner services so that the underlying RPC server
(currently Oslo Messaging based or Kombu based) stops first.
This is needed to make sure that, first of all, no new RPC
calls can arrive, and thereby, let all active DB transactions
finish normally w/o starting new ones. Stopping the RPC server
may be a heavy operation if there are already lots of RPC
messages waiting for processing that are polled from the queue.
So to the great extent the entire functionality of graceful
shutdown will depend on whether an underlying RPC server
implements the corresponding functionality in the proper way,
i.e. after calling stop(graceful=True) it will stop receiving
new calls and wait till all buffered RPC messages are processed
normally.
* The maximum time given to graceful shutdown is controlled via
the "graceful_shutdown_timeout" configuration option, which is
60 seconds, by default.
* Minor refactoring
Implements blueprint: mistral-graceful-scale-in
Change-Id: I6d1234dfa21b1e3420ec9ca2c5235dee973748ee
f61929a3c8