Browse Source

Initial draft specification of introspective instance monitoring.

Specifically VM Heartbeat Monitoring via the QEMU Guest Agent.

Implements-blueprint: introspective-instance-monitoring
Change-Id: Ie41d92651128b41967c1118bbcdaf3656c498801
Signed-off-by: Greg Waines <>
Greg Waines 4 years ago
  1. 240


@ -0,0 +1,240 @@
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
Introspective Instance Monitoring
Currently, Masakari instance monitoring is strictly non-intrusive black-box
type monitoring through qemu and libvirt. There are however a number of
internal instance/VM faults (kernel scheduling and IO, application health),
that if detected by Masakari, could be recovered by existing Masakari auto-recovery
mechanisms; increasing the overall availability of the instance/VM. This blueprint
introduces the capability of performing introspective instance monitoring of VMs, in
order to detect, report and optionally recover VMs from internal VM faults. Specifically,
VM Heartbeat Monitoring via the QEMU Guest Agent is introduced by this spec, in order
to indirectly detect some of these internal VM faults.
Problem description
Currently, Masakari instance monitoring is a strictly non-intrusive black-box
type monitoring through qemu and libvirt. This detects a number of faults
for which Masakari's auto-recovery mechanisms can be used to recover the
However, there are a number of internal instance/VM faults not detected by
this black-box monitoring, that if detected by Masakari, could be recovered
by these same Masakari auto-recovery mechanisms. This includes faults such as
hung Guest OS, failure of the Guest OS to schedule Application process(es), failure
to route basic IO within the Guest, Application-specific process failures or data
corruption, etc. . The exact scope of the proposed monitoring of this blueprint
is described at the end of the 'Proposed change' section.
Monitoring of Internal instance/VM faults requires that the Guest VM
supports software to respond to this monitoring. In the following proposal,
the Guest VM must support the QEMU Guest Agent. Because not all VMs will support
this software, the monitoring of internal instance/VM faults, by the OpenStack Host,
must be optionally enabled per VM or per VM image.
Proposed change
This blueprint introduces introspective instance monitoring; specifically, VM
Heartbeat Monitoring via the QEMU Guest Agent. Any VM Heartbeat fault will be
reported through the Masakari instance-alerter to registered API drivers
(e.g. masakari-api).
The high-level architecture for Introspective Instance Monitoring is shown below::
+--------------------+ instance +-------------+ + - - - - - - +
| instance-alerter |<------------| Masakari | | |
|- - - - - - - - - - | fault | VM | F U T U R E
| driver abstraction | | Heartbeat | | |
| layer | | Agent |
+--------------------+ +-------------+ + - - - - - - +
| | ^ ^
other <--+ | | |
apis | | +----------------+
v | |
+--------------------+ | |
| masakari-api | v v
+--------------------+ +-------------+
| | Libvirtd |
v +-------------+
+--------------------+ ^
| masakari-engine | | unix socket
+--------------------+ v
| +-------------+
| (recovery) | QEMU |
v +-------------+
+--------------------+ ^
| | |
| OpenStack | +--------------------------------------+
| | | VM | virtio serial device |
+--------------------+ | v |
| +--------------------+ |
| | QEMU | |
| | Guest Agent | |
| | ( guest-ping{} ) | |
| +--------------------+ |
| |
| +-------------+ |
| +-------------+ | |
| | | | |
| | Application | | |
| | | + |
| +-------------+ |
VM Heartbeat and Healthcheck Monitoring will leverage the QEMU feature, Guest
Agent [1], for both the transport level
communication between OpenStack Host and the Guest VM, and the built-in
guest ping command (guest-ping{}). A QEMU Guest Agent
daemon, built as part of QEMU, is installed and run inside the Guest and
implements support for QMP commands that are sent to
the guest. Specifically the QEMU Guest Agent daemon
connects to a virtio-serial device (/dev/virtio-ports/org.qemu.guest_agent.0),
feeds the input to a QMP JSON parser, and when a command is received, invokes
the QAPI generated dispatch routine. In the case of VM Heartbeat Monitoring,
the QEMU Guest Agent command, 'guest-ping', will be used as the heartbeat challenge
request from the Host.
On the host, OpenStack Nova already supports an image property,
hw_qemu_guest_agent, that can be used to specify that the VM should
be created with the QEMU guest agent virto-serial-interface. The Masakari
VM Heartbeat Agent will discover VMs with hw_qemu_guest_agent enabled
by monitoring the files representing the socket identifiers for the QEMU Guest
Agents' virtual-serial-interfaces.
libvirt-qemu provides a virDomainQemuAgentCommand() for sending commands
to a selected VM's QEMU guest agent. This command opens the unix socket to
the VM's virtio-serial-interface, sends the command, waits to receive the response
and closes the socket. The command fails if the unix socket is openned by
another process, i.e. another process is sending a command to the same VM.
Masakari VM Heartbeat Agent will leverage virDomainQemuAgentCommand() provided
by libvirtd to send the heartbeat challenge requests (i.e. the QEMU Guest Agent's
guest-ping command) to the VM(s) and report any detected faults to the masakari
The Masakari VM Heartbeat Agent, on the host, will initiate VM Heartbeating as soon
as it discovers the VM has QEMU Guest Agent communication enabled. However, in order
to deal with arbitrary boot times for VMs/Guests, which may delay the Guests ability
to start responding to the heartbeat challenges, the Masakari VM Heartbeat Agent will
not enable reporting of heartbeat failures until after the first successful heartbeat
response is received from the VM/Guest.
This functionality will support a flag in masakari.conf for overall enabling/disabling of
introspective-instance-monitoring. It will also support parameters for configuring
default heartbeat period and default consecutive heartbeat miss threshold (before
declaring fault); in future, flavor extraspecs could be used for VMs to specify
specific values for these.
At a high-level, the scope of this heartbeat monitoring is that the QEMU Guest Agent
is running within the VM. However, just the fact that a Heartbeat message can get
from the Host to the QEMU Guest Agent inside the VM and back, inherently validates
that a lot of basic Guest Kernel functionality is working; i.e. the Guest OS is not
hung or failed, the QEMU heartbeat message was properly routed through basic linux
socket IO, etc. . In the future, the heartbeating can be extended to
do more than just reply/ack the message ... i.e. basic sanity / health tests on key
applications within the VM can be done.
Could simply leverage the virtual hardware watchdog of QEMU/KVM
[2] for Instance monitoring.
However, VM Heartbeat Monitoring:
- provides notification of the Heartbeat status to higher-level cloud
entities through instance-alerter, such as Masakari, Mistral and/or Vitrage,
* which depending on the backend can result in VM auto-recovery (Masakari) or
deduced-state updates in Nova for the VM and resulting Aodh Event generation
due to the VM state change (Vitrage).
- in the future can be extended to provide a higher-level (i.e. application-level)
* i.e. if the Heartbeat requests are being answered by the Application running
within the VM
- in the future can be extended to provide more than just heartbeating, as the
Application can use it to trigger a variety of audits,
- in the future can be extended to provide a mechanism for the Application within the
VM to report a Health Status / Info back to the Host / Cloud.
Only VMs supporting the QEMU Guest Agent can be monitored by the functionality of
this proposal.
Primary assignee:
Target Milestone for completion:
Work Items
- Masakari VM Heartbeat Agent on the Compute
* discovery of VMs with QEMU Guest Agent communication enabled,
* high-level logic for Heartbeat / Healthcheck monitoring,
* reporting of faults to masakari instance-alerter.
- tox and/or tempest test suite updates
- masakari documentation updates
- requires that VMs are installed with and running the QEMU Guest Agent [1]
built as part of QEMU.