Merge "Libvirt: Use the virtlogd deamon"
This commit is contained in:
Jenkins
310
specs/newton/approved/libvirt-virtlogd.rst
Normal file
310
specs/newton/approved/libvirt-virtlogd.rst
Normal file
@@ -0,0 +1,310 @@
|
||||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
================================
|
||||
Libvirt: Use the virtlogd deamon
|
||||
================================
|
||||
|
||||
https://blueprints.launchpad.net/nova/+spec/libvirt-virtlogd
|
||||
|
||||
If the *serial console* feature is enabled on a compute node with
|
||||
``[serial_console].enabled = True`` it deactivates the logging of the
|
||||
boot messages. From a REST API perspective, this means that the two APIs
|
||||
``os-getConsoleOutput`` and ``os-getSerialConsole`` are mutually exclusive.
|
||||
Both APIs can be valuable for cloud operators in the case when something
|
||||
goes wrong during the launch of an instance. This blueprint wants to lift
|
||||
the XOR relationship between those two REST APIs.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
The problem can be seen in the method ``_create_serial_console_devices``
|
||||
in the libvirt driver. The simplified logic is::
|
||||
|
||||
def _create_serial_console_devices(self, guest, instance, flavor,
|
||||
image_meta):
|
||||
if CONF.serial_console.enabled:
|
||||
console = vconfig.LibvirtConfigGuestSerial()
|
||||
console.type = "tcp"
|
||||
guest.add_device(console)
|
||||
else:
|
||||
consolelog = vconfig.LibvirtConfigGuestSerial()
|
||||
consolelog.type = "file"
|
||||
guest.add_device(consolelog)
|
||||
|
||||
This ``if-else`` establishes the XOR relationship between having a log of
|
||||
the guest's boot messages or getting a handle to the guest's serial console.
|
||||
From a driver point of view, this means getting valid return values for the
|
||||
method ``get_serial_console`` or ``get_console_output`` which are used to
|
||||
satisfy the two REST APIs ``os-getConsoleOutput`` and ``os-getSerialConsole``.
|
||||
|
||||
Use Cases
|
||||
----------
|
||||
|
||||
From an end user point of view, this means that, with the current state, it
|
||||
is possible to get the console output of an instance on host A (serial console
|
||||
is not enabled) but after a rebuild on host B (serial console is enabled) it
|
||||
is not possible to get the console output. As an end user is not aware of the
|
||||
host's configuration, this could be a confusing experience. Written that down
|
||||
I'm wondering why the serial console was designed with a compute node scope
|
||||
and not with an instance scope, but that's another discussion I don't want to
|
||||
do here.
|
||||
|
||||
After the implementation, deployers will have both means by hand if there is
|
||||
something wrong during the launch of an instance. The persisted log in case
|
||||
the instance crashed AND the serial console in case the instance launched but
|
||||
has issues, for example a failed establishing of networking so that SSH access
|
||||
is not possible. Also, they will be impacted with a new dependency on the
|
||||
hosts (see `Dependencies`_).
|
||||
|
||||
Developers won't be impacted.
|
||||
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
I'd like to switch from the log file to the ``virtlogd`` deamon. This logging
|
||||
deamon was announced on the libvirt ML [1] and is available with libvirt
|
||||
version 1.3.3 and Qemu 2.6.0. This logging deamon handles the output from the
|
||||
guest's console and writes it into the file
|
||||
``/var/log/libvirt/qemu/guestname-serial0.log`` on the host but
|
||||
truncates/rotates that log so that it doesn't exhaust the hosts disk space
|
||||
(this would solve an old bug [3]).
|
||||
|
||||
Nova would generate::
|
||||
|
||||
<serial type="tcp">
|
||||
<source mode="connect" host="0.0.0.0" service="2445"/>
|
||||
<log file="/var/log/libvirt/qemu/guestname-serial0.log" append="on"/>
|
||||
<protocol type="raw"/>
|
||||
<target port="1"/>
|
||||
</serial>
|
||||
|
||||
For providing the console log data, nova would need to read the console
|
||||
log file from disk directly. As the log file gets rotated automatically
|
||||
we have to ensure that all necessary rotated log files get read to satisfy
|
||||
the upper limit of the ``get_console_output`` driver API contract.
|
||||
|
||||
|
||||
FAQ
|
||||
---
|
||||
|
||||
#. How is the migration/rebuild handled? The 4 cases which are possible
|
||||
(based on the node's patch level):
|
||||
|
||||
#. ``N -> N``: Neither source nor target node is patched. That's what
|
||||
we have today. Nothing to do.
|
||||
|
||||
#. ``N -> N+1``: The target node is patched, which means it can make
|
||||
use of the output from *virtlogd*. Can we "import" the existing log
|
||||
of the source node into the *virtlogd* logs of the target node?
|
||||
|
||||
A: The guest will keep its configuration from the source host
|
||||
and don't make use of the *virtlogd* service until it gets rebuilt.
|
||||
|
||||
#. ``N+1 -> N``: The source node is patched and the instance gets
|
||||
migrated to a target node which cannot utilize the *virtlogd*
|
||||
output. If the serial console is enable on the target node, do
|
||||
we throw away the log because we cannot update it on the target
|
||||
node
|
||||
|
||||
A: In the case of migration to an old host, we try to copy the
|
||||
existing log file across, and configure the guest with the
|
||||
``type=tcp`` backend. This provides ongoing support for interactive
|
||||
console. The log file will remain unchanged if possible. A failed
|
||||
copy operation should not prevent the migration of the guest.
|
||||
|
||||
#. ``N+1 -> N+1``: Source and target node are patched. Will libvirt
|
||||
migrate the existing log from the source node too, which would
|
||||
solve another open bug [4].
|
||||
|
||||
#. Q: Could a stalling of the guest happen if *nova-compute* is reading the
|
||||
log file and *virtlogd* tries to write to the file but is blocked?
|
||||
|
||||
A: No, *virtlogd* will ensure things are fully parallelized
|
||||
|
||||
#. Q: The *virtlogd* deamon has a ``1:1`` relationship to a compute node.
|
||||
It would be interesting how well it performs when, for example,
|
||||
hundreds of instances are running on one compute node.
|
||||
|
||||
A: We could add a I/O rate limit to *virtlogd* so it refuses to read data
|
||||
too quickly from a single guest. This prevents a single guest DOS'ing
|
||||
the host.
|
||||
|
||||
#. Q: Are there architecture dependencies? Right now, a nova-compute node on a
|
||||
s390 architecture depends on the *serial console* feature because it
|
||||
cannot provide the other console types (VNC, SPICE, RDP). Which means it
|
||||
would benefit from having both.
|
||||
|
||||
A: No architecture dependencies.
|
||||
|
||||
#. Q: How are restarts of the *virtlogd* deamon handled? Do we lose
|
||||
information in the timeframe between stop and start?
|
||||
|
||||
A: The *virtlogd* daemon will be able to re-exec() itself while keeping
|
||||
file handles open. This will ensure no data loss during restart of
|
||||
*virtlogd*.
|
||||
|
||||
#. Q: Do we need a version check of libvirt to detect if the *virtlodg* is
|
||||
available on the host? Or is it sufficient to check if the folder
|
||||
``/var/log/virtlogd/`` is present?
|
||||
|
||||
A: We will do a version number check on libvirt to figure out if it is
|
||||
capable to use it.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
#. In case where the *serial console* is enabled, we could establish a
|
||||
connection to the guest with it and execute ``tail /var/log/dmesg.log``
|
||||
and return that output in the driver's ``get_console_output`` method which
|
||||
is used to satisfy the ``os-getConsoleOutput`` REST API.
|
||||
|
||||
**Counter-arguments:** We would need to save the authentication data to
|
||||
the guest, which would not be technically challenging but the customers
|
||||
could be unhappy that Nova can access their guests at any time. A second
|
||||
argument is, that the serial console access is blocking, which means
|
||||
if user A uses the serial console of an instance, user B is not able to do
|
||||
the same.
|
||||
|
||||
#. We could remove the ``if-else`` and create both devices.
|
||||
|
||||
**Counter-arguments:** This was tried in [2] and stopped because this could
|
||||
introduce a backwards incompatibility which could prevent the rebuild
|
||||
of an instance. The root cause for this was, that there is an upper bound
|
||||
of 4 serial devices on a guest, and this upper bound could be exceeded if
|
||||
an instance which already has 4 serial devices gets rebuilt on a compute
|
||||
node which would have patch [2].
|
||||
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
None
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
None
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
* The *virtlogd* service has to run for this functionality and should be
|
||||
monitored.
|
||||
* This would also solve a long-running bug which can cause a host disc space
|
||||
exhaustion (see [3]).
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
Markus Zoeller (https://launchpad.net/~mzoeller)
|
||||
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
* (optional) get a gate job running which has the *serial console* activated
|
||||
* add version check if libvirt supports the *virtlogd* functionality
|
||||
* add "happy path" which creates a guest device which uses *virtlogd*
|
||||
* ensure "rebuild" uses the new functionality when migrated from an old host
|
||||
* add reconfiguration of the guest when migrating from N+1 -> N hosts
|
||||
to keep backwards compatibility
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
* Libvirt 1.3.3 which brings the *libvirt virtlod logging deamon* as
|
||||
described in [1].
|
||||
* Qemu 2.6.0
|
||||
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
The tempest tests which are annotated with
|
||||
``CONF.compute_feature_enabled.console_output`` will have to work with
|
||||
a setup which
|
||||
|
||||
* has the dependency to the *virtlogd deamon* resolved.
|
||||
* AND has the serial console feature enabled (AFAIK there is not job right
|
||||
now which has this enabled)
|
||||
|
||||
* A new functional test for the live-migration case has to be added
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
None
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
[1] libvirt ML, "[libvirt] RFC: Building a virtlogd daemon":
|
||||
http://www.redhat.com/archives/libvir-list/2015-January/msg00762.html
|
||||
|
||||
[2] Gerrit; "libvirt: use log file and serial console at the same time":
|
||||
https://review.openstack.org/#/c/188058/
|
||||
|
||||
[3] Launchpad; " console.log grows indefinitely ":
|
||||
https://bugs.launchpad.net/nova/+bug/832507
|
||||
|
||||
[4] Launchpad; "live block migration results in loss of console log":
|
||||
https://bugs.launchpad.net/nova/+bug/1203193
|
||||
|
||||
[5] A set of patches on the libvirt/qemu ML:
|
||||
|
||||
* [PATCH 0/5] Initial patches to introduce a virtlogd daemon
|
||||
* [PATCH 1/5] util: add API for writing to rotating files
|
||||
* [PATCH 2/5] Import stripped down virtlockd code as basis of virtlogd
|
||||
* [PATCH 3/5] logging: introduce log handling protocol
|
||||
* [PATCH 4/5] logging: add client for virtlogd daemon
|
||||
* [PATCH 5/5] qemu: add support for sending QEMU stdout/stderr to virtlogd
|
||||
|
||||
[6] libvirt ML, "[libvirt] [PATCH v2 00/13] Introduce a virtlogd daemon":
|
||||
https://www.redhat.com/archives/libvir-list/2015-November/msg00412.html
|
||||
|
||||
History
|
||||
=======
|
||||
|
||||
.. list-table:: Revisions
|
||||
:header-rows: 1
|
||||
|
||||
* - Release Name
|
||||
- Description
|
||||
* - Newton
|
||||
- Introduced
|
||||
Reference in New Issue
Block a user