Robustify Compute Node Hostnames backlog spec
This is mostly a brain-dump spec on the topic of compute hosts and how fragile they are in terms of hostname handling. There has long been a requirement that computes can NEVER change hostnames, and we have few tools to even detect the problem before we corrupt the database if it happens. Here I have documented some of the things we could do to make that more robust, should we choose to do so. This is based on a recent near-catastrophe and thus reflects things that would have avoided pain in a real scenario. Per discussion at the PTG, I am adding this as a backlog spec, to be an overarching guide for multiple smaller specs to provide more detailed progress towards the goals described here. Change-Id: I72fa3f605cfcf7c3dd0ff4c791be7df8f19f058b
This commit is contained in:
Dan Smith
@@ -24,4 +24,8 @@ Template:
|
|||||||
|
|
||||||
Approved (but not implemented) backlog specs:
|
Approved (but not implemented) backlog specs:
|
||||||
|
|
||||||
None currently
|
.. toctree::
|
||||||
|
:glob:
|
||||||
|
:maxdepth: 1
|
||||||
|
|
||||||
|
approved/*
|
||||||
|
|||||||
317
specs/backlog/approved/robustify-compute-hostnames.rst
Normal file
317
specs/backlog/approved/robustify-compute-hostnames.rst
Normal file
@@ -0,0 +1,317 @@
|
|||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
========================================
|
||||||
|
Robustify Compute Node Hostname Handling
|
||||||
|
========================================
|
||||||
|
|
||||||
|
Include the URL of your launchpad blueprint:
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/nova/+spec/example
|
||||||
|
|
||||||
|
Nova has long had a dependency on an unchanging hostname on the
|
||||||
|
compute nodes. This spec aims to address this limitation, at least
|
||||||
|
from the perspective of being able to detect an accidental change and
|
||||||
|
avoiding catastrophe in the database that can currently result from a
|
||||||
|
hostname change, whether intentional or not.
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
Currently nova uses the hostname of the compute (specifically
|
||||||
|
``CONF.host``) for a variety of things:
|
||||||
|
|
||||||
|
#. As the routing key for communicating with a compute node over RPC
|
||||||
|
#. As the link between the instance, service and compute node objects
|
||||||
|
in the database
|
||||||
|
#. For neutron to bind ports to the proper hostname (and in some
|
||||||
|
cases, it must match the equivalent setting in the neutron agent
|
||||||
|
config)
|
||||||
|
#. For cinder to export a volume to the proper host
|
||||||
|
#. As the resource provider name in placement (this actually comes
|
||||||
|
from libvirt's notion of the hostname, not ``CONF.host``)
|
||||||
|
|
||||||
|
If the hostname of the compute node changes, all of these links
|
||||||
|
break. Upon starting the compute node with the changed name, we will
|
||||||
|
be unable to find a ``nova-compute`` ``Service`` record in the
|
||||||
|
database that matches, and will create a new one. After that, we will
|
||||||
|
fail to find the matching ``ComputeNode`` record and create a new one
|
||||||
|
of those, with a new UUID. Instances that refer to both the old
|
||||||
|
compute and service records will not be associated with the running
|
||||||
|
host, and thus become unmanageable through the API. Further, new
|
||||||
|
instances that end up created on the compute node after the rename
|
||||||
|
will be able to claim resources that have been promised to the
|
||||||
|
orphaned instances (such as PCI devices and VCPUs) as the tracking of
|
||||||
|
those is associated with the old compute node record.
|
||||||
|
|
||||||
|
If the orphaned instances are relatively static, the first indication
|
||||||
|
that something has gone wrong may be long after the actual rename,
|
||||||
|
where reality has forked and there are instances running on one
|
||||||
|
compute node that refer to two different compute node records and thus
|
||||||
|
are accounted for in two separate locations.
|
||||||
|
|
||||||
|
Further, neutron, cinder, and placement resources will have the old
|
||||||
|
information for existing instances and new information for current
|
||||||
|
instances, which requires reconciliation. This situation may also
|
||||||
|
prevent restarting old instances if the old hostname is no longer
|
||||||
|
reachable.
|
||||||
|
|
||||||
|
Use Cases
|
||||||
|
---------
|
||||||
|
|
||||||
|
* As an operator, I want to make sure my database does not get
|
||||||
|
corrupted due to a temporary or permanent DNS change or outage
|
||||||
|
* As an operator, I may need to change the name of a compute node as
|
||||||
|
my network evolves over many years.
|
||||||
|
* As a deployment tool writer, I want to make sure that changes in
|
||||||
|
tooling and libraries never cause data loss or database corruption.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
There are multiple things we can do here to robustify Nova's handling
|
||||||
|
of this data. Each one increases safety, but we do not have to do all
|
||||||
|
of them.
|
||||||
|
|
||||||
|
Ensure a stable compute node UUID
|
||||||
|
---------------------------------
|
||||||
|
|
||||||
|
For non-ironic virt drivers, whenever we generate a compute node uuid,
|
||||||
|
we should write that to a file on the local disk. Whenever we start,
|
||||||
|
we should look for that UUID file, use that, and under no
|
||||||
|
circumstances should we generate another one. To facilitate
|
||||||
|
pre-generating this by deployment tools, we should use this if we are
|
||||||
|
starting for the first time and create a ComputeNode record in the
|
||||||
|
database.
|
||||||
|
|
||||||
|
We would put the actual lookup of the compute node UUID in the
|
||||||
|
`get_available_nodes()` method of the virt driver (or create a new
|
||||||
|
UUID-specific one). Ironic would override this with its current
|
||||||
|
implementation that returns UUIDs based on the state of Ironic and the
|
||||||
|
hash ring. Thus only non-Ironic computes would read and write the
|
||||||
|
persistent UUID file.
|
||||||
|
|
||||||
|
Single-host virt drivers like libvirt would be able to tolerate a
|
||||||
|
system hostname change, updating ``ComputeNode.hypervisor_hostname``
|
||||||
|
without breaking things.
|
||||||
|
|
||||||
|
Link ComputeNode records with Service records by id
|
||||||
|
---------------------------------------------------
|
||||||
|
|
||||||
|
Currently the ComputeNode and Service records are associated in the
|
||||||
|
database purely by the hostname string. This means that they can
|
||||||
|
become disassociated, and is also not ideal from a performance
|
||||||
|
standpoint. Some other data structures are linked against ComputeNode
|
||||||
|
by id, and thus are not re-associated when the name matches.
|
||||||
|
|
||||||
|
This relationship used to exist, but was `removed`_ in the Kilo
|
||||||
|
timeframe. I believe this was due to the desire to make the process
|
||||||
|
less focused on the service object and more on the compute node
|
||||||
|
(potentially because of Ironic) although the breaking of that tight
|
||||||
|
relationship has serious downsides as well. I think we can keep the
|
||||||
|
tight binding for single-host computes where it makes sense.
|
||||||
|
|
||||||
|
At startup, ``nova-compute`` should resolve its ComputeNode object via
|
||||||
|
the persistent UUID, find the associated Service, and fail to start if
|
||||||
|
the hostname does not match CONF.host. Since this is used with
|
||||||
|
external services, we should not just "fix it" as those other links
|
||||||
|
will be broken as well. This will at least allow us to avoid opening
|
||||||
|
the window for silent data corruption.
|
||||||
|
|
||||||
|
Link Instances to a ComputeNode by id
|
||||||
|
-------------------------------------
|
||||||
|
|
||||||
|
Currently instance records are linked to their Service and ComputeNode
|
||||||
|
objects purely by hostname. We should link them to a ComputeNode by
|
||||||
|
its id. Since we need the Service in order to get the RPC routing key
|
||||||
|
or for hostname resolution when talking to external services, we
|
||||||
|
should find that based on the Instance->ComputeNode->Service id
|
||||||
|
relationship.
|
||||||
|
|
||||||
|
We already link PCI allocations for instances to the compute node by
|
||||||
|
id, even though the instance itself is linked via hostname. This
|
||||||
|
discrepancy makes it easy to get one out of sync with the other.
|
||||||
|
|
||||||
|
Potential Changes in the future
|
||||||
|
-------------------------------
|
||||||
|
|
||||||
|
If the above changes are made, we open ourselves to the future
|
||||||
|
possibility for supporting:
|
||||||
|
|
||||||
|
#. Renaming service objects through the API if a compute host really
|
||||||
|
needs to have its hostname changed. This will require changes to
|
||||||
|
the other services at the same time, but nova would at least have a
|
||||||
|
single source of truth for the hostname, making it feasible.
|
||||||
|
#. If we do all of this, Nova could potentially be confident enough of
|
||||||
|
an intentional rename that it could update port bindings, cinder
|
||||||
|
volume attachments, and placement resources to make it seamless.es
|
||||||
|
#. Moving to the use of the service UUID as the RPC routing key, if
|
||||||
|
desired.
|
||||||
|
#. Dropping quite a bit of duplicate string fields from our database.
|
||||||
|
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
We can always do nothing. Compute hostnames have been unchangeable
|
||||||
|
forever, and the status quo is "don't do that or it will break" which
|
||||||
|
is certainly something we could continue to rely on.
|
||||||
|
|
||||||
|
We could implement part of this (i.e. the persistent ComputeNode UUID)
|
||||||
|
without the rest of the database changes. This would allow us to
|
||||||
|
detect the situation and abort, but without (the work required to get)
|
||||||
|
the benefits of a more robust database schema that could potentially
|
||||||
|
also support voluntary renames.
|
||||||
|
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
Most of the impact here is to the data model for Instance,
|
||||||
|
ComputeNode, Service. Other models that reference compute hostnames
|
||||||
|
may also make sense to change (although it's also reasonable to punt
|
||||||
|
that entirely or to a different phase). Examples:
|
||||||
|
|
||||||
|
* Migration
|
||||||
|
* InstanceFault
|
||||||
|
* InstanceActionEvent
|
||||||
|
* TaskLog
|
||||||
|
* ConsoleAuthToken
|
||||||
|
|
||||||
|
Further, host aggregates use the service name for
|
||||||
|
membership. Migrating those to database IDs is not possible since
|
||||||
|
multiple cells will cause overlap. We could migrate those to UUIDs or
|
||||||
|
simply ignore this case and assume that any *actual* rename operation
|
||||||
|
in the future would involve API operations to fix aggregates (which is
|
||||||
|
doable, unlike changing the host of things like Instance).
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
No specific REST API impact for this, other than the potential for
|
||||||
|
enabling a mutable Service hostname in the future.
|
||||||
|
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
No impact.
|
||||||
|
|
||||||
|
|
||||||
|
Notifications impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
No impact.
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
Not visible to end users.
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
Theoretically some benefit comes from integer-based linkages between
|
||||||
|
these objects that are currently linked by strings. Eventually we
|
||||||
|
could reduce a bunch of string duplication from our DB schema and
|
||||||
|
footprint.
|
||||||
|
|
||||||
|
There will definitely be a one-time performance impact due to the
|
||||||
|
online data migration(s) required to move to the more robust schema.
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
This is really all an (eventual) benefit to the deployer.
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
There will be some churn in the database models during the
|
||||||
|
transition. Looking up the hostname of an instance will require
|
||||||
|
Instance->ComputeNode->Service, but this can probably be hidden with
|
||||||
|
helpers in the Instance object such that not much has to change in the
|
||||||
|
actual workflow.
|
||||||
|
|
||||||
|
Upgrade impact
|
||||||
|
--------------
|
||||||
|
|
||||||
|
There will be some substantial online data migrations required to get
|
||||||
|
things into the new schema, and the benefits will only be achievable
|
||||||
|
in a subsequent release once everything is converted.
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
danms
|
||||||
|
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
|
||||||
|
* Persist the compute node UUID to disk when we generate it. Read the
|
||||||
|
compute node UUID from that location if it exists before we look to
|
||||||
|
see if we need to generate, create, or find an existing node record.
|
||||||
|
* Change the compute startup procedures to abort if we detect a
|
||||||
|
mismatch
|
||||||
|
* Make the schema changes to link database models by id. The
|
||||||
|
ComputeNode and Service objects/tables still have the id fields that
|
||||||
|
we can re-enable without even needing a schema change on those.
|
||||||
|
* Make the data models honor the ID-based linkages, if present
|
||||||
|
* Write an online data migration to construct those links on existing
|
||||||
|
databases
|
||||||
|
|
||||||
|
Later, there will be work items to:
|
||||||
|
* Drop the legacy columns
|
||||||
|
* Potentially implement an actual service rename procedure
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
There should be no external dependencies for the base of this work,
|
||||||
|
but there is a dependency on the release cycle, which affects how
|
||||||
|
quickly we can implement this and drop the old way of doing it.
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
Unit and functional testing for the actual compute node startup
|
||||||
|
behavior should be fine. Existing integration testing should ensure
|
||||||
|
that we haven't broken any of the runtime behavior. Grenade jobs
|
||||||
|
will test the data migration and we can implement some nova status
|
||||||
|
items to help validate things in those upgrade jobs.
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
There will need to be some documentation about the persistent compute
|
||||||
|
node UUID file for deployers and tool authors. Ideally, the only
|
||||||
|
visible result of this would be some additional failure modes if the
|
||||||
|
compute service detects an unexpected rename, so some documentation of
|
||||||
|
what that looks like and what to do about it would be helpful.
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
TODO(danms): There are probably bugs we can reference about compute
|
||||||
|
node renames being not possible, or problematic if/when they happen.
|
||||||
|
|
||||||
|
.. _removed: https://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/detach-service-from-computenode.html
|
||||||
|
|
||||||
|
History
|
||||||
|
=======
|
||||||
|
|
||||||
|
.. list-table:: Revisions
|
||||||
|
:header-rows: 1
|
||||||
|
|
||||||
|
* - Release Name
|
||||||
|
- Description
|
||||||
|
* - Antelope
|
||||||
|
- Introduced
|
||||||
Reference in New Issue
Block a user