8f8667a8dd
Recently we had a customer case where attempts to add new ironic nodes
to an existing undercloud resulted in half of the nodes failing to be
detected and added to nova. Ironic API returned all of the newly added
nodes when called by the driver, but half of the nodes were not
returned to the compute manager by the driver.
There was only one nova-compute service managing all of the ironic
nodes of the all-in-one typical undercloud deployment.
After days of investigation and examination of a database dump from the
customer, we noticed that at some point the customer had changed the
hostname of the machine from something containing uppercase letters to
the same name but all lowercase. The nova-compute service record had
the mixed case name and the CONF.host (socket.gethostname()) had the
lowercase name.
The hash ring logic adds all of the nova-compute service hostnames plus
CONF.host to hash ring, then the ironic driver reports only the nodes
it owns by retrieving a service hostname from the ring based on a hash
of each ironic node UUID.
Because of the machine hostname change, the hash ring contained, for
example: {'MachineHostName', 'machinehostname'} when it should have
contained only one hostname. And because the hash ring contained two
hostnames, the driver was able to retrieve only half of the nodes as
nodes that it owned. So half of the new nodes were excluded and not
added as new compute nodes.
This adds lowercasing of hosts that are added to the hash ring and
ignores case when comparing the CONF.host to the hash ring members
to avoid unnecessary pain and confusion for users that make hostname
changes that are otherwise functionally harmless.
This also adds logging of the set of hash ring members at level DEBUG
to help enable easier debugging of hash ring related situations.
Closes-Bug: #1866380
Change-Id: I617fd59de327de05a198f12b75a381f21945afb0
(cherry picked from commit 7145100ee4)
(cherry picked from commit 588b0484bf)
Team and repository tags
OpenStack Nova
OpenStack Nova provides a cloud computing fabric controller, supporting a wide variety of compute technologies, including: libvirt (KVM, Xen, LXC and more), Hyper-V, VMware, XenServer, OpenStack Ironic and PowerVM.
Use the following resources to learn more.
API
To learn how to use Nova's API, consult the documentation available online at:
For more information on OpenStack APIs, SDKs and CLIs in general, refer to:
Operators
To learn how to deploy and configure OpenStack Nova, consult the documentation available online at:
In the unfortunate event that bugs are discovered, they should be reported to the appropriate bug tracker. If you obtained the software from a 3rd party operating system vendor, it is often wise to use their own bug tracker for reporting problems. In all other cases use the master OpenStack bug tracker, available at:
Developers
For information on how to contribute to Nova, please see the contents of the CONTRIBUTING.rst.
Any new code must follow the development guidelines detailed in the HACKING.rst file, and pass all unit tests.
Further developer focused documentation is available at:
Other Information
During each Summit and Project Team Gathering, we agree on what the whole community wants to focus on for the upcoming release. The plans for nova can be found at: