starlingx
/
openstack-armada-app
Code Issues Proposed changes

Move live-migration traffic to cluster-host-net 

This change updates the application plugins in order to ensure that all
libvirt/live-migration related traffic is happening through the
cluster-host-network. Currently most of the libvirt/live-migration
addresses are being solved through INADDR_ANY (0.0.0.0), and this route
resolution will vary between AIO, routes to oam-network, and Worker,
routes to mgmt-network. Both resolutions are not correct since the
correct network for such traffic should be the cluster-host-network.
Actually, current platform firewall will block any traffic through not
allowed oam-network ports.

The goal will be achieved by setting to the node's cluster-host IP:
* libvirt listen_addr
* nova.conf "live_migration_inbound_addr"

It is important to notice that in the current version of the
openstack-helm nova helm chart, there is a problem with
nova-compute-init.sh for this use case of ours, so an openstack-helm
patch was required to fix it.

Code that was previously implemented only for the Nova plugin and is now
required by the Libvirt plugin, was moved to the parent OpenStack class.

[1] 31be86079d

TEST PLAN:
PASS - Build stx-openstack application
PASS - Apply the application to an AIO-DX system
PASS - "$ sudo netstat -ltnp | grep <libvirtd pid>" to ensure that
       libvirtd is listening on the correct cluster-host-net IP
PASS - Verify that the nova-compute.sh script was populated correctly
PASS - Test a VM live-migration on the controller+worker node
PASS - Verify that live_migration data in LibvirtLiveMigrateData has the
       correct cluster-host-net IP address in its "target_connect_addr"
PASS - Apply the application to a Standard system
PASS - "$ sudo netstat -ltnp | grep <libvirtd pid>" to ensure that
       libvirtd is listening on the correct cluster-host-net IP
PASS - Verify that the nova-compute.sh script was populated correctly
PASS - Test a VM live-migration on the worker node
PASS - Verify that live_migration data in LibvirtLiveMigrateData has the
       correct cluster-host-net IP address in its "target_connect_addr"

Closes-Bug: 2037330

Signed-off-by: Thales Elero Cervi <thaleselero.cervi@windriver.com>
Change-Id: I37db601e4b1b0e397a1b8dbdad1a293ff25c2e55
(cherry picked from commit 310f677d29)
changes/39/896539/1
Thales Elero Cervi 2023-09-18 16:00:28 -03:00
parent d8e0e88108
commit 190502fb6b
6 changed files with 230 additions and 80 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

84
openstack-helm/debian/deb_folder/patches/0019-Make-nova-address-search-optional.patch Normal file
View File

@ -0,0 +1,84 @@
From 66120483da2c8e39a65d14ace0adba96008c615c Mon Sep 17 00:00:00 2001
From: Thales Elero Cervi <thaleselero.cervi@windriver.com>
Date: Fri, 22 Sep 2023 13:38:49 -0300
Subject: [PATCH] Nova: Make address search optional
The recent openstack-helm base commit update [1] brought a recent
change [2] that modifies the nova-compute-init.sh script behavior in
a way that is now affecting anyone that is not actually using it.
When this script was originally introduced [3] it was mainly relying
on a Values specific configuration for libvirt and would add
information to the extra conf file "nova-libvirt.conf" only if this
value was defined. It was used to set the name of the live migration
interface, and since it is delivered through a ConfigMap consumed by
a DaemonSet, it is a "global" config and not something that can be
defined per host.
It was never a problem to stx-openstack, because we were not using the
Value and therefore not populating the nova-libvirt.conf.
Now, after this new merge [2], when this value is not defined a
conditional triggers the IP resolution through the default gateway.
So, even if not using the migration interface Value, we end up having
an extra config file that will overwrite anything that was already
defined to nova.conf per host.
We can not rely on default gateway IP resolution since we want our
live migration traffic to go through an specific network and we also
do not want to define a global name for the related interface, since
this can (although probably will not) differ between hosts.
The solution is similar to what was done in the past for VNC spice
server proxyclient and the hypervisor addresses [4].
This configurable address search was proposed upstream [5].
[1] https://opendev.org/starlingx/openstack-armada-app/
commit/8254cd31bb1f12eebc48b712b33f75b2fc0aa571
[2] https://github.com/openstack/openstack-helm
/commit/31be86079d711c698b2560b4bed654e23373a596
[3] https://github.com/openstack/openstack-helm
/commit/31be86079d711c698b2560b4bed654e23373a596
[4] https://opendev.org/starlingx/openstack-armada-app/src/commit/
7190f7fb5048d014ed46112aeedd4e8622031881/openstack-helm/debian/
deb_folder/patches/0002-Nova-console-ip-address-search-optionality.patch
[5] https://review.opendev.org/c/openstack/openstack-helm/+/896461
Signed-off-by: Thales Elero Cervi <thaleselero.cervi@windriver.com>
Change-Id: I1f84006f71baded82f39d864b1fa38351b3642cf
---
nova/templates/bin/_nova-compute.sh.tpl | 2 ++
nova/values.yaml | 4 +++-
2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/nova/templates/bin/_nova-compute.sh.tpl b/nova/templates/bin/_nova-compute.sh.tpl
index 60b0272d..702e3b92 100644
--- a/nova/templates/bin/_nova-compute.sh.tpl
+++ b/nova/templates/bin/_nova-compute.sh.tpl
@@ -21,7 +21,9 @@ exec nova-compute \
{{- if .Values.console.address_search_enabled }}
--config-file /tmp/pod-shared/nova-console.conf \
{{- end }}
+{{- if .Values.conf.libvirt.address_search_enabled }}
--config-file /tmp/pod-shared/nova-libvirt.conf \
+{{- end }}
{{- if and ( empty .Values.conf.nova.DEFAULT.host ) ( .Values.pod.use_fqdn.compute ) }}
--config-file /tmp/pod-shared/nova-compute-fqdn.conf \
{{- end }}
diff --git a/nova/values.yaml b/nova/values.yaml
index e1dc611f..882470c6 100644
--- a/nova/values.yaml
+++ b/nova/values.yaml
@@ -1341,7 +1341,9 @@ conf:
cpu_allocation_ratio: 1.0
reserved_host_memory_mb: 0
libvirt:
- # Get the IP address to be used as the target for live migration traffic using interface name.
+ address_search_enabled: true
+ # When "address_search_enabled", get the IP address to be used as the target for live migration
+ # traffic using interface name.
# If this option is set to None, the hostname of the migration target compute node will be used.
live_migration_interface:
hypervisor:
--
2.25.1

1
openstack-helm/debian/deb_folder/patches/series
View File

@ -16,3 +16,4 @@
0016-Update-user-in-cinder-related-pods.patch
0017-Support-ceph-dev-version-during-pool-creation.patch
0018-Update-charts-requirements-to-use-local-server.patch
0019-Make-nova-address-search-optional.patch

95
python3-k8sapp-openstack/k8sapp_openstack/k8sapp_openstack/helm/libvirt.py
View File

@ -1,10 +1,12 @@
#
# Copyright (c) 2019-2020 Wind River Systems, Inc.
# Copyright (c) 2019-2023 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
from sysinv.common import constants
from sysinv.common import exception
from sysinv.common import utils
from sysinv.helm import common
from k8sapp_openstack.common import constants as app_constants
@ -19,28 +21,20 @@ class LibvirtHelm(openstack.OpenstackBaseHelm):
SERVICE_NAME = app_constants.HELM_CHART_LIBVIRT
def __init__(self, operator):
super(LibvirtHelm, self).__init__(operator)
self.fallback_conf = {
'listen_addr': '0.0.0.0',
}
self.labels_by_hostid = {}
self.addresses_by_hostid = {}
def get_overrides(self, namespace=None):
admin_keyring = 'null'
if self._is_rook_ceph():
admin_keyring = self._get_rook_ceph_admin_keyring()
self.labels_by_hostid = self._get_host_labels()
self.addresses_by_hostid = self._get_host_addresses()
overrides = {
common.HELM_NS_OPENSTACK: {
'conf': {
'libvirt': {
'listen_addr': '0.0.0.0'
},
'qemu': {
'user': "root",
'group': "root",
'cgroup_controllers': ["cpu", "cpuacct", "cpuset", "freezer", "net_cls", "perf_event"],
'namespaces': [],
'clear_emulator_capabilities': 0
},
'ceph': {
'admin_keyring': admin_keyring,
},
},
'pod': {
'mounts': {
'libvirt': {
@ -48,6 +42,7 @@ class LibvirtHelm(openstack.OpenstackBaseHelm):
}
}
},
'conf': self._get_conf_overrides(),
}
}
@ -58,3 +53,65 @@ class LibvirtHelm(openstack.OpenstackBaseHelm):
namespace=namespace)
else:
return overrides
def _get_conf_overrides(self):
admin_keyring = 'null'
if self._is_rook_ceph():
admin_keyring = self._get_rook_ceph_admin_keyring()
overrides = {
'qemu': {
'user': "root",
'group': "root",
'cgroup_controllers': [
"cpu", "cpuacct", "cpuset",
"freezer", "net_cls", "perf_event"
],
'namespaces': [],
'clear_emulator_capabilities': 0
},
'ceph': {
'admin_keyring': admin_keyring,
},
'overrides': {
'libvirt_libvirt': {
'hosts': self._get_per_host_overrides()
}
},
}
return overrides
def _update_host_addresses(self, host, libvirt_config):
cluster_host_ip = self._get_cluster_host_ip(
host, self.addresses_by_hostid)
if cluster_host_ip is None:
return
libvirt_config.update({'listen_addr': cluster_host_ip})
def _get_per_host_overrides(self):
host_list = []
hosts = self.dbapi.ihost_get_list()
for host in hosts:
host_labels = self.labels_by_hostid.get(host.id, [])
if (host.invprovision in [constants.PROVISIONED,
constants.PROVISIONING] or
host.ihost_action in [constants.UNLOCK_ACTION,
constants.FORCE_UNLOCK_ACTION]):
if (constants.WORKER in utils.get_personalities(host) and
utils.has_openstack_compute(host_labels)):
hostname = str(host.hostname)
libvirt_conf = {}
self._update_host_addresses(host, libvirt_conf)
host_nova = {
'name': hostname,
'conf': {
'libvirt': libvirt_conf if libvirt_conf else self.fallback_conf,
}
}
host_list.append(host_nova)
return host_list

65
python3-k8sapp-openstack/k8sapp_openstack/k8sapp_openstack/helm/nova.py
View File

@ -1,5 +1,5 @@
#
# Copyright (c) 2019-2021 Wind River Systems, Inc.
# Copyright (c) 2019-2023 Wind River Systems, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
@ -76,7 +76,6 @@ class NovaHelm(openstack.OpenstackBaseHelm):
self.cpus_by_hostid = {}
self.interfaces_by_hostid = {}
self.cluster_host_network = None
self.interface_networks_by_ifaceid = {}
self.addresses_by_hostid = {}
self.memory_by_hostid = {}
self.ethernet_ports_by_hostid = {}
@ -92,7 +91,6 @@ class NovaHelm(openstack.OpenstackBaseHelm):
self.interfaces_by_hostid = self._get_host_interfaces()
self.cluster_host_network = self.dbapi.network_get_by_type(
constants.NETWORK_TYPE_CLUSTER_HOST)
self.interface_networks_by_ifaceid = self._get_interface_networks()
self.addresses_by_hostid = self._get_host_addresses()
self.memory_by_hostid = self._get_host_imemory()
self.ethernet_ports_by_hostid = self._get_host_ethernet_ports()
@ -421,39 +419,6 @@ class NovaHelm(openstack.OpenstackBaseHelm):
addresses[address.ifname].append(address)
return addresses
def _get_cluster_host_iface(self, host):
"""
Returns the host cluster interface.
"""
cluster_host_iface = None
for iface in self.interfaces_by_hostid.get(host.id, []):
try:
self._get_interface_network_query(iface.id, self.cluster_host_network.id)
cluster_host_iface = iface
except exception.InterfaceNetworkNotFoundByHostInterfaceNetwork:
pass
LOG.debug("Host: {} Interface: {}".format(host.id, cluster_host_iface))
return cluster_host_iface
def _get_cluster_host_ip(self, host):
"""
Returns the host cluster IP.
"""
cluster_host_iface = self._get_cluster_host_iface(host)
if cluster_host_iface is None:
return
cluster_host_ip = None
for addr in self.addresses_by_hostid.get(host.id, []):
if addr.interface_id == cluster_host_iface.id:
cluster_host_ip = addr.address
LOG.debug("Host: {} Host IP: {}".format(host.id, cluster_host_ip))
return cluster_host_ip
def _update_host_pci_whitelist(self, host, pci_config):
"""
Generate multistring values containing PCI passthrough
@ -499,13 +464,13 @@ class NovaHelm(openstack.OpenstackBaseHelm):
libvirt_config.update({'images_type': 'default'})
def _update_host_addresses(self, host, default_config, vnc_config, libvirt_config):
cluster_host_ip = self._get_cluster_host_ip(host)
cluster_host_ip = self._get_cluster_host_ip(
host, self.addresses_by_hostid)
default_config.update({'my_ip': cluster_host_ip})
vnc_config.update({'server_listen': cluster_host_ip})
libvirt_config.update({'live_migration_inbound_addr': cluster_host_ip})
vnc_config.update({'server_proxyclient_address': cluster_host_ip})
libvirt_config.update({'live_migration_inbound_addr': cluster_host_ip})
def _get_ssh_subnet(self):
address_pool = self.dbapi.address_pool_get(self.cluster_host_network.pool_uuid)
@ -676,28 +641,6 @@ class NovaHelm(openstack.OpenstackBaseHelm):
}
return rbd_config
def _get_interface_networks(self):
"""
Builds a dictionary of interface networks indexed by interface id
"""
interface_networks = {}
db_interface_networks = self.dbapi.interface_network_get_all()
for iface_net in db_interface_networks:
interface_networks.setdefault(iface_net.interface_id, []).append(iface_net)
return interface_networks
def _get_interface_network_query(self, interface_id, network_id):
"""
Return the interface network of the supplied interface id and network id
"""
for iface_net in self.interface_networks_by_ifaceid.get(interface_id, []):
if iface_net.interface_id == interface_id and iface_net.network_id == network_id:
return iface_net
raise exception.InterfaceNetworkNotFoundByHostInterfaceNetwork(
interface_id=interface_id, network_id=network_id)
def _get_datanetworks(self):
"""
Builds a dictionary of datanetworks indexed by datanetwork uuid

63
python3-k8sapp-openstack/k8sapp_openstack/k8sapp_openstack/helm/openstack.py
View File

@ -596,6 +596,69 @@ class OpenstackBaseHelm(FluxCDBaseHelm):
interfaces.setdefault(iface.forihostid, []).append(iface)
return interfaces
def _get_interface_networks(self):
"""
Builds a dictionary of interface networks indexed by interface id
"""
interface_networks = {}
db_interface_networks = self.dbapi.interface_network_get_all()
for iface_net in db_interface_networks:
interface_networks.setdefault(iface_net.interface_id, []).append(iface_net)
return interface_networks
def _get_interface_network_query(self, interface_id, network_id):
"""
Return the interface network of the supplied interface id and network id
"""
interface_networks_by_ifaceid = self._get_interface_networks()
for iface_net in interface_networks_by_ifaceid.get(interface_id, []):
if iface_net.interface_id == interface_id and iface_net.network_id == network_id:
return iface_net
raise exception.InterfaceNetworkNotFoundByHostInterfaceNetwork(
interface_id=interface_id, network_id=network_id)
def _get_cluster_host_iface(self, host, cluster_host_net_id):
"""
Returns the host cluster interface.
"""
cluster_host_iface = None
interfaces_by_hostid = self._get_host_interfaces()
for iface in interfaces_by_hostid.get(host.id, []):
try:
self._get_interface_network_query(iface.id, cluster_host_net_id)
cluster_host_iface = iface
except exception.InterfaceNetworkNotFoundByHostInterfaceNetwork:
LOG.debug("Host: {} Interface: {} is not "
"a cluster-host interface".format(host.id, iface))
LOG.debug("Host: {} Interface: {}".format(host.id, cluster_host_iface))
return cluster_host_iface
def _get_cluster_host_ip(self, host, addresses_by_hostid):
"""
Returns the host cluster IP.
"""
cluster_host_network = self.dbapi.network_get_by_type(
constants.NETWORK_TYPE_CLUSTER_HOST)
cluster_host_iface = self._get_cluster_host_iface(
host, cluster_host_network.id)
if cluster_host_iface is None:
LOG.info("None cluster-host interface "
"found for Host: {}".format(host.id))
return
cluster_host_ip = None
for addr in addresses_by_hostid.get(host.id, []):
if addr.interface_id == cluster_host_iface.id:
cluster_host_ip = addr.address
LOG.debug("Host: {} Host IP: {}".format(host.id, cluster_host_ip))
return cluster_host_ip
def _get_host_labels(self):
"""
Builds a dictionary of labels indexed by host id

2
stx-openstack-helm-fluxcd/stx-openstack-helm-fluxcd/manifests/nova/nova-static-overrides.yaml
View File

@ -252,6 +252,8 @@ conf:
required: false
workarounds:
enable_numa_live_migration: true
libvirt:
address_search_enabled: false
hypervisor:
address_search_enabled: false
ssh: |