Allow VMs to be created via volumes
After STX-Openstack upversioned to Antelope, we noticed that it was not possible to create VMs by volume, as they would be stuck on ERROR status. The first proposed solution was to create a patch containing [1] and [2], because, as specified in [3], Nova now requires a service token in order to be able to manipulate Cinder volumes. This unfortunately did not solve the issue by itself, as now an error message showed up on the nova-conductor pods with the following (not full error message, only important part): "nova.exception.RescheduledException: Build of instance 2f32c7ea-1720-4f61-bce8-dbe970c40b0c was re-scheduled: Secret not found: no secret with matching uuid 'a7f3ae2e-cee7-4f04-9402 -a78047747654". This UUID was not the same one present when issuing `virsh secret-list` on Cinder, Nova and Libvirt containers. Turns out openstack-helm and openstack-helm-infra have a Ceph UUID hardcoded in them, in Cinder [4], Nova [5] [6] and Libvirt [7] values. By changing these values to the UUID that libvirt was trying to find (7f3ae2e-cee7-4f04-9402-a78047747654), and it worked to solve the issue. However, it is not a good practice to use hardcoded values, and, searching on where this UUID was coming from, it turns out it was defined by the platform's Ceph configuration under `/etc/ceph/ceph.conf`. This still leaves the question, why was this working on Ussuri and stopped working on Antelope? First of all, the Ceph official documentation [8] [9] about using it with OpenStack explains the process of adding the secret to libvirt, to store the ceph admin keyring. You can see that the secret uuid is generated "on the fly" and both docs mention that old/hard-coded value (i.e., 457eb676-33da-42ec-9a8c-9293d545c337). This is the reason why it used to work until our upversion to OpenStack Antelope/2023.1: this UUID does not really matter (as long as nova and libvirt have the same value for it)! It is a given UUID to the libvirt secret that will store ceph keyring [10], and the key ring will ensure proper communication between our services and the platform ceph. What changed between Ussuri and Antelope (2023.1), is that now there is a specific method [11] to set a default value (Ceph's Cluster UUID) for this UUID when it is not specified in the driver configuration. What this change does is dynamically read this `/etc/ceph/ceph.conf` file to search for the UUID value, and use it to override the [4] [5] [6] and [7] values. It also adds the patch including the Nova service token configuration. The combination of these 2 changes allows VMs to be created by volumes. [1]91c8a5baf2[2]7d39af25fd[3] https://docs.openstack.org/releasenotes/cinder/2023.1.html#upgrade-notes [4] https://opendev.org/openstack/openstack-helm/src/branch/master/cinder/values.yaml#L942 [5] https://opendev.org/openstack/openstack-helm/src/branch/master/nova/values.yaml#L594 [6] https://opendev.org/openstack/openstack-helm/src/branch/master/nova/values.yaml#L1432 [7] https://opendev.org/openstack/openstack-helm-infra/src/branch/master/libvirt/values.yaml#L100 [8] https://github.com/ceph/ceph/blob/main/doc/rbd/rbd-openstack.rst [9] https://docs.huihoo.com/ceph/v0.80.5/rbd/rbd-openstack/index.html [10] https://opendev.org/starlingx/openstack-armada-app/src/branch/master/python3-k8sapp-openstack/k8sapp_openstack/k8sapp_openstack/helm/libvirt.py#L60 [11]6464d37d16 (diff-9b122c182b4333b747e7fd7e07f73d68ff30256a)Test Plan: PASS: Build openstack-helm, python3-k8sapp-openstack and stx-openstack-helm-fluxcd PASS: Upload / apply / remove STX-Openstack PASS: Create a VM by an image PASS: Create a volume and launch a VM from it PASS: Create a VM using the `boot-from-volume` flag PASS: Delete a VM created by a volume Closes-Bug: 2037463 Change-Id: Ia00bb8dbe3460ce817d69049f97f56a96ad6a298 Signed-off-by: Lucas de Ataides <lucas.deataidesbarreto@windriver.com>
This commit is contained in:
Lucas de Ataides
parent
310f677d29
commit
eaa8b41cb0
@ -0,0 +1,192 @@
|
||||
From b327cf3e379dd1a28e9e1491774c11e1209a1476 Mon Sep 17 00:00:00 2001
|
||||
From: Lucas de Ataides <lucas.deataidesbarreto@windriver.com>
|
||||
Date: Tue, 26 Sep 2023 16:16:04 -0300
|
||||
Subject: [PATCH] Add service tokens for Cinder auth
|
||||
|
||||
Since version 22.1.0 of Cinder, it requires Nova to configure service
|
||||
tokens in order to manipulate volumes [1], meaning that if that's not
|
||||
configured, Nova will not be able to create VMs by volume. The commit
|
||||
configured for StarlingX's openstack-helm package does not include this
|
||||
configuration, which was done later by [2] and [3].
|
||||
|
||||
This patch includes commits [2] and [3] in order to allow Nova to create
|
||||
VMs by volumes.
|
||||
|
||||
[1] https://docs.openstack.org/releasenotes/cinder/2023.1.html#upgrade-notes
|
||||
[2] https://opendev.org/openstack/openstack-helm/commit/91c8a5baf2cf2f0dddded57d88f00ea11dd4ff4a
|
||||
[3] https://opendev.org/openstack/openstack-helm/commit/7d39af25fddbf5fc67e15c92a9265f28567a214e
|
||||
|
||||
Signed-off-by: Lucas de Ataides <lucas.deataidesbarreto@windriver.com>
|
||||
---
|
||||
cinder/values.yaml | 6 ++++--
|
||||
glance/values.yaml | 2 ++
|
||||
neutron/values.yaml | 2 ++
|
||||
nova/templates/configmap-etc.yaml | 23 +++++++++++++++++++++++
|
||||
nova/values.yaml | 18 ++++++++++++++++--
|
||||
placement/values.yaml | 2 ++
|
||||
6 files changed, 49 insertions(+), 4 deletions(-)
|
||||
|
||||
diff --git a/cinder/values.yaml b/cinder/values.yaml
|
||||
index b95bd618..ddc5d632 100644
|
||||
--- a/cinder/values.yaml
|
||||
+++ b/cinder/values.yaml
|
||||
@@ -826,6 +826,8 @@ conf:
|
||||
database:
|
||||
max_retries: -1
|
||||
keystone_authtoken:
|
||||
+ service_token_roles: service
|
||||
+ service_token_roles_required: true
|
||||
auth_version: v3
|
||||
auth_type: password
|
||||
memcache_security_strategy: ENCRYPT
|
||||
@@ -848,7 +850,7 @@ conf:
|
||||
backend_url: file:///var/lib/cinder/coordination
|
||||
service_user:
|
||||
auth_type: password
|
||||
- send_service_user_token: false
|
||||
+ send_service_user_token: true
|
||||
logging:
|
||||
loggers:
|
||||
keys:
|
||||
@@ -1224,7 +1226,7 @@ endpoints:
|
||||
user_domain_name: default
|
||||
project_domain_name: default
|
||||
cinder:
|
||||
- role: admin
|
||||
+ role: admin,service
|
||||
region_name: RegionOne
|
||||
username: cinder
|
||||
password: password
|
||||
diff --git a/glance/values.yaml b/glance/values.yaml
|
||||
index d23674bb..d15b77ac 100644
|
||||
--- a/glance/values.yaml
|
||||
+++ b/glance/values.yaml
|
||||
@@ -256,6 +256,8 @@ conf:
|
||||
oslo_middleware:
|
||||
enable_proxy_headers_parsing: true
|
||||
keystone_authtoken:
|
||||
+ service_token_roles: service
|
||||
+ service_token_roles_required: true
|
||||
auth_type: password
|
||||
auth_version: v3
|
||||
memcache_security_strategy: ENCRYPT
|
||||
diff --git a/neutron/values.yaml b/neutron/values.yaml
|
||||
index 0a84322d..c71a40cf 100644
|
||||
--- a/neutron/values.yaml
|
||||
+++ b/neutron/values.yaml
|
||||
@@ -1809,6 +1809,8 @@ conf:
|
||||
ironic:
|
||||
endpoint_type: internal
|
||||
keystone_authtoken:
|
||||
+ service_token_roles: service
|
||||
+ service_token_roles_required: true
|
||||
memcache_security_strategy: ENCRYPT
|
||||
auth_type: password
|
||||
auth_version: v3
|
||||
diff --git a/nova/templates/configmap-etc.yaml b/nova/templates/configmap-etc.yaml
|
||||
index 5e3b61d3..1d8fbd97 100644
|
||||
--- a/nova/templates/configmap-etc.yaml
|
||||
+++ b/nova/templates/configmap-etc.yaml
|
||||
@@ -228,6 +228,29 @@ limitations under the License.
|
||||
{{- $_ := tuple "oslo_cache" "internal" "memcache" . | include "helm-toolkit.endpoints.host_and_port_endpoint_uri_lookup" | set .Values.conf.nova.ironic "memcache_servers" -}}
|
||||
{{- end -}}
|
||||
|
||||
+{{- if empty .Values.conf.nova.cinder.auth_url -}}
|
||||
+{{- $_ := tuple "identity" "internal" "api" . | include "helm-toolkit.endpoints.keystone_endpoint_uri_lookup" | set .Values.conf.nova.cinder "auth_url" -}}
|
||||
+{{- end -}}
|
||||
+
|
||||
+{{- if empty .Values.conf.nova.cinder.os_region_name -}}
|
||||
+{{- $_ := set .Values.conf.nova.cinder "os_region_name" .Values.endpoints.identity.auth.cinder.region_name -}}
|
||||
+{{- end -}}
|
||||
+{{- if empty .Values.conf.nova.cinder.project_name -}}
|
||||
+{{- $_ := set .Values.conf.nova.cinder "project_name" .Values.endpoints.identity.auth.cinder.project_name -}}
|
||||
+{{- end -}}
|
||||
+{{- if empty .Values.conf.nova.cinder.project_domain_name -}}
|
||||
+{{- $_ := set .Values.conf.nova.cinder "project_domain_name" .Values.endpoints.identity.auth.cinder.project_domain_name -}}
|
||||
+{{- end -}}
|
||||
+{{- if empty .Values.conf.nova.cinder.user_domain_name -}}
|
||||
+{{- $_ := set .Values.conf.nova.cinder "user_domain_name" .Values.endpoints.identity.auth.cinder.user_domain_name -}}
|
||||
+{{- end -}}
|
||||
+{{- if empty .Values.conf.nova.cinder.username -}}
|
||||
+{{- $_ := set .Values.conf.nova.cinder "username" .Values.endpoints.identity.auth.cinder.username -}}
|
||||
+{{- end -}}
|
||||
+{{- if empty .Values.conf.nova.cinder.password -}}
|
||||
+{{- $_ := set .Values.conf.nova.cinder "password" .Values.endpoints.identity.auth.cinder.password -}}
|
||||
+{{- end -}}
|
||||
+
|
||||
{{- if empty .Values.conf.nova.DEFAULT.osapi_compute_listen_port -}}
|
||||
{{- $_ := tuple "compute" "service" "api" . | include "helm-toolkit.endpoints.endpoint_port_lookup" | set .Values.conf.nova.DEFAULT "osapi_compute_listen_port" -}}
|
||||
{{- end -}}
|
||||
diff --git a/nova/values.yaml b/nova/values.yaml
|
||||
index 882470c6..63b8d02f 100644
|
||||
--- a/nova/values.yaml
|
||||
+++ b/nova/values.yaml
|
||||
@@ -1402,6 +1402,10 @@ conf:
|
||||
service_metadata_proxy: True
|
||||
auth_type: password
|
||||
auth_version: v3
|
||||
+ cinder:
|
||||
+ catalog_info: volumev3::internalURL
|
||||
+ auth_url: null
|
||||
+ auth_type: password
|
||||
database:
|
||||
max_retries: -1
|
||||
api_database:
|
||||
@@ -1409,6 +1413,8 @@ conf:
|
||||
cell0_database:
|
||||
max_retries: -1
|
||||
keystone_authtoken:
|
||||
+ service_token_roles: service
|
||||
+ service_token_roles_required: true
|
||||
auth_type: password
|
||||
auth_version: v3
|
||||
memcache_security_strategy: ENCRYPT
|
||||
@@ -1417,7 +1423,7 @@ conf:
|
||||
notify_on_state_change: vm_and_task_state
|
||||
service_user:
|
||||
auth_type: password
|
||||
- send_service_user_token: false
|
||||
+ send_service_user_token: true
|
||||
libvirt:
|
||||
connection_uri: "qemu+unix:///system?socket=/run/libvirt/libvirt-sock"
|
||||
images_type: qcow2
|
||||
@@ -1708,7 +1714,7 @@ endpoints:
|
||||
user_domain_name: default
|
||||
project_domain_name: default
|
||||
nova:
|
||||
- role: admin
|
||||
+ role: admin,service
|
||||
region_name: RegionOne
|
||||
username: nova
|
||||
password: password
|
||||
@@ -1743,6 +1749,14 @@ endpoints:
|
||||
project_name: service
|
||||
user_domain_name: service
|
||||
project_domain_name: service
|
||||
+ cinder:
|
||||
+ role: admin,service
|
||||
+ region_name: RegionOne
|
||||
+ username: cinder
|
||||
+ password: password
|
||||
+ project_name: service
|
||||
+ user_domain_name: service
|
||||
+ project_domain_name: service
|
||||
test:
|
||||
role: admin
|
||||
region_name: RegionOne
|
||||
diff --git a/placement/values.yaml b/placement/values.yaml
|
||||
index aa864620..4a702ace 100644
|
||||
--- a/placement/values.yaml
|
||||
+++ b/placement/values.yaml
|
||||
@@ -82,6 +82,8 @@ conf:
|
||||
placement_database:
|
||||
connection: null
|
||||
keystone_authtoken:
|
||||
+ service_token_roles: service
|
||||
+ service_token_roles_required: true
|
||||
auth_version: v3
|
||||
auth_type: password
|
||||
memcache_security_strategy: ENCRYPT
|
||||
--
|
||||
2.25.1
|
||||
|
||||
@ -17,3 +17,4 @@
|
||||
0017-Support-ceph-dev-version-during-pool-creation.patch
|
||||
0018-Update-charts-requirements-to-use-local-server.patch
|
||||
0019-Make-nova-address-search-optional.patch
|
||||
0020-Add-service-tokens-for-Cinder-auth.patch
|
||||
|
||||
@ -13,6 +13,7 @@ from tsconfig import tsconfig as tsc
|
||||
|
||||
from k8sapp_openstack.common import constants as app_constants
|
||||
from k8sapp_openstack.helm import openstack
|
||||
from k8sapp_openstack.utils import get_ceph_uuid
|
||||
|
||||
|
||||
ROOK_CEPH_BACKEND_NAME = 'ceph-store'
|
||||
@ -226,6 +227,10 @@ class CinderHelm(openstack.OpenstackBaseHelm):
|
||||
constants.SB_TYPE_CEPH_CONF_FILENAME),
|
||||
}
|
||||
|
||||
ceph_uuid = get_ceph_uuid()
|
||||
if ceph_uuid:
|
||||
conf_backends[bk_name]['rbd_secret_uuid'] = ceph_uuid
|
||||
|
||||
return conf_backends
|
||||
|
||||
def _get_endpoints_overrides(self):
|
||||
@ -357,6 +362,12 @@ class CinderHelm(openstack.OpenstackBaseHelm):
|
||||
(constants.CEPH_CONF_PATH +
|
||||
constants.SB_TYPE_CEPH_CONF_FILENAME),
|
||||
}
|
||||
|
||||
ceph_uuid = get_ceph_uuid()
|
||||
if ceph_uuid:
|
||||
conf_backends['rbd1']['rbd_secret_uuid'] = ceph_uuid
|
||||
conf_backends[ROOK_CEPH_BACKEND_NAME]['rbd_secret_uuid'] = ceph_uuid
|
||||
|
||||
return conf_backends
|
||||
|
||||
def _get_ceph_client_rook_overrides(self):
|
||||
|
||||
@ -11,6 +11,7 @@ from sysinv.helm import common
|
||||
|
||||
from k8sapp_openstack.common import constants as app_constants
|
||||
from k8sapp_openstack.helm import openstack
|
||||
from k8sapp_openstack.utils import get_ceph_uuid
|
||||
|
||||
|
||||
class LibvirtHelm(openstack.OpenstackBaseHelm):
|
||||
@ -80,6 +81,12 @@ class LibvirtHelm(openstack.OpenstackBaseHelm):
|
||||
},
|
||||
}
|
||||
|
||||
ceph_uuid = get_ceph_uuid()
|
||||
if ceph_uuid:
|
||||
overrides['ceph']['cinder'] = {
|
||||
'secret_uuid': ceph_uuid,
|
||||
}
|
||||
|
||||
return overrides
|
||||
|
||||
def _update_host_addresses(self, host, libvirt_config):
|
||||
|
||||
@ -18,6 +18,7 @@ from sysinv.helm import common
|
||||
|
||||
from k8sapp_openstack.common import constants as app_constants
|
||||
from k8sapp_openstack.helm import openstack
|
||||
from k8sapp_openstack.utils import get_ceph_uuid
|
||||
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
||||
@ -799,6 +800,15 @@ class NovaHelm(openstack.OpenstackBaseHelm):
|
||||
},
|
||||
}
|
||||
|
||||
ceph_uuid = get_ceph_uuid()
|
||||
if ceph_uuid:
|
||||
overrides['ceph']['cinder'] = {
|
||||
'secret_uuid': ceph_uuid,
|
||||
}
|
||||
overrides['nova']['libvirt'] = {
|
||||
'rbd_secret_uuid': ceph_uuid,
|
||||
}
|
||||
|
||||
if self._is_openstack_https_ready():
|
||||
overrides = self._update_overrides(overrides, {
|
||||
'nova': {
|
||||
|
||||
@ -8,6 +8,7 @@ from grp import getgrnam
|
||||
import os
|
||||
from pathlib import Path
|
||||
from pwd import getpwnam
|
||||
import re
|
||||
import shutil
|
||||
from typing import Generator
|
||||
|
||||
@ -267,3 +268,34 @@ def delete_clients_working_directory(
|
||||
|
||||
shutil.rmtree(working_directory, ignore_errors=True)
|
||||
return True
|
||||
|
||||
|
||||
def get_ceph_uuid():
|
||||
"""Get Ceph secret UUID for Cinder backend configuration
|
||||
|
||||
:returns: str -- The Ceph's secret UUID
|
||||
"""
|
||||
ceph_config_file = os.path.join(constants.CEPH_CONF_PATH,
|
||||
constants.SB_TYPE_CEPH_CONF_FILENAME)
|
||||
|
||||
# If the file doesn't exist, return nothing, as not to change
|
||||
# the default value.
|
||||
if not os.path.isfile(ceph_config_file):
|
||||
LOG.warning(f"`{ceph_config_file}` does not exist. Using "
|
||||
"default configuration.")
|
||||
return None
|
||||
|
||||
# Search for the line that contains the `fsid` parameter, which is
|
||||
# the Ceph UUID required for Cinder's backend configuration.
|
||||
with open(ceph_config_file) as file:
|
||||
line = next((line for line in file if "fsid" in line), None)
|
||||
if not line:
|
||||
LOG.warning(f"`{ceph_config_file}` does not contain the "
|
||||
"'fsid' value. Using default configuration")
|
||||
return None
|
||||
|
||||
# This Regex pattern searches for an UUID
|
||||
pattern = re.compile(r"[0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]"
|
||||
r"{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{12}",
|
||||
re.IGNORECASE)
|
||||
return pattern.findall(line)[0]
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user