Upgrade MAAS to 3.6.4/Noble and improve boot resource import

Region readiness probe:
Replace maas login exec probe with a lightweight Python HTTP check of
/MAAS/api/2.0/version/ on 127.0.0.1, avoiding the ~10s login overhead
that caused probe timeouts. Handles TLS, insecure TLS, and plain HTTP
variants.

Boot resource import:
Redesign configure_boot_sources to guarantee only the desired distro
appears in boot-source-selections before start_import is called. If the
desired selection is missing, update the first existing selection
in-place rather than deleting it — MAAS blocks deletion when the
selection matches commissioning_distro_series. Remaining stale selections
are also updated in-place if delete is blocked. A final assertion aborts
the job if any stale entry survives, preventing import from hanging
indefinitely waiting for a distro the local sstream does not carry.

Add stall detection to start_import: if is-importing stops changing for
stall_threshold seconds, stop-import is issued, the region statefulset
is restarted via the Kubernetes API, and the import is re-issued with
a clean slate.

Wrap start_import + configure_images as a single retriable unit so the
outer timer re-triggers the import on configure failure, rather than
spinning on set-config against an empty boot-resource DB. Post-import,
configure_images switches commissioning_distro_series and
default_distro_series; cleanup_old_boot_source_selections then removes
duplicate selections left from in-place updates.

Add maas_cli wrapper with exponential backoff for transient HTTP errors
(502/503) to improve resilience during region restarts.

Chart and images:
- Bump chart 0.1.8 -> 0.2.0, appVersion 3.6.2 -> 3.6.4
- Switch all image tags to ubuntu_noble variants; postgres 14.5 -> 17.4
- Expose ADMIN_USERNAME from admin secret in region StatefulSet
- Add maas-rack pod readiness as import_resources job dependency
- Generalise Makefile push condition to any non-quay.io registry
- Update HTK_COMMIT to d89bc42b57f8c2898520181550b5019dcf31c06f
- Helm update to v4.1.4
- MAAS version in Jammy up to 1:3.5.12-16413-g.7fb94f378-0ubuntu1~22.04.1
- excluded snakeoil test keys from images

CI:
- Add maas-airskiff-deployment-noble-kubeadm-maas-upgrade job
- Switch all airskiff deployment jobs to 5-node nodeset with
  USE_ARMADA_GO=true
- Disable jammy/focal deployment jobs to reduce opendev infra load
- Drop ubuntu-bionic (airship-maas-single-node) nodeset

Change-Id: Icfb9315dc3eb9a83cb6d2606852e8e86ac3fc629
Signed-off-by: Sergiy Markin <smarkin@mirantis.com>

Change-Id: Icfb9315dc3eb9a83cb6d2606852e8e86ac3fc629
Signed-off-by: Sergiy Markin <smarkin@mirantis.com>
This commit is contained in:
Sergiy Markin
2026-05-14 19:06:04 +00:00
parent 494db96fe9
commit 16f241c2bb
18 changed files with 382 additions and 201 deletions
+135 -116
View File
@@ -21,9 +21,11 @@
- airship-maas-docker-build-gate-ubuntu-jammy
- airship-maas-docker-build-gate-ubuntu-noble
- airship-maas-lint-yaml
- maas-airskiff-deployment-noble-kubeadm
- maas-airskiff-deployment-jammy-kubeadm
- maas-airskiff-deployment-focal-kubeadm
# disabled in order to decrease load on opendev infra
# - maas-airskiff-deployment-noble-kubeadm
# - maas-airskiff-deployment-jammy-kubeadm
# - maas-airskiff-deployment-focal-kubeadm
- maas-airskiff-deployment-noble-kubeadm-maas-upgrade
gate:
jobs:
- airship-maas-lint-ws
@@ -40,95 +42,101 @@
- airship-maas-docker-publish-noble
- maas-upload-git-mirror
- nodeset:
name: airship-maas-single-node
nodes:
- name: primary
label: ubuntu-bionic
- nodeset:
name: airship-maas-single-node-jammy
nodes:
- name: primary
label: ubuntu-jammy
- nodeset:
name: airship-maas-single-node-noble
nodes:
- name: primary
label: ubuntu-noble
- job:
name: airship-maas-lint-ws
# Pinned for now due to Ansible 11 stopped verbosive output.
ansible-version: 9
description: |
Lints all files for trailing whitespace
run: tools/gate/playbooks/zuul-linter.yaml
timeout: 300
nodeset: airship-maas-single-node-jammy
nodeset: airship-maas-single-node-noble
- job:
name: airship-maas-chart-build-gate
# Pinned for now due to Ansible 11 stopped verbosive output.
ansible-version: 9
description: Build charts using pinned Helm toolkit.
run: tools/gate/playbooks/helm-linter.yaml
timeout: 600
nodeset: airship-maas-single-node-jammy
timeout: 3600
parent: airship-chart-build-base
nodeset: airship-maas-single-node-noble
required-projects:
- name: airship/treasuremap
override-checkout: v1.9
vars:
HTK_COMMIT: 80041dfbb5c22aa67dcd8dfa3332a943aa0d366c
HTK_COMMIT: d89bc42b57f8c2898520181550b5019dcf31c06f
- job:
name: airship-maas-chart-build-latest-htk
# Pinned for now due to Ansible 11 stopped verbosive output.
ansible-version: 9
description: Build charts using latest Helm toolkit.
voting: false
run: tools/gate/playbooks/helm-linter.yaml
timeout: 600
nodeset: airship-maas-single-node-jammy
timeout: 3600
parent: airship-chart-build-base
nodeset: airship-maas-single-node-noble
required-projects:
- name: airship/treasuremap
override-checkout: v1.9
vars:
HTK_COMMIT: master
- job:
name: airship-maas-docker-build-gate-ubuntu-noble
# Pinned for now due to Ansible 11 stopped verbosive output.
ansible-version: 9
timeout: 3600
run: tools/gate/playbooks/docker-image-build.yaml
nodeset: airship-maas-single-node-jammy
files:
- "^images/.*"
parent: airship-docker-build-gate-base
nodeset: airship-maas-single-node-noble
required-projects:
- name: airship/treasuremap
override-checkout: v1.9
vars:
publish: false
distro: ubuntu_noble
run_verbose_build: true
trivy_fail_on_vulns: true
trivy_severity: CRITICAL,HIGH
tags:
dynamic:
patch_set: true
- job:
name: airship-maas-docker-build-gate-ubuntu-jammy
# Pinned for now due to Ansible 11 stopped verbosive output.
ansible-version: 9
timeout: 3600
run: tools/gate/playbooks/docker-image-build.yaml
nodeset: airship-maas-single-node-jammy
files:
- "^images/.*"
parent: airship-docker-build-gate-base
nodeset: airship-maas-single-node-noble
required-projects:
- name: airship/treasuremap
override-checkout: v1.9
vars:
publish: false
distro: ubuntu_jammy
run_verbose_build: true
trivy_fail_on_vulns: true
trivy_severity: CRITICAL,HIGH
tags:
dynamic:
patch_set: true
- job:
name: airship-maas-docker-build-gate-ubuntu-focal
# Pinned for now due to Ansible 11 stopped verbosive output.
ansible-version: 9
timeout: 3600
run: tools/gate/playbooks/docker-image-build.yaml
nodeset: airship-maas-single-node-jammy
files:
- "^images/.*"
parent: airship-docker-build-gate-base
nodeset: airship-maas-single-node-noble
required-projects:
- name: airship/treasuremap
override-checkout: v1.9
vars:
publish: false
distro: ubuntu_focal
run_verbose_build: true
trivy_fail_on_vulns: true
trivy_severity: CRITICAL,HIGH
tags:
dynamic:
patch_set: true
@@ -136,27 +144,23 @@
- job:
name: airship-maas-lint-yaml
# Pinned for now due to Ansible 11 stopped verbosive output.
ansible-version: 9
voting: true
timeout: 900
run: tools/gate/playbooks/lint-yaml.yaml
nodeset: airship-maas-single-node-jammy
nodeset: airship-maas-single-node-noble
irrelevant-files:
- "^charts/maas/templates/.*"
- job:
name: airship-maas-docker-publish-focal
# Pinned for now due to Ansible 11 stopped verbosive output.
ansible-version: 9
timeout: 3600
run: tools/gate/playbooks/docker-image-build.yaml
nodeset: airship-maas-single-node-jammy
secrets:
- airship_maas_quay_creds_2026
irrelevant-files:
- "^images/.*"
parent: airship-docker-build-gate-base
nodeset: airship-maas-single-node-noble
required-projects:
- name: airship/treasuremap
override-checkout: v1.9
vars:
quay_creds: "{{ airship_maas_quay_creds_2026 }}"
publish: true
distro: ubuntu_focal
tags:
@@ -168,16 +172,14 @@
- job:
name: airship-maas-docker-publish-jammy
# Pinned for now due to Ansible 11 stopped verbosive output.
ansible-version: 9
timeout: 3600
run: tools/gate/playbooks/docker-image-build.yaml
nodeset: airship-maas-single-node-jammy
secrets:
- airship_maas_quay_creds_2026
irrelevant-files:
- "^images/.*"
parent: airship-docker-build-gate-base
nodeset: airship-maas-single-node-noble
required-projects:
- name: airship/treasuremap
override-checkout: v1.9
vars:
quay_creds: "{{ airship_maas_quay_creds_2026 }}"
publish: true
distro: ubuntu_jammy
tags:
@@ -189,16 +191,14 @@
- job:
name: airship-maas-docker-publish-noble
# Pinned for now due to Ansible 11 stopped verbosive output.
ansible-version: 9
timeout: 3600
run: tools/gate/playbooks/docker-image-build.yaml
nodeset: airship-maas-single-node-jammy
secrets:
- airship_maas_quay_creds_2026
irrelevant-files:
- "^images/.*"
parent: airship-docker-build-gate-base
nodeset: airship-maas-single-node-noble
required-projects:
- name: airship/treasuremap
override-checkout: v1.9
vars:
quay_creds: "{{ airship_maas_quay_creds_2026 }}"
publish: true
distro: ubuntu_noble
tags:
@@ -208,14 +208,75 @@
static:
- latest
# - job:
# name: maas-airskiff-deployment-noble-kubeadm
# description: |
# Deploy maas using Airskiff and submitted Shipyard changes.
# parent: treasuremap-airskiff-deploy-maas-drydock-base
# nodeset: treasuremap-airskiff-5nodes-ubuntu_noble
# required-projects:
# - name: airship/treasuremap
# override-checkout: v1.9
# roles:
# - zuul: airship/treasuremap
# vars:
# treasuremap_ref: v1.9
# ENABLE_SLEEP: false
# ENABLE_MAAS_UPGRADE: false
# CLONE_MAAS: false
# MAKE_MAAS_IMAGES: true
# DISTRO: ubuntu_noble
# gate_scripts_relative_path: ../../airship/treasuremap
# zuul_treasuremap_relative_path: ../../airship/treasuremap
# - job:
# name: maas-airskiff-deployment-jammy-kubeadm
# description: |
# Deploy maas using Airskiff and submitted Shipyard changes.
# parent: treasuremap-airskiff-deploy-maas-drydock-base
# nodeset: treasuremap-airskiff-5nodes-ubuntu_noble
# required-projects:
# - name: airship/treasuremap
# override-checkout: v1.9
# roles:
# - zuul: airship/treasuremap
# vars:
# treasuremap_ref: v1.9
# ENABLE_SLEEP: false
# ENABLE_MAAS_UPGRADE: true
# CLONE_MAAS: false
# MAKE_MAAS_IMAGES: true
# DISTRO: ubuntu_jammy
# gate_scripts_relative_path: ../../airship/treasuremap
# zuul_treasuremap_relative_path: ../../airship/treasuremap
# - job:
# name: maas-airskiff-deployment-focal-kubeadm
# description: |
# Deploy maas using Airskiff and submitted Shipyard changes.
# parent: treasuremap-airskiff-deploy-maas-drydock-base
# nodeset: treasuremap-airskiff-5nodes-ubuntu_noble
# required-projects:
# - name: airship/treasuremap
# override-checkout: v1.9
# roles:
# - zuul: airship/treasuremap
# vars:
# treasuremap_ref: v1.9
# ENABLE_SLEEP: false
# ENABLE_MAAS_UPGRADE: true
# CLONE_MAAS: false
# MAKE_MAAS_IMAGES: true
# DISTRO: ubuntu_focal
# gate_scripts_relative_path: ../../airship/treasuremap
# zuul_treasuremap_relative_path: ../../airship/treasuremap
- job:
name: maas-airskiff-deployment-noble-kubeadm
# Pinned for now due to Ansible 11 stopped verbosive output.
ansible-version: 9
name: maas-airskiff-deployment-noble-kubeadm-maas-upgrade
description: |
Deploy maas using Airskiff and submitted Shipyard changes.
parent: treasuremap-airskiff-deploy-maas-drydock-base
nodeset: treasuremap-airskiff-1node-32GB-ubuntu_jammy
nodeset: treasuremap-airskiff-5nodes-ubuntu_noble
required-projects:
- name: airship/treasuremap
override-checkout: v1.9
@@ -223,56 +284,14 @@
- zuul: airship/treasuremap
vars:
treasuremap_ref: v1.9
ENABLE_SLEEP: false
ENABLE_MAAS_UPGRADE: true
CLONE_MAAS: false
MAKE_MAAS_IMAGES: true
DISTRO: ubuntu_noble
DOCKER_REGISTRY: localhost:5000
gate_scripts_relative_path: ../../airship/treasuremap
zuul_treasuremap_relative_path: ../../airship/treasuremap
- job:
name: maas-airskiff-deployment-jammy-kubeadm
# Pinned for now due to Ansible 11 stopped verbosive output.
ansible-version: 9
description: |
Deploy maas using Airskiff and submitted Shipyard changes.
parent: treasuremap-airskiff-deploy-maas-drydock-base
nodeset: treasuremap-airskiff-1node-32GB-ubuntu_jammy
required-projects:
- name: airship/treasuremap
override-checkout: v1.9
roles:
- zuul: airship/treasuremap
vars:
treasuremap_ref: v1.9
CLONE_MAAS: false
MAKE_MAAS_IMAGES: true
DISTRO: ubuntu_jammy
DOCKER_REGISTRY: localhost:5000
gate_scripts_relative_path: ../../airship/treasuremap
zuul_treasuremap_relative_path: ../../airship/treasuremap
- job:
name: maas-airskiff-deployment-focal-kubeadm
# Pinned for now due to Ansible 11 stopped verbosive output.
ansible-version: 9
description: |
Deploy maas using Airskiff and submitted Shipyard changes.
parent: treasuremap-airskiff-deploy-maas-drydock-base
nodeset: treasuremap-airskiff-1node-32GB-ubuntu_jammy
required-projects:
- name: airship/treasuremap
override-checkout: v1.9
roles:
- zuul: airship/treasuremap
vars:
treasuremap_ref: v1.9
CLONE_MAAS: false
MAKE_MAAS_IMAGES: true
DISTRO: ubuntu_focal
DOCKER_REGISTRY: localhost:5000
gate_scripts_relative_path: ../../airship/treasuremap
zuul_treasuremap_relative_path: ../../airship/treasuremap
- secret:
name: airship_maas_quay_creds
+4 -4
View File
@@ -26,8 +26,8 @@ PUSH_IMAGE ?= false
# use this variable for image labels added in internal build process
LABEL ?= org.airshipit.build=community
COMMIT ?= $(shell git rev-parse HEAD)
DISTRO ?= ubuntu_jammy
DISTRO_ALIAS ?= ubuntu_jammy
DISTRO ?= ubuntu_noble
DISTRO_ALIAS ?= ubuntu_noble
STRIPPED_DISTRO := $(shell echo $(DISTRO) | sed 's/^ubuntu_//')
STRIPPED_DISTRO_ALIAS := $(shell echo $(DISTRO_ALIAS) | sed 's/^ubuntu_//')
IMAGE_NAME := maas-rack-controller maas-region-controller sstream-cache
@@ -104,11 +104,11 @@ build:
$(IMAGE_DIR)
ifneq ($(DISTRO), $(DISTRO_ALIAS))
docker tag $(IMAGE) $(IMAGE_ALIAS)
ifeq ($(DOCKER_REGISTRY), localhost:5000)
ifeq (,$(findstring quay.io,$(DOCKER_REGISTRY)))
docker push $(IMAGE_ALIAS)
endif
endif
ifeq ($(DOCKER_REGISTRY), localhost:5000)
ifeq (,$(findstring quay.io,$(DOCKER_REGISTRY)))
docker push $(IMAGE)
endif
ifeq ($(PUSH_IMAGE), true)
+2 -2
View File
@@ -16,8 +16,8 @@
apiVersion: v1
description: Chart to run Canonical MaaS
name: maas
version: 0.1.8
appVersion: 3.6.2
version: 0.2.0
appVersion: 3.6.4
home: https://docs.ubuntu.com/maas
sources:
- https://git.launchpad.net/maas
@@ -287,7 +287,7 @@ function check_then_set_single {
local raw_val
raw_val=$(maas_cli ${ADMIN_USERNAME} maas get-config name=${option}) || return 1
cur_val=$(echo "$raw_val" | tail -1 | tr -d '"')
cur_val=$(echo "$raw_val" | tail -1 | tr -d '"' | tr ' ' ',')
desired_val=$(echo ${value} | tr -d '"')
if [[ $cur_val != $desired_val ]]; then
@@ -295,7 +295,7 @@ function check_then_set_single {
local result
result=$(maas_cli ${ADMIN_USERNAME} maas set-config name=${option} value=${desired_val}) || return 1
# Check if MAAS rejected the value (e.g. kernel not yet available after import)
if echo "$result" | grep -q '"is not a valid'; then
if echo "$result" | grep -q 'is not a valid'; then
log "WARNING: MAAS rejected value '${desired_val}' for ${option} (images may still be syncing), will retry"
return 1
fi
@@ -335,65 +335,142 @@ function configure_syslog {
}
function configure_images {
check_then_set default_osystem ${MAAS_DEFAULT_OS}
check_then_set commissioning_distro_series ${MAAS_DEFAULT_DISTRO}
check_then_set default_distro_series ${MAAS_DEFAULT_DISTRO}
check_then_set default_min_hwe_kernel ${MAAS_DEFAULT_KERNEL}
# Single-attempt per setting — no internal retry timer.
# If MAAS rejects a value (images not yet in DB), the caller retries the
# entire import+configure cycle via the outer timer, re-triggering start_import
# rather than spinning on set-config alone.
local rc=0
check_then_set_single default_osystem ${MAAS_DEFAULT_OS} || rc=1
check_then_set_single commissioning_distro_series ${MAAS_DEFAULT_DISTRO} || rc=1
check_then_set_single default_distro_series ${MAAS_DEFAULT_DISTRO} || rc=1
check_then_set_single default_min_hwe_kernel ${MAAS_DEFAULT_KERNEL} || true
return $rc
}
function configure_boot_sources {
# Set the boot source URL if using local image cache
if [[ $USE_IMAGE_CACHE == 'true' ]]; then
maas_cli ${ADMIN_USERNAME} boot-source update 1 url=http://localhost:8888/maas/images/ephemeral-v3/daily/ || exit 1
maas_cli ${ADMIN_USERNAME} boot-source update 1 \
url=http://localhost:8888/maas/images/ephemeral-v3/daily/ || exit 1
fi
# Read all selections for boot_source_id 1
maas_cli ${ADMIN_USERNAME} boot-source-selections read 1 || exit 1
local selections
selections=$(maas_cli ${ADMIN_USERNAME} boot-source-selections read 1) || exit 1
# Need to start an import to get the availability data
maas_cli "$ADMIN_USERNAME" boot-resources import || exit 1
sleep 10
maas_cli "$ADMIN_USERNAME" boot-resources stop-import || exit 1
local desired_id
desired_id=$(echo "$selections" | jq -r \
--arg os "${MAAS_DEFAULT_OS}" \
--arg distro "${MAAS_DEFAULT_DISTRO}" \
'.[] | select(.os == $os and .release == $distro) | .id' 2>/dev/null)
# Create a selection for the desired distro
maas_cli ${ADMIN_USERNAME} boot-source-selections create 1 os="${MAAS_DEFAULT_OS}" \
release="${MAAS_DEFAULT_DISTRO}" arches="amd64" subarches='*' labels='*' || exit 1
if [[ -z "$desired_id" ]]; then
# The desired distro is not yet in selections. UPDATE the first existing
# selection in-place rather than deleting it and creating a new one.
# Deletion is blocked by MAAS when the selection is the current
# commissioning OS ("Unable to delete... used in ephemeral environments").
# An in-place update changes the release without triggering that check,
# ensuring the sstream import will only see the desired distro and won't
# hang waiting for a distro the local sstream container doesn't carry.
local first_id
first_id=$(echo "$selections" | jq -r 'first | .id // empty' 2>/dev/null)
# Need to start an import to get the availability data
maas_cli "$ADMIN_USERNAME" boot-resources import || exit 1
sleep 10
if [[ -n "$first_id" ]]; then
log "Updating selection id=${first_id} in-place to ${MAAS_DEFAULT_OS}/${MAAS_DEFAULT_DISTRO}"
maas_cli ${ADMIN_USERNAME} boot-source-selection update 1 "${first_id}" \
os="${MAAS_DEFAULT_OS}" release="${MAAS_DEFAULT_DISTRO}" \
arches="amd64" subarches='*' labels='*' || exit 1
else
log "No existing selections — creating ${MAAS_DEFAULT_OS}/${MAAS_DEFAULT_DISTRO}"
maas_cli ${ADMIN_USERNAME} boot-source-selections create 1 \
os="${MAAS_DEFAULT_OS}" release="${MAAS_DEFAULT_DISTRO}" \
arches="amd64" subarches='*' labels='*' || exit 1
fi
# Set as default
maas_cli ${ADMIN_USERNAME} maas set-config name=default_distro_series value="${MAAS_DEFAULT_DISTRO}" || exit 1
maas_cli ${ADMIN_USERNAME} maas set-config name=commissioning_distro_series value="${MAAS_DEFAULT_DISTRO}" || exit 1
# Wait for MAAS to process the new selection
sleep 10
# Delete any selections that do not match the desired distro
selections_output=$(maas_cli ${ADMIN_USERNAME} boot-source-selections read 1) || exit 1
# Check if output is valid JSON before parsing
if echo "$selections_output" | jq -e . >/dev/null 2>&1; then
for row in $(echo "$selections_output" | jq -r \
--arg distro "$MAAS_DEFAULT_DISTRO" \
'.[] | select(.release != $distro) | "\(.id):\(.boot_source_id)"'); do
id="${row%%:*}"
boot_source_id="${row##*:}"
log "Deleting selection id $id from boot_source_id $boot_source_id"
maas_cli ${ADMIN_USERNAME} boot-source-selection delete "$boot_source_id" "$id" || exit 1
done
# Re-read after update/create so the stale-cleanup below sees fresh state
selections=$(maas_cli ${ADMIN_USERNAME} boot-source-selections read 1) || exit 1
else
log "Warning: Unable to parse boot-source-selections output, skipping cleanup of old selections"
log "Output was: $selections_output"
log "Selection for ${MAAS_DEFAULT_OS}/${MAAS_DEFAULT_DISTRO} already exists (id=${desired_id})"
fi
# Need to re-start an import to get the availability data
sleep 10
maas_cli "$ADMIN_USERNAME" boot-resources stop-import || exit 1
sleep 10
maas_cli "$ADMIN_USERNAME" boot-resources import || exit 1
# Remove all remaining selections not pointing to the desired distro.
# MUST be done before start_import — if any stale selection remains,
# the local sstream (which only carries the desired distro) will never
# satisfy it and is-importing will never return false.
#
# If MAAS refuses to delete a selection because it is the current
# commissioning OS, update it in-place to the desired distro instead.
# After import completes, configure_images will switch commissioning to
# the new distro and cleanup_old_boot_source_selections removes duplicates.
local stale_ids
stale_ids=$(echo "$selections" | jq -r \
--arg distro "${MAAS_DEFAULT_DISTRO}" \
'.[] | select(.release != $distro) | .id' 2>/dev/null)
for id in $stale_ids; do
log "Deleting stale selection id=${id}"
local result
result=$(maas_cli ${ADMIN_USERNAME} boot-source-selection delete 1 "${id}") || true
if echo "$result" | grep -qi "Unable to delete\|ephemeral"; then
log "Delete blocked (commissioning OS restriction) — updating selection id=${id} to ${MAAS_DEFAULT_OS}/${MAAS_DEFAULT_DISTRO} in-place"
local update_result
update_result=$(maas_cli ${ADMIN_USERNAME} boot-source-selection update 1 "${id}" \
os="${MAAS_DEFAULT_OS}" release="${MAAS_DEFAULT_DISTRO}" \
arches="amd64" subarches='*' labels='*') || exit 1
# maas_cli returns 0 even for API-level errors; inspect the response
if echo "$update_result" | grep -qi "error\|not found\|does not exist"; then
log "ERROR: in-place update of selection id=${id} failed: ${update_result}"
exit 1
fi
fi
done
# Final safety check: assert no stale selections remain before start_import.
# Any stale entry would cause the import to hang waiting for a distro the
# local sstream doesn't carry.
local remaining_stale
remaining_stale=$(maas_cli ${ADMIN_USERNAME} boot-source-selections read 1 | jq -r \
--arg distro "${MAAS_DEFAULT_DISTRO}" \
'.[] | select(.release != $distro) | .id' 2>/dev/null)
if [[ -n "$remaining_stale" ]]; then
log "ERROR: stale boot-source selections still present after cleanup (ids: ${remaining_stale}). Aborting to prevent import hang."
exit 1
fi
}
function cleanup_old_boot_source_selections {
# After configure_images has switched commissioning_distro_series to the new
# distro, remove any duplicate selections that were updated in-place from the
# old distro. Keep only the first one found for the desired distro.
local selections
selections=$(maas_cli ${ADMIN_USERNAME} boot-source-selections read 1) || return 1
if ! echo "$selections" | jq -e . >/dev/null 2>&1; then
log "Warning: Unable to parse boot-source-selections output, skipping cleanup"
return 0
fi
local keep_id
keep_id=$(echo "$selections" | jq -r \
--arg distro "$MAAS_DEFAULT_DISTRO" \
'[.[] | select(.release == $distro)] | first | .id // empty' 2>/dev/null)
if [[ -z "$keep_id" ]]; then
log "Warning: no ${MAAS_DEFAULT_DISTRO} selection found in cleanup — nothing to do"
return 0
fi
for row in $(echo "$selections" | jq -r \
--argjson keep "${keep_id}" \
'.[] | select(.id != $keep) | "\(.id):\(.boot_source_id)"'); do
local id="${row%%:*}"
local boot_source_id="${row##*:}"
log "Deleting duplicate selection id=${id} from boot_source_id=${boot_source_id}"
local result
result=$(maas_cli ${ADMIN_USERNAME} boot-source-selection delete "$boot_source_id" "$id") || true
if echo "$result" | grep -qi "Unable to delete\|ephemeral"; then
log "WARNING: Cannot delete selection ${id}: ${result} — skipping"
fi
done
}
function create_extra_commissioning_script {
@@ -455,13 +532,20 @@ configure_syslog
configure_extra_settings
create_extra_commissioning_script
# make call to import images
timer "$RETRY_TIMER" configure_boot_sources
start_import
# Add desired distro selection, then import, then apply configs, then clean up old selections.
# Order matters:
# 1. configure_boot_sources — ensure only the desired distro is in selections
# 2. import_and_configure loop — retry start_import + configure_images as a unit:
# if configure_images fails (images not yet accepted by MAAS), the outer
# timer re-runs start_import on the next attempt rather than spinning on
# set-config alone, which would loop forever against an empty boot-resource DB.
# 3. cleanup_old_boot_source_selections — remove duplicates left by in-place updates
if [[ $? -eq 0 ]]; then
configure_images
else
log "Image import FAILED!"
exit 1
fi
function import_and_configure {
start_import || return 1
configure_images || return 1
}
timer "$RETRY_TIMER" configure_boot_sources
timer "$RETRY_TIMER" import_and_configure || { log "Import and configure FAILED!"; exit 1; }
cleanup_old_boot_source_selections
@@ -0,0 +1,51 @@
#!/bin/bash
# Copyright 2017 The Openstack-Helm Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
{{- if .Values.conf.maas.tls.enabled }}
LOCAL_MAAS_ENDPOINT="https://127.0.0.1:{{ tuple "maas_region" "podport" "region_api" . | include "helm-toolkit.endpoints.endpoint_port_lookup" }}/MAAS/api/2.0/"
{{- else }}
LOCAL_MAAS_ENDPOINT="http://127.0.0.1:{{ tuple "maas_region" "podport" "region_api" . | include "helm-toolkit.endpoints.endpoint_port_lookup" }}/MAAS/api/2.0/"
{{- end }}
function api_is_up {
{{- if (and .Values.conf.maas.tls.enabled .Values.conf.maas.tls.insecure) }}
python3 -c "
import urllib.request, ssl, sys
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
urllib.request.urlopen('${LOCAL_MAAS_ENDPOINT}version/', context=ctx, timeout=5)
"
{{- else if .Values.conf.maas.tls.enabled }}
python3 -c "
import urllib.request, ssl, sys
ctx = ssl.create_default_context(cafile='/usr/local/share/ca-certificates/maas-ca.crt')
urllib.request.urlopen('${LOCAL_MAAS_ENDPOINT}version/', context=ctx, timeout=5)
"
{{- else }}
python3 -c "
import urllib.request
urllib.request.urlopen('${LOCAL_MAAS_ENDPOINT}version/', timeout=5)
"
{{- end }}
}
if ! api_is_up; then
exit 1
fi
+2
View File
@@ -56,6 +56,8 @@ data:
{{ tuple "bin/_maas-vip-configure.sh.tpl" . | include "helm-toolkit.utils.template" | indent 4 }}
start-syslog.sh: |
{{ tuple "bin/_start-syslog.sh.tpl" . | include "helm-toolkit.utils.template" | indent 4 }}
region-readiness.sh: |
{{ tuple "bin/_region-rediness.sh.tpl" . | include "helm-toolkit.utils.template" | indent 4 }}
{{- if .Values.conf.maas.tls.enabled }}
enable-tls.sh: |
{{ tuple "bin/_enable-tls.sh.tpl" . | include "helm-toolkit.utils.template" | indent 4 }}
+17 -2
View File
@@ -155,6 +155,11 @@ spec:
fieldPath: metadata.namespace
- name: MAAS_REGION_SECRET
value: {{ .Values.secrets.maas_region.name }}
- name: ADMIN_USERNAME
valueFrom:
secretKeyRef:
name: {{ .Values.secrets.maas_users.admin }}
key: USERNAME
ports:
- name: region-api
containerPort: {{ tuple "maas_region" "podport" "region_api" $envAll | include "helm-toolkit.endpoints.endpoint_port_lookup" }}
@@ -239,8 +244,14 @@ spec:
- name: region-proxy
containerPort: {{ tuple "maas_region" "podport" "region_proxy" $envAll | include "helm-toolkit.endpoints.endpoint_port_lookup" }}
readinessProbe:
tcpSocket:
port: {{ tuple "maas_region" "podport" "region_api" $envAll | include "helm-toolkit.endpoints.endpoint_port_lookup" }}
exec:
command:
- /bin/bash
- /tmp/region-readiness.sh
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 10
failureThreshold: 3
command:
- /tmp/start.sh
volumeMounts:
@@ -288,6 +299,10 @@ spec:
mountPath: /tmp/start.sh
subPath: start.sh
readOnly: true
- name: maas-bin
mountPath: /tmp/region-readiness.sh
subPath: region-readiness.sh
readOnly: true
- name: maas-etc
mountPath: /etc/nsswitch.conf
subPath: nsswitch.conf
+13 -9
View File
@@ -59,6 +59,10 @@ dependencies:
import_resources:
jobs:
- maas-bootstrap-admin-user
pod:
- labels:
application: maas
component: rack
services:
- service: maas_region
endpoint: internal
@@ -104,19 +108,19 @@ manifests:
images:
tags:
db_init: docker.io/library/postgres:14.5
db_sync: quay.io/airshipit/maas-region-controller:latest
maas_rack: quay.io/airshipit/maas-rack-controller:latest
maas_region: quay.io/airshipit/maas-region-controller:latest
bootstrap: quay.io/airshipit/maas-region-controller:latest
export_api_key: quay.io/airshipit/maas-region-controller:latest
maas_cache: quay.io/airshipit/sstream-cache:latest
db_init: docker.io/library/postgres:17.4
db_sync: quay.io/airshipit/maas-region-controller-noble:latest
maas_rack: quay.io/airshipit/maas-rack-controller-noble:latest
maas_region: quay.io/airshipit/maas-region-controller-noble:latest
bootstrap: quay.io/airshipit/maas-region-controller-noble:latest
export_api_key: quay.io/airshipit/maas-region-controller-noble:latest
maas_cache: quay.io/airshipit/sstream-cache-focal:latest
dep_check: quay.io/airshipit/kubernetes-entrypoint:latest-ubuntu_focal
ingress: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
ingress_vip: docker.io/busybox:latest
error_pages: gcr.io/google_containers/ingress-gce-404-server-with-metrics-amd64:v1.6.0
maas_syslog: quay.io/airshipit/maas-region-controller:latest
enable_tls: quay.io/airshipit/maas-region-controller:latest
maas_syslog: quay.io/airshipit/maas-region-controller-noble:latest
enable_tls: quay.io/airshipit/maas-region-controller-noble:latest
pull_policy: IfNotPresent
local_registry:
# TODO(portdirect): this chart does not yet support local image cacheing
+1 -1
View File
@@ -48,7 +48,7 @@ RUN apt-get -qq update \
&& apt-get install -y \
maas-rack-controller=$MAAS_VERSION \
&& rm -rf /var/lib/apt/lists/*\
&& rm -f /etc/ssl/private/snakeoil.*
&& rm -f /etc/ssl/private/ssl-cert-snakeoil.*
# Preserve the directory structure, permissions, and contents of /var/lib/maas
RUN mkdir -p /opt/maas/ && tar -cvzf /opt/maas/var-lib-maas.tgz /var/lib/maas
+2 -2
View File
@@ -18,7 +18,7 @@ ARG no_proxy
ENV DEBIAN_FRONTEND=noninteractive
ENV container=docker
ENV MAAS_VERSION=1:3.5.11-16404-g.4e3041434-0ubuntu1~22.04.1
ENV MAAS_VERSION=1:3.5.12-16413-g.7fb94f378-0ubuntu1~22.04.1
ENV MAAS_PPA=ppa:maas/3.5
RUN apt-get -qq update \
@@ -53,7 +53,7 @@ RUN apt-get -qq update \
&& apt-get install -y \
maas-rack-controller=$MAAS_VERSION \
&& rm -rf /var/lib/apt/lists/*\
&& rm -f /etc/ssl/private/snakeoil.*
&& rm -f /etc/ssl/private/ssl-cert-snakeoil.*
# Preserve the directory structure, permissions, and contents of /var/lib/maas
RUN mkdir -p /opt/maas/ && tar -cvzf /opt/maas/var-lib-maas.tgz /var/lib/maas
+1 -1
View File
@@ -53,7 +53,7 @@ RUN apt-get -qq update \
&& apt-get install -y \
maas-rack-controller=$MAAS_VERSION \
&& rm -rf /var/lib/apt/lists/*\
&& rm -f /etc/ssl/private/.*snakeoil.*
&& rm -f /etc/ssl/private/ssl-cert-snakeoil.*
# Preserve the directory structure, permissions, and contents of /var/lib/maas
RUN mkdir -p /opt/maas/ && tar -cvzf /opt/maas/var-lib-maas.tgz /var/lib/maas
@@ -47,8 +47,8 @@ RUN apt-get -qq update \
maas-region-api=$MAAS_VERSION \
# tcpdump is required by /usr/lib/maas/beacon-monitor
tcpdump \
&& rm -rf /var/lib/apt/lists/* \
&& rm -f /etc/ssl/private/snakeoil.*
&& rm -rf /var/lib/apt/lists/*\
&& rm -f /etc/ssl/private/ssl-cert-snakeoil.*
# Preserve the directory structure, permissions, and contents of /var/lib/maas
RUN mkdir -p /opt/maas/ && tar -cvzf /opt/maas/var-lib-maas.tgz /var/lib/maas
@@ -18,7 +18,7 @@ ARG no_proxy
ENV DEBIAN_FRONTEND=noninteractive
ENV container=docker
ENV MAAS_VERSION=1:3.5.11-16404-g.4e3041434-0ubuntu1~22.04.1
ENV MAAS_VERSION=1:3.5.12-16413-g.7fb94f378-0ubuntu1~22.04.1
ENV MAAS_PPA=ppa:maas/3.5
RUN apt-get -qq update \
@@ -51,7 +51,7 @@ RUN apt-get -qq update \
# tcpdump is required by /usr/lib/maas/beacon-monitor
tcpdump \
&& rm -rf /var/lib/apt/lists/*\
&& rm -f /etc/ssl/private/snakeoil.*
&& rm -f /etc/ssl/private/ssl-cert-snakeoil.*
# Preserve the directory structure, permissions, and contents of /var/lib/maas
RUN mkdir -p /opt/maas/ && tar -cvzf /opt/maas/var-lib-maas.tgz /var/lib/maas
@@ -51,7 +51,7 @@ RUN apt-get -qq update \
# tcpdump is required by /usr/lib/maas/beacon-monitor
tcpdump \
&& rm -rf /var/lib/apt/lists/*\
&& rm -f /etc/ssl/private/.*snakeoil.*
&& rm -f /etc/ssl/private/ssl-cert-snakeoil.*
# Preserve the directory structure, permissions, and contents of /var/lib/maas
RUN mkdir -p /opt/maas/ && tar -cvzf /opt/maas/var-lib-maas.tgz /var/lib/maas
+3 -1
View File
@@ -27,7 +27,9 @@ RUN apt-get -qq update && \
gpgv \
python3-certifi \
simplestreams \
ubuntu-cloudimage-keyring
ubuntu-cloudimage-keyring \
&& rm -rf /var/lib/apt/lists/*\
&& rm -f /etc/ssl/private/.*snakeoil.*
RUN sstream-mirror --keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg ${SSTREAM_IMAGE} \
/var/www/html/maas/images/ephemeral-v3/daily 'arch=amd64' "release~${SSTREAM_RELEASE}" --max=1 --progress
+3 -1
View File
@@ -28,7 +28,9 @@ RUN apt-get -qq update && \
gpgv \
python3-certifi \
simplestreams \
ubuntu-cloudimage-keyring
ubuntu-cloudimage-keyring \
&& rm -rf /var/lib/apt/lists/*\
&& rm -f /etc/ssl/private/.*snakeoil.*
RUN sstream-mirror --keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg ${SSTREAM_IMAGE} \
/var/www/html/maas/images/ephemeral-v3/daily 'arch=amd64' "release~${SSTREAM_RELEASE}" --max=1 --progress
+3 -1
View File
@@ -28,7 +28,9 @@ RUN apt-get -qq update && \
gpgv \
python3-certifi \
simplestreams \
ubuntu-cloudimage-keyring
ubuntu-cloudimage-keyring \
&& rm -rf /var/lib/apt/lists/*\
&& rm -f /etc/ssl/private/.*snakeoil.*
RUN sstream-mirror --keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg ${SSTREAM_IMAGE} \
/var/www/html/maas/images/ephemeral-v3/daily 'arch=amd64' "release~${SSTREAM_RELEASE}" --max=1 --progress
+1 -1
View File
@@ -17,7 +17,7 @@
set -x
HELM=$1
HELM_ARTIFACT_URL=${HELM_ARTIFACT_URL:-"https://get.helm.sh/helm-v3.18.4-linux-amd64.tar.gz"}
HELM_ARTIFACT_URL=${HELM_ARTIFACT_URL:-"https://get.helm.sh/helm-v4.1.4-linux-amd64.tar.gz"}
function install_helm_binary {