app-upgrade-operator

StarlingX Upgrade Operator FluxCD Application

A Kubernetes operator that provides declarative management and orchestration of deployed software on all hosts of a StarlingX cluster. This includes both Platform (kernel/OS, host, containerized) software and Kubernetes software.

This repository contains the operator source code, Helm chart packaging, FluxCD manifests, and sysinv plugins.

Phase 0.2

In phase 0.2 the upgrade-operator supports:

• Deployment of in-service product (platform) PATCH releases via VIM sw-deploy-strategy
• Deployment of product MAJOR releases via VIM sw-deploy-strategy
• Kubernetes version upgrades via VIM kube-upgrade-strategy
• Multi-step deployment sequencing (e.g. product patch → k8s upgrade → product major) in a single reconcile

Repository Structure

• upgrade-operator/ - Operator Go source code, Dockerfile, tests, and docs
• helm-charts/custom/upgrade-operator-helm/ - Helm chart packaging for STX build
• stx-upgrade-operator-helm/ - FluxCD manifests, app metadata, and image build recipe
• python3-k8sapp-upgrade-operator/ - Sysinv lifecycle/helm plugins

CRD Schema

apiVersion: upgrade.starlingx.io/v1alpha1
kind: Upgrade
metadata:
  name: system
spec:
  release:
    platform: starlingx-13.01.0    # desired product release
    kubernetes: v1.34.1            # desired k8s version
  deployPolicy:
    dryRun: false
    alarmRestriction: strict
    controllerApplyType: serial
    workerApplyType: serial
    maxParallelWorkerHosts: 2
    instanceAction: stop-start
    rollbackPolicy: default
  targetDeployStep: # None | pre-upgrade-deploy | ...
status:
  release: "starlingx-13.01.0 / v1.34.1"   # composite: <product> / <k8s>
  activeDeploy:
    status: None          # None / Deploying / Unknown / Build Failed / Apply Failed / Failed
    deployFromRelease: "" # composite release at start
    deployToRelease: ""   # target composite release
    releaseDeploying: ""  # current sub-step target
    releaseDeployingStatus: "" # phase/stage progress
    statusReason: ""
    timeDuration: ""
    timeStart: ""
  deployPlan:             # computed deploy plan (populated with dryRun or on spec change)
    dryRun: false
    fromRelease: ""
    toRelease: ""
    validationResult: "OK"
    deploySteps: []
  uploadedProductReleases: [...]
  uploadedK8sVersions: [...]

Multi-Step Deployment Sequencing

When spec.release changes, the operator builds a deployment plan:

Change Type	Steps
Patch only	sw-deploy(target)
K8s only	kube-upgrade(target)
Patch + K8s	sw-deploy(target) → kube-upgrade(target)
Major upgrade (to GA)	sw-deploy(highest-patch) → kube-upgrade(highest-k8s) → sw-deploy(major-target) → kube-upgrade(target-k8s)
Major upgrade (to patch)	sw-deploy(highest-patch) → kube-upgrade(highest-k8s) → sw-deploy(GA) → sw-deploy(patch-target) → kube-upgrade(target-k8s)

Printer Columns

$ kubectl get upgrade
NAME     RELEASE                       STATUS      DURATION   DEPLOY-FROM                   DEPLOY-TO
system   starlingx-13.01.2 / v1.33.0   Deploying   18:37      starlingx-13.01.2 / v1.33.0   starlingx-27.03.0 / v1.34.1

$ kubectl get upgrade -o wide
NAME     RELEASE                       STATUS      DURATION   DEPLOY-FROM                   DEPLOY-TO                     DEPLOYING-REL   DEPLOYING-REL-STATUS
system   starlingx-13.01.2 / v1.33.0   Deploying   18:37      starlingx-13.01.2 / v1.33.0   starlingx-27.03.0 / v1.34.1   k8s-v1.34.1     apply(69%)/stg:(3/4) kube-upgrade-complete

Webhook Validation

The admission webhook enforces:

• Only the system Upgrade CR can be created; deletion is rejected
• Product release must be in uploadedProductReleases and not unavailable
• K8s version must be supported by the target product release and not unavailable
• For major upgrades, the ISO entry-point release must be uploaded and deployable
• No changes to spec.release while a deployment is in progress (abort allowed)
• rollbackPolicy: snapshot is only allowed on AIO-SX systems

Kubernetes Events

Event	Description
DeployStarted	Deployment initiated with step count and target release
DeployStepCompleted	Intermediate step completed with cumulative duration
DeployCompleted	All steps finished successfully
SwactInitiated	Swact to controller-0 initiated before a major upgrade step

Key Features

• Singleton CR Bootstrap: Creates system CR on startup matching live system state
• Strategy Management: Manages VIM sw-deploy and kube-upgrade strategies
• Build-Failure Auto-Retry: Auto-retries build-failed VIM strategies
• Robust Restart Recovery: Resumes monitoring after SIGTERM/crash
• Leader Election: Only leader reconciles; all pods serve webhook
• Declarative Reconcile: Each cycle observes and acts — no stored execution plan

Building

Container Image

Via STX build system (build-stx-images.sh):

cd $MY_REPO/build-tools/build-docker-images
./build-stx-images.sh \
  --base starlingx/stx-debian:trixie-amd64-stable-latest \
  --no-pull-base \
  --cache \
  --only upgrade-operator

Manual docker build:

cd $MY_REPO/stx/app-upgrade-operator/upgrade-operator
docker build -t registry.local:9001/upgrade-operator:0.1.0 .

Export and load on target system:

docker save registry.local:9001/upgrade-operator:0.1.0 -o upgrade-operator.tar
scp upgrade-operator.tar sysadmin@<SYSTEM-IP>:/tmp/

# On the target system:
sudo docker load -i /tmp/upgrade-operator.tar
sudo docker push registry.local:9001/upgrade-operator:0.1.0

Debian Packages

Build the three packages in order:

build-pkgs -c -p upgrade-operator-helm
build-pkgs -c -p python3-k8sapp-upgrade-operator
build-pkgs -c -p stx-upgrade-operator-helm

The final package stx-upgrade-operator-helm produces the app tarball (upgrade-operator-<version>.tgz) installed via system application-upload.

Deployment

Upload and Apply

Keystone credentials are injected automatically by the lifecycle plugin (reads from the system keyring during apply). No manual helm-override-update is required for credentials.

source /etc/platform/openrc

# Upload the app tarball
system application-upload /usr/local/share/applications/helm/upgrade-operator-<version>.tgz

# Apply the application (credentials auto-injected from keyring)
system application-apply upgrade-operator

Verify

system application-show upgrade-operator
kubectl get pods -n upgrade-operator-system
kubectl get upgrade
kubectl get upgrade -o wide

Remove

source /etc/platform/openrc
system application-remove upgrade-operator
system application-delete upgrade-operator

Helm Overrides

View current overrides:

system helm-override-show upgrade-operator upgrade-operator upgrade-operator-system

Set custom image:

system helm-override-update upgrade-operator upgrade-operator upgrade-operator-system \
  --set manager.image.repository=registry.local:9001/upgrade-operator \
  --set manager.image.tag=0.1.0

Usage Examples

# Deploy a product patch
kubectl patch upgrade system --type merge \
  -p '{"spec":{"release":{"platform":"starlingx-13.01.1"}}}'

# Deploy a k8s upgrade
kubectl patch upgrade system --type merge \
  -p '{"spec":{"release":{"kubernetes":"v1.34.1"}}}'

# Deploy both product patch and k8s upgrade
kubectl patch upgrade system --type merge \
  -p '{"spec":{"release":{"platform":"starlingx-13.01.2","kubernetes":"v1.34.1"}}}'

# Deploy a major product upgrade
kubectl patch upgrade system --type merge \
  -p '{"spec":{"release":{"platform":"starlingx-27.03.0"}}}'

# Abort an in-progress deployment (revert to deployFromRelease)
kubectl patch upgrade system --type merge \
  -p '{"spec":{"release":{"platform":"starlingx-13.01.0","kubernetes":"v1.33.0"}}}'

# Dry-run: compute deploy plan without executing
kubectl patch upgrade system --type merge \
  -p '{"spec":{"release":{"platform":"starlingx-27.03.0"},"deployPolicy":{"dryRun":true}}}'

Testing

# Verify operator is running
kubectl get pods -n upgrade-operator-system

# Verify CRD and singleton CR
kubectl get upgrade

# Verify webhook (delete must be rejected)
kubectl delete upgrade system
# Expected: "The 'system' Upgrade CR cannot be deleted."

# Verify webhook (only "system" CR allowed)
kubectl apply -f - <<EOF
apiVersion: upgrade.starlingx.io/v1alpha1
kind: Upgrade
metadata:
  name: other
spec:
  release:
    platform: "starlingx-99.99.0"
EOF
# Expected: "only the 'system' Upgrade CR is allowed; cannot create 'other'"

# Verify webhook (unknown release rejected)
kubectl patch upgrade system --type merge \
  -p '{"spec":{"release":{"platform":"starlingx-99.99.0"}}}'
# Expected: "Product release 'starlingx-99.99.0' is not uploaded"

# Verify dry-run
kubectl patch upgrade system --type merge \
  -p '{"spec":{"deployPolicy":{"dryRun":true}}}'
kubectl get upgrade system -o jsonpath='{.status.deployPlan}' | python3 -m json.tool

# Verify events
kubectl describe upgrade system | tail -20

# Verify clean removal
system application-remove upgrade-operator