starlingx/docs
Code Issues Proposed changes

StarlingX: Updated K8S upgrade procedures

Browse Source 
Story: 2011415

Update 12: Addressed Patchset 14 comments.

Update 11: Addresses Patchset 13 comment.

Update 10: Addressed Patchset 12 comments.

Update 9: Addressed some of the Patchset 11 comments.

Update 8: Please review the latest updates based on discussion and email that was sent last week.

Update7: Addressed Patchset 9 comments.

Update6: Addressed Patchset 8 comments.

Update5: Addressed comments received on Patchset 7 onwards.

Update4: Addressed all comments received so far.

Update3: Addressed comments from previous patchset and also added new steps for multi-version upgrades.

Update2: Addressed comments given in two files. Need to clarify comments for the third file.

Update 1: Updated version and also added details for semantic version skew policy checking

Change-Id: I1f689931572bf1f252b4e005ec226f17a761b867
Signed-off-by: Petsy Mathew <petsy.mathew@windriver.com>
This commit is contained in:
Petsy Mathew
2025-09-09 17:48:22 +00:00
parent 15aa8deb0e
commit 6db00e3642
6 changed files with 497 additions and 637 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
doc/source/updates/kubernetes/about-kubernetes-orchestrated-upgrades.rst
View File

@@ -2,24 +2,29 @@
.. xkr1590157116928
.. _about-kubernetes-orchestrated-upgrades:
====================================================
About Kubernetes Version Upgrade Cloud Orchestration
====================================================
.. Kindly highlight the updates needed for 25.09 in this section.
Kubernetes version upgrade cloud orchestration allows the Kubernetes version on
============================================
About Kubernetes Upgrade Cloud Orchestration
============================================
Kubernetes upgrade cloud orchestration allows the Kubernetes version on
all hosts of an entire |prod-long| cloud to be updated with a single operation.
You can configure and run Kubernetes version upgrade cloud orchestration using
the CLI, or the stx-nfv VIM REST API.
.. note::
The ability to abort and rollback during a Kubernetes multi-version upgrade is only supported for |AIO-SX| configuration. For multi-node clusters, while multi-version Kubernetes upgrades are now supported (as of |v_master|), abort and rollback are not available.
.. _xkr1590157116928-section-phk-xls-tlb:
-----------------------------------------------------------
Kubernetes Version Upgrade Cloud Orchestration Requirements
-----------------------------------------------------------
---------------------------------------------------
Kubernetes Upgrade Cloud Orchestration Requirements
---------------------------------------------------
Kubernetes version upgrade orchestration can only be done on a system that
Kubernetes upgrade orchestration can only be done on a system that
meets the following conditions:
@@ -56,5 +61,8 @@ meets the following conditions:
version |kube-ver|.
.. note::
Administrative controller ``swact`` operations should be avoided during
Kubernetes version upgrade orchestration.
Avoid administrative controller ``swact`` operations during Kubernetes
version upgrade orchestration.

137
doc/source/updates/kubernetes/configuring-kubernetes-multi-version-upgrade-orchestration-aio-b0b59a346466.rst
View File

@@ -1,12 +1,20 @@
.. _configuring-kubernetes-multi-version-upgrade-orchestration-aio-b0b59a346466:
=========================================================================
Configure Kubernetes Multi-Version Upgrade Cloud Orchestration for AIO-SX
=========================================================================
============================================
Kubernetes Upgrade using Cloud Orchestration
============================================
You can configure Kubernetes multi-version upgrade orchestration strategy using
the :command:`sw-manager` command. This feature is enabled from
|prod| |k8s-multi-ver-orch-strategy-release| and is supported only for the |AIO-SX| system.
You can upgrade multiple nodes in a single orchestration. The section below
outlines the upgrade steps for |AIO-SX| and multi-nodes.
.. note::
Automated abort and rollback are not supported for multi-node Kubernetes
upgrades. These capabilities are only available on |AIO-SX| platforms.
To perform an orchestrated Kubernetes version upgrade, you must first create a
Kubernetes Upgrade Orchestration Strategy. You can configure this strategy
using the :command:`sw-manager` command.
.. note::
@@ -53,7 +61,7 @@ For example:
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy –help
~(keystone_admin)$ sw-manager kube-upgrade-strategy -help
usage: sw-manager kube-upgrade-strategy [-h] ...
optional arguments:
-h, --help show this help message and exit
@@ -78,7 +86,7 @@ For example:
.. note::
The sysadmin and admin passwords must be set to the same value prior to
starting an upgrade from |prod-long| Release |v_r9| to |prod| Release |v_r10|.
starting an upgrade from |prod-long| Release |v_r10| to |prod| Release |v_master|.
.. only:: partner
@@ -94,10 +102,10 @@ For example:
+-----------------+--------+-------------+
| Version | Target | State |
+-----------------+--------+-------------+
| v1.21.8 | True | active |
| v1.22.5 | False | available |
| v1.23.1 | False | available |
| v1.24.4 | False | available |
| v1.29.2 | True | active |
| v1.30.6 | False | available |
| v1.31.5 | False | available |
| v1.32.2 | False | available |
+-----------------+--------+-------------+
#. Confirm that the system is healthy.
@@ -134,7 +142,7 @@ For example:
#. Create the strategy.
The Kubernetes multi-version upgrade orchestration strategy :command:`create`
The Kubernetes upgrade orchestration strategy :command:`create`
command creates a series of stages with steps that apply the Kubernetes
version upgrade.
@@ -143,7 +151,7 @@ For example:
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy create --to-version v1.24.4
~(keystone_admin)$ sw-manager kube-upgrade-strategy create --to-version v1.32.2
Strategy Kubernetes Upgrade Strategy:
strategy-uuid: f03f5944-ee79-4047-8d2e-68bfa6775210
controller-apply-type: serial
@@ -159,7 +167,7 @@ For example:
where:
``--to-version``
The version of Kubernetes to upgrade to, for example, ``v1.24.4``.
The version of Kubernetes to upgrade to, for example, ``v1.32.2``.
This argument is required.
``--controller-apply-type`` and ``--storage-apply-type``
@@ -212,15 +220,14 @@ For example:
~(keystone_admin)]$ sw-manager kube-upgrade-strategy create --help
usage:sw-manager kube-upgrade-strategy [-h]
--to-version <kubernetesVersion>
[--controller-apply-type {ignore}]
[--storage-apply-type {ignore}]
[--worker-apply-type
{serial,parallel,ignore}]
[--controller-apply-type {serial,ignore}]
[--storage-apply-type {serial,ignore}]
[--worker-apply-type {serial,parallel,ignore}]
[--instance-action {stop-start,migrate}]
[--alarm-restrictions {strict,relaxed}]
[--max-parallel-worker-hosts
{2,3,4,5,6,7,8,9,10}]
[--instance-action {migrate,stop-start}]
[--alarm-restrictions {strict,relaxed}]
--to-version TO_VERSION
optional arguments:
-h, --help show this help message and exit
@@ -230,13 +237,13 @@ For example:
defaults to serial
--worker-apply-type {serial,parallel,ignore}
defaults to serial
--max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10}
maximum worker hosts to update in parallel
--instance-action {migrate,stop-start}
--instance-action {stop-start,migrate}
defaults to stop-start
--alarm-restrictions {strict,relaxed}
defaults to strict
--max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10}
maximum worker hosts to update in parallel
--to-version TO_VERSION <The kubernetes version>
#. |optional| Display the strategy in summary, if required. The Kubernetes
upgrade strategy :command:`show` command displays the strategy in a summary.
@@ -252,31 +259,54 @@ For example:
default-instance-action: stop-start
alarm-restrictions: strict
current-phase: build
current-stage: kube-upgrade-query
current-step: query-kube-host-upgrade
current-phase-completion: 100%
state: ready-to-apply
build-result: success
build-reason:
The :command:`show` strategy subcommand displays a summary of the current
state of the strategy. A complete view of the strategy can be shown using
the ``--details`` option.
The strategy steps and stages are displayed using the ``--details`` option.
For example, see the following output of show command using the
``--details`` for the AIO-DX:
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy show --details | grep -e stage-name
stage-name: kube-upgrade-query
stage-name: kube-upgrade-start
stage-name: kube-upgrade-download-images
stage-name: kube-pre-application-update
stage-name: kube-upgrade-networking
stage-name: kube-upgrade-storage
stage-name: kube-upgrade-first-control-plane v1.30.6
stage-name: kube-upgrade-second-control-plane v1.30.6
stage-name: kube-upgrade-first-control-plane v1.31.5
stage-name: kube-upgrade-second-control-plane v1.31.5
stage-name: kube-upgrade-first-control-plane v1.32.2
stage-name: kube-upgrade-second-control-plane v1.32.2
stage-name: kube-upgrade-kubelet v1.32.2
stage-name: kube-upgrade-kubelet v1.32.2
stage-name: kube-upgrade-complete
stage-name: kube-post-application-update
stage-name: kube-upgrade-cleanup
#. Apply the strategy.
Kubernetes multi-version upgrade orchestration strategy :command:`apply` command
Kubernetes upgrade orchestration strategy :command:`apply` command
runs the strategy stages and steps consecutively until the Kubernetes
upgrade on all the hosts in the strategy is completed.
- Use the ``-stage-id`` option to specify a specific stage to apply one
at a time.
- Use the ``-stage-id`` option to specify a specific stage to apply one at a time.
.. note::
.. note::
When applying a single stage, only the next stage will be applied.
You cannot skip stages.
When applying a single stage, only the next stage will be applied.
You cannot skip stages.
.. code-block:: none
@@ -302,12 +332,12 @@ For example:
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 1272e9cc-1a15-4a53-bb5a-d47494729068 |
| from_version | v1.21.8 |
| to_version | v1.24.4 |
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | downloading-images |
| created_at | 2023-09-25T18:32:10.820488+00:00 |
| updated_at | 2023-09-25T18:32:10.885709+00:00 |
| created_at | 2025-09-25T18:32:10.820488+00:00 |
| updated_at | 2025-09-25T18:32:10.885709+00:00 |
+--------------+--------------------------------------+
You will see the ``state`` property transition through values, such as
@@ -315,7 +345,9 @@ For example:
and ``upgraded-first-master``.
#. |optional| Abort the strategy, if required. This is only used to stop and
abort the entire strategy.
abort the entire strategy.
.. note:: This step is only applicable to AIO-SX.
The Kubernetes version upgrade strategy :command:`abort` command can be
used to abort the Kubernetes version upgrade strategy after the current
@@ -329,30 +361,30 @@ For example:
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 1272e9cc-1a15-4a53-bb5a-d47494729068 |
| from_version | v1.21.8 |
| to_version | v1.24.4 |
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | upgrade-complete |
| created_at | 2023-09-25T18:52:10.885709+00:00 |
| updated_at | 2023-09-25T18:52:11.673259+00:00 |
| created_at | 2025-09-25T18:52:10.885709+00:00 |
| updated_at | 2025-09-25T18:52:11.673259+00:00 |
+--------------+--------------------------------------+
~(keystone_admin)$ system kube-version-list
+-----------------+--------+-------------+
| Version | Target | State |
+-----------------+--------+-------------+
| v1.21.8 | False | unavailable |
| v1.22.5 | False | unavailable |
| v1.23.1 | False | unavailable |
| v1.24.4 | True | active |
| v1.29.2 | False | unavailable |
| v1.30.6 | False | unavailable |
| v1.31.5 | False | unavailable |
| v1.32.2 | True | active |
+-----------------+--------+-------------+
#. Delete the strategy.
.. note::
After the Kubernetes multi-version upgrade orchestration strategy has been
applied (or aborted), it must be deleted before another Kubernetes
After the Kubernetes upgrade orchestration strategy has been
applied, it must be deleted before another Kubernetes
version upgrade strategy can be created. If a Kubernetes version
upgrade strategy application fails, you must address the issue that
caused the failure, then delete and re-create the strategy before
@@ -362,3 +394,8 @@ For example:
~(keystone_admin)$ sw-manager kube-upgrade-strategy delete
Strategy deleted.

321
doc/source/updates/kubernetes/configuring-kubernetes-update-orchestration.rst
View File

@@ -6,7 +6,10 @@
Create Kubernetes Version Upgrade Cloud Orchestration Strategy
==============================================================
You can configure *Kubernetes Version Upgrade Orchestration Strategy* using the
.. I’ve limited this topic to conceptual information and removed the procedure, as the file configuring-kubernetes-multi-version-upgrade-orchestration-aio-b0b59a346466 already contains similar content with the latest procedure updates.
You can configure *Kubernetes Upgrade Orchestration Strategy* using the
:command:`sw-manager` CLI.
.. note::
@@ -90,318 +93,4 @@ For example:
.. include:: /_includes/configuring-kubernetes-update-orchestration.rest
.. rubric:: |proc|
#. List available upgrades, for example:
.. code-block:: none
~(keystone_admin)$ system kube-version-list
+-----------------+--------+-------------+
| version | target | state |
+-----------------+--------+-------------+
| v1.18.1 | True | active |
| v1.19.13 | False | available |
| v1.20.9 | False | unavailable |
| v1.21.8 | False | unavailable |
+-----------------+--------+-------------+
#. Confirm that the system is healthy.
Check the current system health status, resolve any alarms and other issues
reported by the :command:`system health-query-kube-upgrade` command, then
recheck the system health status to confirm that all **System Health**
fields are set to **OK**.
By default, the upgrade process cannot be run and is not recommended to be
run with active alarms present. Use the :command:`system kube-upgrade-start --force`
command to force the upgrade process to start and ignore non-management-affecting
alarms.
.. note::
It is strongly recommended that you clear your system of any and all
alarms before doing an upgrade. While the :command:`--force` option is
available to run the upgrade, it is a best practice to clear any
alarms.
.. code-block:: none
~(keystone_admin)]$ system health-query-kube-upgrade
System Health:
All hosts are provisioned: [OK]
All hosts are unlocked/enabled: [OK]
All hosts have current configurations: [OK]
All hosts are patch current: [OK]
Ceph Storage Healthy: [OK]
No alarms: [OK]
All kubernetes nodes are ready: [OK]
All kubernetes control plane pods are ready: [OK]
Required patches are applied: [OK]
License valid for upgrade: [OK]
No instances running on controller-1: [OK]
All kubernetes applications are in a valid state: [OK]
Active controller is controller-0: [OK]
#. Create the strategy.
The *Kubernetes Version Upgrade Orchestration Strategy* :command:`create`
command creates a series of stages with steps that apply the Kubernetes
version upgrade.
Kubernetes Version upgrade requires a reboot. Therefore, the created strategy
includes steps that automatically lock and unlock the host to bring the new
image function into service.
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy create --to-version v1.19.13
Strategy Kubernetes Upgrade Strategy:
strategy-uuid: f7585178-cea6-4d2f-bda0-e0972145ebcf
controller-apply-type: serial
storage-apply-type: ignore
worker-apply-type: serial
default-instance-action: migrate
alarm-restrictions: strict
current-phase: build
current-phase-completion: 0%
state: building
inprogress: true
where:
``--to-version``
The version of Kubernetes to upgrade to. For example, ``v1.19.13``.
This argument is required.
``--controller-apply-type`` and ``--storage-apply-type``
These options cannot be changed from ``serial`` because Kubernetes
upgrade concurrency is only supported for worker hosts.
.. note::
Setting the Kubernetes version upgrade apply type is only supported
for hosts with only the worker function. Any attempt to modify the
controller or storage apply type will be rejected.
``--worker-apply-type``
This option specifies the host concurrency of the Kubernetes version
upgrade strategy:
- serial (default): worker hosts will be patched one at a time
- parallel: worker hosts will be upgraded in parallel
- At most, ``parallel`` will be upgraded at the same time
- At most, half of the hosts in a host aggregate will be upgraded
at the same time
- ignore: worker hosts will not be upgraded; strategy create will fail
Worker hosts with no instances are upgraded before worker hosts with
instances.
``--max-parallel-worker-hosts``
This option applies to the parallel worker apply type selection to
specify the maximum worker hosts to upgrade in parallel (minimum: 2,
maximum: 10).
``-instance-action``
This option only has significance when the |prefix|-openstack
application is loaded and there are instances running on worker hosts.
It specifies how the strategy deals with worker host instances over the
strategy execution.
``stop-start`` (default)
Instances will be stopped before the host lock operation following the
upgrade and then started again following the host unlock.
.. warning::
Using the ``stop-start`` option will result in an outage for each
instance, as it is stopped while the worker host is locked/unlocked.
In order to ensure this does not impact service, instances MUST be
grouped into anti-affinity (or anti-affinity best effort) server
groups, which will ensure that only a single instance in each server
group is stopped at a time.
``migrate``
Instances will be migrated off a host before it is patched (this
applies to reboot patching only).
``--alarm-restrictions``
This option sets how the how the Kubernetes version upgrade
orchestration behaves when alarms are present.
To display management-affecting active alarms, use the following
command:
.. code-block:: none
~(keystone_admin)$ fm alarm-list --mgmt_affecting
``strict`` (default)
The default strict option will result in patch orchestration failing if
there are any alarms present in the system (except for a small list of
alarms).
``relaxed``
This option allows orchestration to proceed if alarms are present, as
long as none of these alarms are management affecting.
.. code-block:: none
~(keystone_admin)]$ sw-manager kube-upgrade-strategy create --help
usage:sw-manager kube-upgrade-strategy [-h]
--to-version <kubernetesVersion>
[--controller-apply-type {ignore}]
[--storage-apply-type {ignore}]
[--worker-apply-type
{serial,parallel,ignore}]
[--max-parallel-worker-hosts
{2,3,4,5,6,7,8,9,10}]
[--instance-action {migrate,stop-start}]
[--alarm-restrictions {strict,relaxed}]
optional arguments:
-h, --help show this help message and exit
--controller-apply-type {serial,ignore}
defaults to serial
--storage-apply-type {serial,ignore}
defaults to serial
--worker-apply-type {serial,parallel,ignore}
defaults to serial
--max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10}
maximum worker hosts to update in parallel
--instance-action {migrate,stop-start}
defaults to stop-start
--alarm-restrictions {strict,relaxed}
defaults to strict
#. |optional| Display the strategy in summary, if required. The Kubernetes
upgrade strategy :command:`show` command displays the strategy in a summary.
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy show
Strategy Kubernetes Upgrade Strategy:
strategy-uuid: f7585178-cea6-4d2f-bda0-e0972145ebcf
controller-apply-type: serial
storage-apply-type: ignore
worker-apply-type: serial
default-instance-action: migrate
alarm-restrictions: strict
current-phase: build
current-phase-completion: 100%
state: ready-to-apply
build-result: success
build-reason:
The :command:`show` strategy subcommand displays a summary of the current
state of the strategy. A complete view of the strategy can be shown using
the ``--details`` option.
The strategy steps and stages are displayed using the ``--details`` option.
#. Apply the strategy.
*Kubernetes Version Upgrade Orchestration Strategy* :command:`apply` command
executes the strategy stages and steps consecutively until the Kubernetes
upgrade on all the hosts in the strategy is complete.
- Use the ``-stage-id`` option to specify a specific stage to apply; one
at a time.
.. note::
When applying a single stage, only the next stage will be applied;
you cannot skip stages.
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy apply
Strategy Kubernetes upgrade Strategy:
strategy-uuid: 3e43c018-9c75-4ba8-a276-472c3bcbb268
controller-apply-type: ignore
storage-apply-type: ignore
worker-apply-type: serial
default-instance-action: stop-start
alarm-restrictions: strict
current-phase: apply
current-phase-completion: 0%
state: applying
inprogress: true
- Use the :command:`kube-upgrade-show` command to monitor Kubernetes
upgrade state and percentage completion, for example:
.. code-block:: none
~(keystone_admin)$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 3d2da123-bff4-4b3a-a64a-b320c3b498cc |
| from_version | v1.18.1 |
| to_version | v1.19.13 |
| state | downloading-images |
| created_at | 2021-02-23T00:08:24.579257+00:00 |
| updated_at | 2021-02-23T00:09:35.413307+00:00 |
+--------------+--------------------------------------+
You will see the ``state`` property transition through values such as
``downloading-images``, ``downloaded-images``, ``upgrading-first-master``,
``upgraded-first-master``, etc.
#. |optional| Abort the strategy, if required. This is only used to stop, and
abort the entire strategy.
The Kubernetes version upgrade strategy :command:`abort` command can be
used to abort the Kubernetes version upgrade strategy after the current
step of the currently applying stage is completed.
#. Confirm that the upgrade has completed successfully, for example:
.. code-block:: none
~(keystone_admin)$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 426d7e11-2de2-40ba-b482-ed3691625383 |
| from_version | v1.18.1 |
| to_version | v1.19.13 |
| state | upgrade-complete |
| created_at | 2021-04-12T17:58:36.492523+00:00 |
| updated_at | 2021-04-12T18:49:11.673259+00:00 |
+--------------+--------------------------------------+
~(keystone_admin)$ system kube-version-list
+-----------------+--------+-------------+
| version | target | state |
+-----------------+--------+-------------+
| v1.18.1 | True | unavailable |
| v1.19.13 | False | active |
| v1.20.9 | False | available |
| v1.21.8 | False | unavailable |
+-----------------+--------+-------------+
#. Delete the strategy.
.. note::
After the *Kubernetes Version Upgrade Orchestration Strategy* has been
applied (or aborted) it must be deleted before another Kubernetes
version upgrade strategy can be created. If a Kubernetes version
upgrade strategy application fails, you must address the issue that
caused the failure, then delete and re-create the strategy before
attempting to apply it again.
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy delete
Strategy deleted.
For more details, see the section :ref:`Kubernetes Upgrade using Cloud Orchestration for AIO-SX and Multi-Nodes <configuring-kubernetes-multi-version-upgrade-orchestration-aio-b0b59a346466>` .

254
doc/source/updates/kubernetes/manual-kubernetes-components-upgrade.rst
View File

@@ -2,9 +2,9 @@
.. bfd1591638638205
.. _manual-kubernetes-components-upgrade:
=================================
Manual Kubernetes Version Upgrade
=================================
=======================================
Manual Kubernetes Upgrade in Multi-Node
=======================================
You can upgrade the Kubernetes version on a running system from one supported
version to another.
@@ -17,24 +17,27 @@ version to another.
always start with the same Kubernetes version as the highest version from
the release you are upgrading from.
Kubernetes upgrades must be done sequentially without skipping any
release(s).
Kubernetes upgrades previously required sequential version updates, one at
a time. The upgrade system now checks for Kubernetes version skew, allowing
kubelet components to run up to three minor versions behind the control
plane. This enables multi-version upgrades in a single cycle, eliminating
the need to upgrade kubelet through each intermediate version. The system
marks all valid versions as available, simplifying version selection and
reducing the number of upgrades steps.
.. rubric:: |context|
To complete this task, you will apply the following three updates (patches)
To complete this task, you will apply the following updates (patches)
and upgrade various systems.
**Platform update**
The platform update contains metadata for the new Kubernetes version and the
Kubernetes networking pods templates for the new Kubernetes version.
**Kubeadm update**
The kubeadm update upgrades the kubeadm RPM to the new Kubernetes version.
**Kubelet and Kubectl update**
This Kubernetes update upgrades kubelet and kubectl RPMs to the new
Kubernetes version.
**Kubernetes Component Packages**
Each supported Kubernetes version is packaged separately and includes the
corresponding versions of kubeadm, kubelet, and kubectl binaries. The system
uses Debian packages to manage Kubernetes components.
.. rubric:: |prereq|
@@ -71,6 +74,27 @@ and upgrade various systems.
Use the standard :command:`sw-patch`, :command:`upload`, :command:`apply`
and :command:`install` commands to perform these operations.
#. Confirm that the system is healthy.
Check the current system health status, resolve any alarms and other issues
reported by the :command:`system health-query-kube-upgrade` command then
recheck the system health status to confirm that all **System Health**
fields are set to *OK*.
.. code-block:: none
~(keystone_admin)]$ system health-query-kube-upgrade
System Health:
All hosts are provisioned: [OK]
All hosts are unlocked/enabled: [OK]
All hosts have current configurations: [OK]
All hosts are patch current: [OK]
Ceph Storage Healthy: [OK]
No alarms: [OK]
All kubernetes nodes are ready: [OK]
All kubernetes control plane pods are ready: [OK]
All kubernetes applications are in a valid state: [OK]
#. List the available Kubernetes versions.
On a fresh install of |prod| |prod-ver|, the following output appears, for
@@ -82,12 +106,10 @@ and upgrade various systems.
+---------+--------+-------------+
| version | target | state |
+---------+--------+-------------+
| v1.24.4 | False | unavailable |
| v1.25.3 | False | unavailable |
| v1.26.1 | False | unavailable |
| v1.27.5 | False | unavailable |
| v1.28.4 | False | unavailable |
| v1.29.2 | True | active |
| v1.29.2 | False | unavailable |
| v1.30.6 | False | unavailable |
| v1.31.5 | False | unavailable |
| v1.32.2 | True | active |
+---------+--------+-------------+
If |prod| was upgraded to |prod-ver|, the following appears:
@@ -98,14 +120,12 @@ and upgrade various systems.
+---------+--------+-------------+
| version | target | state |
+---------+--------+-------------+
| v1.24.4 | True | active |
| v1.25.3 | False | available |
| v1.26.1 | False | unavailable |
| v1.27.5 | False | unavailable |
| v1.28.4 | False | unavailable |
| v1.29.2 | False | unavailable |
| v1.29.2 | True | active |
| v1.30.6 | False | available |
| v1.31.5 | False | available |
| v1.32.2 | False | available |
+---------+--------+-------------+
The following meanings apply to the output shown:
**Target**
@@ -129,43 +149,18 @@ and upgrade various systems.
downgrade or it requires an intermediate upgrade first. Kubernetes
can be only upgraded one version at a time.
#. Confirm that the system is healthy.
Check the current system health status, resolve any alarms and other issues
reported by the :command:`system health-query-kube-upgrade` command then
recheck the system health status to confirm that all **System Health**
fields are set to *OK*.
.. code-block:: none
~(keystone_admin)]$ system health-query-kube-upgrade
System Health:
All hosts are provisioned: [OK]
All hosts are unlocked/enabled: [OK]
All hosts have current configurations: [OK]
All hosts are patch current: [OK]
Ceph Storage Healthy: [OK]
No alarms: [OK]
All kubernetes nodes are ready: [OK]
All kubernetes control plane pods are ready: [OK]
Required patches are applied: [OK]
License valid for upgrade: [OK]
No instances running on controller-1: [OK]
All kubernetes applications are in a valid state: [OK]
Active controller is controller-0: [OK]
#. Start the Kubernetes upgrade.
For example:
.. code-block:: none
~(keystone_admin)]$ system kube-upgrade-start v1.25.3
~(keystone_admin)]$ system kube-upgrade-start v1.32.2
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| from_version | v1.24.4 |
| to_version | v1.25.3 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | upgrade-started |
+--------------+--------------------------------------+
@@ -189,8 +184,8 @@ and upgrade various systems.
| Property | Value |
+--------------+--------------------------------------+
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.24.4 |
| to_version | v1.25.3 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | downloading-images |
| created_at | 2025-03-26T18:44:46.854319+00:00 |
| updated_at | None |
@@ -205,18 +200,35 @@ and upgrade various systems.
| Property | Value |
+--------------+--------------------------------------+
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.24.4 |
| to_version | v1.25.3 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | downloaded-images |
| created_at | 2025-03-26T18:44:46.854319+00:00 |
| updated_at | 2025-03-26T18:51:29.486448+00:00 |
+--------------+--------------------------------------+
#. Update all applications that have the ``timing: pre`` metadata setting.
#. Update all applications that require updating before performing the
Kubernetes version upgrade. This will update all applications that have the
``timing: pre`` metadata setting.
.. code-block:: none
~(keystone_admin)]$ system kube-pre-application-update
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | pre-updating-apps |
| created_at | 2025-03-26T18:44:46.854319+00:00 |
| updated_at | 2025-03-26T18:51:29.486448+00:00 |
+--------------+--------------------------------------+
.. note::
The state will change to ``pre-updated-apps`` when the app update has
completed.
#. Upgrade Kubernetes networking, for example:
@@ -227,8 +239,8 @@ and upgrade various systems.
| Property | Value |
+--------------+--------------------------------------+
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.24.4 |
| to_version | v1.25.3 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | upgrading-networking |
| created_at | 2025-03-26T18:44:46.854319+00:00 |
| updated_at | 2025-03-26T18:51:29.486448+00:00 |
@@ -246,8 +258,8 @@ and upgrade various systems.
| Property | Value |
+--------------+--------------------------------------+
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.24.4 |
| to_version | v1.25.3 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | upgrading-storage |
| created_at | 2025-03-13T18:44:46.854319+00:00 |
| updated_at | 2025-03-13T18:51:29.486448+00:00 |
@@ -256,7 +268,9 @@ and upgrade various systems.
The state ``upgraded-storage`` will be entered when the storage upgrade has
completed.
#. Upgrade the control plane on the first controller, for example:
.. note:: Repeat the following upgrade steps for each Kubernetes version (up to three minor versions) until you reach the target version or hit the version skew policy limit.
#. Upgrade the control plane on the first controller (controller-1), for example:
.. code-block:: none
@@ -264,21 +278,19 @@ and upgrade various systems.
+-----------------------+-------------------------+
| Property | Value |
+-----------------------+-------------------------+
| control_plane_version | v1.24.4 |
| control_plane_version | v1.29.2 |
| hostname | controller-1 |
| id | 2 |
| kubelet_version | v1.24.4 |
| kubelet_version | v1.29.2 |
| personality | controller |
| status | upgrading-control-plane |
| target_version | v1.25.3 |
| target_version | v1.32.2 |
+-----------------------+-------------------------+
You can upgrade either controller first.
The state ``upgraded-first-master`` will be entered when the first control
plane upgrade has completed.
#. Upgrade the control plane on the second controller, for example:
#. Upgrade the control plane on the second controller (controller-0), for example:
.. code-block:: none
@@ -286,18 +298,21 @@ and upgrade various systems.
+-----------------------+-------------------------+
| Property | Value |
+-----------------------+-------------------------+
| control_plane_version | v1.24.4 |
| control_plane_version | v1.29.2 |
| hostname | controller-0 |
| id | 1 |
| kubelet_version | v1.24.4 |
| kubelet_version | v1.29.2 |
| personality | controller |
| status | upgrading-control-plane |
| target_version | v1.25.3 |
| target_version | v1.32.2 |
+-----------------------+-------------------------+
The state ``upgraded-second-master`` will be entered when the upgrade has
completed.
.. note:: The upgrade order of control plane nodes (Controller 0 and Controller 1) is not critical; either node can be upgraded first. However, both must be upgraded to the target version before proceeding with the next version.
#. Show the Kubernetes upgrade status for all hosts, for example:
.. code-block:: none
@@ -306,13 +321,13 @@ and upgrade various systems.
+----+--------------+-------------+----------------+-----------------------+-----------------+--------+
| id | hostname | personality | target_version | control_plane_version | kubelet_version | status |
+----+--------------+-------------+----------------+-----------------------+-----------------+--------+
| 1 | controller-0 | controller | v1.25.3 | v1.25.3 | v1.24.4 | None |
| 2 | controller-1 | controller | v1.25.3 | v1.25.3 | v1.24.4 | None |
| 3 | worker-0 | worker | v1.24.4 | N/A | v1.24.4 | None |
| 4 | worker-1 | worker | v1.24.4 | N/A | v1.24.4 | None |
| 1 | controller-0 | controller | v1.32.2 | v1.32.2 | v1.29.2 | None |
| 2 | controller-1 | controller | v1.32.2 | v1.32.2 | v1.29.2 | None |
| 3 | worker-0 | worker | v1.29.2 | N/A | v1.29.2 | None |
| 4 | worker-1 | worker | v1.29.2 | N/A | v1.29.2 | None |
+----+--------------+-------------+----------------+-----------------------+-----------------+--------+
The control planes of both controllers are now upgraded to v1.25.3.
The control planes of both controllers are now upgraded to v1.32.2.
#. Upgrade kubelet on both controllers.
@@ -323,7 +338,7 @@ and upgrade various systems.
For each controller, do the following.
#. For non |AIO-SX| systems, lock the controller.
#. Lock the controller.
For example:
@@ -331,10 +346,6 @@ and upgrade various systems.
~(keystone_admin)]$ system host-lock controller-1
.. warning::
For All-In-One Simplex systems, the controller must **not** be
locked.
#. Apply the upgrade.
For example:
@@ -345,16 +356,32 @@ and upgrade various systems.
+-----------------------+-------------------+
| Property | Value |
+-----------------------+-------------------+
| control_plane_version | v1.25.3 |
| control_plane_version | v1.32.2 |
| hostname | controller-1 |
| id | 2 |
| kubelet_version | v1.24.4 |
| kubelet_version | v1.29.2 |
| personality | controller |
| status | upgrading-kubelet |
| target_version | v1.25.3 |
| target_version | v1.32.2 |
+-----------------------+-------------------+
#. For non |AIO-SX| systems, unlock the controller.
#. Confirm the kubelet upgrade status before proceeding.
For example:
.. code-block:: none
~(keystone_admin)]$ system kube-host-upgrade-list
+----+--------------+-------------+----------------+-----------------------+-----------------+--------------------+
| id | hostname | personality | target_version | control_plane_version | kubelet_version | status |
+----+--------------+-------------+----------------+-----------------------+-----------------+--------------------+
| 1 | controller-0 | controller | v1.32.2 | v1.32.2 | v1.29.2 | None |
| 2 | controller-1 | controller | v1.32.2 | v1.32.2 | v1.29.2 | upgraded-kubelet |
| 3 | worker-0 | worker | v1.29.2 | N/A | v1.29.2 | None |
| 4 | worker-1 | worker | v1.29.2 | N/A | v1.29.2 | None |
+----+--------------+-------------+----------------+-----------------------+-----------------+--------------------+
#. Unlock the controller.
For example:
@@ -362,6 +389,32 @@ and upgrade various systems.
~(keystone_admin)]$ system host-unlock controller-1
#. Switch the active controller.
.. code-block:: none
~(keystone_admin)]$ swact
#. Lock the controller-0.
.. code-block:: none
~(keystone_admin)]$ system host-lock controller-0
#. Apply the upgrade.
.. code-block:: none
~(keystone_admin)]$ system kube-host-upgrade controller-0 kubelet
#. Unlock the controller-0.
For example:
.. code-block:: none
~(keystone_admin)]$ system host-unlock controller-0
#. Show the Kubernetes upgrade status.
.. code-block:: none
@@ -371,20 +424,19 @@ and upgrade various systems.
| Property | Value |
+--------------+--------------------------------------+
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.24.4 |
| to_version | v1.25.3 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | upgrading-kubelets |
| created_at | 2025-03-26T18:44:46.854319+00:00 |
| updated_at | 2025-03-26T19:41:57.936617+00:00 |
+--------------+--------------------------------------+
#. Upgrade kubelet on all worker hosts.
#. Upgrade the kubelet on each worker node by repeating the following steps
for each worker host in the system:
Multiple worker hosts can be upgraded simultaneously provided there is
sufficient capacity remaining on other worker hosts.
For each worker host, do the following:
#. Lock the host.
For example:
@@ -403,13 +455,13 @@ and upgrade various systems.
+-----------------------+-------------------+
| Property | Value |
+-----------------------+-------------------+
| control_plane_version | v1.25.3 |
| control_plane_version | v1.32.2 |
| hostname | worker-1 |
| id | 3 |
| kubelet_version | v1.24.4 |
| kubelet_version | v1.29.2 |
| personality | worker |
| status | upgrading-kubelet |
| target_version | v1.25.3 |
| target_version | v1.32.2 |
+-----------------------+-------------------+
#. Unlock the host.
@@ -420,6 +472,8 @@ and upgrade various systems.
~(keystone_admin)]$ system host-unlock worker-1
.. note:: If your upgrade includes four or more Kubernetes versions, return to the "Upgrade the control plane" step after every three minor version upgrades. For example, if your target version is 1.32.2, and your starting version is more than three versions behind, you will need to repeat the control plane upgrade process to stay within the Kubernetes version skew policy.
#. Complete the Kubernetes upgrade.
.. code-block:: none
@@ -429,21 +483,23 @@ and upgrade various systems.
| Property | Value |
+--------------+--------------------------------------+
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.24.4 |
| to_version | v1.25.3 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | upgrade-complete |
| created_at | 2025-03-26T18:44:46.854319+00:00 |
| updated_at | 2025-03-26T19:06:47.515747+00:00 |
+--------------+--------------------------------------+
#. |Optional| Update all applications that have the ``timing: post`` metadata
setting.
#. Update all applications that require updating after the K8S Version
Upgrade. This will update all applications that have the ``timing: post``
metadata setting.
.. code-block:: none
~(keystone_admin)]$ system kube-post-application-update
#. Remove the alarm 900.007 (Kubernetes upgrade in progress) if it is still
#. Delete temporary resources associated with a Kubernetes upgrade
procedure. This step will also help clear Alarm 900.007 if it is still
running after the upgrade.
.. code-block:: none

352
doc/source/updates/kubernetes/manual-kubernetes-multi-version-upgrade-in-aio-sx-13e05ba19840.rst
View File

@@ -1,22 +1,12 @@
.. _manual-kubernetes-multi-version-upgrade-in-aio-sx-13e05ba19840:
=================================================
Manual Kubernetes Multi-Version Upgrade in AIO-SX
=================================================
===================================
Manual Kubernetes Upgrade in AIO-SX
===================================
You can upgrade the Kubernetes version on a running system from one supported
version to another.
|AIO-SX| now supports multi-version Kubernetes upgrades. In this model,
Kubernetes is upgraded by two or more versions after disabling applications and
then applications are enabled again. This is faster than upgrading Kubernetes
one version at a time. Also, the upgrade can be aborted and reverted to the
original version. This feature is supported only for |AIO-SX|.
|AIO-SX| supports the Kubernetes multi-version upgrade. Thus, Kubernetes can be
upgraded from the lowest version to the highest version available in the
system. This feature is not supported in the system which is not |AIO-SX|.
.. note::
Each |prod| release supports two or more consecutive Kubernetes releases.
@@ -25,8 +15,32 @@ system. This feature is not supported in the system which is not |AIO-SX|.
always start with the same Kubernetes version as the highest version from
the release you are upgrading from.
Kubernetes upgrades previously required sequential version updates, one at
a time. The upgrade system now checks for Kubernetes version skew, allowing
kubelet components to run up to three minor versions behind the control
plane. This enables multi-version upgrades in a single cycle, eliminating
the need to upgrade kubelet through each intermediate version. The system
marks all valid versions as available, simplifying version selection and
reducing the number of upgrades steps.
.. rubric:: |context|
To complete this task, you will apply the following updates (patches)
and upgrade various systems.
**Platform update**
The platform update contains metadata for the new Kubernetes version and the
Kubernetes networking pods templates for the new Kubernetes version.
**Kubernetes Component Packages**
Each supported Kubernetes version is packaged separately and includes the
corresponding versions of kubeadm, kubelet, and kubectl binaries. The system
uses Debian packages to manage Kubernetes components.
.. rubric:: |prereq|
.. _manual-kubernetes-components-upgrade-ul-jbr-vcn-ylb:
- The system must be clear of alarms.
- All hosts must be unlocked, enabled, and available.
@@ -49,54 +63,9 @@ system. This feature is not supported in the system which is not |AIO-SX|.
.. rubric:: |proc|
#. List the available Kubernetes versions, for example:
#. Upload, apply and install the platform update.
On a fresh install of |prod| |prod-ver|, the following output appears:
.. code-block:: none
~(keystone_admin)]$ system kube-version-list
+---------+--------+-------------+
| Version | Target | State |
+---------+--------+-------------+
| v1.21.8 | False | unavailable |
| v1.22.5 | False | unavailable |
| v1.23.1 | False | unavailable |
| v1.24.4 | True | active |
+---------+--------+-------------+
If |prod| was upgraded to |prod-ver|, the following output appears:
.. code-block:: none
~(keystone_admin)]$ system kube-version-list
+---------+--------+-------------+
| Version | Target | State |
+---------+--------+-------------+
| v1.21.8 | True | active |
| v1.22.5 | False | available |
| v1.23.1 | False | available |
| v1.24.4 | False | available |
+---------+--------+-------------+
The following meanings apply to the output shown:
**Target**
Target is either true or false. Target will be true only for the active
Kubernetes version.
**State**
State can be one of the following:
*active*: The version is running everywhere.
*partial*: The version is running somewhere.
*available*: The version can be upgraded.
*unavailable*: The version is not available for upgrading.
For more details, see the section :ref:`Updates and Upgrade section <index-updates-e3b970bb69ce>`.
#. Confirm that the system is healthy.
@@ -122,7 +91,56 @@ system. This feature is not supported in the system which is not |AIO-SX|.
All kubernetes applications are in a valid state: [OK]
Active controller is controller-0: [OK]
#. Start the Kubernetes multi-version upgrade.
#. List the available Kubernetes versions, for example:
On a fresh install of |prod| |prod-ver|, the following output appears:
.. code-block:: none
~(keystone_admin)]$ system kube-version-list
+---------+--------+-------------+
| Version | Target | State |
+---------+--------+-------------+
| v1.29.2 | False | unavailable |
| v1.30.6 | False | unavailable |
| v1.31.5 | False | unavailable |
| v1.32.2 | True | active |
+---------+--------+-------------+
If |prod| was upgraded to |prod-ver|, the following output appears:
.. code-block:: none
~(keystone_admin)]$ system kube-version-list
+---------+--------+-------------+
| Version | Target | State |
+---------+--------+-------------+
| v1.29.2 | True | active |
| v1.30.6 | False | available |
| v1.31.5 | False | available |
| v1.32.2 | False | available |
+---------+--------+-------------+
The following meanings apply to the output shown:
**Target**
Target is either true or false. Target will be true only for the active
Kubernetes version.
**State**
State can be one of the following:
*active*: The version is running everywhere.
*partial*: The version is running somewhere.
*available*: The version can be upgraded.
*unavailable*: The version is not available for upgrading.
#. Start the Kubernetes upgrade.
Specify the desired target version available to upgrade.
@@ -130,16 +148,19 @@ system. This feature is not supported in the system which is not |AIO-SX|.
.. code-block:: none
~(keystone_admin)]$ system kube-upgrade-start v1.24.4
~(keystone_admin)]$ system kube-upgrade-start v1.32.2
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| from_version | v1.21.8 |
| to_version | v1.24.4 |
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | upgrade-started |
| created_at | 2025-08-24T02:33:47.049826+00:00 |
| updated_at | None |
+--------------+--------------------------------------+
The upgrade process checks the applied/available updates, the upgrade
The upgrade process checks the *applied*/*available* updates, the upgrade
path, the system health, the installed applications compatibility, and
validates that the system is ready for an upgrade.
@@ -158,11 +179,11 @@ system. This feature is not supported in the system which is not |AIO-SX|.
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8 |
| to_version | v1.24.4 |
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | downloading-images |
| created_at | 2023-08-24T02:33:47.049826+00:00 |
| created_at | 2025-08-24T02:33:47.049826+00:00 |
| updated_at | None |
+--------------+--------------------------------------+
@@ -174,19 +195,36 @@ system. This feature is not supported in the system which is not |AIO-SX|.
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8 |
| to_version | v1.24.4 |
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | downloaded-images |
| created_at | 2023-08-24T02:33:47.049826+00:00 |
| updated_at | 2023-08-24T02:38:16.374677+00:00 |
| created_at | 2025-08-24T02:33:47.049826+00:00 |
| updated_at | 2025-08-24T02:38:16.374677+00:00 |
+--------------+--------------------------------------+
#. Update all applications that have the ``timing: pre`` metadata setting.
#. Update all applications that require updating before performing the
Kubernetes version upgrade. This will update all applications that have the
``timing: pre`` metadata setting.
.. code-block:: none
~(keystone_admin)]$ system kube-pre-application-update
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | pre-updating-apps |
| created_at | 2025-03-26T18:44:46.854319+00:00 |
| updated_at | 2025-03-26T18:51:29.486448+00:00 |
+--------------+--------------------------------------+
.. note::
The state will change to ``pre-updated-apps`` when the app update has
completed.
#. Upgrade Kubernetes networking, for example:
@@ -199,11 +237,11 @@ system. This feature is not supported in the system which is not |AIO-SX|.
| Property | Value |
+--------------+--------------------------------------+
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.21.8 |
| to_version | v1.24.4 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | upgrading-networking |
| created_at | 2023-08-24T02:33:47.049826+00:00 |
| updated_at | 2023-08-24T02:38:16.374677+00:00 |
| created_at | 2025-08-24T02:33:47.049826+00:00 |
| updated_at | 2025-08-24T02:38:16.374677+00:00 |
+--------------+--------------------------------------+
The state **upgraded-networking** will be entered when the networking
@@ -215,17 +253,50 @@ system. This feature is not supported in the system which is not |AIO-SX|.
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8 |
| to_version | v1.24.4 |
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | upgraded-networking |
| created_at | 2023-08-24T02:33:47.049826+00:00 |
| updated_at | 2023-08-24T02:42:40.543522+00:00 |
| created_at | 2025-08-24T02:33:47.049826+00:00 |
| updated_at | 2025-08-24T02:42:40.543522+00:00 |
+--------------+--------------------------------------+
#. |optional| Cordon
#. Update the Kubernetes storage components to align with the target version.
This ensures compatibility and stability for persistent volumes and
storage-related services throughout the upgrade process.
The :command:`kube-host-cordon` command will evict the regular pods from
.. code-block:: none
~(keystone_admin)]$ system kube-upgrade-storage
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | upgrading-storage |
| created_at | 2025-08-24T02:33:47.049826+00:00 |
| updated_at | 2025-08-24T02:42:40.543522+00:00 |
+--------------+--------------------------------------+
The state **upgraded-storage** will be entered when the storage upgrade has
completed.
.. code-block:: none
~(keystone_admin)]$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | upgraded-storage |
| created_at | 2025-08-24T02:33:47.049826+00:00 |
| updated_at | 2025-08-24T02:42:40.543522+00:00 |
+--------------+--------------------------------------+
#. Run the :command:`kube-host-cordon` command to evict the regular pods from
the host. This command will prevent the application from running on
intermediate versions.
@@ -239,12 +310,12 @@ system. This feature is not supported in the system which is not |AIO-SX|.
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8 |
| to_version | v1.24.4 |
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | cordon-started |
| created_at | 2023-08-24T02:45:32.257231+00:00 |
| updated_at | 2023-08-24T02:45:32.257231+00:00 |
| created_at | 2025-08-24T02:45:32.257231+00:00 |
| updated_at | 2025-08-24T02:45:32.257231+00:00 |
+--------------+--------------------------------------+
.. code-block:: none
@@ -253,17 +324,19 @@ system. This feature is not supported in the system which is not |AIO-SX|.
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8 |
| to_version | v1.24.4 |
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | cordon-complete |
| created_at | 2023-08-24T02:45:32.257231+00:00 |
| updated_at | 2023-08-24T11:47:56.178266+00:00 |
| created_at | 2025-08-24T02:45:32.257231+00:00 |
| updated_at | 2025-08-24T11:47:56.178266+00:00 |
+--------------+--------------------------------------+
The state **cordon-complete** will be entered when the host cordon has
completed.
.. note:: Repeat the following upgrade steps for each Kubernetes version (up to three minor versions) until you reach the target version or hit the version skew policy limit.
#. Upgrade the control plane on controller-0.
.. _upgrade_control_plane:
@@ -274,13 +347,13 @@ system. This feature is not supported in the system which is not |AIO-SX|.
+-----------------------+-------------------------+
| Property | Value |
+-----------------------+-------------------------+
| control_plane_version | v1.21.8 |
| control_plane_version | v1.29.2 |
| hostname | controller-0 |
| id | 1 |
| kubelet_version | v1.21.8 |
| kubelet_version | v1.29.2 |
| personality | controller |
| status | upgrading-control-plane |
| target_version | v1.24.4 |
| target_version | v1.32.2 |
+-----------------------+-------------------------+
Check if the control plane version upgrade status is changed to *None*.
@@ -293,7 +366,7 @@ system. This feature is not supported in the system which is not |AIO-SX|.
+----+---------------+------------+----------------+-----------------------+-----------------+--------------+
| id | hostname | personality| target_version | control_plane_version | kubelet_version | status |
+----+---------------+------------+----------------+-----------------------+-----------------+--------------+
| 1 | controller-0 | controller | v1.22.5 | v1.22.5 | v1.21.8 | None |
| 1 | controller-0 | controller | v1.30.6 | v1.30.6 | v1.29.2 | None |
+----+---------------+---+--------+----------------+-----------------------+-----------------+--------------+
#. Upgrade kubelet on controller-0.
@@ -306,13 +379,13 @@ system. This feature is not supported in the system which is not |AIO-SX|.
+-----------------------+-------------------------+
| Property | Value |
+-----------------------+-------------------------+
| control_plane_version | v1.22.5 |
| control_plane_version | v1.30.6 |
| hostname | controller-0 |
| id | 1 |
| kubelet_version | v1.21.8 |
| kubelet_version | v1.29.2 |
| personality | controller |
| status | upgrading-kubelet |
| target_version | v1.22.5 |
| target_version | v1.30.6 |
+-----------------------+-------------------------+
Check the status of the kubelet upgrade.
@@ -323,23 +396,16 @@ system. This feature is not supported in the system which is not |AIO-SX|.
+----+---------------+------------+----------------+-----------------------+-----------------+------------------+
| id | hostname | personality| target_version | control_plane_version | kubelet_version | status |
+----+---------------+------------+----------------+-----------------------+-----------------+------------------+
| 1 | controller-0 | controller | v1.22.5 | v1.22.5 | v1.22.5 | upgraded-kubelet |
| 1 | controller-0 | controller | v1.30.6 | v1.30.6 | v1.30.6 | upgraded-kubelet |
+----+---------------+---+--------+----------------+-----------------------+-----------------+------------------+
The status **upgraded-kubelet** will be entered when the kubelet upgrade
has completed.
Repeat steps :ref:`Upgrade control plane <upgrade_control_plane>` and
:ref:`Upgrade Kubelet <upgrade_kubelet>` to reach the target Kubernetes
version. For example, in this case, repeat steps
:ref:`Upgrade control plane <upgrade_control_plane>` and
:ref:`Upgrade Kubelet <upgrade_kubelet>` twice for the remaining versions
v1.23.1 and v1.24.4.
Repeat the upgrade steps from :ref:`Upgrade control plane <upgrade_control_plane>` for each version in the sequence until you reach the target Kubernetes version. This approach keeps the upgrade within the Kubernetes version skew policy, which supports up to three minor version upgrades at a time.
#. |optional| Run the ``uncordon`` subcommand.
Skip this step if you did not perform step :ref:`Upgrade Kubernetes Networking <upgrade_kubernetes_networking>`.
The :command:`kube-host-uncordon` command will allow the regular pods on the
host again.
@@ -349,28 +415,28 @@ system. This feature is not supported in the system which is not |AIO-SX|.
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8 |
| to_version | v1.24.4 |
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | uncordon-started |
| created_at | 2023-08-24T11:56:56.178266+00:00 |
| updated_at | 2023-08-24T11:56:56.178266+00:00 |
| created_at | 2025-08-24T11:56:56.178266+00:00 |
| updated_at | 2025-08-24T11:56:56.178266+00:00 |
+--------------+--------------------------------------+
~(keystone_admin)]$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8 |
| to_version | v1.24.4 |
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | uncordon-complete |
| created_at | 2023-08-24T11:56:56.178266+00:00 |
| updated_at | 2023-08-24T11:58:35.136866+00:00 |
| created_at | 2025-08-24T11:56:56.178266+00:00 |
| updated_at | 2025-08-24T11:58:35.136866+00:00 |
+--------------+--------------------------------------+
The state **uncordon-complete** will be entered when the host uncordon has
completed.
The state **uncordon-complete** will be entered when the host uncordon has
completed.
#. Complete the Kubernetes upgrade.
@@ -380,22 +446,24 @@ system. This feature is not supported in the system which is not |AIO-SX|.
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8 |
| to_version | v1.24.4 |
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | upgrade-complete |
| created_at | 2023-08-24T02:33:47.049826+00:00 |
| updated_at | 2023-08-24T02:55:18.122620+00:00 |
| created_at | 2025-08-24T02:33:47.049826+00:00 |
| updated_at | 2025-08-24T02:55:18.122620+00:00 |
+--------------+--------------------------------------+
#. |Optional| Update all applications that have the ``timing: post`` metadata
setting.
#. |Optional| Update all applications that require updating after the K8S
Version Upgrade. This step will update all applications that have the
``timing: post`` metadata setting.
.. code-block:: none
~(keystone_admin)]$ system kube-post-application-update
#. Remove the alarm 900.007 (Kubernetes upgrade in progress) if it is still
#. Delete the temporary resources associated with a Kubernetes upgrade
procedure. This step also help clear Alarm 900.007 if it is still
running after the upgrade.
.. code-block:: none
@@ -415,12 +483,12 @@ the following command:
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8 |
| to_version | v1.22.4 |
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | upgrade-aborting |
| created_at | 2023-06-26T18:44:46.854319+00:00 |
| updated_at | 2023-08-24T02:55:18.122620+00:00 |
| created_at | 2025-06-26T18:44:46.854319+00:00 |
| updated_at | 2025-08-24T02:55:18.122620+00:00 |
+--------------+--------------------------------------+
To check the status of the abort operation, run the following command:
@@ -431,12 +499,12 @@ To check the status of the abort operation, run the following command:
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 065e683a-13a3-4229-b3c7-701f90216a3d |
| from_version | v1.21.8 |
| to_version | v1.24.4 |
| uuid | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2 |
| to_version | v1.32.2 |
| state | upgrade-aborted |
| created_at | 2023-08-24T07:10:02.578787+00:00 |
| updated_at | 2023-08-24T07:24:00.429794+00:00 |
| created_at | 2025-08-24T07:10:02.578787+00:00 |
| updated_at | 2025-08-24T07:24:00.429794+00:00 |
+--------------+--------------------------------------+
.. note::

42
doc/source/updates/kubernetes/the-kubernetes-update-orchestration-process.rst
View File

@@ -2,15 +2,17 @@
.. htb1590431033292
.. _the-kubernetes-update-orchestration-process:
=======================================================
Kubernetes Version Upgrade Cloud Orchestration Overview
=======================================================
===============================================
Kubernetes Upgrade Cloud Orchestration Overview
===============================================
For an orchestrated Kubernetes version upgrade you need to first create a
.. Adding this file to the review. Please review this section, as the content may need to be updated to reflect recent changes that support multi-node upgrades in a single orchestration.
For an orchestrated Kubernetes upgrade you need to first create a
*Kubernetes Upgrade Orchestration Strategy*, or plan for the automated
Kubernetes version upgrade procedure.
You can customize the Kubernetes version upgrade orchestration by specifying
You can customize the Kubernetes upgrade orchestration by specifying
the following parameters:
@@ -39,9 +41,9 @@ You must use the :command:`sw-manager`` CLI tool to create, and then apply the
upgrade strategy. A created strategy can be monitored with the **show** command.
Kubernetes version upgrade orchestration automatically iterates through all
*unlocked-enabled* hosts on the system looking for hosts with the worker
function that need Kubernetes version upgrades and then proceeds to upgrade them
on the strategy :command:`apply` action.
*unlocked-enabled* hosts on the system looking for hosts that need Kubernetes
version upgrades and then proceeds to upgrade them on the strategy
:command:`apply` action.
.. note::
Controllers (including |AIO| controllers) are upgraded before worker-only
@@ -53,19 +55,17 @@ either apply the entire strategy automatically, or manually apply individual
stages to control and monitor the Kubernetes version upgrade progress one stage
at a time.
When the Kubernetes version upgrade strategy is applied, if the system is
All-in-one, the controllers are upgraded first, one after the other with a
swact in between, followed by the remaining worker hosts according to the
selected worker apply concurrency (**serial** or **parallel**) method.
When the Kubernetes version upgrade strategy is applied, the controllers are
upgraded first, one after the other with a swact in between, followed by the
remaining worker hosts according to the selected worker apply concurrency
(**serial** or **parallel**) method.
By default, strategies upgrade the worker hosts serially unless the **parallel**
worker apply type option is specified, which configures the Kubernetes version
upgrade process for worker hosts to be in parallel (up to a maximum parallel
number). This reduces the overall Kubernetes version upgrade installation time.
The upgrade takes place in two phases. The first phase upgrades the patches
(controllers, storage and then workers), and the second phase upgrades
Kubernetes based on those patches (controllers, then hosts).
The upgrade takes place in the following sequence:
.. _htb1590431033292-ol-a1b-v5s-tlb:
@@ -75,11 +75,13 @@ Kubernetes based on those patches (controllers, then hosts).
#. Download Kubernetes Images.
#. Upgrade the first Control Plane.
#. Upgrade Kubernetes networking.
#. Upgrade the second control plane (on duplex environments only).
#. Upgrade Kubernetes storage.
#. Upgrade the first Control Plane.
#. Upgrade the second control plane (on multi-node environments).
#. Patch the hosts.
@@ -94,11 +96,11 @@ Kubernetes based on those patches (controllers, then hosts).
#. Swact if the host is the active controller.
#. Lock the host.
#. Lock the host, or cordon the host, in the case of AIO-SX.
#. Upgrade kubelet.
#. Unlock the host.
#. Unlock the host, or uncordon the host, in the case of AIO-SX.
#. Restore |VMs| (if applicable).