starlingx/docs
Code Issues Proposed changes

Platform k8s upgrades

Browse Source 
Initial draft based on downstream content.
Implemented patchset 1 review comments.
Implemented patchset 2 rewiew comments.
Implemented patchset 3 rewiew comments.
Implemented patchset 4 rewiew comments.
Implemented patchset 5 rewiew comments.
Implemented patchset 6 rewiew comments.

Story: 2008055
Task: 42401

Signed-off-by: Ron Stone <ronald.stone@windriver.com>
Change-Id: I6262018778fae44726985853ec4e01f1abf5b890
Signed-off-by: Ron Stone <ronald.stone@windriver.com>
This commit is contained in:
Ron Stone 2021-04-29 12:41:33 -04:00
parent a4da81318e
commit c782df8892
10 changed files with 1144 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

0
doc/source/_includes/configuring-kubernetes-update-orchestration.rest Normal file
View File

0
doc/source/_includes/handling-kubernetes-update-orchestration-failures.rest Normal file
View File

2
doc/source/shared/abbrevs.txt
View File

@ -7,6 +7,8 @@
.. |ACL| replace:: :abbr:`ACL (Access Control List)`
.. |AE| replace:: :abbr:`AE (Aggregated Ethernet)`
.. |AIO| replace:: :abbr:`AIO (All-In-One)`
.. |AIO-DX| replace:: :abbr:`AIO-DX (All-In-One Duplex)`
.. |AIO-SX| replace:: :abbr:`AIO-SX (All-In-One Simplex)`
.. |AVP| replace:: :abbr:`AVP (Accelerated Virtual Port)`
.. |AVPs| replace:: :abbr:`AVPs (Accelerated Virtual Ports)`
.. |AWS| replace:: :abbr:`AWS (Amazon Web Services)`

25
doc/source/updates/index.rst
View File

@ -1,11 +1,26 @@
=======
Updates
=======
====================
Updates and Upgrades
====================
----------
Openstack
Kubernetes
----------
.. check what put here
Kubernetes version upgrade cloud orchestration allows the Kubernetes version on
all hosts of an entire |prod-long| cloud to be updated with a single operation.
.. toctree::
:maxdepth: 2
kubernetes/index
---------
Openstack
---------
The system application-update -n |prefix|-openstack -v <new-app-version>
command is used for corrective content \(bug fixes\) -type updates to the
running containerized openstack application.
.. toctree::
:maxdepth: 2

51
doc/source/updates/kubernetes/about-kubernetes-orchestrated-upgrades.rst Normal file
View File

@ -0,0 +1,51 @@
.. xkr1590157116928
.. _about-kubernetes-orchestrated-upgrades:
====================================================
About Kubernetes Version Upgrade Cloud Orchestration
====================================================
Kubernetes version upgrade cloud orchestration allows the Kubernetes version on
all hosts of an entire |prod-long| cloud to be updated with a single operation.
You can configure and run Kubernetes version upgrade cloud orchestration using
the CLI, or the stx-nfv VIM REST API.
.. _xkr1590157116928-section-phk-xls-tlb:
-----------------------------------------------------------
Kubernetes Version Upgrade Cloud Orchestration Requirements
-----------------------------------------------------------
Kubernetes version upgrade orchestration can only be done on a system that
meets the following conditions:
.. _xkr1590157116928-ul-frz-yls-tlb:
- The system is clear of alarms \(with the exception of alarms for locked
hosts, stopped instances, and Kubernetes version upgrade cloud
orchestration in progress\).
.. note::
When configuring Kubernetes version upgrade cloud orchestration, you
have the option to ignore alarms that are not management-affecting
severity. For more information, see :ref:`Kubernetes Version Upgrade
Cloud Orchestration <configuring-kubernetes-update-orchestration>`.
- There are unlocked-enabled worker function hosts in the system that
requires Kubernetes Version Upgrade Cloud Orchestration. The *Kubernetes
Upgrade Orchestration Strategy* creation step will fail if there are no
qualified hosts detected.
A Kubernetes version upgrade is a reboot required operation. Therefore, in
systems that have the |prefix|-openstack application applied with running
instances, if the migrate option is selected there must be spare
openstack-compute \(worker\) capacity to move instances off the
openstack-compute \(worker\) host\(s\) being upgraded.
.. note::
Administrative controller ``swact`` operations should be avoided during
Kubernetes version upgrade orchestration.

354
doc/source/updates/kubernetes/configuring-kubernetes-update-orchestration.rst Normal file
View File

@ -0,0 +1,354 @@
.. noc1590162360081
.. _configuring-kubernetes-update-orchestration:
==============================================
Kubernetes Version Upgrade Cloud Orchestration
==============================================
You can configure *Kubernetes Version Upgrade Orchestration Strategy* using the
:command:`sw-manager` CLI.
.. note::
You require administrator privileges to use :command:`sw-manager`. You must
log in to the active controller as **user sysadmin** and source the script
by using the command, source /etc/platform/openrc to obtain administrator
privileges. Do not use :command:`sudo`.
.. note::
Management-affecting alarms cannot be ignored using relaxed alarm rules
during an orchestrated Kubernetes version upgrade operation. For a list of
management-affecting alarms, see |fault-doc|: :ref:`Alarm Messages
<100-series-alarm-messages>`. To display management-affecting active
alarms, use the following command:
.. code-block:: none
~(keystone_admin)$ fm alarm-list --mgmt_affecting
During an orchestrated Kubernetes version upgrade operation, the following
alarms are ignored even when the default strict restrictions are selected:
.. _noc1590162360081-ul-vhg-jxs-tlb:
- 100.103: Memory threshold exceeded.
- 200.001: Locked host.
- 280.001: Subcloud resource off-line.
- 280.002: Subcloud resource out-of-sync.
- 700.004: |VM| stopped.
- 750.006: Configuration change requires reapply of cert-manager.
- 900.001: Patch in progress.
- 900.007: Kube upgrade in progress.
- 900.401: kube-upgrade-auto-apply-inprogress.
You can use ``help`` for the overall commands and also for each sub-command.
For example:
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy –help
usage: sw-manager kube-upgrade-strategy [-h] ...
optional arguments:
-h, --help show this help message and exit
Kubernetes Update Commands:
create Create a strategy
delete Delete a strategy
apply Apply a strategy
abort Abort a strategy
show Show a strategy
.. rubric:: |prereq|
.. _noc1590162360081-ul-ls2-pxs-tlb:
- Hosts that need to be upgraded must be in the ``unlocked-enabled`` state.
.. only:: partner
.. include:: ../../_includes/configuring-kubernetes-update-orchestration.rest
.. rubric:: |proc|
#. List available upgrades.
.. code-block: none
~(keystone_admin)$ system kube-version-list
+-----------------+--------+-----------+
| version | target | state |
+-----------------+--------+-----------+
| v1.18.1 | True | active |
| v1.18.1-upgrade | False | available |
+-----------------+--------+-----------+
#. Create the strategy.
The *Kubernetes Version Upgrade Orchestration Strategy* :command:`create`
command creates a series of stages with steps that apply the Kubernetes
version upgrade.
Kubernetes Version upgrade requires a reboot. Therefore, the created strategy
includes steps that automatically lock and unlock the host to bring the new
image function into service.
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy create --to-version v1.18.1-upgrade
Strategy Kubernetes Upgrade Strategy:
strategy-uuid: f7585178-cea6-4d2f-bda0-e0972145ebcf
controller-apply-type: serial
storage-apply-type: ignore
worker-apply-type: serial
default-instance-action: migrate
alarm-restrictions: strict
current-phase: build
current-phase-completion: 0%
state: building
inprogress: true
where:
``--to-version``
The version of Kubernetes to upgrade to. For example,
``v1.18.1-upgrade``. This argument is required.
``--controller-apply-type`` and ``--storage-apply-type``
These options cannot be changed from ``serial`` because Kubernetes
upgrade concurrency is only supported for worker hosts.
.. note::
Kubernetes version upgrade is currently only supported for hosts with
worker function. Any attempt to modify the controller or storage
apply type is rejected.
``--worker-apply-type``
This option specifies the host concurrency of the Kubernetes version
upgrade strategy:
- serial \(default\): worker hosts will be patched one at a time
- parallel: worker hosts will be upgraded in parallel
- At most, ``parallel`` will be upgraded at the same time
- At most, half of the hosts in a host aggregate will be upgraded
at the same time
- ignore: worker hosts will not be upgraded; strategy create will fail
Worker hosts with no instances are upgraded before worker hosts with
instances.
``--max-parallel-worker-hosts``
This option applies to the parallel worker apply type selection to
specify the maximum worker hosts to upgrade in parallel \(minimum: 2,
maximum: 10\).
``–instance-action``
This option only has significance when the |prefix|-openstack
application is loaded and there are instances running on worker hosts.
It specifies how the strategy deals with worker host instances over the
strategy execution.
``stop-start`` \(default\)
Instances will be stopped before the host lock operation following the
upgrade and then started again following the host unlock.
.. warning::
Using the ``stop-start`` option will result in an outage for each
instance, as it is stopped while the worker host is locked/unlocked.
In order to ensure this does not impact service, instances MUST be
grouped into anti-affinity \(or anti-affinity best effort\) server
groups, which will ensure that only a single instance in each server
group is stopped at a time.
``migrate``
Instances will be migrated off a host before it is patched \(this
applies to reboot patching only\).
``--alarm-restrictions``
This option sets how the how the Kubernetes version upgrade
orchestration behaves when alarms are present.
To display management-affecting active alarms, use the following
command:
.. code-block:: none
~(keystone_admin)$ fm alarm-list --mgmt_affecting
``strict`` \(default\)
The default strict option will result in patch orchestration failing if
there are any alarms present in the system \(except for a small list of
alarms\).
``relaxed``
This option allows orchestration to proceed if alarms are present, as
long as none of these alarms are management affecting.
.. code-block:: none
~(keystone_admin)]$ sw-manager kube-upgrade-strategy create --help
usage:sw-manager kube-upgrade-strategy [-h]
--to-version <kubernetesVersion>
[--controller-apply-type {ignore}]
[--storage-apply-type {ignore}]
[--worker-apply-type
{serial,parallel,ignore}]
[--max-parallel-worker-hosts
{2,3,4,5,6,7,8,9,10}]
[--instance-action {migrate,stop-start}]
[--alarm-restrictions {strict,relaxed}]
optional arguments:
-h, --help show this help message and exit
--controller-apply-type {serial,ignore}
defaults to serial
--storage-apply-type {serial,ignore}
defaults to serial
--worker-apply-type {serial,parallel,ignore}
defaults to serial
--max-parallel-worker-hosts {2,3,4,5,6,7,8,9,10}
maximum worker hosts to update in parallel
--instance-action {migrate,stop-start}
defaults to stop-start
--alarm-restrictions {strict,relaxed}
defaults to strict
#. Optional: Display the strategy in summary, if required. The Kubernetes
upgrade strategy :command:`show` command displays the strategy in a summary.
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy show
Strategy Kubernetes Upgrade Strategy:
strategy-uuid: f7585178-cea6-4d2f-bda0-e0972145ebcf
controller-apply-type: serial
storage-apply-type: ignore
worker-apply-type: serial
default-instance-action: migrate
alarm-restrictions: strict
current-phase: build
current-phase-completion: 100%
state: ready-to-apply
build-result: success
build-reason:
The :command:`show` strategy subcommand displays a summary of the current
state of the strategy. A complete view of the strategy can be shown using
the ``--details`` option.
The strategy steps and stages are displayed using the ``--details`` option.
#. Apply the strategy.
*Kubernetes Version Upgrade Orchestration Strategy* :command:`apply` command
executes the strategy stages and steps consecutively until the Kubernetes
upgrade on all the hosts in the strategy is complete.
- Use the ``-stage-id`` option to specify a specific stage to apply; one
at a time.
.. note::
When applying a single stage, only the next stage will be applied;
you cannot skip stages.
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy apply
Strategy Kubernetes upgrade Strategy:
strategy-uuid: 3e43c018-9c75-4ba8-a276-472c3bcbb268
controller-apply-type: ignore
storage-apply-type: ignore
worker-apply-type: serial
default-instance-action: stop-start
alarm-restrictions: strict
current-phase: apply
current-phase-completion: 0%
state: applying
inprogress: true
- Use the :command:`kube-upgrade-show` command to monitor Kubernetes
upgrade state and percentage completion.
.. code-block:: none
~(keystone_admin)$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 3d2da123-bff4-4b3a-a64a-b320c3b498cc |
| from_version | v1.18.1 |
| to_version | v1.18.1-upgrade |
| state | downloading-images |
| created_at | 2021-02-23T00:08:24.579257+00:00 |
| updated_at | 2021-02-23T00:09:35.413307+00:00 |
+--------------+--------------------------------------+
You will see the ``state`` property transition through values such as
``downloading-images``, ``downloaded-images``, ``upgrading-first-master``,
``upgraded-first-master``, etc.
#. Optional: Abort the strategy, if required. This is only used to stop, and
abort the entire strategy.
The Kubernetes version upgrade strategy :command:`abort` command can be
used to abort the Kubernetes version upgrade strategy after the current
step of the currently applying stage is completed.
#. Confirm that the upgrade has completed successfully.
.. code-block:: none
~(keystone_admin)$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 426d7e11-2de2-40ba-b482-ed3691625383 |
| from_version | v1.18.1 |
| to_version | v1.18.1-upgrade |
| state | upgrade-complete |
| created_at | 2021-04-12T17:58:36.492523+00:00 |
| updated_at | 2021-04-12T18:49:11.673259+00:00 |
+--------------+--------------------------------------+
~(keystone_admin)$ system kube-version-list
+-----------------+--------+-----------+
| version | target | state |
+-----------------+--------+-----------+
| v1.18.1 | False | available |
| v1.18.1-upgrade | True | active |
+-----------------+--------+-----------+
#. Delete the strategy.
.. note::
After the *Kubernetes Version Upgrade Orchestration Strategy* has been
applied \(or aborted\) it must be deleted before another Kubernetes
version upgrade strategy can be created. If a Kubernetes version
upgrade strategy application fails, you must address the issue that
caused the failure, then delete and re-create the strategy before
attempting to apply it again.
.. code-block:: none
~(keystone_admin)$ sw-manager kube-upgrade-strategy delete
Strategy deleted.

125
doc/source/updates/kubernetes/handling-kubernetes-update-orchestration-failures.rst Normal file
View File

@ -0,0 +1,125 @@
.. jkf1590184623714
.. _handling-kubernetes-update-orchestration-failures:
========================================================
Handle Kubernetes Version Upgrade Orchestration Failures
========================================================
The creation or application of a strategy could fail for any of the listed
reasons described in this section. Follow the suggested actions in each case to
resolve the issue.
.. _jkf1590184623714-section-fhk-nnq-5lb:
-------------------------
Strategy creation failure
-------------------------
.. _jkf1590184623714-ul-fvs-vnq-5lb:
- **Reason**: build failed with no reason.
- **Action**:
- Verify that the ``--worker-apply-type`` was not set to ``ignore``.
- Check recent logs added to /var/log/nfv-vim.log.
- **Reason**: alarms from platform are present.
- **Action**:
- Query for management affecting alarms and take actions to clear
them.
.. code-block:: none
~(keystone_admin)$ fm alarm-list --mgmt_affecting
- If there are no management affecting alarms present, take actions
to clear other reported alarms or try creating the strategy with
the ``relaxed`` alarms restrictions option ``--alarm-restrictions
relaxed``.
- **Reason**: no Kubernetes version upgrade required.
- **Action**:
- Verify that the Kubernetes patches have been uploaded and applied.
Verify the version of Kubernetes on the hosts by executing "system
kube-host-upgrade-list.
.. note::
If the strategy create failed, first you must resolve it. You
must delete the failed strategy before you create another
strategy.
.. _jkf1590184623714-section-ppt-gpq-5lb:
----------------------
Strategy Apply Failure
----------------------
.. _jkf1590184623714-ul-rdf-4pq-5lb:
- **Reason**: alarms from platform are present.
- **Action**: suggests that an alarm has been raised since the creation
of the strategy. Address the cause of the new alarm, delete the
strategy and try creating and applying a new strategy.
- **Reason**: unable to migrate instances.
- **Action**: See :ref:`Kubernetes Version Upgrade Operations Requiring
Manual Migration
<kubernetes-update-operations-requiring-manual-migration>` for steps to
resolve migration issues.
- **Reason**: Kubernetes version upgrade failed. Suggests that the Kubernetes
upgrade for the specified host has failed.
.. only:: starlingx
- **Action**: Consult the `StarlingX community
<https://www.starlingx.io/community/>`_.
.. only:: partner
.. include:: ../../_includes/handling-kubernetes-update-orchestration-failures.rest
- **Reason**: lock host failed.
- **Action**:
- Investigate the /var/log/sysinv.log, and /var/log/nfv-vim.log
files.
- Address the underlying issue.
- Manually lock and unlock the host.
- Try recreating and re-applying the Kubernetes version upgrade
strategy to automatically finish the upgrade process.
- **Reason**: unlock host failed.
- **Action**:
- Investigate /var/log/mtcAgent.log file for cause logs files.
- Address the underlying issue.
- Manually lock and unlock the host to recover.
- Try recreating and re-applying the Kubernetes version upgrade
strategy to automatically finish the upgrade process.
.. note::
If the strategy :command:`apply` fails, you must resolve the
strategy:command:`apply` failure, and delete the failed strategy before
trying to create and apply another strategy.

29
doc/source/updates/kubernetes/index.rst Normal file
View File

@ -0,0 +1,29 @@
==========
Kubernetes
==========
---------------------------------
Manual Kubernetes Version Upgrade
---------------------------------
.. toctree::
:maxdepth: 1
:caption: Contents:
manual-kubernetes-components-upgrade
----------------------------------------------
Kubernetes Version Upgrade Cloud Orchestration
----------------------------------------------
.. toctree::
:maxdepth: 1
:caption: Contents:
about-kubernetes-orchestrated-upgrades
the-kubernetes-update-orchestration-process
configuring-kubernetes-update-orchestration
handling-kubernetes-update-orchestration-failures

381
doc/source/updates/kubernetes/manual-kubernetes-components-upgrade.rst Normal file
View File

@ -0,0 +1,381 @@
.. bfd1591638638205
.. _manual-kubernetes-components-upgrade:
===========================
Kubernetes Version Upgrade
===========================
You can upgrade the Kubernetes version on a running system from one
supported version to another.
.. rubric:: |context|
To complete this task, you will apply the following three updates \(patches\)
and upgrade various systems.
**Platform update**
The platform update contains metadata for the new Kubernetes version and the
Kubernetes networking pods templates for the new Kubernetes version.
**Kubeadm update**
The kubeadm update upgrades the kubeadm RPM to the new Kubernetes version.
**Kubelet and Kubectl update**
This Kubernetes update upgrades kubelet and kubectl RPMs to the new
Kubernetes version.
.. rubric:: |prereq|
.. _manual-kubernetes-components-upgrade-ul-jbr-vcn-ylb:
- The system must be clear of alarms.
- All hosts must be unlocked, enabled and available.
- All Kubernetes pods must be ready.
- The installed applications must be compatible with the new Kubernetes
version.
- All updates required for the new Kubernetes version must be transferred to
the active controller.
.. rubric:: |proc|
#. Upload, apply and install the platform update.
Use the standard :command:`sw-patch`, :command:`upload`, :command:`apply`
and :command:`install` commands to perform these operations.
#. List the available Kubernetes versions.
.. code-block:: none
~(keystone_admin)]$ system kube-version-list
+---------+--------+-----------+
| Version | Target | State |
+---------+--------+-----------+
| v1.16.1 | True | active |
| v1.16.2 | False | available |
+---------+--------+-----------+
The following meanings apply to the output shown:
**Target**
A value of True means that the target is currently selected for
installation.
**State**
Can be one of:
**active**
The version is running everywhere.
**partial**
The version is running somewhere.
**available**
The version can be upgraded to.
#. Upload, apply and install the kubeadm update.
Use the standard :command:`sw-patch`, :command:`upload`, :command:`apply`
and :command:`install` commands to perform these operations.
#. Upload the kubelet update.
.. note::
Run the :command:`upload` command only:
.. code-block:: none
~(keystone_admin)]$ sw-patch upload <kubelet-patch>
The kubelet update cannot be applied before upgrading kubelet.
#. Start the Kubernetes upgrade.
For example:
.. code-block:: none
~(keystone_admin)]$ system kube-upgrade-start v1.16.2
+-------------------+-------------------+
| Property | Value |
+-------------------+-------------------+
| from_version | v1.16.1 |
| to_version | v1.16.2 |
| state | upgrade-started |
+-------------------+-------------------+
The upgrade process checks the applied/available updates, the upgrade path,
the health of the system, the installed applications compatibility and
validates the system is ready for an upgrade.
.. warning::
If you use the command :command:`system kube-upgrade-start --force` to
cause the upgrades process to ignore management affecting alarms and
start, first determine that these alarms will not compromise the
upgrade process.
#. Download the Kubernetes images.
For example:
.. code-block:: none
~(keystone_admin)]$ system kube-upgrade-download-images
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | b5f7dada-2537-4416-9d2c-f9ca9fcd0e22 |
| from_version | v1.16.1 |
| to_version | v1.16.2 |
| state | downloading-images |
| created_at | 2020-02-20T16:08:48.854869+00:00 |
| updated_at | None |
+--------------+--------------------------------------+
#. Confirm that the download has completed.
.. code-block:: none
~(keystone_admin)]$ system-kube-upgrade-show
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | b5f7dada-2537-4416-9d2c-f9ca9fcd0e22 |
| from_version | v1.16.1 |
| to_version | v1.16.2 |
| state | downloaded-images |
| created_at | 2020-02-20T16:08:48.854869+00:00 |
| updated_at | 2020-02-20T16:10:37.858661+00:00 |
+--------------+--------------------------------------+
#. Upgrade the control plane on the first controller.
.. code-block:: none
~(keystone_admin)]$ system kube-host-upgrade controller-1 control-plane
+-----------------------+-------------------------+
| Property | Value |
+-----------------------+-------------------------+
| control_plane_version | v1.16.1 |
| hostname | controller-1 |
| id | 2 |
| kubelet_version | v1.16.1 |
| personality | controller |
| status | upgrading-control-plane |
| target_version | v1.16.2 |
+-----------------------+-------------------------+
You can upgrade either controller first.
The state **upgraded-first-master** will be entered when the first control
plane upgrade has completed.
#. Upgrade Kubernetes networking.
This step must be completed after the first control plane has been upgraded
and before upgrading the second control plane.
.. code-block:: none
~(keystone_admin)]$ system kube-upgrade-networking
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | b5f7dada-2537-4416-9d2c-f9ca9fcd0e22 |
| from_version | v1.16.1 |
| to_version | v1.16.2 |
| state | upgrading-networking |
| created_at | 2020-02-20T16:08:48.854869+00:00 |
| updated_at | 2020-02-20T16:18:11.459736+00:00 |
+--------------+--------------------------------------+
The state **upgraded-networking** will be entered when the networking
upgrade has completed.
#. Upgrade the control plane on the second controller.
.. code-block:: none
~(keystone_admin)]$ system kube-host-upgrade controller-0 control-plane
+-----------------------+-------------------------+
| Property | Value |
+-----------------------+-------------------------+
| control_plane_version | v1.16.1 |
| hostname | controller-0 |
| id | 1 |
| kubelet_version | v1.16.1 |
| personality | controller |
| status | upgrading-control-plane |
| target_version | v1.16.2 |
+-----------------------+-------------------------+
The state **upgraded-second-master** will be entered when the upgrade has
completed.
#. Show the Kubernetes upgrade status for all hosts.
.. code-block:: none
~(keystone_admin)]$ system kube-host-upgrade-list
+----+--------------+-------------+----------------+-----------------------+-----------------+--------+
| id | hostname | personality | target_version | control_plane_version | kubelet_version | status |
+----+--------------+-------------+----------------+-----------------------+-----------------+--------+
| 1 | controller-0 | controller | v1.16.2 | v1.16.2 | v1.16.1 | None |
| 2 | controller-1 | controller | v1.16.2 | v1.16.2 | v1.16.1 | None |
| 3 | storage-0 | storage | v1.16.1 | N/A | N/A | None |
| 4 | storage-1 | storage | v1.16.1 | N/A | N/A | None |
| 5 | worker-0 | worker | v1.16.1 | N/A | v1.16.1 | None |
| 6 | worker- 1 | worker | v1.16.1 | N/A | v1.16.1 | None |
+----+--------------+-------------+----------------+-----------------------+-----------------+--------+
The control planes of both controllers are now upgraded to v1.16.2.
#. Apply and install the kubectl update.
Use the standard :command:`sw-patch`, :command:`apply` and
:command:`install` commands to perform these operations.
This places the new version of kubelet binary on each host, but will not
restart kubelet.
#. Upgrade kubelet on both controllers.
Either controller can be upgraded first.
The kubelets on all controller hosts must be upgraded before upgrading
kubelets on worker hosts.
For each controller, do the following.
#. For non |AIO-SX| systems, lock the controller.
For example:
.. code-block:: none
~(keystone_admin)]$ system host-lock controller-1
.. note::
For All-In-One Simplex systems, the controller must **not** be
locked.
#. Apply the upgrade.
For example:
.. code-block:: none
~(keystone_admin)]$ system kube-host-upgrade controller-1 kubelet
+-----------------------+-------------------+
| Property | Value |
+-----------------------+-------------------+
| control_plane_version | v1.16.2 |
| hostname | controller-1 |
| id | 2 |
| kubelet_version | v1.16.1 |
| personality | controller |
| status | upgrading-kubelet |
| target_version | v1.16.2 |
+-----------------------+-------------------+
#. For non |AIO-SX| systems, unlock the controller.
For example:
.. code-block:: none
~(keystone_admin)]$ system host-unlock controller-1
#. Show the Kubernetes upgrade status.
.. code-block:: none
~(keystone_admin)]$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | b5f7dada-2537-4416-9d2c-f9ca9fcd0e22 |
| from_version | v1.16.1 |
| to_version | v1.16.2 |
| state | upgrading-kubelets |
| created_at | 2020-02-20T16:08:48.854869+00:00 |
| updated_at | 2020-02-20T21:53:16.347406+00:00 |
+--------------+--------------------------------------+
#. Upgrade kubelet on all worker hosts.
Multiple worker hosts can be upgraded simultaneously provided there is
sufficient capacity remaining on other worker hosts.
For each worker host, do the following:
#. Lock the host.
For example:
.. code-block:: none
~(keystone_admin)]$ system host-lock worker-1
#. Perform the upgrade.
For example:
.. code-block:: none
~(keystone_admin)]$ system kube-host-upgrade worker-1 kubelet
+-----------------------+-------------------+
| Property | Value |
+-----------------------+-------------------+
| control_plane_version | v1.16.2 |
| hostname | worker-1 |
| id | 3 |
| kubelet_version | v1.16.1 |
| personality | worker |
| status | upgrading-kubelet |
| target_version | v1.16.2 |
+-----------------------+-------------------+
#. Unlock the host.
For example:
.. code-block:: none
~(keystone_admin)]$ system host-unlock worker-1
#. Complete the Kubernetes upgrade.
.. code-block:: none
~(keystone_admin)]$ system kube-upgrade-complete
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 4e942297-465e-47d4-9e1b-9fb1630be33c |
| from_version | v1.16.1 |
| to_version | v1.16.2 |
| state | upgrade-complete |
| created_at | 2020-02-19T20:59:51.079966+00:00 |
| updated_at | 2020-02-24T15:03:34.572199+00:00 |
+--------------+--------------------------------------+
.. from step 1
.. For more
information, see the :ref:`Managing Software Updates
<managing-software-updates>`.

182
doc/source/updates/kubernetes/the-kubernetes-update-orchestration-process.rst Normal file
View File

@ -0,0 +1,182 @@
.. htb1590431033292
.. _the-kubernetes-update-orchestration-process:
=======================================================
Kubernetes Version Upgrade Cloud Orchestration Overview
=======================================================
For an orchestrated Kubernetes version upgrade you need to first create a
*Kubernetes Upgrade Orchestration Strategy*, or plan for the automated
Kubernetes version upgrade procedure.
You can customize the Kubernetes version upgrade orchestration by specifying
the following parameters:
.. _htb1590431033292-ul-pdh-5ms-tlb:
- The host types to be upgraded; only **worker-apply-type** is supported.
- The alarm restriction mode; **strict** or **relaxed** where the **relaxed**
mode allows the strategy to be created while there are alarms present that
are not management-affecting.
- The concurrency of the upgrade process; whether to Kubernetes version
upgrade hosts **serially** or in **parallel**.
- The maximum number of worker hosts that can be upgraded together when
**parallel** mode is selected.
For hosts that have the |prefix|-openstack application running with active
instances and since the Kubernetes version upgrade is a reboot required
operation for a host, the strategy offers **stop/start** or **migrate** options
for managing instances over the **lock/unlock** \(reboot\) steps in the upgrade
process.
You must use the **sw-manager** CLI tool to **create**, and then **apply** the
upgrade strategy. A created strategy can be monitored with the **show**
command.
Kubernetes version upgrade orchestration automatically iterates through all
**unlocked-enabled** hosts on the system looking for hosts with the worker
function that need Kubernetes version upgrade and then proceeds to upgrade them
on the strategy :command:`apply` action.
.. note::
Controllers (including |AIO| controllers) are upgraded before worker only
hosts. Storage hosts do not run Kubernetes so Kubernetes is not upgraded
on them, although they still may be patched.
After creating the *Kubernetes Version Upgrade Orchestration Strategy*, you can
either apply the entire strategy automatically, or manually apply individual
stages to control and monitor the Kubernetes version upgrade progress one stage
at a time.
When the Kubernetes version upgrade strategy is **applied**, if the system is
All-in-one, the controllers are upgraded first, one after the other with a
swact in between, followed by the remaining worker hosts according to the
selected worker apply concurrency \(**serial** or **parallel**\) method.
The strategy creation default is to upgrade the worker hosts serially unless
the **parallel** worker apply type option is specified which configures the
Kubernetes version upgrade process for worker hosts to be in parallel \(up to a
maximum parallel number\) to reduce the overall Kubernetes version upgrade
installation time.
The upgrade takes place in two phases. The first phase upgrades the patches
(controllers, storage and then workers), and the second phase upgrades
Kubernetes based on those patches (controllers, then hosts).
.. _htb1590431033292-ol-a1b-v5s-tlb:
#. Alarm query. This step is an upgrade pre-check.
#. Initiate the Kubernetes version upgrade.
#. Download Kubernetes Images.
#. Upgrade the first Control Plane.
#. Upgrade Kubernetes networking.
#. Upgrade the second control plane (on duplex environments only).
#. Patch the hosts.
#. Patch controller nodes if they are not |AIO|.
#. Patch storage nodes.
#. Patch worker nodes (this includes |AIO| controllers).
#. Upgrade Kubelets on the hosts (controllers then workers. If controllers
are |AIO| they are processed before regular workers).
#. Swact if the host is the active controller.
#. Lock the host.
#. Upgrade kubelet.
#. Unlock the host.
#. Restore |VMs| (if applicable).
#. Complete the Kubernetes version upgrade.
Systems with |prefix|-openstack application enabled could include additional
instance management steps. For more information, see :ref:`Kubernetes Version
Upgrade Operations Requiring Manual Migration
<kubernetes-update-operations-requiring-manual-migration>`.
On systems with |prefix|-openstack application, the *Kubernetes Version Upgrade
Orchestration Strategy* considers any configured server groups and host
aggregates when creating the stages to reduce the impact to running instances.
The *Kubernetes Version Upgrade Orchestration Strategy* automatically manages
the instances during the strategy application process. The instance management
options include **start-stop** or **migrate**.
.. _htb1590431033292-ul-vcp-dvs-tlb:
- **start-stop**: where instances are stopped following the actual Kubernetes
upgrade but before the lock operation and then automatically started again
after the unlock completes. This is typically used for instances that do
not support migration or for cases where migration takes too long. To
ensure this does not impact the high-level service being provided by the
instance, the instance\(s\) should be protected and grouped into an
anti-affinity server group\(s\) with its standby instance.
- **migrate**: where instances are moved off a host following the Kubernetes
upgrade but before the host is locked. Instances with **Live Migration**
support are **Live Migrated**. Otherwise, they are **Cold Migrated**.
.. _kubernetes-update-operations-requiring-manual-migration:
----------------------------------------
Manually Migrating Application Instances
----------------------------------------
On systems with the |prefix|-openstack application there may be some instances
that cannot be migrated automatically by orchestration. In these cases the
instance migration must be managed manually.
Do the following to manage the instance re-location manually:
.. _rbp1590431075472-ul-mgr-kvs-tlb:
- Manually perform Kubernetes version upgrade at least one openstack-compute worker host. This
assumes that at least one openstack-compute worker host does not have any
instances, or has instances that can be migrated. For more information on
manually updating a host, see :ref:`Kubernetes Version Upgrade
<manual-kubernetes-components-upgrade>`.
- If the migration is prevented by limitations in the VNF or virtual
application, perform the following:
- Create new instances on an already upgraded openstack-compute worker
host.
- Manually migrate the data from the old instances to the new instances.
.. note::
This is specific to your environment and depends on the virtual
application running in the instance.
- Terminate the old instances.
- If the migration is prevented by the size of the instances local disks:
- For each openstack-compute worker host that has instances that cannot
be migrated, manually move the instances using the CLI.
Once all openstack-compute worker hosts containing instances that cannot be
migrated have been Kubernetes version upgraded, Kubernetes version upgrade
orchestration can then be used to upgrade the remaining worker hosts.