From b963950c47c8b3ee1372283f0d9323b7d30df209 Mon Sep 17 00:00:00 2001 From: Ron Stone Date: Tue, 20 Apr 2021 15:18:33 -0400 Subject: [PATCH] Kubernetes upgrade orchestration on DC Initial content submit Implemented review comments on patchset 1 Implemented review comments on patchset 2 Implemented review comments on patchset 3 (Note: This feature cannot be finished until https://review.opendev.org/c/starlingx/docs/+/789644 has merged.) Story: 2008055 Task: 42228 Signed-off-by: Ron Stone Change-Id: Id80dddf036d2e646daae3e356963470acfeb8a85 Signed-off-by: Ron Stone --- ...te-orchestration-on-distributed-cloud.rest | 0 ...ernetes-upgrade-for-distributed-cloud.rest | 0 ...ate-orchestration-on-distributed-cloud.rst | 264 ++++++++++++++++++ doc/source/dist_cloud/index.rst | 10 + ...ted-cloud-update-orchestration-process.rst | 36 +++ 5 files changed, 310 insertions(+) create mode 100644 doc/source/_includes/configuring-kubernetes-update-orchestration-on-distributed-cloud.rest create mode 100644 doc/source/_includes/performing-an-orchestred-kubernetes-upgrade-for-distributed-cloud.rest create mode 100644 doc/source/dist_cloud/configuring-kubernetes-update-orchestration-on-distributed-cloud.rst create mode 100644 doc/source/dist_cloud/the-kubernetes-distributed-cloud-update-orchestration-process.rst diff --git a/doc/source/_includes/configuring-kubernetes-update-orchestration-on-distributed-cloud.rest b/doc/source/_includes/configuring-kubernetes-update-orchestration-on-distributed-cloud.rest new file mode 100644 index 000000000..e69de29bb diff --git a/doc/source/_includes/performing-an-orchestred-kubernetes-upgrade-for-distributed-cloud.rest b/doc/source/_includes/performing-an-orchestred-kubernetes-upgrade-for-distributed-cloud.rest new file mode 100644 index 000000000..e69de29bb diff --git a/doc/source/dist_cloud/configuring-kubernetes-update-orchestration-on-distributed-cloud.rst b/doc/source/dist_cloud/configuring-kubernetes-update-orchestration-on-distributed-cloud.rst new file mode 100644 index 000000000..2370afe4e --- /dev/null +++ b/doc/source/dist_cloud/configuring-kubernetes-update-orchestration-on-distributed-cloud.rst @@ -0,0 +1,264 @@ + +.. ccf1617821548674 +.. _configuring-kubernetes-update-orchestration-on-distributed-cloud: + +==================================================================== +Kubernetes Version Upgrade Distributed Cloud Orchestration Procedure +==================================================================== + +You can configure a *Kubernetes Version Upgrade Distributed Cloud +Orchestration Strategy* using the **dcmanager** CLI. + +.. note:: + Management-affecting alarms cannot be ignored using relaxed alarm rules + during an orchestrated Kubernetes version upgrade operation. For a list of + management-affecting alarms, see |fault-doc|: + :ref:`Alarm Messages <100-series-alarm-messages>`. To display + management-affecting active alarms, use the following command: + +.. code-block:: none + + ~(keystone_admin)$ fm alarm-list --mgmt_affecting + +During an orchestrated Kubernetes version upgrade operation, the following +alarms are ignored even when the default strict restrictions are selected: + + +.. _ccf1617821548674-ul-vhg-jxs-tlb: + +**100.103** + Memory threshold exceeded + +**200.001** + Locked Host + +**280.001** + Subcloud resource off-line + +**280.002** + Subcloud resource out-of-sync + +**700.004** + VM stopped + +**750.006** + Automatic Application Re-Apply Is Pending + +**900.001** + Patching operation in progress + +**900.007** + Kubernetes upgrade in progress + +**900.401** + Kubernetes upgrade auto-apply inprogress + + +.. rubric:: |prereq| + + +.. _ccf1617821548674-ul-ls2-pxs-tlb: + +Hosts that need to be upgraded must be in the **unlocked-enabled** state. + +.. only:: starlingx + + Kubernetes has been upgraded on the system controller. + The managed subclouds are all unlocked and online. + +.. only:: partner + + .. include:: ../_includes/configuring-kubernetes-update-orchestration-on-distributed-cloud.rest + +All :command:`dcmanager` strategies must be deleted. + +The system controller should already be upgraded to the new version of +Kubernetes. This will cause the 'audit' for the subclouds note that they are +now out of sync. + +Hosts that need to be upgraded must be in the unlocked-enabled state. + +Patches related to Kubernetes version upgrade must be uploaded to the central +controller for access by subclouds. For example: + +.. code-block:: none + + ~(keystone_admin)]$ sw-patch --os-region-name SystemController upload PLATFORM.1.patch + ~(keystone_admin)]$ sw-patch --os-region-name SystemController upload KUBE.1.patch + ~(keystone_admin)]$ sw-patch --os-region-name SystemController upload KUBE.2.patch + + +.. rubric:: |proc| + +#. Create the strategy. + + The *Kubernetes Version Upgrade Distributed Cloud Orchestration Strategy* + :command:`create` command creates a series of stages with steps that apply + the Kubernetes version upgrade to the hardware. + + A Kubernetes version upgrade requires a reboot. Therefore, the created + strategy includes steps that automatically lock and unlock the host to + bring the new image function into service. + + .. code-block:: none + + ~(keystone_admin)$ dcmanager kube-upgrade-strategy create + +------------------------+----------------------------+ + | Field | Value | + +------------------------+----------------------------+ + | strategy type | kubernetes | + | subcloud apply type | None | + | max parallel subclouds | None | + | stop on failure | False | + | state | initial | + | created_at | 2020-11-09T23:00:07.210958 | + | updated_at | None | + +------------------------+----------------------------+ + + The syntax for the :command:`dcmanager kube-upgrade-strategy create` command + is: + + .. code-block:: none + + dcmanager kube-upgrade-strategy create [-h] + [-f {json,shell,table,value,yaml}] + [-c COLUMN] + [--max-width ] + [--fit-width] [--print-empty] + [--noindent] [--prefix PREFIX] + [--subcloud-apply-type {parallel,serial}] + [--max-parallel-subclouds MAX_PARALLEL_SUBCLOUDS] + [--stop-on-failure] [--force] + [--group GROUP] + [cloud_name] + + + where the following are significant for performing an orchestrated Kubernetes + upgrade: + + **--subcloud-apply-type** + Determines whether the subclouds are upgraded in parallel, or serially. If + this is not specified using the CLI, the values for subcloud\_update\_type + defined for each subcloud group will be used by default. + + **--max-parallel-subclouds** + Sets the maximum number of subclouds that can be upgraded in parallel + \(default 20\). If this is not specified using the CLI, the values for + max\_parallel\_subclouds defined for each subcloud group will be used by + default. + + **--stop-on-failure** + **true** \(default\) or **false** — determines whether upgrade + orchestration failure for a subcloud prevents application to subsequent + subclouds. + + **--group** + Optionally pass the name or ID of a subcloud group to the + :command:`dcmanager kube-upgrade-strategy` command. This results in a + strategy that is only applied to all subclouds in the specified group. If + not specified, all subcloud groups are upgraded. + + +#. Optional: Display the strategy in summary, if required. The Kubernetes + upgrade strategy :command:`show` command displays the strategy in a summary. + + .. code-block:: none + + ~(keystone_admin)$ dcmanager kube-upgrade-strategy show + +------------------------+----------------------------+ + | Field | Value | + +------------------------+----------------------------+ + | strategy type | kubernetes | + | subcloud apply type | None | + | max parallel subclouds | None | + | stop on failure | False | + | state | complete | + | created_at | 2020-11-09 23:00:07.210958 | + | updated_at | 2020-11-09 23:01:15.697763 | + +------------------------+----------------------------+ + + The strategy steps are displayed using the ``--details`` option. + +#. Apply the strategy. + + The scope of the subcloud orchestration is restricted through the 'group' + argument. + + .. code-block:: none + + ~(keystone_admin)$ dcmanager kube-upgrade-strategy apply + +------------------------+----------------------------+ + | Field | Value | + +------------------------+----------------------------+ + | strategy type | kubernetes | + | subcloud apply type | None | + | max parallel subclouds | None | + | stop on failure | False | + | state | applying | + | created_at | 2021-04-13T21:00:19.067829 | + | updated_at | 2021-04-13T21:00:38.527080 | + +------------------------+----------------------------+ + + - Use the :command:`show` command to monitor Kubernetes version upgrade + state and percentage completion. + + + .. code-block:: none + + ~(keystone_admin)$ dcmanager kube-upgrade-strategy show + +------------------------+----------------------------+ + | Field | Value | + +------------------------+----------------------------+ + | strategy type | kubernetes | + | subcloud apply type | None | + | max parallel subclouds | None | + | stop on failure | False | + | state | complete | + | created_at | 2020-11-09 23:00:07.210958 | + | updated_at | 2020-11-09 23:01:15.697763 | + +------------------------+----------------------------+ + +#. Monitor the progress of the strategy. + + .. code-block:: + + ~(keystone_admin)$ dcmanager strategy-step list + + +------------------+-------+---------+---------+------------+-------------+ + | cloud | stage | state | details | started_at | finished_at | + +------------------+-------+---------+---------+------------+-------------+ + | subcloud1 | 2 | initial | | None | None | + +------------------+-------+---------+---------+------------+-------------+ + +#. Optional: Abort the strategy, if required. This is only used to stop, and + abort the entire strategy. + + The Kubernetes Version Upgrade Distributed Cloud Orchestration strategy + :command:`abort` command can be used to abort the Kubernetes Version + Upgrade Distributed Cloud Orchestration strategy after the current step of + the currently applying stage is completed. + +#. Delete the strategy. + + .. note:: + After the *Kubernetes Version Upgrade Distributed Cloud Orchestration + Strategy* has been applied \(or aborted\) it must be deleted before + another Kubernetes Version Upgrade Distributed Cloud Orchestration + strategy can be created. If a Kubernetes upgrade strategy application + fails, you must address the issue that caused the failure, then delete + and re-create the strategy before attempting to apply it again. + + .. code-block:: none + + ~(keystone_admin)$ dcmanager kube-upgrade-strategy delete + +------------------------+----------------------------+ + | Field | Value | + +------------------------+----------------------------+ + | strategy type | kubernetes | + | subcloud apply type | None | + | max parallel subclouds | None | + | stop on failure | False | + | state | deleting | + | created_at | 2020-11-09T23:00:07.210958 | + | updated_at | 2020-11-09T23:01:52.620362 | + +------------------------+----------------------------+ diff --git a/doc/source/dist_cloud/index.rst b/doc/source/dist_cloud/index.rst index 6c11ed9cf..8b392f90d 100644 --- a/doc/source/dist_cloud/index.rst +++ b/doc/source/dist_cloud/index.rst @@ -49,6 +49,16 @@ Operation changing-the-admin-password-on-distributed-cloud updating-docker-registry-credentials-on-a-subcloud +---------------------------------------------------------- +Kubernetes Version Upgrade Distributed Cloud Orchestration +---------------------------------------------------------- + +.. toctree:: + :maxdepth: 1 + :caption: Contents: + + the-kubernetes-distributed-cloud-update-orchestration-process + configuring-kubernetes-update-orchestration-on-distributed-cloud -------- Appendix diff --git a/doc/source/dist_cloud/the-kubernetes-distributed-cloud-update-orchestration-process.rst b/doc/source/dist_cloud/the-kubernetes-distributed-cloud-update-orchestration-process.rst new file mode 100644 index 000000000..1e5fe125e --- /dev/null +++ b/doc/source/dist_cloud/the-kubernetes-distributed-cloud-update-orchestration-process.rst @@ -0,0 +1,36 @@ + +.. fez1617811988954 +.. _the-kubernetes-distributed-cloud-update-orchestration-process: + +==================================================================== +Kubernetes Version Upgrade Distributed Cloud Orchestration Overviews +==================================================================== + +For an orchestrated Kubernetes version upgrade across a |prod-dc|, you need to +first create a *Kubernetes Version Upgrade Distributed Cloud Orchestration +Strategy*, or plan, for the automated Kubernetes version upgrade procedure +orchestrated across all subclouds of the Distributed Cloud system. + +You must use the :command:`dcmanager` CLI tool to **create**, and then +**apply** the upgrade strategy. A created strategy can be monitored with the +**show** command. For more information, see :ref:`Kubernetes Version Upgrade +Distributed Cloud Orchestration Procedure +`. + +Kubernetes upgrade orchestration automatically iterates through all managed +online subclouds. + +The specific steps involved in a Kubernetes Version Upgrade Distributed Cloud +Orchestration for a single or group of hosts in each subcloud includes: + +.. _fez1617811988954-ol-a1b-v5s-tlb: + +#. Create the subcloud patch strategy. + +#. Apply the subcloud patch strategy. + +#. Delete the subcloud patch strategy. + +#. Create the subcloud kube upgrade strategy. + +#. Apply the subcloud kube upgrade strategy.