94fd67c34a
Distributed Upgrade Orchestration Process Using the CLI - Modified this topic to include 2 prerequisites Changing Case on file extensions. Updated comments for Patchset 3 Fixed merge conflicts Updated comments for Patchset 4 Fixed merge conflicts Story: 2008055 Task: 42387 Signed-off-by: Juanita-Balaraj <juanita.balaraj@windriver.com> Change-Id: Ia2c44812052c4f70f4742923fa847698cc0d6fa6 Signed-off-by: Juanita-Balaraj <juanita.balaraj@windriver.com>
336 lines
15 KiB
ReStructuredText
336 lines
15 KiB
ReStructuredText
|
||
.. pek1594745988225
|
||
.. _distributed-upgrade-orchestration-process-using-the-cli:
|
||
|
||
=======================================================
|
||
Distributed Upgrade Orchestration Process Using the CLI
|
||
=======================================================
|
||
|
||
Distributed upgrade orchestration can be initiated after the upgrade and
|
||
stability of the SystemController cloud. Upgrade orchestration automatically
|
||
iterates through each of the subclouds, installing the new software load on
|
||
each one.
|
||
|
||
.. rubric:: |context|
|
||
|
||
The user first creates a distributed upgrade orchestration strategy, or plan,
|
||
for the automated upgrade procedure. This customizes the upgrade orchestration,
|
||
using parameters to specify:
|
||
|
||
|
||
.. _distributed-upgrade-orchestration-process-using-the-cli-ul-eyw-fyr-31b:
|
||
|
||
- whether to stop on failure of a subcloud upgrade or continue with the next
|
||
subcloud
|
||
|
||
- whether to upgrade hosts serially or in parallel
|
||
|
||
|
||
Based on these parameters, and the state of the subclouds, distributed upgrade
|
||
orchestration creates a number of stages for the overall upgrade strategy. All
|
||
the subclouds that are included in the same stage will be upgraded in parallel.
|
||
|
||
.. rubric:: |prereq|
|
||
|
||
Distributed upgrade orchestration can only be done on a system that meets the
|
||
following conditions:
|
||
|
||
.. _distributed-upgrade-orchestration-process-using-the-cli-ul-blp-gcx-ry:
|
||
|
||
- The subclouds must use the Redfish platform management service if it is
|
||
an |AIO-SX| subcloud.
|
||
|
||
- Duplex \(|AIO-DX|/Standard\) upgrades are supported, and they do not
|
||
require remote install using Redfish.
|
||
|
||
- Redfish |BMC| is required for orchestrated subcloud upgrades. The install
|
||
values, and :command:`bmc\_password` for each |AIO-SX| subcloud controller
|
||
must be provided using the following |CLI| command on the SystemController:
|
||
|
||
.. code-block:: none
|
||
|
||
~(keystone_admin)]$ dcmanager subcloud update subcloud1 --install-values\
|
||
install-values.yaml --bmc-password <password>
|
||
|
||
For more information on :command:`install-values.yaml` file, see
|
||
:ref:`Installing a Subcloud Using Redfish Platform Management Service
|
||
<installing-a-subcloud-using-redfish-platform-management-service>`.
|
||
|
||
- All subclouds are clear of alarms \(with the exception of the alarm upgrade
|
||
in progress\).
|
||
|
||
- All hosts of all subclouds must be unlocked, enabled, and available.
|
||
|
||
- No distributed update orchestration strategy exists, to verify use the
|
||
command :command:`dcmanager upgrade-stratagy-show`. An upgrade cannot be
|
||
orchestrated while update orchestration is in progress.
|
||
|
||
- Verify the size and format of the platform-backup filesystem on each
|
||
subcloud. From the shell on each subcloud, use the following command to view
|
||
the details of the file system:
|
||
|
||
:command:`df -Th /opt/platform-backup`
|
||
|
||
The type must be ext4 and the size must be 9.5GB. For example, on
|
||
controller-0, run the following command:
|
||
|
||
.. code-block:: none
|
||
|
||
~(keystone_admin)]$ df -Th /opt/platform-backup/ Filesystem Type Size Used Avail Use% Mounted on /dev/sda2 ext4 9.5G 51M 9.0G 1% /opt/platform-backup
|
||
|
||
- **If a previous upgrade has been done on the subcloud**, from the shell on
|
||
each subcloud, use the following command to remove the previous upgrade
|
||
data:
|
||
|
||
:command:`sudo rm /opt/platform-backup/upgrade\_data\*`
|
||
|
||
.. rubric:: |proc|
|
||
|
||
.. _distributed-upgrade-orchestration-process-using-the-cli-steps-vcm-pq4-3mb:
|
||
|
||
#. Review the upgrade status for the subclouds.
|
||
|
||
After the SystemController upgrade is completed, wait for 10 minutes for
|
||
the **load\_sync\_status** of all subclouds to be updated.
|
||
|
||
To identify which subclouds are upgrade-current \(in-sync\), use the
|
||
:command:`subcloud list` command. For example:
|
||
|
||
.. code-block:: none
|
||
|
||
~(keystone_admin)]$ dcmanager subcloud list
|
||
+----+-----------+--------------+--------------------+-------------+
|
||
| id | name | management | availability | sync |
|
||
+----+-----------+--------------+--------------------+-------------+
|
||
| 1 | subcloud1 | managed | online | out-of-sync |
|
||
| 2 | subcloud2 | managed | online | out-of-sync |
|
||
| 3 | subcloud3 | managed | online | out-of-sync |
|
||
| 4 | subcloud4 | managed | online | out-of-sync |
|
||
+----+-----------+--------------+--------------------+-------------+
|
||
|
||
.. note::
|
||
The sync status is the rolled up sync status of platform, patching,
|
||
identity, etc.
|
||
|
||
To see synchronization details for a subcloud, use the following command:
|
||
|
||
.. code-block:: none
|
||
|
||
~(keystone_admin)]$ dcmanager subcloud show subcloud1
|
||
+-----------------------------+----------------------------+
|
||
| Field | Value |
|
||
+-----------------------------+----------------------------+
|
||
| id | 1 |
|
||
| name | subcloud1 |
|
||
| description | None |
|
||
| location | None |
|
||
| software_version | nn.nn |
|
||
| management | managed |
|
||
| availability | online |
|
||
| deploy_status | complete |
|
||
| management_subnet | fd01:82::0/64 |
|
||
| management_start_ip | fd01:82::2 |
|
||
| management_end_ip | fd01:82::11 |
|
||
| management_gateway_ip | fd01:82::1 |
|
||
| systemcontroller_gateway_ip | fd01:81::1 |
|
||
| group_id | 1 |
|
||
| created_at | 2020-07-15 19:23:50.966984 |
|
||
| updated_at | 2020-07-17 12:36:28.815655 |
|
||
| dc-cert_sync_status | in-sync |
|
||
| identity_sync_status | in-sync |
|
||
| load_sync_status | in-sync |
|
||
| patching_sync_status | in-sync |
|
||
| platform_sync_status | in-sync |
|
||
+-----------------------------+----------------------------+
|
||
|
||
#. To create an upgrade strategy, use the :command:`dcmanager upgrade-strategy create`
|
||
command.
|
||
|
||
The upgrade strategy for a |prod-dc| system controls how upgrades are
|
||
applied to subclouds.
|
||
|
||
.. code-block:: none
|
||
|
||
~(keystone_admin)]$ dcmanager upgrade-strategy create \
|
||
[--subcloud-apply-type <type>] \
|
||
[–-max-parallel-subclouds <i>] \
|
||
[–-stop-on-failure <level>] \
|
||
[--group group] \
|
||
[<subcloud>]
|
||
|
||
where:
|
||
|
||
**subcloud-apply-type**
|
||
**parallel** or **serial**— determines whether the subclouds are
|
||
upgraded in parallel, or serially.
|
||
|
||
If this is not specified using the CLI, the values for
|
||
:command:`subcloud\_update\_type` defined for each subcloud group will
|
||
be used by default.
|
||
|
||
**max-parallel-subclouds**
|
||
Sets the maximum number of subclouds that can be upgraded in parallel
|
||
\(default 20\).
|
||
|
||
If this is not specified using the CLI, the values for
|
||
:command:`max\_parallel\_subclouds` defined for each subcloud group
|
||
will be used by default.
|
||
|
||
**stop-on-failure**
|
||
**true**\(default\) or **false**— determines whether upgrade
|
||
orchestration failure for a subcloud prevents application to subsequent
|
||
subclouds.
|
||
|
||
**group**
|
||
Optionally pass the name or ID of a subcloud group to the
|
||
:command:`dcmanager upgrade-strategy create` command. This results in a
|
||
strategy that is only applied to all subclouds in the specified group.
|
||
The subcloud group values are used for subcloud apply type and max
|
||
parallel subclouds parameters.
|
||
|
||
For example:
|
||
|
||
.. code-block:: none
|
||
|
||
~(keystone_admin)]$ dcmanager upgrade-strategy create
|
||
+------------------------+----------------------------+
|
||
| Field | Value |
|
||
+------------------------+----------------------------+
|
||
| strategy type | upgrade |
|
||
| subcloud apply type | parallel |
|
||
| max parallel subclouds | 10 |
|
||
| stop on failure | False |
|
||
| state | initial |
|
||
| created_at | 2020-06-10T17:16:51.857207 |
|
||
| updated_at | None |
|
||
+------------------------+----------------------------+
|
||
|
||
#. To show the settings for the upgrade strategy, use the
|
||
:command:`dcmanager upgrade-strategy show` command.
|
||
|
||
For example:
|
||
|
||
.. code-block:: none
|
||
|
||
~(keystone_admin)]$ dcmanager upgrade-strategy show
|
||
+------------------------+----------------------------+
|
||
| Field | Value |
|
||
+------------------------+----------------------------+
|
||
| subcloud apply type | parallel |
|
||
| max parallel subclouds | 20 |
|
||
| stop on failure | False |
|
||
| state | initial |
|
||
| created_at | 2020-02-02T14:42:13.822499 |
|
||
| updated_at | None |
|
||
+------------------------+----------------------------+
|
||
|
||
.. note::
|
||
A value of **None** for :command:`subcloud apply type`, and
|
||
:command:`max parallel subclouds` indicates that subcloud group values
|
||
are being used.
|
||
|
||
#. Review the upgrade strategy for the subclouds.
|
||
|
||
To show the subclouds that will be upgraded when the upgrade strategy is
|
||
applied, use the :command:`dcmanager strategy-step list` command. For
|
||
example:
|
||
|
||
.. code-block:: none
|
||
|
||
~(keystone_admin)]$ dcmanager strategy-step list
|
||
+------------------+-------+---------+---------+------------+-------------+
|
||
| cloud | stage | state | details | started_at | finished_at |
|
||
+------------------+-------+---------+---------+------------+-------------+
|
||
| subcloud-1 | 1 | initial | | None | None |
|
||
| subcloud-4 | 1 | initial | | None | None |
|
||
| subcloud-5 | 2 | initial | | None | None |
|
||
| subcloud-6 | 2 | initial | | None | None |
|
||
+------------------+-------+---------+---------+------------+-------------+
|
||
|
||
.. note::
|
||
All the subclouds that are included in the same stage will be upgraded
|
||
in parallel.
|
||
|
||
#. To apply the upgrade strategy, use the :command:`dcmanager upgrade-strategy apply`
|
||
command.
|
||
|
||
.. code-block:: none
|
||
|
||
~(keystone_admin)]$ dcmanager upgrade-strategy apply
|
||
+------------------------+----------------------------+
|
||
| Field | Value |
|
||
+------------------------+----------------------------+
|
||
| subcloud apply type | parallel |
|
||
| max parallel subclouds | 20 |
|
||
| stop on failure | False |
|
||
| state | applying |
|
||
| created_at | 2020-02-02T14:42:13.822499 |
|
||
| updated_at | 2020-02-02T14:42:19.376688 |
|
||
+------------------------+----------------------------+
|
||
|
||
#. To show the step currently being performed on each of the subclouds, use
|
||
the :command:`dcmanager strategy-step list` command.
|
||
|
||
For example:
|
||
|
||
.. code-block:: none
|
||
|
||
~(keystone_admin)]$ dcmanager strategy-step list
|
||
+------------------+-------+-------------+-----------------------------+----------------------------+----------------------------+
|
||
| cloud | stage | state | details | started_at | finished_at |
|
||
+------------------+-------+-------------+-----------------------------+----------------------------+----------------------------+
|
||
| subcloud-1 | 2 | applying... | apply phase is 66% complete | 2020-03-13 14:12:12.262001 | 2020-03-13 14:15:52.450908 |
|
||
| subcloud-4 | 2 | applying... | apply phase is 83% complete | 2020-03-13 14:16:02.457588 | None |
|
||
| subcloud-5 | 2 | finishing | | 2020-03-13 14:16:02.463213 | None |
|
||
| subcloud-6 | 2 | applying... | apply phase is 66% complete | 2020-03-13 14:16:02.473669 | None |
|
||
+------------------+-------+-------------+-----------------------------+----------------------------+----------------------------+
|
||
|
||
#. To show the step currently being performed on a subcloud, use the
|
||
:command:`dcmanager strategy-step show` <subcloud> command.
|
||
|
||
.. code-block:: none
|
||
|
||
~(keystone_admin)]$ dcmanager strategy-step show <subcloud>
|
||
|
||
#. When the distributed upgrade orchestration complete, delete the upgrade
|
||
strategy, using the :command:`dcmanager upgrade-strategy delete` command.
|
||
|
||
.. code-block:: none
|
||
|
||
~(keystone_admin)]$ dcmanager upgrade-strategy delete
|
||
+------------------------+----------------------------+
|
||
| Field | Value |
|
||
+------------------------+----------------------------+
|
||
| subcloud apply type | parallel |
|
||
| max parallel subclouds | 20 |
|
||
| stop on failure | False |
|
||
| state | deleting |
|
||
| created_at | 2020-03-23T20:04:50.992444 |
|
||
| updated_at | 2020-03-23T20:05:14.157352 |
|
||
+------------------------+----------------------------+
|
||
|
||
.. rubric:: |postreq|
|
||
|
||
.. _distributed-upgrade-orchestration-process-using-the-cli-ul-lx1-zcv-3mb:
|
||
|
||
- Check and update docker registry credentials for **ALL** subclouds. For
|
||
each subcloud:
|
||
|
||
.. code-block:: none
|
||
|
||
REGISTRY="docker-registry"
|
||
SECRET_UUID='system service-parameter-list | fgrep
|
||
$REGISTRY | fgrep auth-secret | awk '{print $10}''
|
||
SECRET_REF='openstack secret list | fgrep ${SECRET_UUID}|
|
||
awk '{print $2}''
|
||
openstack secret get ${SECRET_REF} --payload -f value
|
||
|
||
The secret payload should be, "username: sysinv password:<password>". If
|
||
the secret payload is, "username: admin password:<password>", see,
|
||
:ref:`Updating Docker Registry Credentials on a Subcloud
|
||
<updating-docker-registry-credentials-on-a-subcloud>` for more information.
|
||
|
||
.. only:: partner
|
||
|
||
.. include:: ../_includes/distributed-upgrade-orchestration-process-using-the-cli.rest
|