.. Greg updates required for -High Security Vulnerability Document Updates .. pek1594745988225 .. _distributed-upgrade-orchestration-process-using-the-cli: ======================================================= Distributed Upgrade Orchestration Process using the CLI ======================================================= Distributed upgrade orchestration can be initiated after the System Controller has been successfully upgraded. For more information on Prestaging Subcloud Orchestration see, :ref:`prestage-subcloud-orchestration-eb516473582f`. .. rubric:: |context| The user first creates a distributed upgrade orchestration strategy, or plan, for the automated upgrade procedure. This customizes the upgrade orchestration, using parameters to specify: .. _distributed-upgrade-orchestration-process-using-the-cli-ul-eyw-fyr-31b: - whether to stop on failure of a subcloud upgrade or continue with the next subcloud - whether to upgrade hosts serially or in parallel Based on these parameters, and the state of the subclouds, the upgrade orchestration creates a number of stages for the overall upgrade strategy. All the subclouds that are included in the same stage will be upgraded in parallel. .. rubric:: |prereq| Distributed upgrade orchestration can only be done on a system that meets the following conditions: .. _distributed-upgrade-orchestration-process-using-the-cli-ul-blp-gcx-ry: - The subclouds must use the Redfish platform management service if it is an |AIO-SX| subcloud. The install values as well as `bmc_password` for each |AIO-SX| subcloud must have already been saved in the system controller database before the start of orchestrated subcloud upgrade. .. note:: The following command is only required if the |AIO-SX| subcloud install values and `bmc_password` have never been provided using :command:`dcmanager CLI` command. .. code-block:: none ~(keystone_admin)]$ dcmanager subcloud update subcloud1 --install-values\ install-values.yaml --bmc-password For more information on :command:`install-values.yaml` file, see :ref:`Installing a Subcloud Using Redfish Platform Management Service `. - Duplex (|AIO-DX|/Standard) upgrades are supported, and they do not require remote install using Redfish. - All subclouds are clear of management-affecting alarms (with the exception of the alarm upgrade in progress). - All hosts of all subclouds must be unlocked, enabled, and available. - No distributed upgrade orchestration strategy exists, to verify use the command :command:`dcmanager upgrade-strategy-show`. An upgrade cannot be orchestrated while upgrade orchestration is in progress. - The size and format of the persistent filesystem, /opt/platform-backup, of each subcloud must be 30GiB (or larger) and ext4 respectively. From the shell on a subcloud, use the following command to view the details of its persistent file system: :command:`df -Th /opt/platform-backup` The type must be ext4 and the size must be 30GiB. For example, on controller-0, run the following command: .. code-block:: none ~(keystone_admin)]$ df -Th /opt/platform-backup Filesystem Type Size Used Avail Use% Mounted on /dev/sda1 ext4 29G 6.9G 21G 26% /opt/platform-backup - **If a previous upgrade has been done on the subcloud**, from the shell on each subcloud, use the following command to remove the previous upgrade data: :command:`sudo rm /opt/platform-backup/upgrade_data\*` You can configure an upgrade Distributed Cloud orchestration strategy using the dcmanager CLI or the Horizon web interface. If you prefer to use Horizon, see :ref:`create-an-upgrade-orchestration-using-horizon-9f8c6c2f3706`. .. rubric:: |proc| .. _distributed-upgrade-orchestration-process-using-the-cli-steps-vcm-pq4-3mb: #. Review the upgrade status for the subclouds. After the System Controller upgrade is completed, wait for 10 minutes for the **load_sync_status** of all subclouds to be updated. To identify which subclouds are upgrade-current (in-sync), use the :command:`subcloud list` command. For example: .. code-block:: none ~(keystone_admin)]$ dcmanager subcloud list +----+-----------+--------------+--------------------+-------------+ | id | name | management | availability | sync | +----+-----------+--------------+--------------------+-------------+ | 1 | subcloud-1| managed | online | out-of-sync | | 2 | subcloud-2| managed | online | out-of-sync | | 3 | subcloud-3| managed | online | out-of-sync | | 4 | subcloud-4| managed | online | out-of-sync | +----+-----------+--------------+--------------------+-------------+ .. note:: The subclouds are out-of-sync because the load-sync-status is out-of-sync. All of the above subclouds are not upgrade-current and, therefore, need to be upgraded. To see synchronization details for a subcloud, use the following command: .. code-block:: none ~(keystone_admin)]$ dcmanager subcloud show subcloud1 +-----------------------------+------------------------------+ | Field | Value | +-----------------------------+------------------------------+ | id | 1 | | name | subcloud-1 | | description | None | | location | None | | software_version | nn.nn | | management | managed | | availability | online | | deploy_status | complete | | management_subnet | fd01:82::0/64 | | management_start_ip | fd01:82::2 | | management_end_ip | fd01:82::11 | | management_gateway_ip | fd01:82::1 | | systemcontroller_gateway_ip | fd01:81::1 | | group_id | 1 | | created_at | 2021-06-07 21:05:16.224664 | | updated_at | 2021-06-09 20:01:37.525012 | | dc-cert_sync_status | in-sync | | firmware_sync_status | in-sync | | identity_sync_status | in-sync | | kubernetes_sync_status | in-sync | | load_sync_status | out-of-sync | | patching_sync_status | in-sync | | platform_sync_status | in-sync | +-----------------------------+------------------------------+ #. To create an upgrade strategy, use the :command:`dcmanager upgrade-strategy create` command. The upgrade strategy for a |prod-dc| system controls how upgrades are applied to subclouds. .. code-block:: none ~(keystone_admin)]$ dcmanager upgrade-strategy create \ [--subcloud-apply-type ] \ [–-max-parallel-subclouds ] \ [–-stop-on-failure ] \ [--group group] \ [--force] \ [] where: **subcloud-apply-type** **parallel** or **serial**— determines whether the subclouds are upgraded in parallel, or serially. If this is not specified using the CLI, the values for :command:`subcloud_update_type` defined for each subcloud group will be used by default. **max-parallel-subclouds** Sets the maximum number of subclouds that can be upgraded in parallel (default 2). If this is not specified using the CLI, the values for :command:`max_parallel_subclouds` defined for each subcloud group will be used by default. **stop-on-failure** **true**\(default) or **false**— determines whether upgrade orchestration failure for a subcloud prevents application to subsequent subclouds. **group** Optionally pass the name or ID of a subcloud group to the :command:`dcmanager upgrade-strategy create` command. This results in a strategy that is only applied to all subclouds in the specified group. The subcloud group values are used for subcloud apply type and max parallel subclouds parameters. **force** Upgrade both online and offline subclouds. Can be used for a single subcloud, or a subcloud group. For example: .. code-block:: none ~(keystone_admin)]$ dcmanager upgrade-strategy create +------------------------+----------------------------+ | Field | Value | +------------------------+----------------------------+ | strategy type | upgrade | | subcloud apply type | parallel | | max parallel subclouds | 2 | | stop on failure | False | | state | initial | | created_at | 2020-06-10T17:16:51.857207 | | updated_at | None | +------------------------+----------------------------+ #. To show the settings for the upgrade strategy, use the :command:`dcmanager upgrade-strategy show` command. For example: .. code-block:: none ~(keystone_admin)]$ dcmanager upgrade-strategy show +------------------------+----------------------------+ | Field | Value | +------------------------+----------------------------+ | subcloud apply type | parallel | | max parallel subclouds | 2 | | stop on failure | False | | state | initial | | created_at | 2020-02-02T14:42:13.822499 | | updated_at | None | +------------------------+----------------------------+ .. note:: The values for `subcloud apply type` and `max parallel subclouds` will be taken from the subcloud group if specified through the ``--group`` parameter. #. Review the upgrade strategy for the subclouds. To show the subclouds that will be upgraded when the upgrade strategy is applied, use the :command:`dcmanager strategy-step list` command. For example: .. code-block:: none ~(keystone_admin)]$ dcmanager strategy-step list +------------------+-------+---------+---------+------------+-------------+ | cloud | stage | state | details | started_at | finished_at | +------------------+-------+---------+---------+------------+-------------+ | subcloud-1 | 1 | initial | | None | None | | subcloud-4 | 1 | initial | | None | None | | subcloud-5 | 2 | initial | | None | None | | subcloud-6 | 2 | initial | | None | None | +------------------+-------+---------+---------+------------+-------------+ .. note:: All the subclouds that are included in the same stage will be upgraded in parallel. #. To apply the upgrade strategy, use the :command:`dcmanager upgrade-strategy apply` command. .. code-block:: none ~(keystone_admin)]$ dcmanager upgrade-strategy apply +------------------------+----------------------------+ | Field | Value | +------------------------+----------------------------+ | subcloud apply type | parallel | | max parallel subclouds | 2 | | stop on failure | False | | state | applying | | created_at | 2020-02-02T14:42:13.822499 | | updated_at | 2020-02-02T14:42:19.376688 | +------------------------+----------------------------+ #. To show the step currently being performed on each of the subclouds, use the :command:`dcmanager strategy-step list` command. For example: .. code-block:: none ~(keystone_admin)]$ dcmanager strategy-step list +------------------+-------+-----------------------+-------------------+----------------------------+----------------------------+ | cloud | stage | state | details | started_at | finished_at | +------------------+-------+-----------------------+-------------------+----------------------------+----------------------------+ | subcloud-1 | 1 | complete | | 2021-06-11 14:12:12.262001 | 2021-06-11 14:15:52.450908 | | subcloud-4 | 1 | activating upgrade | | 2021-06-11 14:16:02.457588 | None | | subcloud-5 | 2 | initial | | None | None | | subcloud-6 | 2 | initial | | None | None | +------------------+-------+-------------+-----------------------------+----------------------------+----------------------------+ #. To show the step currently being performed on a subcloud, use the :command:`dcmanager strategy-step show` command. .. code-block:: none ~(keystone_admin)]$ dcmanager strategy-step show #. When all the subclouds within the distributed upgrade orchestration indicate they have entered the complete state, delete the upgrade strategy, using the :command:`dcmanager upgrade-strategy delete` command. .. code-block:: none ~(keystone_admin)]$ dcmanager upgrade-strategy delete +------------------------+----------------------------+ | Field | Value | +------------------------+----------------------------+ | subcloud apply type | parallel | | max parallel subclouds | 2 | | stop on failure | False | | state | deleting | | created_at | 2020-03-23T20:04:50.992444 | | updated_at | 2020-03-23T20:05:14.157352 | +------------------------+----------------------------+ .. only:: partner .. include:: /_includes/distributed-upgrade-orchestration-process-using-the-cli.rest :start-after: DMupgrade-begin :end-before: DMupgrade-end