docs/doc/source/dist_cloud/distributed-upgrade-orchestration-process-using-the-cli.rst

.. pek1594745988225
.. _distributed-upgrade-orchestration-process-using-the-cli:

=======================================================
Distributed Upgrade Orchestration Process Using the CLI
=======================================================

Distributed upgrade orchestration can be initiated after the upgrade and
stability of the System Controller cloud. Distributed upgrade orchestration can
be initiated after the system controller has been successfully upgraded.

.. rubric:: |context|

The user first creates a distributed upgrade orchestration strategy, or plan,
for the automated upgrade procedure. This customizes the upgrade orchestration,
using parameters to specify:


.. _distributed-upgrade-orchestration-process-using-the-cli-ul-eyw-fyr-31b:

-   whether to stop on failure of a subcloud upgrade or continue with the next
    subcloud

-   whether to upgrade hosts serially or in parallel


Based on these parameters, and the state of the subclouds, the upgrade
orchestrator creates a number of stages for the overall upgrade strategy. All
the subclouds that are included in the same stage will be upgraded in parallel.

.. rubric:: |prereq|

Distributed upgrade orchestration can only be done on a system that meets the
following conditions:

.. _distributed-upgrade-orchestration-process-using-the-cli-ul-blp-gcx-ry:

-   The subclouds must use the Redfish platform management service if it is
    an |AIO-SX| subcloud.

-   Duplex \(|AIO-DX|/Standard\) upgrades are supported, and they do not
    require remote install using Redfish.

-   Redfish |BMC| is required for orchestrated subcloud upgrades. The install
    values, and :command:`bmc\_password` for each |AIO-SX| subcloud controller
    must be provided using the following |CLI| command on the System Controller:

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager subcloud update subcloud1 --install-values\
        install-values.yaml --bmc-password <password>

    For more information on :command:`install-values.yaml` file, see
    :ref:`Installing a Subcloud Using Redfish Platform Management Service
    <installing-a-subcloud-using-redfish-platform-management-service>`.

-   All subclouds are clear of management-affecting alarms \(with the exception of the alarm upgrade
    in progress\).

-   All hosts of all subclouds must be unlocked, enabled, and available.

-   No distributed upgrade orchestration strategy exists, to verify use the
    command :command:`dcmanager upgrade-strategy-show`. An upgrade cannot be
    orchestrated while upgrade orchestration is in progress.

-   Verify the size and format of the platform-backup filesystem on each
    subcloud. From the shell on each subcloud, use the following command to view
    the details of the file system:

    :command:`df -Th /opt/platform-backup`

    The type must be ext4 and the size must be 9.5GB. For example, on
    controller-0, run the following command:

    .. code-block:: none

        ~(keystone_admin)]$ df -Th /opt/platform-backup/ Filesystem Type Size Used Avail Use% Mounted on /dev/sda2 ext4 9.5G 51M 9.0G 1% /opt/platform-backup

-   **If a previous upgrade has been done on the subcloud**, from the shell on
    each subcloud, use the following command to remove the previous upgrade
    data:

    :command:`sudo rm /opt/platform-backup/upgrade\_data\*`

.. rubric:: |proc|

.. _distributed-upgrade-orchestration-process-using-the-cli-steps-vcm-pq4-3mb:

#.  Review the upgrade status for the subclouds.

    After the System Controller upgrade is completed, wait for 10 minutes for
    the **load\_sync\_status** of all subclouds to be updated.

    To identify which subclouds are upgrade-current \(in-sync\), use the
    :command:`subcloud list` command. For example:

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager subcloud list
        +----+-----------+--------------+--------------------+-------------+
        | id | name      | management   | availability       | sync        |
        +----+-----------+--------------+--------------------+-------------+
        |  1 | subcloud1 | managed      | online             | out-of-sync |
        |  2 | subcloud2 | managed      | online             | out-of-sync |
        |  3 | subcloud3 | managed      | online             | out-of-sync |
        |  4 | subcloud4 | managed      | online             | out-of-sync |
        +----+-----------+--------------+--------------------+-------------+

    .. note::
        The sync status is the rolled up sync status of platform, patching,
        identity, etc.

    To see synchronization details for a subcloud, use the following command:

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager subcloud show subcloud1
        +-----------------------------+----------------------------+
        | Field                       | Value                      |
        +-----------------------------+----------------------------+
        | id                          | 1                          |
        | name                        | subcloud1                  |
        | description                 | None                       |
        | location                    | None                       |
        | software_version            | nn.nn                      |
        | management                  | managed                    |
        | availability                | online                     |
        | deploy_status               | complete                   |
        | management_subnet           | fd01:82::0/64              |
        | management_start_ip         | fd01:82::2                 |
        | management_end_ip           | fd01:82::11                |
        | management_gateway_ip       | fd01:82::1                 |
        | systemcontroller_gateway_ip | fd01:81::1                 |
        | group_id                    | 1                          |
        | created_at                  | 2020-07-15 19:23:50.966984 |
        | updated_at                  | 2020-07-17 12:36:28.815655 |
        | dc-cert_sync_status         | in-sync                    |
        | identity_sync_status        | in-sync                    |
        | load_sync_status            | in-sync                    |
        | patching_sync_status        | in-sync                    |
        | platform_sync_status        | in-sync                    |
        +-----------------------------+----------------------------+

#.  To create an upgrade strategy, use the :command:`dcmanager upgrade-strategy create`
    command.

    The upgrade strategy for a |prod-dc| system controls how upgrades are
    applied to subclouds.

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager upgrade-strategy create \
        [--subcloud-apply-type <type>] \
        [–-max-parallel-subclouds <i>] \
        [–-stop-on-failure <level>] \
        [--group group] \
        [<subcloud>]

    where:

    **subcloud-apply-type**
        **parallel** or **serial**— determines whether the subclouds are
        upgraded in parallel, or serially.

        If this is not specified using the CLI, the values for
        :command:`subcloud\_update\_type` defined for each subcloud group will
        be used by default.

    **max-parallel-subclouds**
        Sets the maximum number of subclouds that can be upgraded in parallel
        \(default 20\).

        If this is not specified using the CLI, the values for
        :command:`max\_parallel\_subclouds` defined for each subcloud group
        will be used by default.

    **stop-on-failure**
        **true**\(default\) or **false**— determines whether upgrade
        orchestration failure for a subcloud prevents application to subsequent
        subclouds.

    **group**
        Optionally pass the name or ID of a subcloud group to the
        :command:`dcmanager upgrade-strategy create` command. This results in a
        strategy that is only applied to all subclouds in the specified group.
        The subcloud group values are used for subcloud apply type and max
        parallel subclouds parameters.

    For example:

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager upgrade-strategy create
        +------------------------+----------------------------+
        | Field                  | Value                      |
        +------------------------+----------------------------+
        | strategy type          | upgrade                    |
        | subcloud apply type    | parallel                   |
        | max parallel subclouds | 10                         |
        | stop on failure        | False                      |
        | state                  | initial                    |
        | created_at             | 2020-06-10T17:16:51.857207 |
        | updated_at             | None                       |
        +------------------------+----------------------------+

#.  To show the settings for the upgrade strategy, use the
    :command:`dcmanager upgrade-strategy show` command.

    For example:

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager upgrade-strategy show
        +------------------------+----------------------------+
        | Field                  | Value                      |
        +------------------------+----------------------------+
        | subcloud apply type    | parallel                   |
        | max parallel subclouds | 20                         |
        | stop on failure        | False                      |
        | state                  | initial                    |
        | created_at             | 2020-02-02T14:42:13.822499 |
        | updated_at             | None                       |
        +------------------------+----------------------------+

    .. note::
        A value of **None** for :command:`subcloud apply type`, and
        :command:`max parallel subclouds` indicates that subcloud group values
        are being used.

#.  Review the upgrade strategy for the subclouds.

    To show the subclouds that will be upgraded when the upgrade strategy is
    applied, use the :command:`dcmanager strategy-step list` command. For
    example:

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager strategy-step list
        +------------------+-------+---------+---------+------------+-------------+
        | cloud            | stage | state   | details | started_at | finished_at |
        +------------------+-------+---------+---------+------------+-------------+
        | subcloud-1       |     1 | initial |         | None       | None        |
        | subcloud-4       |     1 | initial |         | None       | None        |
        | subcloud-5       |     2 | initial |         | None       | None        |
        | subcloud-6       |     2 | initial |         | None       | None        |
        +------------------+-------+---------+---------+------------+-------------+

    .. note::
        All the subclouds that are included in the same stage will be upgraded
        in parallel.

#.  To apply the upgrade strategy, use the :command:`dcmanager upgrade-strategy apply`
    command.

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager upgrade-strategy apply
        +------------------------+----------------------------+
        | Field                  | Value                      |
        +------------------------+----------------------------+
        | subcloud apply type    | parallel                   |
        | max parallel subclouds | 20                         |
        | stop on failure        | False                      |
        | state                  | applying                   |
        | created_at             | 2020-02-02T14:42:13.822499 |
        | updated_at             | 2020-02-02T14:42:19.376688 |
        +------------------------+----------------------------+

#.  To show the step currently being performed on each of the subclouds, use
    the :command:`dcmanager strategy-step list` command.

    For example:

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager strategy-step list
        +------------------+-------+-------------+-----------------------------+----------------------------+----------------------------+
        | cloud            | stage | state       | details                     | started_at                 | finished_at                |
        +------------------+-------+-------------+-----------------------------+----------------------------+----------------------------+
        | subcloud-1       |     2 | applying... | apply phase is 66% complete | 2020-03-13 14:12:12.262001 | 2020-03-13 14:15:52.450908 |
        | subcloud-4       |     2 | applying... | apply phase is 83% complete | 2020-03-13 14:16:02.457588 | None                       |
        | subcloud-5       |     2 | finishing   |                             | 2020-03-13 14:16:02.463213 | None                       |
        | subcloud-6       |     2 | applying... | apply phase is 66% complete | 2020-03-13 14:16:02.473669 | None                       |
        +------------------+-------+-------------+-----------------------------+----------------------------+----------------------------+

#.  To show the step currently being performed on a subcloud, use the
    :command:`dcmanager strategy-step show` <subcloud> command.

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager strategy-step show <subcloud>

#.  When the distributed upgrade orchestration complete, delete the upgrade
    strategy, using the :command:`dcmanager upgrade-strategy delete` command.

    .. code-block:: none

        ~(keystone_admin)]$ dcmanager upgrade-strategy delete
        +------------------------+----------------------------+
        | Field                  | Value                      |
        +------------------------+----------------------------+
        | subcloud apply type    | parallel                   |
        | max parallel subclouds | 20                         |
        | stop on failure        | False                      |
        | state                  | deleting                   |
        | created_at             | 2020-03-23T20:04:50.992444 |
        | updated_at             | 2020-03-23T20:05:14.157352 |
        +------------------------+----------------------------+

.. rubric:: |postreq|

.. _distributed-upgrade-orchestration-process-using-the-cli-ul-lx1-zcv-3mb:

The secret payload should be, "username: sysinv password:<password>". If
the secret payload is, "username: admin password:<password>", see,
:ref:`Update Docker Registry Credentials on a Subcloud
<updating-docker-registry-credentials-on-a-subcloud>` for more information.

.. only:: partner

   .. include:: ../_includes/distributed-upgrade-orchestration-process-using-the-cli.rest