docs/doc/source/updates/kubernetes/manual-rollback-host-software-deployment-9295ce1e6e29.rst

Manual Rollback Host Software Deployment

For a major release software deployment, you can abort and rollback the manual-host-software-deployment-ee17ec6f71a4 procedure at any time between software deploy start and software deploy delete.

Note

After the software deploy deletion step, aborting and rolling back of the major release deployment is not possible.

Note

This section also covers the abort and rollback of a new patched major release deployment.

This section describes different rollback procedures based on where you currently are in the software deployment procedure.

Rollback After Successful Software Deploy Activate

You can roll back the manual host software deployment after a successful software deploy activate in the following scenarios:

• A Patch Release deployment

  Immediately after the successful software deploy activate

• A major release deployment

  • Immediately after the successful software deploy activate, before attempting the Kubernetes upgrade.

  • After attempting a Kubernetes upgrade which failed and was rolled back.

    Note

    A rollback of a host major release software deployment can only be done if Kubernetes has been successfully rolled back to its original version.

1. Verify the current deploy state and Kubernetes current version.

   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy show
   +--------------+------------+------+------------------+
   | From Release | To Release |  RR  |      State       |
   +--------------+------------+------+------------------+
   | 24.09.400    | 25.09.0    | True | deploy-completed |
   +--------------+------------+------+------------------+
   
   [sysadmin@controller-0 ~(keystone_admin)]$ system kube-version-list
   +---------+--------+-----------+
   | version | target | state     |
   +---------+--------+-----------+
   | v1.29.2 | True   | active    |
   | v1.30.6 | False  | available |
   | v1.31.5 | False  | available |
   | v1.32.2 | False  | available |
   | v1.33.0 | False  | available |
   +---------+--------+-----------+

2. Abort the current host software deployment.

   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy abort
   Deployment has been aborted

3. Verify the deploy state.

   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy show
   +--------------+------------+------+----------------------------------+
   | From Release | To Release |  RR  |              State               |
   +--------------+------------+------+----------------------------------+
   | 25.09.0      | 24.09.400  | True | deploy-activate-rollback-pending |
   +--------------+------------+------+----------------------------------+

4. Perform deploy activate-rollback and check the deploy and host deploy states.

   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy activate-rollback
   Deploy activate-rollback has started
   
   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy show
   +--------------+------------+------+-------------------------------+
   | From Release | To Release |  RR  |             State             |
   +--------------+------------+------+-------------------------------+
   | 25.09.0      | 24.09.400  | True | deploy-activate-rollback-done |
   +--------------+------------+------+-------------------------------+
   
   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy host-list
   +--------------+--------------+------------+------+------------------------------+
   | Host         | From Release | To Release |  RR  |            State             |
   +--------------+--------------+------------+------+------------------------------+
   | controller-0 | 25.09.0      | 24.09.400  | True | deploy-host-rollback-pending |
   | controller-1 | 25.09.0      | 24.09.400  | True | deploy-host-rollback-pending |
   +--------------+--------------+------------+------+------------------------------+

5. Roll back host.

   Perform the following commands in the order: all worker nodes, all storage nodes, controller-1, and controller-0:

   [sysadmin@controller-0 ~(keystone_admin)]$ system host-lock <host>
   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy host-rollback <host>
   [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock <host>

   Note

   For controllers, a system host-swact controller-# will be required to lock and roll back the active controller.

6. Verify the deploy state and host deploy state.

   All the hosts must be in the deploy-host-rollback-deployed state and the deploy state should be deploy-host-rollback-done.

   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy host-list
   +--------------+--------------+------------+------+-------------------------------+
   | Host         | From Release | To Release |  RR  |             State             |
   +--------------+--------------+------------+------+-------------------------------+
   | controller-0 | 25.09.0      | 24.09.400  | True | deploy-host-rollback-deployed |
   | controller-1 | 25.09.0      | 24.09.400  | True | deploy-host-rollback-deployed |
   +--------------+--------------+------------+------+-------------------------------+
   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy show
   +--------------+------------+------+---------------------------+
   | From Release | To Release |  RR  |           State           |
   +--------------+------------+------+---------------------------+
   | 25.09.0      | 24.09.400  | True | deploy-host-rollback-done |
   +--------------+------------+------+---------------------------+

7. Delete the current deployment.

   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy delete
   Deploy deleted with success
   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy show
   No deploy in progress

Now, the system is ready for the next host software deployment.

Rollback After Software Deploy Activate Fails

When software deploy activate fails, follow the steps below to abort and roll back the manual host software deployment.

1. Verify current deploy state.

   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy show
   +--------------+------------+------+------------------------+
   | From Release | To Release |  RR  |          State         |
   +--------------+------------+------+------------------------+
   | 24.09.400    | 25.09.0    | True | deploy-activate-failed |
   +--------------+------------+------+------------------------+

2. Follow the steps from step 2 <aborthostsoftwaredeploy> to step 7 <deletecurrentdeploy> in Rollback After Successful Software Deploy Activate <rollbackafterdeployactive>.

Rollback After Software Deploy Host <hostname> Fails

When deploying software to a host (software deploy host <hostname>) fails, follow the steps below to abort and roll back the manual host software deployment.

1. Verify the current deploy state.

   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy show
   +--------------+------------+------+--------------------+
   | From Release | To Release |  RR  |        State       |
   +--------------+------------+------+--------------------+
   | 24.09.400    | 25.09.0    | True | deploy-host-failed |
   +--------------+------------+------+--------------------+
   
   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy host-list
   +--------------+--------------+------------+------+---------------------+
   | Host         | From Release | To Release |  RR  |        State        |
   +--------------+--------------+------------+------+---------------------+
   | controller-0 | 25.09.0      | 24.09.400  | True | deploy-host-failed  |
   | controller-1 | 25.09.0      | 24.09.400  | True | deploy-host-done    |
   | compute-0    | 25.09.0      | 24.09.400  | True | deploy-host-pending |
   +--------------+--------------+------------+------+---------------------+

2. Abort the deploy.

   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy abort
   Deployment has been aborted

3. Verify the deploy state.

   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy host-list
   +--------------+--------------+------------+------+------------------------------+
   | Host         | From Release | To Release |  RR  |            State             |
   +--------------+--------------+------------+------+------------------------------+
   | controller-0 | 25.09.0      | 24.09.400  | True | deploy-host-rollback-pending |
   +--------------+--------------+------------+------+------------------------------+
   
   [sysadmin@controller-0 ~(keystone_admin)]$ software deploy host-list
   +--------------+--------------+------------+------+-------------------------------+
   | Host         | From Release | To Release |  RR  |            State              |
   +--------------+--------------+------------+------+-------------------------------+
   | controller-0 | 25.09.0      | 24.09.400  | True | deploy-host-rollback-pending  |
   | controller-1 | 25.09.0      | 24.09.400  | True | deploy-host-rollback-pending  |
   | compute-0    | 25.09.0      | 24.09.400  | True | deploy-host-rollback-deployed |
   +--------------+--------------+------------+------+-------------------------------+

Follow step 5 <rollbackhost> in Rollback After Successful Software Deploy Activate <rollbackafterdeployactive>. Roll back all the hosts that are in deploy-host-rollback-pending until all the hosts are in deploy-host-rollback-deployed. Then, follow step 6 <verifydeploystate> and step 7 <deletecurrentdeploy> to complete the rollback.

Rollback Before Software Deploy Host <hostname> to Any Host

If a host software deployment needs to be cancelled before the software is deployed to any host, that is, if the deploy state is either deploy-start-done or deploy-start-failed, you can delete the pending deployment directly using software deploy delete.