From aa31a230b376229b308d2494accb37f0e8c493cd Mon Sep 17 00:00:00 2001 From: Goutham Pacha Ravi Date: Fri, 10 Jul 2020 00:47:45 -0700 Subject: [PATCH] Add upgrade documentation This was missing for manila, and is required for us to apply for the assertion that the service supports accessible upgrades. Change-Id: Ib3f4c917bdd22209e6b76e2175e3b4b28f05bdba Signed-off-by: Goutham Pacha Ravi --- doc/source/admin/index.rst | 1 + .../admin/shared-file-systems-upgrades.rst | 118 ++++++++++++++++++ 2 files changed, 119 insertions(+) create mode 100644 doc/source/admin/shared-file-systems-upgrades.rst diff --git a/doc/source/admin/index.rst b/doc/source/admin/index.rst index d7ae518926..8c448d3dad 100644 --- a/doc/source/admin/index.rst +++ b/doc/source/admin/index.rst @@ -54,6 +54,7 @@ deployment. shared-file-systems-multi-backend.rst shared-file-systems-networking.rst shared-file-systems-troubleshoot.rst + shared-file-systems-upgrades.rst share_back_ends_feature_support_mapping capabilities_and_extra_specs group_capabilities_and_extra_specs diff --git a/doc/source/admin/shared-file-systems-upgrades.rst b/doc/source/admin/shared-file-systems-upgrades.rst new file mode 100644 index 0000000000..0f4b7e6600 --- /dev/null +++ b/doc/source/admin/shared-file-systems-upgrades.rst @@ -0,0 +1,118 @@ +Upgrading the Shared File System service +======================================== + +This document outlines steps and notes for operators for reference when +upgrading their Shared File System service (manila) from previous versions of +OpenStack. The service aims to provide a minimal downtime upgrade experience. +Since the service does not operate in the data plane, the accessibility of any +provisioned resources such as shares, share snapshots, share groups, share +replicas, share servers, security services and share networks will not be +affected during an upgrade. Clients can continue to actively use these +resources while the service control plane is being upgraded. + +Plan the upgrade +---------------- + +It is highly recommended that you: + +* update the Shared File System service to the latest code from the release + you are currently using. +* read the `Shared File System service release notes + `_ for the release that + you intended to upgrade to. Pay special attention to the deprecations and + upgrade notes. +* consider the impact of the service control plane upgrade to your cloud's + users. The upgrade process interrupts provisioning of new shared + file systems and associated resources. It also prevents management + operations on existing shared file systems and associated resources. Data + path access to shared file systems will remain uninterrupted. +* take a backup of the shared file system service database so you can + rollback any failed upgrades to a previous version of the software. + Although the ``manila-manage`` command offers a database downgrade + command, it is not supported for production use. The only way to recover + from a failed update is to restore the database from a backup. +* identify your Shared File System service back end storage systems/solutions + and their drivers. Ensure that the version of each storage system is + supported by the respective driver in the target release. If you're using + a storage solution from a third party vendor, consult their product pages to + determine if the solution is supported by the release of OpenStack that you + are upgrading to. Many vendors publish a support matrix either within this + service administration guide, or on their own websites. If you find an + incompatibility, stop, and determine if you have to upgrade the storage + solution first. +* develop an upgrade procedure and assess it thoroughly by using a test + environment similar to your production environment. + +Graceful service shutdown +------------------------- +Shared File System service components (scheduler, share-manager, +data-manager) are python processes listening for messages on a AMQP queue. +When the operator sends SIGTERM signal to the process, they stop getting new +work from the queue, complete any outstanding work and then terminate. + +Database Migration +------------------ +The Shared File System service only supports cold upgrades, meaning that the +service plane is expected to be down during the database upgrade. Database +upgrades include schema changes as well as data migrations to accommodate +newer versions of the schema. Once upgraded, downgrading the database is not +supported. When the database has been upgraded, older services may misbehave +when accessing database objects, so ensure all ``manila-*`` services are down +before you upgrade the database. + +Prune deleted database rows +--------------------------- +Shared File System service resources are soft deleted in the database, so +users are able to track instances in the DB that are created and destroyed +in production. Soft-deletion also helps cloud operators adhere to data +retention policies. Not purging soft-deleted entries affects DB performance as +indices grow very large and data migrations take longer as there is more +data to migrate. It is recommended that you prune the service database before +upgrading to prevent unnecessary data migrations. Pruning permanently +deletes soft deleted database records. + +.. code:: + + manila-manage db purge + +Upgrade procedure +----------------- + +#. Ensure you're running the latest Shared File System service packages for + the OpenStack release that you currently use. +#. Run the ``manila-status upgrade check`` command to validate that the service + is ready for upgrade. +#. Backup the manila database +#. Gracefully stop all Shared File System service processes. We recommend in + this order: manila-api, manila-scheduler, manila-share and manila-data. + +.. note:: + + The manila-data service may be processing time consuming data migrations. + Shutting it down will interrupt any ongoing migrations, and these will not + be automatically started when the service comes back up. You can check + the status on ongoing migrations with ``manila migration-get-progress`` + command; issue ``manila migration-complete`` for any ongoing migrations + that have completed their data copy phase. + +#. Upgrade all the service packages. If upgrading from distribution packages, + your system package manager is expected to handle this automatically. +#. Fix any deprecated configuration options used. +#. Fix any deprecated api policies used. +#. Run ``manila-manage db sync`` from any node with the latest manila + packages. +#. Start all the Shared File System service processes. +#. Inspect the ``services`` by running ``manila service-list``. If there are + any orphaned records, run ``manila-manage service cleanup`` to delete them. + +Upgrade testing +--------------- + +The Shared File System service code is continually tested for upgrade from +a previous release to the current release using `Grenade `_. Grenade is an OpenStack test harness project +that validates upgrade scenarios between releases. It uses DevStack to +initially perform a base OpenStack install and then upgrade to a target +version. Tests include the creation of a variety of Shared File System +service resources on the prior release, and verification for their existence +and functionality after the upgrade.