From 2a75cb0a7a9b52eb9138d52831377a2572887cfc Mon Sep 17 00:00:00 2001 From: Litao Gao Date: Thu, 7 Dec 2023 14:43:49 +0800 Subject: [PATCH] Add Distributed Cloud GEO Redundancy docs (r9, dsr8MR3) - Overview of the feature - Procedure of the feature configuration Story: 2010852 Task: 48493 Change-Id: If5fd6792adbb7e77ab2e92f29527c951be0134ee Signed-off-by: Litao Gao Signed-off-by: Ngairangbam Mili --- doc/source/_vendor/vendor_strings.txt | 2 + ...louds-using-dcmanager-cli-f12020a8fc42.rst | 6 + ...controller-geo-redundancy-e3a31d6bf662.rst | 617 ++++++++++++++++++ .../kubernetes/figures/dcg1695034653874.png | Bin 0 -> 34912 bytes .../index-dist-cloud-kub-95bef233eef0.rst | 10 + ...ew-of-distributed-cloud-geo-redundancy.rst | 118 ++++ .../kubernetes/rehoming-a-subcloud.rst | 8 +- 7 files changed, 760 insertions(+), 1 deletion(-) create mode 100644 doc/source/dist_cloud/kubernetes/configure-distributed-cloud-system-controller-geo-redundancy-e3a31d6bf662.rst create mode 100644 doc/source/dist_cloud/kubernetes/figures/dcg1695034653874.png create mode 100644 doc/source/dist_cloud/kubernetes/overview-of-distributed-cloud-geo-redundancy.rst diff --git a/doc/source/_vendor/vendor_strings.txt b/doc/source/_vendor/vendor_strings.txt index 74a94fe3f..575911ddb 100755 --- a/doc/source/_vendor/vendor_strings.txt +++ b/doc/source/_vendor/vendor_strings.txt @@ -23,6 +23,8 @@ .. |os-prod-hor| replace:: OpenStack |prod-hor| .. |prod-img| replace:: https://mirror.starlingx.windriver.com/mirror/starlingx/ .. |prod-abbr| replace:: StX +.. |prod-dc-geo-red| replace:: Distributed Cloud Geo Redundancy +.. |prod-dc-geo-red-long| replace:: Distributed Cloud System controller Geographic Redundancy .. Guide names; will be formatted in italics by default. .. |node-doc| replace:: :title:`StarlingX Node Configuration and Management` diff --git a/doc/source/dist_cloud/kubernetes/backup-a-subcloud-group-of-subclouds-using-dcmanager-cli-f12020a8fc42.rst b/doc/source/dist_cloud/kubernetes/backup-a-subcloud-group-of-subclouds-using-dcmanager-cli-f12020a8fc42.rst index 2abd63f0b..1dfa66495 100644 --- a/doc/source/dist_cloud/kubernetes/backup-a-subcloud-group-of-subclouds-using-dcmanager-cli-f12020a8fc42.rst +++ b/doc/source/dist_cloud/kubernetes/backup-a-subcloud-group-of-subclouds-using-dcmanager-cli-f12020a8fc42.rst @@ -16,6 +16,12 @@ system data backup file has been generated on the subcloud, it will be transferred to the system controller and stored at a dedicated central location ``/opt/dc-vault/backups//``. +.. note:: + + Enabling the GEO Redundancy function will affect some of the subcloud + backup functions. For more information on GEO Redundancy and its + restrictions, see :ref:`configure-distributed-cloud-system-controller-geo-redundancy-e3a31d6bf662`. + Backup data creation requires the subcloud to be online, managed, and in healthy state. diff --git a/doc/source/dist_cloud/kubernetes/configure-distributed-cloud-system-controller-geo-redundancy-e3a31d6bf662.rst b/doc/source/dist_cloud/kubernetes/configure-distributed-cloud-system-controller-geo-redundancy-e3a31d6bf662.rst new file mode 100644 index 000000000..01c6f94b1 --- /dev/null +++ b/doc/source/dist_cloud/kubernetes/configure-distributed-cloud-system-controller-geo-redundancy-e3a31d6bf662.rst @@ -0,0 +1,617 @@ +.. _configure-distributed-cloud-system-controller-geo-redundancy-e3a31d6bf662: + +============================================================ +Configure Distributed Cloud System Controller GEO Redundancy +============================================================ + +.. rubric:: |context| + +You can configure a distributed cloud System Controller GEO Redundancy +using DC manager |CLI| commands. + +System administrators can follow the procedures below to enable and +disable the GEO Redundancy feature. + +.. Note:: + + In this release, the GEO Redundancy feature supports only two + distributed clouds in one protection group. + +.. contents:: + :local: + :depth: 1 + +--------------------- +Enable GEO Redundancy +--------------------- + +Set up a protection group for two distributed clouds, making these two +distributed clouds operational in 1+1 active GEO Redundancy mode. + +For example, let us assume we have two distributed clouds, site A and site B. +When the operation is performed on site A, the local site is site A and the +peer site is site B. When the operation is performed on site B, the local +site is site B and the peer site is site A. + +.. rubric:: |prereq| + +The peer system controller's |OAM| network is accessible to each other and can +access the subclouds via both |OAM| and management networks. + +For security of production system, it is important to ensure the safety and +identification of peer site queries. To meet this objective, it is essential to +have an HTTPS-based system API in place. This necessitates the presence of a +well-known and trusted |CA| to enable secure HTTPS communication between peers. +If you are using an internally trusted |CA|, ensure that the system trusts the |CA| by installing +its certificate with the following command. + +.. code-block:: none + + ~(keystone_admin)]$ system certificate-install --mode ssl_ca + +where: + +```` + is the path to the intermediate or Root |CA| certificate associated + with the |prod| REST API's Intermediate or Root |CA|-signed certificate. + +.. rubric:: |proc| + +You can enable the GEO Redundancy feature between site A and site B from the +command line. In this procedure, the subclouds managed by site A will be +configured to be managed by GEO Redundancy protection group that consists of site +A and site B. When site A is offline for some reasons, an alarm notifies the +administrator, who initiates the group based batch migration +to rehome the subclouds of site A to site B for centralized management. + +Similarly, you can also configure the subclouds managed by site B to be +taken over by site A when site B is offline by following the same procedure where +site B is local site and site A is peer site. + +#. Log in to the active controller node of site B and get the required + information about the site B to create a protection group. + + * Unique |UUID| of the central cloud of the peer system controller + * URI of Keystone endpoint of peer system controller + * Gateway IP address of the management network of peer system controller + + For example: + + .. code-block:: bash + + # On site B + sysadmin@controller-0:~$ source /etc/platform/openrc + ~(keystone_admin)]$ system show | grep -i uuid + | uuid | 223fcb30-909d-4edf-8c36-1aebc8e9bd4a | + + ~(keystone_admin)]$ openstack endpoint list --service keystone \ + --interface public --region RegionOne -c URL + +-----------------------------+ + | URL | + +-----------------------------+ + | http://10.10.10.2:5000 | + +-----------------------------+ + + ~(keystone_admin)]$ system host-route-list controller-0 | awk '{print $10}' | grep -v "^$" + gateway + 10.10.27.1 + +#. Log in to the active controller node of the central cloud of site A. Create + a System Peer instance of site B on site A so that site A can access information of + site B. + + .. code-block:: bash + + # On site A + ~(keystone_admin)]$ dcmanager system-peer add \ + --peer-uuid 223fcb30-909d-4edf-8c36-1aebc8e9bd4a \ + --peer-name siteB \ + --manager-endpoint http://10.10.10.2:5000 \ + --peer-controller-gateway-address 10.10.27.1 + Enter the admin password for the system peer: + Re-enter admin password to confirm: + + +----+--------------------------------------+-----------+-----------------------------+----------------------------+ + | id | peer uuid | peer name | manager endpoint | controller gateway address | + +----+--------------------------------------+-----------+-----------------------------+----------------------------+ + | 2 | 223fcb30-909d-4edf-8c36-1aebc8e9bd4a | siteB | http://10.10.10.2:5000 | 10.10.27.1 | + +----+--------------------------------------+-----------+-----------------------------+----------------------------+ + +#. Collect the information from site A. + + .. code-block:: bash + + # On site A + sysadmin@controller-0:~$ source /etc/platform/openrc + ~(keystone_admin)]$ system show | grep -i uuid + ~(keystone_admin)]$ openstack endpoint list --service keystone --interface public --region RegionOne -c URL + ~(keystone_admin)]$ system host-route-list controller-0 | awk '{print $10}' | grep -v "^$" + +#. Log in to the active controller node of the central cloud of site B. Create + a System Peer instance of site A on site B so that site B has information about site A. + + .. code-block:: bash + + # On site B + ~(keystone_admin)]$ dcmanager system-peer add \ + --peer-uuid 3963cb21-c01a-49cc-85dd-ebc1d142a41d \ + --peer-name siteA \ + --manager-endpoint http://10.10.11.2:5000 \ + --peer-controller-gateway-address 10.10.25.1 + Enter the admin password for the system peer: + Re-enter admin password to confirm: + +#. Create a |SPG| for site A. + + .. code-block:: bash + + # On site A + ~(keystone_admin)]$ dcmanager subcloud-peer-group add --peer-group-name group1 + +#. Add the subclouds needed for redundancy protection on site A. + + Ensure that the subclouds bootstrap data is updated. The bootstrap data is + the data used to bootstrap the subcloud, which includes the |OAM| and + management network information, system controller gateway information, and docker + registry information to pull necessary images to bootstrap the system. + + For an example of a typical bootstrap file, see :ref:`installing-and-provisioning-a-subcloud`. + + #. Update the subcloud information with the bootstrap values. + + .. code-block:: bash + + ~(keystone_admin)]$ dcmanager subcloud update subcloud1 \ + --bootstrap-address \ + --bootstrap-values + + #. Update the subcloud information with the |SPG| created locally. + + .. code-block:: bash + + ~(keystone_admin)]$ dcmanager subcloud update \ + --peer-group + + For example, + + .. code-block:: bash + + ~(keystone_admin)]$ dcmanager subcloud update subcloud1 --peer-group group1 + + #. If you want to remove one subcloud from the |SPG|, run the + following command: + + .. code-block:: bash + + ~(keystone_admin)]$ dcmanager subcloud update --peer-group none + + For example, + + .. code-block:: bash + + ~(keystone_admin)]$ dcmanager subcloud update subcloud1 --peer-group none + + #. Check the subclouds that are under the |SPG|. + + .. code-block:: bash + + ~(keystone_admin)]$ dcmanager subcloud-peer-group list-subclouds + +#. Create an association between the System Peer and |SPG|. + + .. code-block:: bash + + # On site A + ~(keystone_admin)]$ dcmanager peer-group-association add \ + --system-peer-id \ + --peer-group-id \ + --peer-group-priority + + The ``peer-group-priority`` parameter can accept an integer value greater + than 0. It is used to set the priority of the |SPG|, which is + created in peer site using the peer site's dcmanager API during association + synchronization. + + * The default priority in the |SPG| is 0 when it is created + in the local site. + + * The smallest integer has the highest priority. + + During the association creation, the |SPG| in the association + will be synchronized from the local site to the peer site, and the subclouds + belonging to the |SPG|. + + Confirm that the local |SPG| and its subclouds have been synchronized + into site B with the same name. + + * Show the association information just created in site A and ensure that + ``sync_status`` is ``in-sync``. + + .. code-block:: bash + + # On site A + ~(keystone_admin)]$ dcmanager peer-group-association list + + +----+---------------+----------------+---------+-----------------+---------------------+ + | id | peer_group_id | system_peer_id | type | sync_status | peer_group_priority | + +----+---------------+----------------+---------+-----------------+---------------------+ + | 1 | 1 | 2 | primary | in-sync | 2 | + +----+---------------+----------------+---------+-----------------+---------------------+ + + * Show ``subcloud-peer-group`` in site B and ensure that it has been created. + + * List the subcloud in ``subcloud-peer-group`` in site B and ensure that all + the subclouds have been synchronized as secondary subclouds. + + .. code-block:: bash + + # On site B + ~(keystone_admin)]$ dcmanager subcloud-peer-group show + ~(keystone_admin)]$ dcmanager subcloud-peer-group list-subclouds + + When you create the primary association on site A, a non-primary association + on site B will automatically be created to associate the synchronized |SPG| + from site A and the system peer pointing to site A. + + You can check the association list to confirm if the non-primary association + was created on site B. + + .. code-block:: bash + + # On site B + ~(keystone_admin)]$ dcmanager peer-group-association list + +----+---------------+----------------+-------------+-------------+---------------------+ + | id | peer_group_id | system_peer_id | type | sync_status | peer_group_priority | + +----+---------------+----------------+-------------+-------------+---------------------+ + | 2 | 26 | 1 | non-primary | in-sync | None | + +----+---------------+----------------+-------------+-------------+---------------------+ + +#. (Optional) Update the protection group related configuration. + + After the peer group association has been created, you can still update the + related resources configured in the protection group: + + * Update subcloud with bootstrap values + * Add subcloud(s) into the |SPG| + * Remove subcloud(s) from the |SPG| + + After any of the above operations, ``sync_status`` is changed to ``out-of-sync``. + + After the update has been completed, you need to use the :command:`sync` + command to push the |SPG| changes to the peer site that + keeps the |SPG| the same status. + + .. code-block:: bash + + # On site A + dcmanager peer-group-association sync + + .. warning:: + + The :command:`dcmanager peer-group-association sync` command must be run + after any of the following changes: + + - Subcloud is removed from the |SPG| for the subcloud name change. + + - Subcloud is removed from the |SPG| for the subcloud management network + reconfiguration. + + - Subcloud updates one or both of these parameters: + ``--bootstrap-address``, ``--bootstrap-values parameters``. + + Similarly, you need to check the information has been synchronized by + showing the association information just created in site A, ensuring that + ``sync_status`` is ``in-sync``. + + .. code-block:: bash + + # On site A + ~(keystone_admin)]$ dcmanager peer-group-association show + + +----+---------------+----------------+---------+-----------------+---------------------+ + | id | peer_group_id | system_peer_id | type | sync_status | peer_group_priority | + +----+---------------+----------------+---------+-----------------+---------------------+ + | 1 | 1 | 2 | primary | in-sync | 2 | + +----+---------------+----------------+---------+-----------------+---------------------+ + +.. rubric:: |result| + +You have configured a GEO Redundancy protection group between site A and site B. +If site A is offline, the subclouds configured in the |SPG| can be +migrated in batch to site B for central management manually. + +---------------------------- +Health Monitor and Migration +---------------------------- + +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Peer monitoring and alarming +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +After the peer protection group is formed, if site A cannot be connected to +site B, there will be an alarm message on site B. + +For example: + +.. code-block:: bash + + # On site B + ~(keystone_admin)]$ fm alarm-list + +----------+--------------------------------------------------------------------------------------------------------------------------+--------------------------------------+----------+--------------------------+ + | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp | + +----------+--------------------------------------------------------------------------------------------------------------------------+--------------------------------------+----------+--------------------------+ + | 280.004 | Peer siteA is in disconnected state. Following subcloud peer groups are impacted: group1. | peer=223fcb30-909d-4edf- | major | 2023-08-18T10:25:29. | + | | | 8c36-1aebc8e9bd4a | | 670977 | + | | | | | | + +----------+--------------------------------------------------------------------------------------------------------------------------+--------------------------------------+----------+--------------------------+ + +Administrator can suppress the alarm with the following command: + +.. code-block:: bash + + # On site B + ~(keystone_admin)]$ fm event-suppress --alarm_id 280.004 + +----------+------------+ + | Event ID | Status | + +----------+------------+ + | 280.004 | suppressed | + +----------+------------+ + +--------- +Migration +--------- + +If site A is down, after receiving the alarming message the administrator +can choose to perform the migration on site B, which will migrate the +subclouds under the |SPG| from site A to site B. + +.. note:: + + Before initiating the migration operation, ensure that ``sync-status`` of the + peer group association is ``in-sync`` so that the latest updates from site A + have been successfully synchronized to site B. If ``sync_status`` is not + ``in-sync``, the migration may fail. + +.. code-block:: bash + + # On site B + ~(keystone_admin)]$ dcmanager subcloud-peer-group migrate + + # For example: + ~(keystone_admin)]$ dcmanager subcloud-peer-group migrate group1 + +During the batch migration, you can check the status of the migration of each +subcloud in the |SPG| by showing the details of the |SPG| being migrated. + +.. code-block:: bash + + # On site B + ~(keystone_admin)]$ dcmanager subcloud-peer-group status + +After successful migration, the subcloud(s) should be in +``managed/online/complete`` status on site B. + +For example: + +.. code-block:: bash + + # On site B + ~(keystone_admin)]$ dcmanager subcloud list + +----+---------------------------------+------------+--------------+---------------+-------------+---------------+-----------------+ + | id | name | management | availability | deploy status | sync | backup status | backup datetime | + +----+---------------------------------+------------+--------------+---------------+-------------+---------------+-----------------+ + | 45 | subcloud3-node2 | managed | online | complete | in-sync | None | None | + | 46 | subcloud1-node6 | managed | online | complete | in-sync | None | None | + +----+---------------------------------+------------+--------------+---------------+-------------+---------------+-----------------+ + +-------------- +Post Migration +-------------- + +If site A is restored, the subcloud(s) should be adjusted to +``unmanaged/secondary`` status in site A. The administrator can receive an +alarm on site A that notifies that the |SPG| is managed by a peer site (site +B), because this |SPG| on site A has the higher priority. + +.. code-block:: bash + + ~(keystone_admin)]$ fm alarm-list + +----------+-------------------------------------------------------------------------------------------------------------------------+----------------------------------+----------+-----------------------+ + | Alarm ID | Reason Text | Entity ID | Severity | Time Stamp | + +----------+-------------------------------------------------------------------------------------------------------------------------+----------------------------------+----------+-----------------------+ + | 280.005 | Subcloud peer group (peer_group_name=group1) is managed by remote system | subcloud_peer_group=7 | warning | 2023-09-04T04:51:58. | + | | (peer_uuid=223fcb30-909d-4edf-8c36-1aebc8e9bd4a) with lower priority. | | | 435539 | + | | | | | | + +----------+-------------------------------------------------------------------------------------------------------------------------+----------------------------------+----------+-----------------------+ + +Then, the administrator can decide if and when to migrate the subcloud(s) back. + +.. code-block:: bash + + # On site A + ~(keystone_admin)]$ dcmanager subcloud-peer-group migrate + + # For example: + ~(keystone_admin)]$ dcmanager subcloud-peer-group migrate group1 + +After successful migration, the subcloud status should be back to the +``managed/online/complete`` status. + +For example: + +.. code-block:: bash + + +----+---------------------------------+------------+--------------+---------------+---------+---------------+-----------------+ + | id | name | management | availability | deploy status | sync | backup status | backup datetime | + +----+---------------------------------+------------+--------------+---------------+---------+---------------+-----------------+ + | 33 | subcloud3-node2 | managed | online | complete | in-sync | None | None | + | 34 | subcloud1-node6 | managed | online | complete | in-sync | None | None | + +----+---------------------------------+------------+--------------+---------------+---------+---------------+-----------------+ + +Also, the alarm mentioned above will be cleared after migrating back. + +.. code-block:: bash + + ~(keystone_admin)]$ fm alarm-list + +---------------------- +Disable GEO Redundancy +---------------------- + +You can disable the GEO Redundancy feature from the command line. + +Ensure that you have a stable environment to disable the GEO Redundancy +feature, ensuring that the subclouds are managed by the expected site. + +.. rubric:: |proc| + +#. Delete the primary association on both the sites. + + .. code-block:: bash + + # site A + ~(keystone_admin)]$ dcmanager peer-group-association delete + +#. Delete the |SPG|. + + .. code-block:: bash + + # site A + ~(keystone_admin)]$ dcmanager subcloud-peer-group delete group1 + +#. Delete the system peer. + + .. code-block:: bash + + # site A + ~(keystone_admin)]$ dcmanager system-peer delete siteB + # site B + ~(keystone_admin)]$ dcmanager system-peer delete siteA + +.. rubric:: |result| + +You have torn down the protection group between site A and site B. + +--------------------------- +Backup and Restore Subcloud +--------------------------- + +You can backup and restore a subcloud in a distributed cloud environment. +However, GEO redundancy does not support the replication of subcloud backup +files from one site to another. + +A subcloud backup is valid only for the current system controller. When a +subcloud is migrated from site A to site B, the existing backup becomes +unavailable. In this case, you can create a new backup of that subcloud on site +B. Subsequently, you can restore the subcloud from this newly created backup +when it is managed under site B. + +For information on how to backup and restore a subcloud, see +:ref:`backup-a-subcloud-group-of-subclouds-using-dcmanager-cli-f12020a8fc42` +and :ref:`restore-a-subcloud-group-of-subclouds-from-backup-data-using-dcmanager-cli-f10c1b63a95e`. + +------------------------------------------- +Operations Performed by Protected Subclouds +------------------------------------------- + +The table below lists the operations that can/cannot be performed on the protected subclouds. + +**Primary site**: The site where the |SPG| was created. + +**Secondary site**: The peer site where the subclouds in the |SPG| can be migrated to. + +**Protected subcloud**: The subcloud that belongs to a |SPG|. + +**Local/Unprotected subcloud**: The subcloud that does not belong to any |SPG|. + ++------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------------------+ +| Operation | Allow (Y/N/Maybe) | Note | ++==========================================+==================================+=================================================================================================+ +| Unmanage | N | Subcloud must be removed from the |SPG| before it can be manually unmanaged. | ++------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------------------+ +| Manage | N | Subcloud must be removed from the |SPG| before it can be manually managed. | ++------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------------------+ +| Delete | N | Subcloud must be removed from the |SPG| before it can be manually unmanaged | +| | | and deleted. | ++------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------------------+ +| Update | Maybe | Subcloud can only be updated while it is managed in the primary site because the sync command | +| | | can only be issued from the system controller where the |SPG| was created. | +| | | | +| | | .. warning:: | +| | | | +| | | The subcloud network cannot be reconfigured while it is being managed by the secondary | +| | | site. If this operation is necessary, perform the following steps: | +| | | | +| | | #. Remove the subcloud from the |SPG| to make it a local/unprotected | +| | | subcloud. | +| | | #. Update the subcloud. | +| | | #. (Optional) Manually rehome the subcloud to the primary site after it is restored. | +| | | #. (Optional) Re-add the subcloud to the |SPG|. | ++------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------------------+ +| Rename | Yes | - If the subcloud in the primary site is already a part of |SPG|, we need to remove it from the | +| | | |SPG| and then unmanage, rename, and manage the subcloud, and add it back to |SPG| and perform| +| | | the sync operation. | +| | | | +| | | - If the subcloud is in the secondary site, perform the following steps: | +| | | | +| | | #. Remove the subcloud from the |SPG| to make it a local/unprotected subcloud. | +| | | | +| | | #. Unmange the subcloud. | +| | | | +| | | #. Rename the subcloud. | +| | | | +| | | #. (Optional) Manually rehome the subcloud to the primary site after it is restored. | +| | | | +| | | #. (Optional) Re-add the subcloud to the |SPG|. | ++------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------------------+ +| Patch | Y | .. warning:: | +| | | | +| | | There may be a patch out-of-sync alarm when the subcloud is migrated to another site. | ++------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------------------+ +| Upgrade | Y | All the system controllers in the protection group must be upgraded first before upgrading | +| | | any of the subclouds. | ++------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------------------+ +| Rehome | N | Subcloud cannot be manually rehomed while being part of the |SPG| | ++------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------------------+ +| Backup | Y | | +| | | | ++------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------------------+ +| Restore | Maybe | - If the subcloud in the primary site is already a part of |SPG|, we need to remove it from the | +| | | |SPG| and then unmanage and restore the subcloud, and add it back to |SPG| and perform | +| | | the sync operation. | +| | | | +| | | - If the subcloud is in the secondary site, perform the following steps: | +| | | | +| | | #. Remove the subcloud from the |SPG| to make it a local/unprotected subcloud. | +| | | | +| | | #. Unmange the subcloud. | +| | | | +| | | #. Restore the subcloud from the backup. | +| | | | +| | | #. (Optional) Manually rehome the subcloud to the primary site after it is restored. | +| | | | +| | | #. (Optional) Re-add the subcloud to the |SPG|. | ++------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------------------+ +| Prestage | Y | .. warning:: | +| | | | +| | | The prestage data will get overwritten because it is not guaranteed that both the system | +| | | controllers always run on the same patch level (ostree repo) and/or have the same images | +| | | list. | +| | | | ++------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------------------+ +| Reinstall | Y | | +| | | | ++------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------------------+ +| Remove from |SPG| | Maybe | Subcloud can be removed from the |SPG| in the primary site. Subcloud can | +| | | only be removed from the |SPG| in the secondary site if the primary site is | +| | | currently down. | ++------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------------------+ +| Add to |SPG| | Maybe | Subcloud can only be added to the |SPG| in the primary site as manual sync is required. | +| | | | +| | | | ++------------------------------------------+----------------------------------+-------------------------------------------------------------------------------------------------+ + + + + + diff --git a/doc/source/dist_cloud/kubernetes/figures/dcg1695034653874.png b/doc/source/dist_cloud/kubernetes/figures/dcg1695034653874.png new file mode 100644 index 0000000000000000000000000000000000000000..3e149b83ed14501b1689e8175c0b596ed63eaf10 GIT binary patch literal 34912 zcmb5VWmr{R*Y_==gtRnBY*0eF8)Sok(gxirpmaCFCZt0`NAHj z?}GEZuIIjw^)Dwg(S9=!`#G*s9oASUWn<32S@MxzY)9veLN$KfPG* zj|6%C|BrdN1pYnapTFN7>w399PQrNbASn2awB!d@z5NFCda4PhgQkhOkFvzXQ7ReV z=oAu~bEdmHF(~H}f0`r7=h`hgdv9NOztJ&jT5n(y?2-Q(e*KPCNyo^Rvn;$NC&Wye z;3MtxZvpO2>KLcD*D#S?%cVjghrt>|ipT5$@}l3|?5uxprrg}Du;EOrJhv-DEd?Kp z0Sdqek8_E|*8JypHf~R#>8{-=5}W4k&ln)-1^PDG`%^Jg5#5Ib|6V5c|8%OQZwMh{ zEzQ+3dXLr=hy(6<*n%D#*J|Js;c+^l9vinq-^kyqwO8SNoV?@+P4loWi%*7NkzUq^ zp9@a1=<2J{+&?xv0y`c16KDMBG85?q1p>mqUWe8DPH|&fN&Kud@&c*)HcuYrBl!LD z8R)G3V>t@zT@MK8+j3DZwqzxC)Qa`?H~W)fe>u`**r|n)=BU&j&yuidtq*+<BmL~JJ8Fv#n6)eEWXdsRq8DlQ=x5%bzM4d3@Zx=$AACbMFKr1#%NC`rPz z{PQ&g1DU>BG5{UW-mAzghdVKi@dqR1wTm6SRB1N$Oo%ol10%Q8ZjNYfM+C+R$XpiP z%wt}Dd)(nW+|}LE{G?^8jfoH(Z24<86Thr`s+!{$Xh~PYT%#f`SA@&>cidr0jJe=< zWX*e0jO?g>;$_o8dp=+-YA-dl>V+B4oBRjkkAwQ@siH_6~`=PN-A z=;DVcE*-?`&p;_XjEZ@QoKRN=kz&QTvF?}Uh8K|c6h}3yyZyQK`=SOLt@I9KiUJp= zJ99aShd;)eecS6k7MQcBCoxNTdZkFD_<9uqrqkp(@|yJGgEkb184VnstZv3n(!f6OBlJ460x^IZSRO_@GlRfI|eK`{8 zJx=|`@O=%l>4#uB*+6B|5;T+Zcf%NAlw}3)(@of~< zT)eI}4X2Z5*VTa~N6De-Hz&DziTIZg17jgxIA>rtPor~y;KQi3#?wNZ3kNMBoBE^9 zl#Tw5rJMbwN}8d7%-{p^Z5{J!nkh0$bcoT{6sw$m)~r{XL+JU-O)h9i$)+k>6%;G?|xOLq5PObo$QaaR8-)3tg?90FCtvD z2RM_itx>(Ly}?|@D!3Fy7AnqGQQB@;+lHBlNAyDmNxc5`Og3Z>Nd@3SF4G#7wJe<~ zVFw*Hw`Vp;hjv>IDM10Jk8dGHGN>{c(L*JrHlI5qX%3*-ojXs5QVfTz*hkYqFdPKW znZeD?!L4`M^8oy9+90r4XB>ma*cnfhd`;zdEMebO+_LJh?)JaewCoNUmuL+k;*tVc zvBs%jy>%)n`lVh1kG(bA79CSuP!ngQ#tr3BP3!JoxY@STps`!fM%mTP3e643oO*?? zOGWnYK76}AQOC3rN&Zw{1Ub#qTGN(?-v+N@W^bAZ+DP5K48TrTIzC^EtP;?Y!WR~G zG^^Ga@)CD#-uKHvnMsNqlTwxw+~76{V@z{x#p<1|v+X1|e=biIGRCNo7{$5~nDC=4 zAlw2DVk06VlgcnO(#ss4M#*PkV458G0uJmEmC0CsA*SvXce&=BhL^_ZZyt2qHT8@z zq4>kJ+fhH!+amK%qrcwBiuTo{v48>|P<-v)n68UN7aAwEKte7n@ejOfGV0@^1OI+U zbxWU|={%*CZ-bM(3=%r)=NUTblYSaNmDX(v|0cqu$l(KXNI$-`8Ti4s4>ijStHDm- zWs(9Bu=4gsieGM>UGFUh3RjbD`SMG3z^|^L5(lTBuGs8c2P}wY&30LJM7}&@$$hQz z(?v74e;mDUvQ#PQclsfDX3?65lq~nqgSk+?c?wK&aa~{1D*UpuwU!M@iQR(oS(;Ce zKDfLOJ{l!fI6s8jHz=6d27JYA3l`>-4Ec)DAZn!y|>1%lRhCYkH2_mCtF@ToTWJ_sj@YPNU7bLFQ+u^k;)p*)2 zg6T9?uUUnCRnFdhrN-HP&}yaRA9V|!b58X$=N>;8S3UpyEBXxkD5ZTw)RA|yPac>u zPa*E6_dP_xg}i$4GdDMd*0X?=K1B}eaqOuAh>2hr!}9nk=t`6E5!(mH4ue1jXc0*@ zNpFA)OXDW*-fWvnAefm!bPz^>$ckDNJS#VDW&W@Z#)yXS!N*K!8)mv?gDH_Uv=Zwq zu}+o5qtS#h@|9esEpF3{Y@d@vjWN`pFn`d?bz|<9I60g?8^*}M(2Ur79rUBLMVjJ0(=L~HKN%3F9tJY!%Ort-TwBxQPi9rkzq&em(L9-NH zHD_(Fhva)upHYx4Q2Ah6m`Vq4a!-GIESa3oW%HGJFR9GNhtsweAWhUlA#;vbTjB3H(jLDSIS8-`)vJV3q% z7jf2uU&*RTJ(9<`96!0kv)$dhUc(TfWO9weSnv1G{`a=wGxX? zVQpqOXM!V1EsRk$BK0)rtb_)(YAfMz%Ke0dBRl&zMlCZafSV4#@EI9h_7+uEQMW)o z*AV|%sXL!>FgFDJ&2fjnp7E&L>Vj5+0XZ2wJ@a?hgi0;+-mK zVBu?fg7m49+EUyG)&-2MPban+X^DyFdbzs!k~4{|?h)ylfJ4x9k*IST0DXG_F+ z4O;_D6yuWBP+Ye{^{Fd?P9$ z!NS7evad3lbk|U;(f+{&DA_^MJEbpp!*#a8R%x{LdURc4BP~2z2@C$Vt1_ za07R2mAmZOBn~ogZiYmkaaX}E6#{)z2RkP_%MFuJ)19e+K)YMhQcLD#5@mJJ&Y*&A zI}!?^#|nM)1@BAX0aKl)fo>fa!Smf~^!88BibCKen%*`7-bcNd?957s1D1m`1txE^y69`V>WkS@YaQ$6m~zS8T)~uW zmFiN7JnGIM`E6v{ZmhN^AP$i!v1JdWsJcGn@1Q><#u|EznA@45lcSvi4A*lyom-Rq z%Qy6&7%R1{UcX)P)#(Rgxgi;T7OJ4QAkS9wV5sm^)I41>wT47b*(a4xj<-GOS8sx* z_2%oDD2QIOXS{riZfMN3cSsEc)_KaGUUh}*sZw)(9R*juHl}Nnb;WNzu?=Y7%{(zU z)WFkJ_oxQ~*$)zq{G-Oj5gjl|s!%%Vz{vaPoc!mAV&2D3%flnDF%l_OH)LgUf}>aH z?*d;{w8JgqtANA(Fz|eDD7k1eg3FVU{?Y%4;VC8*|8FDvSfT0`3#<4S@?ZL4Q|pa9 zmVflE0k=J%iLBS>Cy5kW3_n84{XNd?@ggLc zH66tD;R4naZeewwnpK*v3aNG%s*(lHO=(m2OPp`@1Z=M5y=Q3k;X%6sg^u!wzvreH zeU8wgLRjA%?rl}z?y`GWHG3ZV4l2pM#%|+OeGf+)vA;}pF>d-+g8pLFoY&_z7VjBX z_tpZFo{KMg#;nhT7j;Ek{JdifC>z0xIa^ujMot zcA{!|t4WG5S1+e{?8w9jzicH5DopYaEQyQ=whx6YCbh{$RA!j3a=J6j4)fS07MRne zTMzJhcK2JS-Hk+TgkwH1NN70bZ1;InK%U1*I50*Zd^rI}1mse>jQ8iaYKQ^{+4K>w zNyGi`E&(j2mT&AZM56mCaGvy3lr=Fpxc8@uz<=_uSQo)v+i9`8t0dtSb|Vd>HPwhk zkA;{Q@-Y?sonho_*3~P4CIdBhF%z3!{dB6&%>`^Vl;g}Il!VP*UDxS*GPmhB{R38) zdAlI(%7V4wmd^3t?-!h=bQ20NRa4hvi|@0!U~B2xkH75ZzEstGXHlBBfdmf`&>0+^efn?T-l_LV;czXXJ_ zqe4dij9@goy|+aN17u0Htw!JvYu{=KC8XqVwWpnITEH~@!DkgeSF&AKYD);ZTW42cQ6 z*`{Ffm!)=VM{t8|85Mu52jbDBeFhd(kY)Opxk@)>tjqoLHN4wWYB{N)ACl5}b1sd? zR8>B=-agp7&GvceTyck^^FFuN-tODk3G+$jY1>sBMi=l6y{Q=40;?Idx7X~_FNT-V zRxxayr1l174Ct`8zwfZKPEPk4?SjXJDF(GL3=PJIXM!VG7%Yu$t}Yhk9|h#j2ncT3 zwJuG|$TASh$eYu1Zw37N>;N59d>4?XRLh%kT7Sy%LOzo~a=3%mM2gXnY)6KPLMFO{ zF38$#Zis=W2rlXE>fwvWQa85sG&5xjJtsWWF<;Jjr4_Px&P8SU-W)v7`EEFg{~QIM zTXqq$9+b<(in?23+=VManyIxtil;CITS{gPkuhow!1Rvz$uj;5rJfr@>yg0M(MLt( z(kC*}gU>R;xZ)neJOWMtI&ghiZMXJ>WENOC3gxQ4=@?ukrtu!mN2xK9T=F|?Do6o0n;wyNeodAE7-)2#?Da`NblSU)S2ZS8Yyi0mgoWe3# zQEDBx-*J3mW0=AIP0XTpeg_*=BY!d~9%wOXHx7c^>bbGqsCH}^gz=Ju{UJ3Y?D&~# zzzX)RWtvmcg68eRUC0xCjdHW_^dOisCp}Zbij*)KA@wi($v5hev3X7;xK!ytK2DL! z`B@b0`c1b>DH;}4Voq1vn>N)@cZT~p*C_01RQ8;L99a~Exe}V&sk42-DM|H2mm!qY z+S>Z;m~2<|o0!sBc%*xZbfLrkvT9GvC*EJ$ufrhO8>5BT3a=hXgEYD_okRnF%7$*& zV{<9TK-#cW!`H_OlLCi;73jHqW_0({umoM_q;yiGpYf;q0799M6l8*AnigG&M=t{K zD}o*KgF(Fy7i3(uuBcd+dPYNB7u}#Qm}d;m@qbIA8zv@PV-3l8vye0%V73hb6XJx1 z#_rJ-y02}S!~4!1AAplw3-B|hdzOh@Ze=Z$4~g&IN*LV24aJ^8ea3PWVFLn7C$|rb zeEjv;ka&KlO5~qCRv@JF7P+a;?flJKna?b`o~W{T7Q4lYWH0?RQN$S8GEn(B#mTw1 zB%(k@dG&tiCcoTyU#mFiqL@lJD#XTUB=xMA`sOhabRtU@50TB`)dP*jLPp}ryw^Zu zz%F09y{Hx8r#(}tim6B7!5J%&-T7oFI|$Me5BDCTZ7`Q0Xuz~c{d3pH~j&IOo83jqtEhHpi5K`jgj|d^1eBzoaB$Xg@~bwg_5gt*Z0;gs-bd?{Oq-eyGu>Z zfpwIa?qarjYY_`1?bgkx&zDkhw-l3mp*@$sTaB3>UpQLjtnk$78%i-g0gfXMB9qd9 z$s$$z5d+jKCBE>3dclWhGqUfRWBEnI*>9+Khlk^EbYa;@z5tZBq=eBLtkwRhh>wrpF(aU0(JI z@dp|0UFiKXTl@6RzudW&OxV54Cw|qYiFM~DUff$f;FZNS0iUA!_5^jT^YuR+OYk-j zZgg2Pd2@GN_N%+bm;QI%%Y5kRHn;O{JMH}qtmonG|8#V|#>xtqKseiKz^B5y|8x$3 zo9~Iad;VL$e>@+g78AH9(6B>IE)F~vrHD%O<=qQ_q?x;OwRv#Thj6K5fw>yM^a3j= zLOEVK3$*Yu0Dl8?|FnC%0X{1@_#N$C^LNP%>$|(e!z|v!-`!XB$v-whKLp&-t=fC6 z*W?W^#ec1ZI!NnHeB|<9zP|Y}LbDfE9d|&hhLXeDQai~l+`$XIiY=(XYrJM)=miVr znE3jXl=fTY_uP-|c`)8w-CXG#XAVHp_w78fl{vv~B(}Up?|=rmTZX)RlAQJ;sQfj>7;S zw`VVQ;Lu$}bainYO9Zv9+vm4g4(P-~;!yB*FW9!>czaF!`3RfTwkgzJ%dNwv*;WTh zzhIJXQrobdtmyYeL7gw{qTW>!-hMt+WuuQc8B?y)zgi`QK_zZ$JT{Zeb<)MXuK3Rk z^?^<*-Fu^P4IP&SCvWeA2E^qBmCO7J)a;}=j`{t3foA#qXtsPTk*cCx(-Y92@AuJR zjm0`w&$}AtkXT(m1rC=knG^b+0 zp0xpo13QLBl*p0;fDO;nM~bdedmc)rPci4FtjdIany1zTruxNX#3^7q6Co%00BZhc zDbZ5tKamzYYA1w2Nc9fIf-q)F5IwKX;TbOo`?x0MUsDfB+}1~GGS~nXh`wYkF8Wn7 zt3}Rc^VCoUm7$yYxu$RL{9#~p!n5ixbNmRvOfR6JP~S?K-zW9>n&XanbB8N!*L~$SOO4MrI59p3 zPK)mU9lMIk3w!?8pyi=NsRG5^G$YAYiTMz96%aoPm~=-y&vpkn`Wni&ZGTtSJ9@~# zE6)FAcD;(gT7{|fC=Lt$+-6YK@?5a8jtlj7O-+qd!@H?p<+|gfymU>Msrt1vi^mEv zbiF8Tnrr(t=c}RDwKz@#KAzZNX)omCtB;2yB;F8Mcy7e&*VfT^32Cyhh3Dxj!g3WY zQWYzGGTfgI_dpz<*o*HvKEE{@iiMEVX8UrJj0#=CT$<*Blrp6NE^qBJB!)P5ZY%b= zwsJx4S0NoM6aKV)2-X>|Uy-V}!){PGVHiy0Y6B*ho%L#aNF(DPmxD!bBR7Emde8MI z@mym_*$ojz32+#(!PjGzFBjvoBE@S>TFX?MDWy|YeK4T-lU#uxa!()2423V%k9i>& zBN>@D%$=LUd`iX%Jh3C8oeb`-Ycz>vN@Q{SZ4#cG?)`#KYc5Wve73AF7g2k(wm+F% zf4jjJsZTtEE$oIza%wf7Wo4DB0NwNDvIWwZ7gpK+a2J;b2ccRRmUGc0}zb92X0 z5?pyCQHG5(J*DR{Ff>)5xptQ^Q5Lty)2G-C@ce!0;D3cb351FcR#Qca@AC0vdR8dL zR#%o>*KnGs1i0r2#P2F}*Rt84QQf@z`BDW$|sd*rkqks=o6Xw#({xsW&;bZpBop z!=TXJmk(Gea6@OWKWxP%w6JJL!a362v-4|-hl@o78aMTHDq6%QFZZ~0z3dLh#J8t1 z$tAq&r|<$+J8B=Tp=l~V0;`*F*aux=ntF;e5t_=gsF&L6yH<5G?$6a4l`1u}uGyUJ z%`Dah;r2TziH43_XM3g#nD{6O2dw6agbXW=ErD=-Fla{Qmx~f`zay7%_xuF z=UkH;fo`0pqPWvA?eSuk^ZLi@Y#tb{)rbBN#;=w{W2aVtWjUxu=zL9c-&LN1;+dJao_EKkbd8= z)t}=?8O5mTf(}6-yv}??Y1~g`LU8}u1SFO#2@icKje^dYr|$#4eshj|y+=R}@|Rfp z%e(Rn>yvNCPYu75K6w~=i%;|f?6x~ucmv7%sG?W%Q+yT8I|JtQ=#dE~HypQ#(Ma&a zAUOqND#~k=b#sGp6^1xA+P^AKGgJdJzN+-evYYB+V5kOgLNRv(?N!o@iWZ7^>S@N! z<5d}+Kd*n&zHObonaP;Hi?V5n9-uYN;UCY ztuSlsiy+$mqea_;+J_CY7}**^Kc!V^bwZgL9vR7L^}92-nBpF}6+G+Esna!A&?(M5 zj&u0H3VPY`14kq#dk}Q}19uUklZcCBDq&B93flASgFxPIaJxV7?{UOLmWJVSsQGjm zoKGVJ(E5HEl07)qK{}G^u&jKy^D0vko(qN8%n0&ZvoO^cIwzI$i?j>8|1+Si->`%~ z+`Q2-v=KOTdM!gybM|`6A;QSL^iO!f3V3^l69sI8hlcYMJ3VGV(a zE{f#hjG@}VL|QiOYp26)WuMvWLk`mCRzVlL66e0!4d=>pxp&3xrlWmz$Ib<5hE z%k;N)2X5>lo?~OZA@4hD^{*|u=&MN%=X*rm)6#TF2` zg6V(RqwKxYZb@^g)kYVDs zV5=v6R7!?JisnWB^#cxZXxn634lyc!@#}JQIulMKh-c*|HBl9aI)xaO&u}_0vw|3y z!Q%{~eG*rFyYuZ_x_DHv-inO+xfD~OCMV^Lrh$++BR@kis$d_T@LaadBh z_Z-|Hht30y#caE%TpY>7g|8%(e%I_*&h`F~h`W3?9^wU6;qbidEuVL}J)yVm{y9oU zFj~Bws1qs10SfSeq}p~L4!ZgZOR2XGw%Ww5_G}dpob2QmU)Ff}C!@JX95ih0^^L|_ z)Y3emdX9>xj4@vf=US)Zxh9_58Q8D*9a^H@+>ZW)i2*9bqu z8j(#wMI~*+q@wU})Tr{qZs$6+rc(Rdx+P^`9P{LrW{|eXf#XymVo- za+Mi{_&vb?qm=iqV(vl!_6lIDzqn0SVFw{rk8v_^)D;TC2QbbZ3-r~;T9)(+qf4%w>OSot4 z;=;=Ur$YUt$@h>i=5q0wNTI>I@u!}rme37P=VuRv>YaOA+P%uWMsI3b#4;B@(+nyv zW{M8`ibCUH(>-$gFx9yrHv1-Y!_CJ z3Sy~cstkrz_+iQWvbya{$A@d&>?-&F+I*x0KL!4Wi@^#OUYOtyjzfdc&MFq7TWKoN=+M0sYg24OZzc z<}q4C?z0uHxNxUF%GTW{W5*M++0ou?V!UA%4|4sEp5E@XoaJVPMX<&jOZj)!ljFQ< zo2A2ZHN$*U74eqGz_+u?iA1ZBOvstFgxaMQa)D~PrqSEQJ>1;&0L4Rw^Vce_U$ue( z8m$VW(l^(j!I9ZSB(NP(&qm3oJpAI_B zK0tH4v3+(L`<-#>^e#E|d6zv=+N}(_taw*jv57;oEV&V6Q7z}DA;OYE(2L=AbA|+h zEDx7@{7muRe^(sV;-~uH^zzuaNNYOlf}lX$;jMHLmm-_6(P079#~K1s&0%CBgV=c+ zeF;c3`A>Tf+)5Or+6R@pXI48x-y`_^2g1lEdlHNX4nrEm4TW(iuqpaPM8(BzigwGc z3$Qg$RdN2w>EBX_0}#zDZ|*dC*1h%*6|46n<_0>(6kG7%$fvSbOzEER9zCwNOjw}a z0k68cv*PbM!z#JaN>Vg!zNS3fW`ij+))C#TGl+{85`<1A#YII0*wclxcoceo6`krphosV#S5m~ zXf>MbpK6XQcQ0rea-ZgE)-r0_eIUe8(}6OkI=-v3Whjga@e;y)%#^_zkcZw+*8CvT zFH*J*8!P^(gyM%{-u1oZXDz>!~~)rLs~Bmi5I3oa)q%)s^{7gDHp?6iucfMdo(-Megy!;=I49MZH5Nf5Akr9eg_y3 z`FbUTheVc|mLI=d_aMd&6MTey@MOiTquH%(xKi+>5dX8mS58rm@WZwXowV%p+EMVO zwCo*(not6`vBi0u6c>PDZD9^l$F~Yq8s~E91XKOTEdd10^Ua@{k>IQ*mGgMCtY-MS zhC4je@7Ud76_&ediS{{tw|K7|(tN0T8WU!(hW9N6*{V zDB~27%~6vVbks7%7WEGA7t;%*RnU5kPN??g`>kTW2NO#;Te}=~9qAQ5*=tvR&O(q6 zvoA9JQWg_s6+AM55UJY#Zt}v9TO~fRB2u{OqQ>Nf;cQR}Wzo`kWCr zwmT#y68zC>KSaxe;hS&z@tMbker&~~3NF0&f##VyS+yHdcR>3Nkq6hx-X8sy5;lIl ztLG$<3P0bXb`sgx`TnE)%nec(qD1dDeC?=4vIYEcw0J`KK0ky4U_0i;9kl;-vF!Ih z0jk>nVZ8U?{huc=9#hiHA1Ny*elH+aUQF!K$x;TexeGz$AfalJ;5yRzz?Kw;>_a#G?lUui_r+&He-RUW;T(v`%6)xS!ecFe=$&EjolCM`a~oo9P~D$->qtB8YRs z`S4?j@t41f?H!baXK=^`Ifa17)+X2p*vJ6|1=WB1GNEuk6M<^i`?3#|*fy5-qyxee zY*)iU3(zGWh={`_wm^x#MAQEFs>$kjmqp;c&K$cUsGvoS21UrVNS!rtVJNt1F`rNO zj$sQ_+%Q1mJ|goKa_27HfdZ2%>6a?D4UcS z^F@JO5h~m3SHoW-I`WI04}Ng*%+TGHG49y0m?~)?Uc=j5WwAk=tvP`=LjCcnA`4|7 zSr#Gy%G#&iw5?_{yN)aEypo ziJ&|{FfM8N!f!ShB~lBVXwu!5>d$H`n2X8_JjRG$It^rcjHpazlH{3RUv}h+@0q$% zWZ*IpjbU~l0$065Ul>UDx~iJpoy@5oEQ2Yk=Z*7)!szddGDT#(beezkWU#467f~yW zr`Qo3i241+*|M20Ru;}x*E`e*b%~n{WN{a@U9tY*XEU zMAXgEcz3hu9(|LbcDb@!i#+R_zzyHWOZV}cI~^49?jA)?(~kfAo1;h+I&bnD`D!}N zv%j=a@5|-T3%nX3#?;lv&NaW=fp>!DM0WwD){(r&3s15~iaC&LR&Sb&PJW{5EtNDD z1wE2jtKRN;hcHO3z~7mI>iG=#P|7qUB*PlbMy~&<)v5<6E5!dv=*g0;yLqEbMBlGt zJKED2eF?q2u_9j8Ep}VSHNuJ8vkky8Qq#=EDmc*`BkFA&dP_2h8c_s zTI~xC(Sxdn2J=YL^);DiFt6RI++vTbjScIio9j1kO@-{>CKA$Hp*D@%i__Jhv8@bA z@3wlp^Iao-|IVxVSmh%pPNPEum$Ux0Yv;3utA3Y@?f&ht>ofdE!&5~_n|&lF1QA;H z^BhC}y!}PO#l9lyWgTKbe9Y@`&tc)PMUiT-rIK{XlU}G6r}u2-82Ss=R+%MBy>WR4 z^J?eUnFvstwu@WoQCC0zf+i+kDK;ez(!rzI6R*7Lpzp=}r5YGM=W>1-dVsUxwY29_ z?t0zX%Av-aA9KT(583K>ED>p%xivK6u1(0g z+uAlNVaL1*g7d#fiQ5Z*wUxOpyf(M1_BXIb)1l|Gm6F9IU-K!LvvY$=*hI-TEjwc+ z+h&$^s?!E0hT6F~eCFwSO!owvu5G;S>yRT-c$a@A3!lVXRv6ovIw-;U@Oe|F&Pr{$ z9bCp(+Jy#Cjosc48h3`BLg`b5dS8akWv8U@^e20j)${Q%C>$Q=>Um8Jjs3*d#bo&v zw!+8bH{qpTL_Q7eQP)30dO}Ghew;?WuR}?!(3m`HF}{J=Az( zD$x@CTl&U*>Bz4=SKoJM10`h#1r8P$yjfA@e-`67@(M>e&ULl+D4@p%rg2f(xH&un ztR~tyXj{oD$X2%!&n-57qtGx4d(a#j0-{=&CBA-p4Vh9%RaI9*{Bqj-fdk80m`WGD zl#6-!C;UF1y_gQHw9)_by|gXpg4yIWC)|2{85BWannNSj5>ClAx-a26EP#K3Xud&3 zpWp6MJXVu<=(=|HMV#%CeB58^``3yec1H$Db2fXzMyHoPJ|SOD%FQNYlzHfvDwK3wAbpG zrcOWfjD;p!96w~xtahG_5`Rg{R7uMUQp{~d9TI~=bD9)ogWMt4BVt=;I*ESx&OcOT zdM%ea!>A|ER$M>*jB@!T_sh>v-8!#HKs#O4AFfhVQe^a`#?1e++^WA;O?6@XB+nWc zW_ZK9l7{RKiQwHz>r*_a{?(zxynqjUO$~Hbc@W;((CEEOau9#pLoYwC-sF<+V~5zA zgT^;h_WG2#l{iTxay!@G(n#|8{amlb|H=Dx-*lff!}bB8ky)ZGF3%@06Y3mzbbd1G z1sO*AE7zb_!hQ`d6l&v*P#m)3(G>4_yPNv;@C-5l3HqG^LN4PM0}k<_;G&u3T0n4- zA1B{Lhxnt~l&M|AAMdc84i3pe12>qtB}*D7EpeJ7Q38{y({vbx?G z{qQrOY@F)S8UnML@@n{_D0rOumVNIv^chB#ZRSV>=hn;h79yyP9^3B6 zZ(PL+o+&jK7l|Z?nY8BwaDW#XeiHNI|E4Ome3O(A%B?+91~bi=0`@dzXX;!!Ip!-A z@n3%%@&C+^7qXo(_*5WMN} zf`!@+gvm$3&E7zrR-wMcZEOtuM>?|~o>WU%xbbu>h9~|tP6Q|LXSl^HZK1Ss_Mw_I z*l!xX@n3TwSAm?NjSy1AlUFCy#6=TO6Dnz1d#W~w3f}xOc@w;o6@6mGBvokqa*{Y3 z!w(CyDH=1EQ?!s}_$VP>;(ffKOqb;>mx5scT54$gU=8QrGSOFLolz4#K`qgJzAryC z*>e=?ICC63xs$jLWb*$Ldgd}Z4lHO}d!P1?!AOQeDI69D<6II>Cv0xoc^X>i+G=f0WW&aG3#e`m z5jyL_5$?;?I#X6lw`N|9R-L8*xhN=F!*6DVzZRKp=!f$(=$v%$o8maYL@^M$F3{<%tRgCrLDw{1`qSi|AKF6Tuv2 zgMAmVvNUH%9k)R@Z(Pbsn`58U#WDx;n0HjuP9VJDFxltk)0hCXn25tZF<8CO?l{lVc_?OJN5Ju zIUnD@;g1r#?&FMS{1;SH{yi?SW0k3=DtPU}Pgm==K46(vZO5oUA71(&n6^d)`VfVm6V3{`CmC1mh>!CsZ z>CI1%)$a+{m%%Mk?;zZV<_3hlb6Q$g=lf54@3KLEt<#VCCoSaVPKD{#utaaie$`a8%0`z9A-?5#LK$;4G z91(9>1`7Xo0YY!@G&xCcA+;NT_LDjtBE=V$4)6?AaJKdJ#IM^lL#=cEn~xLYn(NNA z;7Du!FOuVrO`~dAYif(CYA?(J$Ovm`wx@P--_?HnA9&$zcfbeVqXs0y|B^dq4f)D~=SD6-Dp8%pV_wVrKIdF$ z7f?a`mVj+#MJk<<0K@lIzT&p|!=1?#0|h{B2ZwTG@$Dr?6`-GWIb4=R42m55RjzY6 zaK=^(ipqxeByt*l6#)cOUPH0_b%4x90v+yla{*ldp!&vCUGKs5;=zR~8~#lPyrS4^ zfToYv=jQ5pg{N)fU|e~*iok-MuBNT`PC}W<(Qjq1-UAn)+Vq?`Gv5($tQO8XTgtBq zd^#Y-{Dy7#l4Lz%TP?@F4+K9@5UK<7b*o$5u8$r2Ty^>EAqcZGo(^eO{8Q1nCwYIA zDs7fv-e=|MHSzHz2VguQ+hyO~vD`?H1COQMA3uJ~-zm-jwVgr}4$3^BZfnBcL!8ao z)S=UO4r6U_-^}LG;uHC!;n?2-lKtf%`UYV8Zw%W;%ixsmp^;LWsE}s9s3>Jk7|qTm zAS-(_Db2hEgBEJt8UCFGP84Zp=$*1;xYs%F!)w1=07AQ-czE+gZLDQ%u`3Sqdy1_E z6x2U+NKLEz-u{!vr`A<6$WQbE5`lRR?<1@`Isa6?dhzoLoJTNuj)rw;^=uNMz5>TM ze4}rn(F^HGaeSwe!@9Ke#YY|S`V8pMSYW&uW3JbyQ?Y!2Do?oLr*;PQX}+*-E$Imh zD9Yle%M81;%R2!H&KHa4ch#f@MTP3QbfEf$7?#4jz|YQ*3c5^Va8HxVB2AZJdk7Se zXn>tS361Cxi`b%+uGU*0GoPvJxn(2oO2SYyjvM`}4*he}`9?$flrIwJEqL>QI?xT# zip5b)s4O~?TYjZ{d!oGP(OQW=XEr=%510p|)fbHw|Fdf$C=So`0d- z&?qwr;Se|;#_Di-w<&=v2lA~5(cBFyCxhYRPCGUAqhnvJ`J%1}oZXh*7&W|$G&Yv3 z1-wmBN;J2-s_yCA`7^AlE|^=LOk1zGS$`rff|p%ghY^2@D*&3cSj~8`Bj#Dz4EyM6 z<7eex^31|?xdww0*TN~rX`m*p>g~$z2MJE%1%<-v=D!rD`HEs@AOaN*jMqUvLx|N@krarD*XrN6h3Q+1W+Ka};9tkO`nQuFQ ziKKM#D`l&xBQ_fC*!$*sW0Z@38Jhk@5=exuzhUH{ZqqjCv-~`S?Nk7Vml!gNJ>CUG4H;ILJ8Ix=@h;96}dSG+hRs#804`DbB2f ztzWIy(`>1i>kYFSRPYDa--y?btKsP+q(++R2?x9Z(Te6MCdU_4*a0ei_bkRh(Lz5P zXcnt;IA|Q?Hto&y`u%kCHJ70oQiWrjW)6E$mi!A%yc7zy+QG{4D>6w}@mEY)pIhsh z+Fm5+jHY!~M5}A~Onxfy62$9*Yy=co4Z1H8bb<`H(U#5Xs`2z zdKk~JwHkt;^amQya^<|+2Mgr8^U9v9gx-`yV}U)jn+4Fq=zjumEA+^>@s|F%Y3{vs z<9FgU@6&w!8lZGl<&)mx--#7$Yn-PPz%9msaR>_}?PvAc6wMQ@zc7&SUkgs%K*6K2 zE{f*Wb4y_Afnc0)gStliomBJZ zqD#OI!lv`%jwfqHb+EVbvqe;}8Yr-&zn80LF)*c-@IST{Piu?voMN3l-#I)v#8bT; z|6vW$8Y#|rCWQ-Pbbx3<5})xlv{rzYzzgQ~BPV?J;Z}GVJREToHNmHDpy5}W^JI_F zPSY5IM6lo#1qvNo-*wtIq(}`!vvplS!Xw>N@tFfH;vi=){#x37^g+^{W0fJl36cH8 zLr;r+inOYdqCN4TeZsT~6-JV=!0pe-(tkfQM5>EXz5>#f703cJ2h5e1|X zkp>YZq#H>E1p@^{3CR&@hVE{VmPTR-Dd`-#M!Gv@hVB@8hUV*ln}symuRX7aR$9db;z4v>zKfd9q%4d-yBiSk~qH^2vp&90}`*-rC~ zQj&KO=r?Q!3Rytg^MJBQv%H-zRxdwVaHEauVJ}Xq*qxU?;+q%vHpi8#B@9!u4wCAe4u{B(Sjy2N-<{->?cD2jYpl#ZgW!3j#79b(vlr!>dI)AW`yDST|9gt*xP3!x zTN%S{-FNXA*F%nS`!l@l9}M+fgzjmQXT=D|c#+T}H>ioFKa_6w5e>z zy(>*>GdS@Ry^}_ZSr4>l!d^sIrJEvuogoj;wLQifv z*02|oON?DmeQOs0KYQTj$?bsaONl|zJXE$gqrN{GO|&#JVDpU-xyw?{&Pff@aS(UR z30Ke@bP08T^x8|NW-%W7%Ew;JpT}HNCTTyE>W+OKd1vp?5O%ov^v=Z2CvQYVf7+lr z#%Bzr5v|MO%uq>7BHMyEeVi8U&dK|x{E3$v;<9B)EB|gJhywhdWZ)#FURDOmrIaQy zK?HAd$DC6bn|9w1>sL&!eL`y<+oC80dOegZ)YU>$O-RW0(6n*q%#A=v(=jLU@e?V? zy#$wxTiYF+x$H&c9Uxvax$zifNn!^QP0i#$z}K>eotC9OVDJ<-L9oq#2L{DV3i*BUb6 zekxV@5gju&w{S#o1p!(I7Go3cUns-lb_tIeA8q zpGw&k|7ibM0RJk@9X_~Eg$=P$r6Hl`;`4-Qi+~))_cFh!3O#mkVcahj(teRw9c5~& zkdJ>52%VF8Q|*ydE4E^dA-8d^w7RfK*Ez8|=?%~*`&?qz*LApd{#?J@>iS!{7=U_W zdm7gSj*N$)!rv2523cEvmIM4)JBhfu@C>uFfQW6Hh|Kk=*VTEo@{7_5Dh-`Q zcW6NVLtulpGtQkPfWYMpC>JB-zScgTcXFK2e)cD^Uj7&MFIIR$3y4>&z%33>e z7R?jc(|`A%*Q1EGVFf4i3Sb5rdwH-23|ZN0V`H9XM_9~245@o z?%`o__oz^pLQ?qake>QfhNCmcv)%!H3boKZsaIe?lJ!sEVxIdcNco>;um-DcDkzUu z0jfk*Ub-((KP(?3W*jgT0Ln%z{;SGC#SBP>S)RQMltW@Q?0`W1Ga#7!fx;?@fsgOX zA^#)D8PwHAV#TEv9kCkxVHLy4!EXWaA17L7lj%#4-93|$S_feJD=#r)T3Q(Z(c?{v zx@`PtadM`WE>%BvZySK<6zBusc`gHn093aCo)floRs0V$ZqarA-(EnKA-H1vYCrL+ z)MJ%PQ3;@ky+d;_MY7L-0*Z=4oc|^IJK9|%!Pe^`Zmhy?NeeY_tD_^gg&*wc`%~5C z6MvrMNc?a0dU}CcUOfr@I0rRw))1`kKnNi}wgpk(CJ^=GL+1Ju;zXjRR|!@Ddek)( zADxc|#GuUX00s1k``RF7J6Nem+wE(5s`bHAhZh8>l&iNIv5WW)yV7Q-{iwXmE1-|S z;NjhW3~W-f&Mt>u$T(bI6t;MIG=%BAV^2%~ule3{UkocvE-omr(H$8bT>`T1$@KLK zqdpHe_4FEWo8Mh8I97BeG$a*k`C-VuH;z+=Z$W&r@b3RU@ibMnSSFV4p-lbr^K_0x?!0ja^1on{|Mi& zo(aipAvhtnR>RXZKa57-bqFL@uWD_1(Ep^Qu9{$F@YI6~o@l?@p%q8&=Q94^VzSfYN46QCVnwXy&oRrS>CGEiA$ld#w#BN-1BLi#Z`XWX^!m{BSc}5QaOKA!qf1xa zfa~|M+*kVrFV*;Iggxf%F-YeN$L8#|>9Uu1ETeC)9n{$P+$It+l+|qZsDrGtat$Tz zU!-#@L^33Ef1!3P)ou$BSj(W}?D#pe| zTUBJ{?rnaYW`Ks8_X7@=rhYzP*o4U%l2Z;S=sXJT7&|N^ksa2 z$&Z>}#}=*vr6Z^h;3IDH%d}0~{P^!@1UA&t(javp?f_0Ku?+`HypekHhU`0UyXZVm z*ui~T@Y)~33eL?V#vQl$28p0GC%3ifY#78+t|)?ol|+Uk|mO+QSoA z8jTy67n>^1LrFfPanP0Ll^0C&8jVJ4v}NuuuL6gH;#nSCFWP@Uv05iA?a2t2lHGxc z1a+CH?07!wk$wl;8rb3j1Qk(8N$n^SRd=>s&#L;{hs4`|QSu(V{?7NZI|j{M4^US6 zGfwmE?&Pw}i#4b95^qiB^5i!Dcu-Zp=R|qFHmYfI{ylSHvFF0g+#MU)D4a&!(t^Im z6`XN!;;uAAGA9gt4^QLry4jsp`Do9J7Gq5FAMEvuK@=9%_l|lx$+o*2olw=N`Yh+0 zprPlLCRyZl%v+XR@|O&OLrF-(42DG!+PH%geF?EYC!U#?iwCryBfP5JD$gq%3n7}y zf#9F}$yLs+D54{2IN8*_^O3n9r#c3E_otup@Xg&_Xb!|Y@{(x>A2>Stz1y7JLZ1fQ z4((lSTj;bIb4hLoBgkuPh2+jjEZ~D-gCr}OdM*tkE3{&tW+1^4p2eB3>Fg)p4&kpZ zc&Ljqk>8OcAmf35FrdEe_$gxgy_w@{&&om1Zz&?FGYV-(qa7zB?l0g@D3X&7TLit{ z#|$P|y+m0a*c;8lN6F;mHTah4Y++S_Lvv<_mZ+_|4=wE5cSc11_sjb}ZSYZUHhgXE z=^_&6J-bJEy&$OTZ_`Va;31=0=>Ye?l5IkMKI?Ny!!?acHuOyw)nIvp6zcJG{B#~v zytKyeqkb(d^i$?mwe)Ad>4GaClDy*lE;`-N)zDqO@zJ8cf+sE_b5LH%^Xc~94GMFx5dy|}wEYBL$u#RTF0 zLcr}YeQgd;O^l)U85X97*LTgXA-FC0gT>mJBrFiCFf=}$k5n{UZE|HgpGPN?l~4&s z3TIHXT;FFR;SZ)M+Pbz;wHgQ^xUm9}nb`@5cT!HLm>i|3w_@f8A2eDYyxSg<=&o@H zb&FJP=sM0F=W-e#Fi6+k!c}?!cjsTouK{!L`FGx(ZXiSi)1Uxt)U42bSs{+E1XNBm zlapL#Ng0n?dre3LAK(n7-m~ITga{rl#ru3c^)28aqnavKAxs8KaF3^GObEEw zjEU1gZT#q5Yge*bc6AMP=!-NpOlD>hx&)+D!K@HEVRYI5yE+Oc`k>36k60qBE!;F+ zOehX?gQ|JH>-Q}Cy(~ZDe1mnxXO^O+u`&e}IxVrsTthTW0m61UUHc25HX$w6;=<2` zKHXb9eX|dn0|Wv^H7gG}s;HwnE0KkAd)Csv3#+suZIwk4COx4Vo#bhSvNnuZ*)YfUiywJs3QBpjVMNh6Dip<4)%KOJ zPI)=`;(_}yJ7z8EAqec*h^Ef(wCc;@OMmMouzyDLL6kJ?=2M8Ou<4Z(_GT`D8GO zmr&WxIgzxrv*Agaad(^?X4OK&;Wdi>A+M3zbUx`zND$^(Us>VPqGj@Bf2T}WnGY>X z#eIMJirjBi>31hyO1cDpjUw7kD~QHSl^OOEQLXFe{HIus(AHBi1a(hLcdODfCpnr51!N?H(==cE>XmS4DkdRn(|)&0q7lD|CY{b{(Z{*GSPJbKzK z-WKqF!ZL`S0U$w)pycFFgL;JAoE=DT%B_!+Z?&eP^AS`od7yrih`=3LNY(qEzV%p} zgG;jTf-r~6ZhqFL#``opr>%)(?@Pg~T|C=mk?+3X*7(BpOC60DX-DY+7kR#sm6*=R zCW;$3{~Br*M^;g8%u;PI{W9tW=zM1;z~o0K;-%?;Rsc{-=SSZ?!27V3>6Z8vaPRt z@=fGs5hBkcF7T?@5*;pbA3-$hV{pzf2_2_$Gg5kldZw|JkfQ?Gw1=hWS;3Z&)mvWY|UKk3K zFJ(;i*!cRiNWW$7#zi=QIq=&sP)_9=_H!BG@_z@Vw>={v`%mgSL^>#B2~eLM*a>F& zC#9($>Zx@sX8yA``QwjO-B@bfzny9WctE(35&m7Y42Yc(UB>!knJIw?Qg$uW=pLRw z+@plBll~++;uRJy9mNv-|By-$-ZSgN&+ekY)5uxO{O7oGRG0Z&Mi7uhyz|L;N-6=tJx~E>uIe@<2r3?Fpi^nZ z!BL#2`1aRtn`UFk_hW2;y-D)8gKZ}O5(xy9|r;kq~dsIbfCJOMzbyfj@#?9 zC*x|{bNdA%Psq&u>BU;sQ10xfW{!Ri7LJ*~1<@7sSS#|JlNOf&%#9si9kt=gi0-cf zarnV{CkD0MEbCTJ4+uwK%#!$@ub?Q?--_fG(l;r0WZs;H z)MuOx0}_ppS%bOr#DAg zV)gE$VQg>CLMM;sZh^aFv;x-?(}_kXA? zTepEWn0bWtuIbU=IQsB7px;yJNz3|FN6ead?%`4GG?Wk9NhWnMAP!HKi-ftf?cX8} z=O?R412mXDU$w6@6?@Pku)pZ)6GllV7x^HITEN)E^Vfj9|G;PvDOgDKSQlX3jPv~q zBxAavb+an0^-jZ0f4m%`M$eq=wNJlP`;_~b1pH7DCk~bCc21>bAh*`8`2O~yJX0*6 zhQF1YEca`MD!_YU8pm^j97i>7DeR5?{Fi?yS97kR7J_+FkWaVujsc#USE*~P{=Hjv=KJ4pMB5E_g z==JJix{Y=F94WGJOKwkX*N>WA^b;YA3=n0ozJ5k>{YxsM;cWiCq)DMT{$WR2rYR(9 z15#l>SrysdXEj9sPk2uQ{N7tJ+Qq9^oc@Tb-AKotnEZzc&0{fJOIMq4c+68(GrJ~b z0+U6=j#pq{j3c#&ynHQf5Dah#Y6N{_Z8!Sq$S{9BeG89R!eD;)7Tj_HmknL{bC$3J z?^#*&bNY?eW>D|lu0_r)rSr8scOazwd#`LxA+Re6)54mR$|~7p+YEGSAP$=vVyk_a ziq!_c-lt+|WEW-sfcfLrZOg}Yyt(crVQdU^X74H3;nEXls+s{WjL}jNxsr>_i_f}m zTAY3X;bY`<4PA|zOkg#9BFk@6dodgGw&zzqo*v*v3NTJ~?paaY0k}=oTW+KD@!C!$ z&SHB~huU~e$`SroQ6|9qajw!Y_->R^wYNX(JHUGjJp2-h_Xes?Jz9!gr&}mwl%>l| zv1}RNTU#sBAw5){VU)AK_Ig(R0&YK_we~Ivq#1~4^Yjvf|N6*ZKI86bW4;Ar3iJyd z-G)vnPh_h78VfLP{Dxhv1m+Ib%Gr;vzDibYPpadc>}*=mNU7IX6aEA&1peuLCQz^ zrPruchR$U_zX}G3NwdQj4p06)2do|3<~Vwnj`)~5EcDC?*x1IDMuef7^6!?+SB-ks zR0|Z7i>bZ`JV*gI+Ht94o8J*~tO5C?mPwmc1%=aOp96s|sK-Ck+7TXZ&9V&t5g~xE zl68Z`n%p`(q?O59``M*9{`g;gFT#uUx1kqhcrK_az%Q37g>4vP-Rv$q^C%P;?^h3b zHaMR`3PonU1p@dFBhnwz;fbaSr$6*ss*z+P2K=2B|>k(6>GOiC@bSZ2B~kW=N1q9f%A|a^wQFrM_19)k%b&@< z1Dly$G`PAjf*1jV_&%~WU0%RZk|)F3+hKNRphN88_ z!L+-Q1csa4S_6`bjx6_U^BI9jP~!Pr*$5BL6BC8sBZy*w$Z3^-3lvKr4rk)xT5lUJ zuc)vzYRwQ>etKs)q(bWr63BQY0!D3{0JqT-xcCqd=$6N3*o#=V+6J{r{~d%Wp=uPz zII!IJ2eTe7?p@{FE)8bm(>Km+@7~HNRNcNLfY0Yd=3h=ySsTnvNT#e8I`nD}>4b1?D z ztbO}or!gK?=dZM>8ZBB&ND{y`bRqzU$>j95gMCCOaF<^yw_C70!qdlM@fY!7Rm;s7D|IkyfYC4L z^;0Mo${IKW{<5GgIoV(>#>|uz54aiL8v2`BJ#=B@FY%k#clz(}2kJ@LP0-Hc^hm6z zx9B;t}+a-DzNac(PN~-%Vmk$bn?S$GMj?nvfv6j$dAloqbAt`B56uDwv5@; zJ6$tE!(aGc^qT?$cbdD}^7*ZQxC#L6xlsX4Q$%of%gCu1>plNA!0`5oT?Cz?3=O@4 z3olIJJL_K_QMu{)K`KaBFJS_{jaNyc#a$^yC%tf1dJo@iB1Ok#8g1y+PHh=tu{seq zU93qc_$LE68#n?k{74&prV*#2^n22jD%)7Ja6L3d&Ji<=OdG*ELSAIH$ zB>00k(;W8WEE<4FYtGat{ntI{0E$iWHlzq0DNjPAfY4PQbsGq9z-Qfo0Iiq${}?I$ z{P(}^ZETE-BPC&YvPA&|I;&_OEcrBmye|4b7T-WF=>8qZH(&gh`ums&`>lVs`oOny?i}fpe%h7 z2gjVMS?$FRu@%UT<4coP$8Zy=+CNHV#&Qwz7XhyYCtzw%i9Yt|UfE)Yzn~^G^gm3d z0<;-O1FC)-5zQhW^MJ*i+STLLfS*D0&GiL5g2@GN$B`hg#2&;0ymO(+%neNmn!tSg zl>-V9A_X!g2#s4#q$H~Ld$C|ZXOj5+31zR(if5{ftLtgbb8iz~(YXahJ}LgF5XH3X zwGH*c%Ke;x^D|G-2_jP%3b;oZQ{DT*)TcXQi(U3l_GNzL;aH3J&G)x}pJqO*i#w|M z0`+~DAF$2N0)jzbj0kXizzP`8)Y<1<08Vp2<+BE0#ZHv$AC)G7V+1mBVD`$*ny3{b zsi>MZc;gFzM~+JXZ08S^ZYSm#z>Cp^1zBmW@7x1S$w}wLo|gk@7AKQ_!)y~WFR|e| z^*P=2i*XX*7rJGpU0=mYOqiVkyWzK?#^7&(trT$s3MoiqOTytHCY6xl}{sTQiwz;mo*!cqK>x`}ijYZjCxg9%rx`|H> zoj7>9ADgC;4qXIp7sKI`v7ECIkJ&ZpuSXWhscP^x(rr#2pRE4s0&}rxP~ibkr&&k2 zku`yREp+33VALw&Vrp0FG1*#*_E!W-9v|6XpoUnr=QD3GVEMxUtP(G93%-&_6X1OY zI+X%^7_9)5)(QYqtpNPAhe=mHC@TNK9-s%i##FZ0M%B=fNq;3t7=T_D!kaF?K#y_~ zb&o`tmCzDOV_;|lF7$Hmd-28di}d*??}KM-XYZj}%i65|dIE-f!J8xImL$yL-6Ztl z!;J3ZswG`*@~HMYd*nSK6E=4O)ifYE*c}c#+Erw-86rXL)tpGJoRcsv1qqX&hAUSR z;qGa(ZiyaIpGSozf_HbDRBH6M8^0U}Z%-CZEHAW?O1}|}8ao8*OZ|@8hTix{I@`Yw z_Wb4Cvg@>i+;*$BYFKHnvzoOXiz%ea6391(u>;=w`Rp>sr%hJd`4p()r&0CDnHehd(?wUI^)#@W)*n@^?f@Z=irkDM3{6-9e0^|Md;RidxEENEegfuAhWwY_{ zwCjM+TP{i?=I3_bgD*L$ar0FohgB*CI~8I~El+dTm$&ttv>{%TyBD#rASKJ=a4Dw> z`btm=z0;+G*kJ&zRd>GX7YuS}N$&!Yn;wi%bYhs}1AD)>m}`QAEzO*)_mJBq$e%s*9q4&4G-F0p zz0q6nE=KpZI;-(kHt11vFcRLUZ%h9%X)Bvc?hni5?r&&KJuqY2AYaunFngON`|1wG~ z5{wN!$cDv5k)J<_o0|tu9%m?QHyGy=f$;+YEf-n?(QyBJp@BxBdo8wErQW?x==(yk>fZ-J~DabBCcKVkZIQD{+Uy>kaU?-`0_f5CSSAMw74tnRKh zhN)a%9J(4&-0rGv;cvYaPK)}*70Ccn3~|epl$HsUW1^8;?u_7?UXCS_hbKfSeH24(><`QZ3!80{~#?HUoaT(#~0XgOG+oueL`k6e2Orxx-_>!_QAdV$enbB>J^gE$MllX`u!i1geRAlx}0 zzO@>NQAFmhmR99-NUKuB3wzATSjAY#fPT^2ub0M*w}WT~lMOrT$37;n4oa`50x|u( zm2TTC$rg9gr7PQ40M}9ocKb~i4M9Xw*$DF*H6=<>#zpm8HZd%KTt|qTUxwQw(k_|e zfxen)SwOTkFRUy4VXLO0Prz^rW#Rxql)te5dtqMt#Oq$3mYbJQ=WX&hfnI$#4Z%opZ2DRMy+Uk^zlo7WOZ z6W3ZZ~9^Gr>$0) z9QhB|-ZHkzY930nhy-Y3Q{OIFdj1~Jd5;Si1(kSRt3r=A^APC7 zSqWm07u0&d>^S_i-_dF%SY~08+w)IT%lr3Q~9*Y(MT)q?ERqqk<1f4aPmpFyvZ z*AX>|a2b^Jrwt}Mc{TL=%LFQClRXwF&-UmR*(`i(yS? zuRUKz36kU`P}!KogE2H26g*ds+(pfd4WvqR=*nQSR>I}t0COLb{tJCC1kcT=7$SXM zYIjP&JSbIsEJ9-U0rYU*Yhx@@*e#XbeUf$;WD|eQDuQxB912VdeF@boNY3u875XTP zO!HJ<4LW$3_2~hfI$MD0^5ImvZ6CAF=ruD$)fu@qdAY;HtV{9j+_t~id#|Rst{%7q z1Si&~1I2-D@!KhI7Ro3zE`0!lxyZ+wBF!1pK~L6ukd22kyfU%D2F2jq4&+JFKo`5I z`93~>SP*FDml*O7SR0X^2eF|3MI+Oo=Z}vHS9`B8k|83O=9l)-^z`VCTh*HLajh*@ z!qD@Og$`Fr67co{T|7~aBA@AJm|jUB-CY05qB>00SQ+uZ#D8DY{6%PX_! zt~}qfv)pRbO=ESt^ij6tC4M6R;ucx`0GZ5|8{c_?ey~dZR~ns3<^s@gv8TH_+5y#TVuHJ7(x0GI|^DWOsY;m-vkCSsU4T=20LA74U4O&bu$)#iYcS z)+X*X&6g@)?JA#TitC16Alet~u71+FBy~1djYcVrdAGiR8c)2FC9a83!Yv<0g}!HM zld?Sg{k@o^QCeHy9f|p5G?KIDt-!Pz(ea#G_cy2vD1sTp^I0ak}<$d{Gl@Nvxl!-gW%t9TW=dlo_Pk@jQ zm}hIMa&5PCfXX*hLk3jBy)L2FXN4u0fCI1R5C2>#FvcH6jd!l#_#;NVC@!}ypgdl=p9S6KkYDUU?%Jwt$sIlXk(N3n8Fe{v7}y2XN%$W9%-rq zrMWLO>o+JexpMx!w(|Gj^IcC-fqtyEBb5d|l_Z8$X0}hh&<im@6E*6+*qI^EKI2!R8laAnzroCBONCb5Uycy;Vwc`l zC#v$X(_cQCTcHJ1ZC zR(-B{>;=IE8kvn2dwQ6qAV(Cn^L*NG0+w&5T4p88De8p1@^zuUEKLGV!ma)s2`v(f zN@7bDVTdyC)37HwB*c)#LQ?0{z;g=gVN`@R-Q`F(`>nK+Or}l=q--_&vu(Ex>TeHf z6wm>u+4XV)Re4%4B%r41nMV>ZsY<)W%hR2eG0IouPVUFF*|rP4bwI*O}G~OIGkyiS90ZN4$4pe0*GQvdDtJJJF687_Cqm zd{G|Z*lRjPn=pg!;Hq>vSR$3TftSB{Uf{1YWi*Ayt?iYlWA~#0=&i)?45m4ZzD1o( z={plq^D2FsMMH0xX|;@m+wDWom&&CFP^F?Ro9I%nKIzjjV4mLWB^~lGaw=`REqv!&u>s=OHm>ojr9i(YwwzqQP~-&bezQI5-z_Gu?zb0E6e^tM@KyA+T7u zSOb~Vc^|rJ%1NZ*i0QIG|6_JseRbDd6UuKnw?X#QU-k{~ywL$&RZ{FsxNGxv2j_mo z5inPURgz!=iBDGRYp4K7Sf|l*Lb>$=rA=#z@`tq2i|DV$yXalJuCSOF%PzwiFPm|~ z7Y#_f4d{dJ!jnw%=WP|?&OkOr7PZVQ6&=q$JZ(ZxVjIb#FNv=(0wa@6-P!bjT>)0} zCvek7LvdOL=ZP;G)y&S$q3puk?37)a&kZ^9cDr16zmkj!xV<6kv%L7`U~oL`9l~GB zq46uTLqOdrTzG9fzBqWDNz5u$%-qLQ=5mqjLgH@-&Kd(Hoph_~^A=j^fF96sH1+zA zZMAXgR~nb_(`96!kg@}FDGLKSauLp+7uDB;A;rb;x z=6XLjj8yiAq}!XhOQ@wfP3}hqeXX1JFo5j6q|PcFz6B92-tF>qdEYOIW9&d<$x1RB zsa>8mV388f9?s}~8g=NRAUE`SqYZ}+wI=<~l3(;#w$f8B0MhcYUp;4erVs|w zdxpCVdo;BFYkF!cuyG~`n5vyX_iX%xr(yhaaSC`bf+37tYhI zC>&|GdOl*w*{i|)xBpr;sevtf#sjIFY`PA<^e*Fz5Zi3#%I3P=@DW_t#qm+Sz<9RJ zdcD|{sKZ*hKc2hwu-L~W_apz7&*&W{c{dT9Tkj1fMNyuRzi|$CN8548uH^h?` zlC-xRa1u4+`kS+*Po>UUV)bLoFw-){D7saJC{#MB(g+RE^?DBC7)&|SJxpna5P}BI?<x_iXB>@56J9rL*Y`&isriaHV{OqsX=)sZvPKt)AdZ ziJ*v0l~H;5JfNh`V~~ZBn0c=to61^TCYy!7Jzy0YKHFSbH{pys=joSB)Q&oSPV zdhbP^!L6UTSJBFN+^9Xv#{U)-F_e^g zTyw?zuXqEbV0INsbo*e^&bxSz&CKQ*9fO1oqb>zXb`WL(K~Gp*Y2K5? zFI3f#)rM?!YIE)dl1HPMT3Vo%yvI3@aO&JR^R{l>ImvSM7>=r8+V0~=mYtyXPl@S7 z`x(caeHs@0YW5-nuzA?45f7)8JIr{_OK0o#uE3oh+(sf}GaT{DOQ%9{DuO9QTIN`4 z1UFFC|5g`Ci@8RnvSIsEYrh6y6^X;dGp)UWi~M#W+nH|5MH@T9ku<`zX7PmIn)qTy zj0@e?49->s&kufIrIpuAB6(l~j8<_cquj;|8pI`&T7a5(3dM%irW%+DWUg~Irqkew z6JU&H@T6Olk^rmZ`KTk6HJu=J0fQaTZxzT|LHT~o{zW!T4Jb(!@6f;i7f(C`5hrrl z{7#r{a^|#}BG$uXSwvp3VNCY;rP#@A;T8JR#+3PLYkHH_kNUq{_ z$Y=)OJCay?vAc^JxN=AQx^hRBPBv}UTmuOOG9T143HyF;^(*t>`ENk|bs!&9XHine z_Ld^cL6cQF(RP?tB89zcF_U9*dXt#uHjQpwc$)bgTmt!!3L@2a3A@PEYuh99DUSso z$+ANc=(mnSRD{_4{(Rf-&8c^rUpUmEOtUTX5n#F{wg{$GwkXDRJoMh)9M(?YU>UAZ zMVS=`Jd#hBd!%_#WK+d9ecqR9P|4Hj=laT@dFW+6z=G^TI!Z|G zQI|lithJSMO}!7zQZS|irH=YjHO0=`&&e32HqcNycEB=k6}K2qGu3#rhDWc)b4e;m z=uuA@2!;UW03iWs+$^tph#&w~#l=~JwE+*e*{zzJHBQXsDEkqew56;~-_6I#`ot0% znG1KcP2LB{V&pD-KFy~6UbFMNkWU6dKYZIpCxRRb4&xSM>BHY|DPVD~d_4e(it&Gc zy|%lp^WMI4+jE{3b%M`qJM@8RSND*`_`yWzqJ6%JveD`P%i&~tl59=GUNP<7|9NmNEN{#BjbPyV2tVKOOsU5ENx=eW=E0FrIPa3 zK%oSnYS>AwB6m9PDiF=@hFH>A{i-Kq5320s07ll0xL_FvA$WIKGsbc8_m@8$7FR*p zi5d+7KudX20v}ht8xCM|)c{`x0QeL_6irxvheN{eH+Lut;P5G)Que14bTJB3{-Zo^0{USZV&Umz(`aoDYLY`>ermOKAK_6& zS(%RBqsAwe>z6cyG}5hS0EKD|`XOCvCa3`9KmntICiwha9Y?z{xN}&A)zCeYWd@7MqJ!<|)i(a@0OkjNuEv^B zyO@VT09_C6b3{v{WgK9TYVc%LLe@2D1Vrg*V1(6%4W(Gz?(G#=v4C^$VYvpb#3!_q zvaSJ+3|84ii;lp&!42Tn1b|6Ke?=JGRc7Ar!ng{Ys7wkv_RLI5e?>wvB9J!?dnSm& zaPBEv`_@9^(cqCi>+xg&D4_MAv->A&Y)+&Kneq6`TEL?TuqeVpsgA@XUKZ0r9rKM2 z3Bw#4&)z5Up5TSY3oz3t(Ad)OoJoDb!=s!_0{m4=%e<2MaUak*dsN2ZoeU!osYP|9 zeE?AU2LNkKk>kS6>@jcYOP6-HHVzQ7N)WY zXHR-=pDRnfeg{vYb-yj;JKS>GCcIKOf0@6Eht!i2*qHRc+m_DH`lD&%a$Pruk6D4* z@UO_xcmCm*8bKg7G6i!(exIT}x6r^n6MmX*W?bCtXMLyTI5ZV?KJyhp93 z?W~<5wpR2TJ{E>4t|8M2eXu|QDz^yD8tV}3)ujF>RekkwKgfXQGn~Iayg7(MExap_ zOHm!I2^U4yuwtVLT90$rQ3hy*a^dq&2;;>fLX>17Yyw~`{+2z_K9}> zmvVdc_t`jrKOX`R`#)ToKotWF`v!K0-5>yS|L4R1PM7@myw@xKziv|hH?IAE22uY1 feAPbn;s%GdDLk^E#FOwA@bB&GkFQGP4E+8dSR3^& literal 0 HcmV?d00001 diff --git a/doc/source/dist_cloud/kubernetes/index-dist-cloud-kub-95bef233eef0.rst b/doc/source/dist_cloud/kubernetes/index-dist-cloud-kub-95bef233eef0.rst index b44a68f1e..8cb05739d 100644 --- a/doc/source/dist_cloud/kubernetes/index-dist-cloud-kub-95bef233eef0.rst +++ b/doc/source/dist_cloud/kubernetes/index-dist-cloud-kub-95bef233eef0.rst @@ -175,6 +175,16 @@ Upgrade Orchestration for Distributed Cloud SubClouds failure-prior-to-the-installation-of-n-plus-1-load-on-a-subcloud failure-during-the-installation-or-data-migration-of-n-plus-1-load-on-a-subcloud +-------------------------------------------------- +Distributed Cloud System Controller GEO Redundancy +-------------------------------------------------- + +.. toctree:: + :maxdepth: 1 + + overview-of-distributed-cloud-geo-redundancy + configure-distributed-cloud-system-controller-geo-redundancy-e3a31d6bf662 + -------- Appendix -------- diff --git a/doc/source/dist_cloud/kubernetes/overview-of-distributed-cloud-geo-redundancy.rst b/doc/source/dist_cloud/kubernetes/overview-of-distributed-cloud-geo-redundancy.rst new file mode 100644 index 000000000..f0d6cd013 --- /dev/null +++ b/doc/source/dist_cloud/kubernetes/overview-of-distributed-cloud-geo-redundancy.rst @@ -0,0 +1,118 @@ + +.. eho1558617205547 +.. _overview-of-distributed-cloud-geo-redundancy: + +============================================ +Overview of Distributed Cloud GEO Redundancy +============================================ + +|prod-long| |prod-dc-geo-red| configuration supports the ability to recover from +a catastrophic event that requires subclouds to be rehomed away from the failed +system controller site to the available site(s) which have enough spare capacity. +This way, even if the failed site cannot be restored in short time, the subclouds +can still be rehomed to available peer system controller(s) for centralized +management. + +In this configuration, the following items are addressed: + +* 1+1 GEO redundancy + + - Active-Active redundancy model + - Total number of subcloud should not exceed 1K + +* Automated operations + + - Synchronization and liveness check between peer systems + - Alarm generation if peer system controller is down + +* Manual operations + + - Batch rehoming from alive peer system controller + +--------------------------------------------- +Distributed Cloud GEO Redundancy Architecture +--------------------------------------------- + +1+1 Distributed Cloud GEO Redundancy Architecture consists of two local high +availability Distributed Cloud clusters. They are the mutual peers that form a +protection group illustrated in the figure below: + +.. image:: figures/dcg1695034653874.png + +The architecture features a synchronized distributed control plane for +geographic redundancy, where system peer instance is created in each local +Distributed Cloud cluster pointing to each other via keystone endpoints to +form a system protection group. + +If the administrator wants the peer site to take over the subclouds where local +system controller is in failure state, |SPG| needs to be created and subclouds +need to be assigned to it. Then, a Peer Group Association needs to be created +to link the system peer and |SPG| together. The |SPG| information and the +subclouds in it will be synchronized to the peer site via the endpoint information +stored in system peer instance. + +The peer sites do health checks via the endpoint information stored in the system peer +instance. If the local site detects that the peer site is not reachable, +it will raise an alarm to alert the administrator. + +If the failed site cannot be restored quickly, the administrator needs to +initiate batch subcloud migration by performing migration on the |SPG| from the +healthy peer of the failed site. + +When the failed site has been restored and is ready for service, administrator can +initiate the batch subcloud migration from the restored site to migrate back +all the subclouds in the |SPG| for geographic proximity. + +**Protection Group** A group of peer sites, which is configured to monitor each +other and decide how to take over the subclouds (based on predefined |SPG|) if +any peer in the group fails. + +**System Peer** +A logic entity, which is created in a system controller site. System controller +site uses the information (keystone endpoint, credential) stored in the system +peer for the health check and data synchronization. + +**Subcloud Secondary Deploy State** +This is a newly introduced state for a subcloud. If a subcloud is in the secondary +deploy state, the subcloud instance is only a placeholder holding the configuration +parameters, which can be used to migrate the corresponding subcloud from the peer +site. After rehoming, the subcloud's state will be changed from secondary to complete, +and is managed by the local site. The subcloud instance on the peer site is changed to secondary. + +**Subcloud Peer Group** +Group of locally managed subclouds, which is supposed to be duplicated into a +peer site as secondary subclouds. The |SPG| instance will also be created in +peer site and it will contain all the secondary subclouds just duplicated. + +Multiple |SPGs| are supported and the membership of the |SPG| is decided by +administrator. This way, administrator can divide local subclouds into different groups. + +|SPG| can be used to initiate subcloud batch migration. For example, when the +peer site has been detected to be down, and the local site is supposed to take +over the management of the subclouds in failed peer site, administrator can +perform |SPG| migration to migrate all the subclouds in the |SPG| to the local +site for centralized management. + +**Subcloud Peer Group Priority** +The priority is an attribute of |SPG| instance, and the |SPG| is designed to be +synchronized to each peer sites in the protection group with different priority +value. + +In a Protection Group, there can be multiple System Peers. The site which owns +the |SPG| with the highest priority (smallest value) is the +leader site, which needs to initiate the batch migration to take over the +subclouds grouped by the |SPG|. + +**Subcloud Peer Group and System Peer Association** +Association refers to the binding relationship between |SPG| and system peer. +When the association between a |SPG| and system peer is created on the local site, +the |SPG| and the subclouds in the group will be duplicated to the peer site to +which the system peer in this association is pointing. This way, when the local +site is down, the peer site has enough information to initiate the |SPG| based batch +migration to take over the centralized management for subclouds previously +managed by the failed site. + +One system peer can be associated with multiple |SPGs|. One |SPG| can be associated +with multiple system peers, with priority specified. This priority is used to +decide which |SPG| has the higher priority to take over the subclouds when batch migration +should be performed. diff --git a/doc/source/dist_cloud/kubernetes/rehoming-a-subcloud.rst b/doc/source/dist_cloud/kubernetes/rehoming-a-subcloud.rst index f3f97191d..6bfc56f26 100644 --- a/doc/source/dist_cloud/kubernetes/rehoming-a-subcloud.rst +++ b/doc/source/dist_cloud/kubernetes/rehoming-a-subcloud.rst @@ -17,6 +17,12 @@ controller using the rehoming playbook. The rehoming playbook does not work with freshly installed/bootstrapped subclouds. +.. note:: + + Manual rehoming is not possible if a subcloud is included in an |SPG|. + Use the :command:`dcmanager subcloud-peer-group migrate` command for automatic + rehoming. To get more information, see :ref:`configure-distributed-cloud-system-controller-geo-redundancy-e3a31d6bf662`. + .. note:: The system time should be accurately configured on the system controllers @@ -27,7 +33,7 @@ controller using the rehoming playbook. Do not rehome a subcloud if the RECONCILED status on the system resource or any host resource of the subcloud is FALSE. To check the RECONCILED status, run the :command:`kubectl -n deployment get system` and :command:`kubectl -n deployment get hosts` commands. - + Use the following procedure to enable subcloud rehoming and to update the new subcloud configuration (networking parameters, passwords, etc.) to be compatible with the new system controller.