Merge "Doc: Add doc for replication in OpenStack"
This commit is contained in:
Zuul
@@ -55,6 +55,7 @@ Amazon EC2 Elastic Block Storage (EBS) offering.
|
||||
default-volume-types
|
||||
api-configuration
|
||||
upgrades
|
||||
replication-in-openstack
|
||||
|
||||
.. _`Storage Decisions`: https://docs.openstack.org/arch-design/design-storage/design-storage-arch.html
|
||||
.. _`OpenStack Operations Guide`: https://wiki.openstack.org/wiki/OpsGuide
|
||||
|
||||
391
doc/source/admin/replication-in-openstack.rst
Normal file
391
doc/source/admin/replication-in-openstack.rst
Normal file
@@ -0,0 +1,391 @@
|
||||
========================
|
||||
Replication in OpenStack
|
||||
========================
|
||||
|
||||
Replication provides a Disaster Recovery (DR) solution for mission-critical
|
||||
workloads.
|
||||
This guide will provide you step by step procedure on how to configure/utilize
|
||||
the Cinder Replication feature in your own deployment. There are two parts
|
||||
to the feature, Cinder side and Driver side. The Cinder side steps should be
|
||||
common, however, the driver side steps may differ. This guide will use RBD
|
||||
as the reference driver for the procedure.
|
||||
|
||||
Prerequisites
|
||||
-------------
|
||||
|
||||
- Should have 2 backend clusters
|
||||
- Cinder driver should support replication
|
||||
|
||||
See :doc:`../reference/support-matrix` to know about which backends support
|
||||
replication.
|
||||
|
||||
Enable Replication
|
||||
------------------
|
||||
|
||||
CEPH
|
||||
^^^^
|
||||
|
||||
Reference: https://docs.ceph.com/en/latest/rbd/rbd-mirroring
|
||||
|
||||
*NOTE*: These steps are Ceph specific and are tested against Pacific release
|
||||
of Ceph. Make sure that:
|
||||
|
||||
- A pool with the same name exists on both storage clusters.
|
||||
- A pool contains journal-enabled images you want to mirror.
|
||||
|
||||
STEPS
|
||||
"""""
|
||||
|
||||
* Get shell access for primary and secondary ceph clusters
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
site-a # sudo cephadm shell --fsid <PRIMARY_FSID> -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
|
||||
site-b # sudo cephadm shell --fsid <SECONDARY_FSID> -c /etc/ceph2/ceph.conf -k /etc/ceph2/ceph.client.admin.keyring
|
||||
|
||||
* Enable RBD mirroring on both hosts
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
site-a # ceph orch apply rbd-mirror --placement=<Primary Host>
|
||||
site-b # ceph orch apply rbd-mirror --placement=<Secondary Host>
|
||||
|
||||
* Enable image level mirroring
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
site-a # rbd mirror pool enable volumes image
|
||||
site-b # rbd mirror pool enable volumes image
|
||||
|
||||
* Bootstrap Peers
|
||||
|
||||
*NOTE*: These commands needs to be executed outside the cephadm shell.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
site-a # sudo cephadm shell --fsid <PRIMARY_FSID> -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring -- rbd mirror pool peer bootstrap create --site-name <FSID of site-a> <pool_name> | awk 'END{print}' > "$HOME/token_file"
|
||||
site-b # sudo cephadm shell --fsid <SECONDARY_FSID> -c /etc/ceph2/ceph.conf -k /etc/ceph2/ceph.client.admin.keyring -- rbd mirror pool peer bootstrap import --site-name <FSID of site-b> <pool_name> - < "$HOME/token_file"
|
||||
|
||||
Verification
|
||||
""""""""""""
|
||||
|
||||
Verify that **Mode: image** and **Direction: rx-tx** are set in the below
|
||||
output.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
site-a # rbd mirror pool info volumes
|
||||
|
||||
Mode: image
|
||||
Site Name: 55b6325e-e6b3-4b7c-91fd-64b5720c1685
|
||||
|
||||
Peer Sites:
|
||||
|
||||
UUID: 544777e2-4418-4dba-8f10-03238f63990d
|
||||
Name: 69cc3310-8dd4-4656-a75b-64d4890b0ca6
|
||||
Mirror UUID:
|
||||
Direction: rx-tx
|
||||
Client: client.rbd-mirror-peer
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
site-b # rbd mirror pool info volumes
|
||||
|
||||
Mode: image
|
||||
Site Name: 69cc3310-8dd4-4656-a75b-64d4890b0ca6
|
||||
|
||||
Peer Sites:
|
||||
|
||||
UUID: a102dd15-cc37-4df6-acf1-266ec0248a37
|
||||
Name: 55b6325e-e6b3-4b7c-91fd-64b5720c1685
|
||||
Mirror UUID:
|
||||
Direction: rx-tx
|
||||
Client: client.rbd-mirror-peer
|
||||
|
||||
CINDER
|
||||
^^^^^^
|
||||
|
||||
STEPS
|
||||
"""""
|
||||
|
||||
* Set the ``replication_device`` values in ``cinder.conf`` file.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
replication_device = backend_id:<unique_identifier>,conf:<ceph.conf path for site-b>,user:<user for site-b>,secret_uuid: <libvirt secret UUID>
|
||||
|
||||
* Create a replicated volume type. Note that we've used the
|
||||
``volume_backend_name=ceph`` here which can be different for your
|
||||
deployment.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
openstack volume type create --property replication_enabled='<is> True' --property volume_backend_name='ceph' ceph
|
||||
|
||||
Verification
|
||||
""""""""""""
|
||||
|
||||
- Create a volume with replicated volume type
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
openstack volume create --type ceph --size 1 replicated-volume
|
||||
|
||||
- Confirm on RBD side that a replica is created
|
||||
|
||||
On site-a, you will see **mirroring primary: true**
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
site-a # rbd info volumes/volume-d217e292-0a98-4572-ae68-a4c40b73a278
|
||||
|
||||
rbd image 'volume-d217e292-0a98-4572-ae68-a4c40b73a278':
|
||||
size 1 GiB in 256 objects
|
||||
order 22 (4 MiB objects)
|
||||
snapshot_count: 0
|
||||
id: a9ebeef62570
|
||||
block_name_prefix: rbd_data.a9ebeef62570
|
||||
format: 2
|
||||
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling
|
||||
op_features:
|
||||
flags:
|
||||
create_timestamp: Thu May 15 14:15:04 2025
|
||||
access_timestamp: Thu May 15 14:15:04 2025
|
||||
modify_timestamp: Thu May 15 14:15:04 2025
|
||||
journal: a9ebeef62570
|
||||
mirroring state: enabled
|
||||
mirroring mode: journal
|
||||
mirroring global id: e8f583ed-abab-489c-b9d5-ef68c0a1b56f
|
||||
mirroring primary: true
|
||||
|
||||
On site-b, you will see **mirroring primary: false**
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
rbd ls volumes
|
||||
volume-d217e292-0a98-4572-ae68-a4c40b73a278
|
||||
|
||||
rbd info volumes/volume-d217e292-0a98-4572-ae68-a4c40b73a278
|
||||
rbd image 'volume-d217e292-0a98-4572-ae68-a4c40b73a278':
|
||||
size 1 GiB in 256 objects
|
||||
order 22 (4 MiB objects)
|
||||
snapshot_count: 0
|
||||
id: 6a993924cde
|
||||
block_name_prefix: rbd_data.6a993924cde
|
||||
format: 2
|
||||
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, journaling
|
||||
op_features:
|
||||
flags:
|
||||
create_timestamp: Thu May 15 14:15:06 2025
|
||||
access_timestamp: Thu May 15 14:15:06 2025
|
||||
modify_timestamp: Thu May 15 14:15:06 2025
|
||||
journal: 6a993924cde
|
||||
mirroring state: enabled
|
||||
mirroring mode: journal
|
||||
mirroring global id: e8f583ed-abab-489c-b9d5-ef68c0a1b56f
|
||||
mirroring primary: false
|
||||
|
||||
|
||||
Failover of a Boot From Volume (BFV) Server
|
||||
-------------------------------------------
|
||||
|
||||
* Create a bootable replicated volume
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
openstack volume create --type ceph --image <Image-UUID> --size 1 test-bootable-replicated
|
||||
|
||||
* Launch a server from the volume
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
openstack server create --flavor c1 --nic=none --volume <Volume-UUID> test-bfv-server
|
||||
|
||||
* Create a file to write data to the VM disk
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ cat > failover-dr <<EOF
|
||||
> # Before failover
|
||||
> this should be consistent before/after failover
|
||||
> EOF
|
||||
|
||||
* Failover the replicated cinder backend
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
cinder failover-host <host>@<backend>
|
||||
|
||||
* Shelve/unshelve the server. (This is required to remove the connection
|
||||
from the volume in primary backend and create a new connection to the volume
|
||||
replica in secondary backend)
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
openstack server shelve <server-UUID>
|
||||
openstack server unshelve <server-UUID>
|
||||
|
||||
Verification
|
||||
^^^^^^^^^^^^
|
||||
|
||||
* Verify that the connection is now made from secondary cluster
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# In cinder-volume logs, we can see the ``hosts``, ``cluster_name`` and ``auth_username`` fields will point to secondary cluster
|
||||
Connection info returned from driver {'name': 'volumes/volume-e310359c-6587-4454-9a9c-a590b50dd4a5', 'hosts': ['127.0.0.1'], 'ports': ['6789'], 'cluster_name': 'ceph2', 'auth_enabled': True, 'auth_username': 'cinder2', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': 'e310359c-6587-4454-9a9c-a590b50dd4a5', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': 'b691cd50-83a1-4484-8081-7120a5cad054', 'enforce_multipath': True}
|
||||
|
||||
* Confirm that the data written before failover is persistent.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ cat failover-dr
|
||||
# Before failover
|
||||
this should be consistent before/after failover
|
||||
|
||||
Failback of a Boot From Volume (BFV) Server
|
||||
-------------------------------------------
|
||||
|
||||
* Create a file and write data to the VM disk. (Note that the volume backend
|
||||
is in failover mode and we are writing to the replica disk in secondary
|
||||
backend.)
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ cat > failover-dr <<EOF
|
||||
> # Before Failback
|
||||
> this should be consistent before/after failback
|
||||
> EOF
|
||||
|
||||
* Failback to primary backend
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
cinder failover-host <host>@<backend> --backend_id default
|
||||
|
||||
* Shelve/Unshelve the server (This is required to remove the connection
|
||||
from the replica volume in secondary backend and create a new connection to
|
||||
the original volume in primary backend)
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
openstack server shelve <server UUID>
|
||||
openstack server unshelve <server UUID>
|
||||
|
||||
Verification
|
||||
^^^^^^^^^^^^
|
||||
|
||||
* Verify that the connection is now made from primary cluster
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# In cinder-volume logs, we can see the ``hosts``, ``cluster_name`` and ``auth_username`` fields will point to primary cluster
|
||||
Connection info returned from driver {'name': 'volumes/volume-e310359c-6587-4454-9a9c-a590b50dd4a5', 'hosts': ['10.0.79.218'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': 'e310359c-6587-4454-9a9c-a590b50dd4a5', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': '2c8bb96b-5d5c-444c-aba5-13272b673b34', 'enforce_multipath': True}
|
||||
|
||||
* Confirm that the data written before failback is persistent.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ cat failback-dr
|
||||
# Before Failback
|
||||
this should be consistent before/after failback
|
||||
|
||||
Failover of a External Data Volume
|
||||
----------------------------------
|
||||
|
||||
* Create a test server
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
openstack server create --flavor c1 --nic=none --image <Image UUID> test-server
|
||||
|
||||
* Create and attach data volume to it
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
openstack volume create --type ceph --size 1 replicated-vol
|
||||
openstack server add volume <Server UUID> <Volume UUID>
|
||||
|
||||
* Write data to the volume. Note that creating a filesystem and mounting the
|
||||
device are implied here.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ cat > failover-dr <<EOF
|
||||
> # Before failover
|
||||
> this should be consistent before/after failover
|
||||
> EOF
|
||||
|
||||
* Detach and attach the external data volume
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
openstack server remove volume <Server UUID> <Volume UUID>
|
||||
openstack server add volume <Server UUID> <Volume UUID>
|
||||
|
||||
Verification
|
||||
^^^^^^^^^^^^
|
||||
|
||||
* Verify that the connection is now made from secondary cluster
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# In cinder-volume logs, we can see the ``hosts``, ``cluster_name`` and ``auth_username`` fields will point to secondary cluster
|
||||
Connection info returned from driver {'name': 'volumes/volume-437573fd-08e2-42c9-b658-2f982bc0cdd2', 'hosts': ['127.0.0.1'], 'ports': ['6789'], 'cluster_name': 'ceph2', 'auth_enabled': True, 'auth_username': 'cinder2', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': '437573fd-08e2-42c9-b658-2f982bc0cdd2', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': '595bd265-4212-4d9a-8d48-ba6fb59d19fe', 'enforce_multipath': True}
|
||||
|
||||
* Verify that the data exists after failover.
|
||||
NOTE that in some cases, the data might/might not be persistent depending
|
||||
on the type of replication i.e. async or sync.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ cat failover-dr
|
||||
# Before failover
|
||||
this should be consistent before/after failover
|
||||
|
||||
Failback of a External Data Volume
|
||||
----------------------------------
|
||||
|
||||
* Create a file and write data to the external data volume. (Note that the
|
||||
volume backend is in failover mode and we are writing to the replica disk
|
||||
in secondary backend.)
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ cat > failback-dr <<EOF
|
||||
> # Before Failback
|
||||
> this should be consistent before/after failback
|
||||
> EOF
|
||||
|
||||
* Failback to primary backend
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
cinder failover-host <host>@<backend> --backend_id default
|
||||
|
||||
* Detach and attach the external data volume
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
openstack server remove volume <Server UUID> <Volume UUID>
|
||||
openstack server add volume <Server UUID> <Volume UUID>
|
||||
|
||||
Verification
|
||||
^^^^^^^^^^^^
|
||||
|
||||
* Verify that the connection is now made from primary cluster
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# In cinder-volume logs, we can see the ``hosts``, ``cluster_name`` and ``auth_username`` fields will point to primary cluster
|
||||
Connection info returned from driver {'name': 'volumes/volume-437573fd-08e2-42c9-b658-2f982bc0cdd2', 'hosts': ['10.0.79.218'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': '***', 'secret_uuid': '***', 'volume_id': '437573fd-08e2-42c9-b658-2f982bc0cdd2', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False, 'driver_volume_type': 'rbd', 'attachment_id': 'b4e0c0a6-50b6-4ff3-83a5-a3da7be0e18c', 'enforce_multipath': True}
|
||||
|
||||
* Confirm that the data written before failback is persistent.
|
||||
NOTE that in some cases, the data might/might not be persistent depending
|
||||
on the type of replication i.e. async or sync.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ cat failback-dr
|
||||
# Before Failback
|
||||
this should be consistent before/after failback
|
||||
Reference in New Issue
Block a user