diff --git a/doc/source/operations/ceph_cluster_migration.rst b/doc/source/operations/ceph_cluster_migration.rst new file mode 100644 index 000000000..892c0786d --- /dev/null +++ b/doc/source/operations/ceph_cluster_migration.rst @@ -0,0 +1,633 @@ +====================== +Ceph cluster migration +====================== + +This guide contains step by step instructions for migrating StarlingX R3.0 +with standard controller storage Ceph cluster to the containerized Ceph +cluster deployed by Rook. + +.. contents:: + :local: + :depth: 1 + +------------ +Introduction +------------ + +In StarlingX 3.0 or lower versions, Ceph as the backend storage cluster +solution was deployed directly on the host platform. The intent is that +starting from 4.0 Ceph cluster will be containerized and managed by Rook, +for the sake of operation and maintenance efficiency. Therefore, in the +context of StarlingX upgrade from 3.0 to 4.0, we are here introducing a +method to migrate the original Ceph cluster deployed by users at 3.0 +provisioning stage to the newly containerized Ceph cluster at 4.0, while +keeping user data (in OSDs) uncorrupted. + +--------------------- +Prepare for migration +--------------------- + +StarlingX has some HA mechanisms for critical service monitoring and +recovering. To migrate Ceph monitor(s) and Ceph OSD(s), the first step is to +disable the monitoring and recovering for Ceph services, otherwise the migration +procedure might be interfered due to the continuous service restarting. + +************************************* +Disable StarlingX HA for ceph service +************************************* + +Disable monitoring and recovering for Ceph service by pmon and service manager. + +#. Disable pmon's monitoring for Ceph mon and Ceph osd on every host. + + :: + + $ sudo rm -f /etc/pmon.d/ceph.conf + $ sudo /usr/local/sbin/pmon-restart pmon_cmd_port + +#. Disable service manager's monitoring for Ceph manager on controller host. + + :: + + $ sudo sm-unmanage service mgr-restful-plugin + Service (mgr-restful-plugin) is no longer being managed. + $ sudo sm-unmanage service ceph-manager + Service (ceph-manager) is no longer being managed. + +********************************** +Enable ceph service authentication +********************************** + +StarlingX disables Ceph authentication, but authentication is must for rook. +Before migration, enable authentication for each of daemons. + +#. Enabled authentication for Ceph mon and osd service. + + :: + + $ ceph config set mon.storage-0 auth_cluster_required cephx + $ ceph config set mon.storage-0 auth_supported cephx + $ ceph config set mon.storage-0 auth_service_required cephx + $ ceph config set mon.storage-0 auth_client_required cephx + $ ceph config set mon.controller-0 auth_cluster_required cephx + $ ceph config set mon.controller-0 auth_supported cephx + $ ceph config set mon.controller-0 auth_service_required cephx + $ ceph config set mon.controller-0 auth_client_required cephx + $ ceph config set mon.controller-1 auth_cluster_required cephx + $ ceph config set mon.controller-1 auth_supported cephx + $ ceph config set mon.controller-1 auth_service_required cephx + $ ceph config set mon.controller-1 auth_client_required cephx + $ ceph config set mgr.controller-0 auth_supported cephx + $ ceph config set mgr.controller-0 auth_cluster_required cephx + $ ceph config set mgr.controller-0 auth_client_required cephx + $ ceph config set mgr.controller-0 auth_service_required cephx + $ ceph config set mgr.controller-1 auth_supported cephx + $ ceph config set mgr.controller-1 auth_cluster_required cephx + $ ceph config set mgr.controller-1 auth_client_required cephx + $ ceph config set mgr.controller-1 auth_service_required cephx + $ ceph config set osd.0 auth_supported cephx + $ ceph config set osd.0 auth_cluster_required cephx + $ ceph config set osd.0 auth_service_required cephx + $ ceph config set osd.0 auth_client_required cephx + $ ceph config set osd.1 auth_supported cephx + $ ceph config set osd.1 auth_cluster_required cephx + $ ceph config set osd.1 auth_service_required cephx + $ ceph config set osd.1 auth_client_required cephx + +#. Generate client.admin key. + + :: + + $ ADMIN_KEY=$(ceph auth get-or-create-key client.admin mon 'allow *' osd 'allow *' mgr 'allow *' mds 'allow *') + $ echo $ADMIN_KEY + AQDRGqFea0cYERAAwYdhhle5zEbLLkYHWF+sDw== + $ MON_KEY=$(ceph auth get-or-create-key mon. mon 'allow *') + $ echo $MON_KEY + AQBbs79eM4/FMRAAbu4jwdBFVS1hOmlCdoCacQ== + +*********************************************** +Create configmap and secret for rook deployment +*********************************************** + +Rook will read secret rook-ceph-mon and configmap rook-ceph-mon-endpoint +to get cluster info in deployment. + +#. Create configmap and secret for rook deployment. + + :: + + $ export NAMESPACE=kube-system + $ export ROOK_EXTERNAL_CEPH_MON_DATA=a=192.188.204.3:6789 + $ export ROOK_EXTERNAL_FSID=$(ceph fsid) + $ export ROOK_EXTERNAL_CLUSTER_NAME=$NAMESPACE + $ export ROOK_EXTERNAL_MAX_MON_ID=0 + + $ kubectl -n "$NAMESPACE" create secret generic rook-ceph-mon \ + > --from-literal=cluster-name="$ROOK_EXTERNAL_CLUSTER_NAME" \ + > --from-literal=fsid="$ROOK_EXTERNAL_FSID" \ + > --from-literal=admin-secret="$ADMIN_KEY" \ + > --from-literal=mon-secret="$MON_KEY" + secret/rook-ceph-mon created + + $ kubectl -n "$NAMESPACE" create configmap rook-ceph-mon-endpoints \ + > --from-literal=data="$ROOK_EXTERNAL_CEPH_MON_DATA" \ + > --from-literal=mapping="$ROOK_EXTERNAL_MAPPING" \ + > --from-literal=maxMonId="$ROOK_EXTERNAL_MAX_MON_ID" + configmap/rook-ceph-mon-endpoint created + +********************** +Remove rbd-provisioner +********************** + +Application platform-integ-apps deploys the helm chart rbd-provisioner. This +chart will be unncesssary after rook deployed, remove before rook deployment. + +#. remove rbd-provisioner. + + :: + + $ sudo rm -rf /opt/platform/sysinv/20.01/.crushmap_applied + $ source /etc/platform/openrc + $ system application-remove platform-integ-apps + +---------------+----------------------------------+ + | Property | Value | + +---------------+----------------------------------+ + | active | True | + | app_version | 1.0-8 | + | created_at | 2020-04-22T14:56:19.148562+00:00 | + | manifest_file | manifest.yaml | + | manifest_name | platform-integration-manifest | + | name | platform-integ-apps | + | progress | None | + | status | removing | + | updated_at | 2020-04-22T15:46:24.018090+00:00 | + +---------------+----------------------------------+ + +************************************************************************ +Disable ceph osd on all storage hosts and create configmap for migration +************************************************************************ + +Login to storage host with osd provisioned, disable Ceph osd service and create +journal file + +#. Disable Ceph osd service. + + :: + + $ sudo service ceph -a stop osd.1 + === osd.1 === + Stopping Ceph osd.1 on storage-0...kill 213077... + done + 2020-04-26 23:36:56.988 7f1d647bb1c0 -1 journal do_read_entry(585007104): bad header magic + 2020-04-26 23:36:56.988 7f1d647bb1c0 -1 journal do_read_entry(585007104): bad header magic + 2020-04-26 23:36:56.994 7f1d647bb1c0 -1 flushed journal /var/lib/ceph/osd/ceph-1/journal for object store /var/lib/ceph/osd/ceph-1 + +#. Remove journal link and create a blank journal file + + :: + + $ sudo rm -f /var/lib/ceph/osd/ceph-1/journal + $ sudo touch /var/lib/ceph/osd/ceph-1/journal + $ sudo dd if=/dev/zero of=/var/lib/ceph/osd/ceph-1/journal bs=1M count=1024 + $ sudo ceph-osd --id 1 --mkjournal --no-mon-config + $ sudo umount /dev/sdc1 + +#. Mount to host patch /var/lib/ceph/osd, which could be access by rook's osd pod + + :: + + $ sudo mkdir -p /var/lib/ceph/ceph-1/osd1 + $ sudo mount /dev/sdc1 /var/lib/ceph/ceph-1/osd1 + $ sudo ls /var/lib/ceph/ceph-1/osd1 -l + total 1048640 + -rw-r--r-- 1 root root 3 Apr 26 12:57 active + -rw-r--r-- 1 root root 37 Apr 26 12:57 ceph_fsid + drwxr-xr-x 388 root root 12288 Apr 27 00:01 current + -rw-r--r-- 1 root root 37 Apr 26 12:57 fsid + -rw-r--r-- 1 root root 1073741824 Apr 27 00:49 journal + -rw-r--r-- 1 root root 37 Apr 26 12:57 journal_uuid + -rw------- 1 root root 56 Apr 26 12:57 keyring + -rw-r--r-- 1 root root 21 Apr 26 12:57 magic + -rw-r--r-- 1 root root 6 Apr 26 12:57 ready + -rw-r--r-- 1 root root 4 Apr 26 12:57 store_version + -rw-r--r-- 1 root root 53 Apr 26 12:57 superblock + -rw-r--r-- 1 root root 0 Apr 26 12:57 sysvinit + -rw-r--r-- 1 root root 10 Apr 26 12:57 type + -rw-r--r-- 1 root root 2 Apr 26 12:57 wanttobe + -rw-r--r-- 1 root root 2 Apr 26 12:57 whoami + +For every host with osd device, create a configmap. Configmap name is +rook-ceph-osd--config. In configmap, it specified osd data folder. +For example, this data will info rook osd0 data path is /var/lib/ceph/osd0 + + :: + + osd-dirs: '{"/var/lib/ceph/ceph-0/":0}' + + $ system host-stor-list storage-0 + +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+--------------+-----------+ + | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_size | tier_name | + | | | | | | | de | _gib | | + +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+--------------+-----------+ + | 21a90d60-2f1e-4f46-badc-afa7d9117622 | osd | 1 | configured | a13c6ac9-9d59-4063-88dc-2847e8aded85 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | + | | | | | | 00:03.0-ata-3.0-part2 | | | | + | | | | | | | | | | + | d259a366-3633-4c03-9268-0cd35b2b274d | osd | 0 | configured | 54b3cb9d-4527-448a-9051-62b250c2a03f | /dev/disk/by-path/pci-0000: | /dev/sdb2 | 1 | storage | + | | | | | | 00:03.0-ata-2.0-part2 | | | | + | | | | | | | | | | + +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+--------------+-----------+ + + $ system host-stor-list storage-1 + +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+--------------+-----------+ + | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_size | tier_name | + | | | | | | | de | _gib | | + +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+--------------+-----------+ + | 17f2db8e-c80e-4df7-9525-1f0cb5b54cd3 | osd | 3 | configured | 89637c7d-f959-4c54-bfe1-626b5c630d96 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | + | | | | | | 00:03.0-ata-3.0-part2 | | | | + | | | | | | | | | | + | 64b9d56c-c384-4bd6-a437-89d6bfda4ec5 | osd | 2 | configured | ad52345c-254c-48c1-9034-778738c7e23b | /dev/disk/by-path/pci-0000: | /dev/sdb2 | 1 | storage | + | | | | | | 00:03.0-ata-2.0-part2 | | | | + | | | | | | | | | | + +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+--------------+-----------+ + + +#. Sample osd-configmap.yaml + :: + + apiVersion: v1 + kind: ConfigMap + metadata: + name: rook-ceph-osd-storage-0-config + namespace: kube-system + data: + osd-dirs: '{"/var/lib/ceph/ceph-0":0,"/var/lib/ceph/ceph-1":1}' + --- + apiVersion: v1 + kind: ConfigMap + metadata: + name: rook-ceph-osd-storage-1-config + namespace: kube-system + data: + osd-dirs: '{"/var/lib/ceph/ceph-2":2,"/var/lib/ceph/ceph-3":3}' + +#. Apply yaml file for configmap + + :: + + $ kubectl apply -f osd-configmap.yaml + configmap/rook-ceph-osd-storage-0-config created + configmap/rook-ceph-osd-storage-1-config created + +************************** +Ceph monitor data movement +************************** + +For Ceph monitor migration, Rook deployed monitor pod will read monitor data +for host path /var/lib/ceph/mon-/data. For example, if only deployed one +monitor pod, a monitor process named "mon.a" in monitor pod will be created +and monitor data in host path /var/lib/ceph/mon-a/data. So before migration, +one monitor service should be disable and launch another monitor which will +be specified with parameter "--mon-data /var/lib/ceph/mon-a/data" to make +monitor data migrating to /var/lib/ceph/mon-a/data. + +#. Login host controller-0, disable service monitor.controller-0. + + :: + + $ sudo service ceph -a stop mon.controller-0 + === mon.controller-0 === + Stopping Ceph mon.controller-0 on controller-0...kill 291101...done + +#. Login host controller-1, disable service monitor.controller-1. + + :: + + $ sudo service ceph -a stop mon.controller-1 + === mon.controller-1 === + Stopping Ceph mon.controller-1 on controller-1...kill 385107... + done + +#. Login host storage-0, disable service monitor.storage-0. + + :: + + $ sudo service ceph -a stop mon.storage-0 + === mon.storage-0 === + Stopping Ceph mon.storage-0 on storage-0...kill 31394... + done + +#. Copy mon data to folder /var/lib/ceph/mon-a/data. + + :: + + $ sudo mkdir -p /var/lib/ceph/mon-a/data/ + $ sudo ceph-monstore-tool /var/lib/ceph/mon/ceph-controller-0/ store-copy /var/lib/ceph/mon-a/data/ + +#. Update monmap in this copy of monitor data, update monitor info. + + :: + + $ sudo ceph-mon --extract-monmap monmap --mon-data /var/lib/ceph/mon-a/data/ + 2020-05-21 06:01:39.477 7f69d63b2140 -1 wrote monmap to monmap + + $ monmaptool --print monmap + monmaptool: monmap file monmap + epoch 2 + fsid 6c9e9e4b-599e-4a4f-931e-2c09bec74a2a + last_changed 2020-05-21 04:29:59.164965 + created 2020-05-21 03:50:51.893155 + 0: 192.188.204.3:6789/0 mon.controller-0 + 1: 192.188.204.4:6789/0 mon.controller-1 + 2: 192.188.204.41:6789/0 mon.storage-0 + + $ sudo monmaptool --rm controller-0 monmap + monmaptool: monmap file monmap + monmaptool: removing controller-0 + monmaptool: writing epoch 2 to monmap (2 monitors) + + $ sudo monmaptool --rm controller-1 monmap + monmaptool: monmap file monmap + monmaptool: removing controller-1 + monmaptool: writing epoch 2 to monmap (1 monitors) + + $ sudo monmaptool --rm storage-0 monmap + monmaptool: monmap file monmap + monmaptool: removing storage-0 + monmaptool: writing epoch 2 to monmap (0 monitors) + + $ sudo monmaptool --add a 192.188.204.3 monmap + monmaptool: monmap file monmap + monmaptool: writing epoch 2 to monmap (1 monitors) + + $ monmaptool --print monmap + monmaptool: monmap file monmap + epoch 2 + fsid 6c9e9e4b-599e-4a4f-931e-2c09bec74a2a + last_changed 2020-05-21 04:29:59.164965 + created 2020-05-21 03:50:51.893155 + 0: 192.188.204.3:6789/0 mon.a + + $ sudo ceph-mon --inject-monmap monmap --mon-data /var/lib/ceph/mon-a/data/ + +---------------------- +Deploy Rook helm chart +---------------------- + +StarlingX already creates a application for Rook deployment, after finish the +above preparation, apply the application to deploy rook. To make live migration +and keep Ceph service always readiness, Ceph service should migrate in turn. +First Ceph monitor, which is mon.a, exits and launch rook cluster with one monitor +pod. At this time, 2 monitor daemons and 1 monitor pod is running and then migrate +osd one by one. At last, migrate 2 monitor daemon and migration is done. + +************************************** +Disable Ceph monitors and Ceph manager +************************************** + +Disable Ceph manager on host controller-0 and controller-1 + +#. Disable Ceph manager + + :: + + $ ps -aux | grep mgr + root 97971 0.0 0.0 241336 18488 ? S< 03:54 0:02 /usr/bin/python /etc/init.d/mgr-restful-plugin start + root 97990 0.5 0.0 241468 18916 ? S< 03:54 0:38 /usr/bin/python /etc/init.d/mgr-restful-plugin start + root 186145 1.2 0.3 716488 111328 ? S