diff --git a/doc/source/storage/kubernetes/index-storage-kub-e797132c87a8.rst b/doc/source/storage/kubernetes/index-storage-kub-e797132c87a8.rst index 477a29c44..93b8b82ef 100644 --- a/doc/source/storage/kubernetes/index-storage-kub-e797132c87a8.rst +++ b/doc/source/storage/kubernetes/index-storage-kub-e797132c87a8.rst @@ -141,6 +141,7 @@ Rook Ceph Application install-rook-ceph-a7926a1f9b70 uninstall-rook-ceph-cbb046746782 + rook-ceph-osd-management-674d728af874 deployment-models-for-rook-ceph-b855bd0108cf performance-configurations-rook-ceph-9e719a652b02 diff --git a/doc/source/storage/kubernetes/rook-ceph-osd-management-674d728af874.rst b/doc/source/storage/kubernetes/rook-ceph-osd-management-674d728af874.rst new file mode 100644 index 000000000..8a700e02a --- /dev/null +++ b/doc/source/storage/kubernetes/rook-ceph-osd-management-674d728af874.rst @@ -0,0 +1,226 @@ +.. WARNING: Add no lines of text between the label immediately following +.. and the title. + +.. _rook-ceph-osd-management-674d728af874: + +======================== +Rook Ceph OSD Management +======================== + +This section describes the steps to add, remove, and replace |OSDs| in Rook +Container-based Ceph. + + +.. _addosd: + +Add OSDs +-------- + +.. rubric:: |proc| + +#. Obtain the |UUIDs| of the target disk(s). + + .. code-block:: none + + [sysadmin@controller-1 ~(keystone_admin)]$ system host-disk-list compute-0 + +--------------------------------------+-----------+---------+---------+-------+------------+--------------+--------------------+--------------------------------------------+ + | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path | + | | de | num | type | gib | gib | | | | + +--------------------------------------+-----------+---------+---------+-------+------------+--------------+--------------------+--------------------------------------------+ + | bc48b2ba-068d-4fc4-8c5a-f75cbd26eee4 | /dev/sda | 2048 | HDD | 292. | 0.0 | Undetermined | VB734fe32a- | /dev/disk/by-path/pci-0000:00:0d.0-ata-1.0 | + | | | | | 968 | | | 9839f8bf | | + | | | | | | | | | | + | 1aa1e19b-474a-46ef-9877-12afe2759046 | /dev/sdb | 2064 | HDD | 9.765 | 0.0 | Undetermined | VB29b38d6e- | /dev/disk/by-path/pci-0000:00:0d.0-ata-2.0 | + | | | | | | | | 28b7e2ae | | + | | | | | | | | | | + | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/sdc | 2080 | HDD | 5.859 | 5.855 | Undetermined | VB816fb0b6-2ace762 | /dev/disk/by-path/pci-0000:00:0d.0-ata-3.0 | + | | | | | | | | a | | + | | | | | | | | | | + | ed4c51a7-372f-4460-915d-43049cf23dd5 | /dev/sdd | 2096 | HDD | 5.859 | 5.855 | Undetermined | VBc059f8a9-953642e | /dev/disk/by-path/pci-0000:00:0d.0-ata-4.0 | + | | | | | | | | e | | + | | | | | | | | | | + | 68f4ec86-3d3c-41e0-8356-ef75275f9c88 | /dev/sde | 2112 | HDD | 5.859 | 5.855 | Undetermined | VB627293e5-cdeb561 | /dev/disk/by-path/pci-0000:00:0d.0-ata-5.0 | + | | | | | | | | 4 | | + | | | | | | | | | | + | 65e2238c-a355-460e-ab08-8e5d6ccd97aa | /dev/sdf | 2128 | HDD | 5.859 | 5.855 | Undetermined | VB93e0fe15-45d4840 | /dev/disk/by-path/pci-0000:00:0d.0-ata-6.0 | + | | | | | | | | d | | + | | | | | | | | | | + +--------------------------------------+-----------+---------+---------+-------+------------+--------------+--------------------+--------------------------------------------+ + +#. Add the target disk(s) as |OSDs|. + + .. code-block:: none + + [sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-add compute-0 ed4c51a7-372f-4460-915d-43049cf23dd5 + [sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-add compute-0 ab35233a-9952-4437-87ed-80cc4d6817d7 + + At this point, the |OSDs| will be in the `configuring-with-app` state. + + .. code-block:: none + + [sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-list compute-0 + +--------------------------------------+----------+-------+----------------------+--------------------------------------+-----------------------------+------------+----------+-----------+ + | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_ | tier_name | + | | | | | | | de | size_gib | | + +--------------------------------------+----------+-------+----------------------+--------------------------------------+-----------------------------+------------+----------+-----------+ + | 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397 | osd | None | configuring-with-app | ed4c51a7-372f-4460-915d-43049cf23dd5 | /dev/disk/by-path/pci-0000: | /dev/sdd2 | 1 | storage | + | | | | | | 00:0d.0-ata-4.0-part2 | | | | + | | | | | | | | | | + | 61e6566b-52b4-4ba8-b8db-e31b092523a6 | osd | None | configuring-with-app | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | + | | | | | | 00:0d.0-ata-3.0-part2 | | | | + | | | | | | | | | | + +--------------------------------------+----------+-------+----------------------+--------------------------------------+-----------------------------+------------+----------+-----------+ + +#. Apply rook-ceph application and wait for the application to become applied. + + .. code-block:: + + [sysadmin@controller-1 ~(keystone_admin)]$ system application-apply rook-ceph + + The rook-ceph application may take several minutes to apply the changes and + provision the |OSDs| before reaching the applied state. + + Use the :command:`system application-show rook-ceph` command to monitor progress. + +#. List the |OSDs| and verify that they are now in the `configured` state. + + .. code-block:: none + + [sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-list compute-0 + +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ + | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_size_g | tier_name | + | | | | | | | de | ib | | + +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ + | 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397 | osd | 0 | configured | ed4c51a7-372f-4460-915d-43049cf23dd5 | /dev/disk/by-path/pci-0000: | /dev/sdd2 | 1 | storage | + | | | | | | 00:0d.0-ata-4.0-part2 | | | | + | | | | | | | | | | + | 61e6566b-52b4-4ba8-b8db-e31b092523a6 | osd | 1 | configured | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | + | | | | | | 00:0d.0-ata-3.0-part2 | | | | + | | | | | | | | | | + +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ + +.. _removeosd: + +Remove OSDs +----------- + +.. rubric:: |proc| + +#. Obtain the |UUIDs| of the |OSDs| to be removed. + + .. code-block:: none + + [sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-list compute-0 + +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ + | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_size_g | tier_name | + | | | | | | | de | ib | | + +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ + | 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397 | osd | 0 | configured | ed4c51a7-372f-4460-915d-43049cf23dd5 | /dev/disk/by-path/pci-0000: | /dev/sdd2 | 1 | storage | + | | | | | | 00:0d.0-ata-4.0-part2 | | | | + | | | | | | | | | | + | 61e6566b-52b4-4ba8-b8db-e31b092523a6 | osd | 1 | configured | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | + | | | | | | 00:0d.0-ata-3.0-part2 | | | | + | | | | | | | | | | + +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ + +#. Mark the |OSDs| for removal. + + .. code-block:: none + + [sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-delete 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397 + + At this point, the |OSDs| will be in the `deleting-with-app` state. + + .. code-block:: none + + [sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-list compute-0 + +--------------------------------------+----------+-------+-------------------+--------------------------------------+-----------------------------+------------+----------+-----------+ + | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_ | tier_name | + | | | | | | | de | size_gib | | + +--------------------------------------+----------+-------+-------------------+--------------------------------------+-----------------------------+------------+----------+-----------+ + | 4c612b3b-93f1-42b1-9f65-7ebd0cb3d397 | osd | 0 | deleting-with-app | ed4c51a7-372f-4460-915d-43049cf23dd5 | /dev/disk/by-path/pci-0000: | /dev/sdd2 | 1 | storage | + | | | | | | 00:0d.0-ata-4.0-part2 | | | | + | | | | | | | | | | + | 61e6566b-52b4-4ba8-b8db-e31b092523a6 | osd | 1 | configured | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | + | | | | | | 00:0d.0-ata-3.0-part2 | | | | + | | | | | | | | | | + +--------------------------------------+----------+-------+-------------------+--------------------------------------+-----------------------------+------------+----------+-----------+ + +#. Apply rook-ceph application and wait for the application to become applied. + + .. code-block:: none + + [sysadmin@controller-1 ~(keystone_admin)]$ system application-apply rook-ceph + + The rook-ceph application may take several minutes to apply the changes and + provision the |OSDs| before reaching the applied state. + + Use the :command:`system application-show rook-ceph` command to monitor progress. + +#. Wait until the |OSDs| are removed. + + The |OSDs| will be removed asynchronously after rook-ceph reaches the + applied state. Then, once the |OSDs| entry disappears from system + host-stor-list, the |OSDs| are successfully removed. + + .. code-block:: none + + [sysadmin@controller-1 ~(keystone_admin)]$ system host-stor-list 3 + +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ + | uuid | function | osdid | state | idisk_uuid | journal_path | journal_no | journal_size_g | tier_name | + | | | | | | | de | ib | | + +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ + | 61e6566b-52b4-4ba8-b8db-e31b092523a6 | osd | 1 | configured | ab35233a-9952-4437-87ed-80cc4d6817d7 | /dev/disk/by-path/pci-0000: | /dev/sdc2 | 1 | storage | + | | | | | | 00:0d.0-ata-3.0-part2 | | | | + | | | | | | | | | | + +--------------------------------------+----------+-------+------------+--------------------------------------+-----------------------------+------------+----------------+-----------+ + +Replace OSDs +------------ + +.. rubric:: |proc| + +Repeat the following steps for each target host, one at a time so that it does +not impact Ceph availability. + +#. Follow the steps in the :ref:`Remove OSDs ` section to remove the target + |OSDs| from the host. + + .. note:: + + If after several minutes the |OSD| entry is reverted to the `configured` + state, then the |OSD| removal is considered unsafe and is aborted to + avoid data loss. + + You can proceed using :command:`system host-stor-delete --force `. + This will skip data integrity checks and will forcefully remove the |OSD|. + However, this may cause loss of data. Thus, use this only when you are + sure that there is enough replicas of data. Otherwise, before using the + `--force` option, always attempt the standard deletion first. + +#. Lock the host. + + .. code-block:: none + + [sysadmin@controller-1 ~(keystone_admin)]$ system host-lock + +#. After the host is locked, power off the host. + + .. code-block:: none + + [sysadmin@controller-1 ~(keystone_admin)]$ system host-power-off + +#. Replace the disk(s) while the host is powered off. + +#. Power on the host and wait for the host to become online. + +#. Unlock the host. + + .. code-block:: none + + [sysadmin@controller-1 ~(keystone_admin)]$ system host-unlock + + Wait for the host to become unlocked/available. + +#. Follow the steps in :ref:`Add OSDs ` to add the new replaced + disk(s) as |OSDs|.