fd1157e47a
After merger of osh-infra we have to update the home urls for all charts to point to the openstack-helm repo. Also cleanup references to osh-infra in other places. Change-Id: Iefeba8024b6720d25013c60cfd3a835950675801
247 lines
7.0 KiB
ReStructuredText
247 lines
7.0 KiB
ReStructuredText
================================================================
|
|
Ceph - upgrade monolithic ceph-osd chart to multiple ceph charts
|
|
================================================================
|
|
|
|
This document captures the steps to move from installed monolithic ceph-osd chart
|
|
to mutlitple ceph osd charts.
|
|
|
|
this work will bring flexibility on site update as we will have more control on osds.
|
|
|
|
|
|
Install single ceph-osd chart:
|
|
==============================
|
|
|
|
step 1: Setup:
|
|
==============
|
|
|
|
- Follow OSH single node or multinode guide to bring up OSH environment.
|
|
|
|
.. note::
|
|
we will have single ceph osd chart and here are the override values for ceph disks
|
|
osd:
|
|
- data:
|
|
type: block-logical
|
|
location: /dev/vdb
|
|
journal:
|
|
type: block-logical
|
|
location: /dev/vda1
|
|
- data:
|
|
type: block-logical
|
|
location: /dev/vdc
|
|
journal:
|
|
type: block-logical
|
|
location: /dev/vda2
|
|
|
|
|
|
Step 2: Setup the OSH environment and check ceph cluster health
|
|
=================================================================
|
|
|
|
.. note::
|
|
Make sure we have healthy ceph cluster running
|
|
|
|
``Ceph status:``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph -s
|
|
cluster:
|
|
id: 61a4e07f-8b4a-4c47-8fc7-a0e7345ac0b0
|
|
health: HEALTH_OK
|
|
|
|
services:
|
|
mon: 3 daemons, quorum k8smaster,k8sslave1,k8sslave2
|
|
mgr: k8sslave2(active), standbys: k8sslave1
|
|
mds: cephfs-1/1/1 up {0=mds-ceph-mds-5bf9fdfc6b-8nq4p=up:active}, 1 up:standby
|
|
osd: 6 osds: 6 up, 6 in
|
|
data:
|
|
pools: 18 pools, 186 pgs
|
|
objects: 377 objects, 1.2 GiB
|
|
usage: 4.2 GiB used, 116 GiB / 120 GiB avail
|
|
pgs: 186 active+clean
|
|
|
|
- Ceph cluster is in HEALTH_OK state with 3 MONs and 6 OSDs.
|
|
|
|
.. note::
|
|
Make sure we have single ceph osd chart only
|
|
|
|
``Helm status:``
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ helm list | grep -i osd
|
|
ceph-osd 1 Tue Mar 26 03:21:07 2019 DEPLOYED ceph-osd-vdb-0.1.0
|
|
|
|
- single osd chart deployed sucessfully.
|
|
|
|
|
|
Upgrade to multiple ceph osd charts:
|
|
====================================
|
|
|
|
step 1: setup
|
|
=============
|
|
|
|
- create multiple ceph osd charts as per requirement
|
|
|
|
.. note::
|
|
copy ceph-osd folder to multiple ceph osd charts in openstack-helm folder
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ cp -r ceph-osd ceph-osd-vdb
|
|
ubuntu@k8smaster:/opt/openstack-helm$ cp -r ceph-osd ceph-osd-vdc
|
|
|
|
.. note::
|
|
make sure to correct chart name in each osd chart folder created above, need to
|
|
update it in Charts.yaml .
|
|
|
|
- create script to install multiple ceph osd charts
|
|
|
|
.. note::
|
|
create new installation scripts to reflect new ceph osd charts.
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ cp ./tools/deployment/multinode/030-ceph.sh
|
|
./tools/deployment/multinode/030-ceph-osd-vdb.sh
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ cp ./tools/deployment/multinode/030-ceph.sh
|
|
./tools/deployment/multinode/030-ceph-osd-vdc.sh
|
|
|
|
.. note::
|
|
make sure to delete all other ceph charts from above scripts and have only new ceph osd chart.
|
|
and also have correct overrides as shown below.
|
|
|
|
example1: for CHART in ceph-osd-vdb; do
|
|
helm upgrade --install ${CHART} ${OSH_PATH}/${CHART} \
|
|
--namespace=ceph \
|
|
--values=/tmp/ceph.yaml \
|
|
${OSH_EXTRA_HELM_ARGS} \
|
|
${OSH_EXTRA_HELM_ARGS_CEPH_DEPLOY}
|
|
|
|
osd:
|
|
- data:
|
|
type: block-logical
|
|
location: /dev/vdb
|
|
journal:
|
|
type: block-logical
|
|
location: /dev/vda1
|
|
|
|
example2: for CHART in ceph-osd-vdc; do
|
|
helm upgrade --install ${CHART} ${OSH_PATH}/${CHART} \
|
|
--namespace=ceph \
|
|
--values=/tmp/ceph.yaml \
|
|
${OSH_EXTRA_HELM_ARGS} \
|
|
${OSH_EXTRA_HELM_ARGS_CEPH_DEPLOY}
|
|
|
|
osd:
|
|
- data:
|
|
type: block-logical
|
|
location: /dev/vdc
|
|
journal:
|
|
type: block-logical
|
|
location: /dev/vda2
|
|
|
|
step 2: Scale down applications using ceph pvc
|
|
===============================================
|
|
|
|
.. note::
|
|
Scale down all the applications who are using pvcs so that we will not
|
|
have any writes on ceph rbds .
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ sudo kubectl scale statefulsets -n openstack
|
|
mariadb-server --replicas=0
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ sudo kubectl scale statefulsets -n openstack
|
|
rabbitmq-rabbitmq --replicas=0
|
|
|
|
- just gave one example but we need to do it for all the applications using pvcs
|
|
|
|
|
|
step 3: Setup ceph cluster flags to prevent rebalance
|
|
=====================================================
|
|
|
|
.. note::
|
|
setup few flags on ceph cluster to prevent rebalance during this process.
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd set
|
|
noout
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd set
|
|
nobackfill
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd set
|
|
norecover
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd set
|
|
pause
|
|
|
|
step 4: Delete single ceph-osd chart
|
|
====================================
|
|
|
|
.. note::
|
|
Delete the single ceph-osd chart.
|
|
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ helm delete --purge ceph-osd
|
|
|
|
|
|
step 5: install new ceph-osd charts
|
|
===================================
|
|
|
|
.. note::
|
|
Now we can install multiple ceph osd releases.
|
|
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ ./tools/deployment/multinode/030-ceph-osd-vdb.sh
|
|
ubuntu@k8smaster:/opt/openstack-helm$ ./tools/deployment/multinode/030-ceph-osd-vdc.sh
|
|
ubuntu@k8smaster:/opt/openstack-helm# helm list | grep -i osd
|
|
ceph-osd-vdb 1 Tue Mar 26 03:21:07 2019 DEPLOYED ceph-osd-vdb-0.1.0
|
|
ceph-osd-vdc 1 Tue Mar 26 03:22:13 2019 DEPLOYED ceph-osd-vdc-0.1.0
|
|
|
|
- wait and check for healthy ceph cluster , if there are any issues need to sort out until we see
|
|
healthy ceph cluster.
|
|
|
|
step 6: Unset ceph cluster flags
|
|
================================
|
|
|
|
.. note::
|
|
unset the flags we set on the ceph cluster in above steps.
|
|
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd unset
|
|
noout
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd unset
|
|
nobackfill
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd unset
|
|
norecover
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd unset
|
|
pause
|
|
|
|
step 7: Scale up the applications using pvc
|
|
===========================================
|
|
|
|
.. note::
|
|
Since ceph cluster is back to healthy status, now scale up the applications.
|
|
|
|
|
|
.. code-block:: console
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ sudo kubectl scale statefulsets -n openstack
|
|
mariadb-server --replicas=3
|
|
|
|
ubuntu@k8smaster:/opt/openstack-helm$ sudo kubectl scale statefulsets -n openstack
|
|
rabbitmq-rabbitmq --replicas=3
|