diff --git a/doc/source/testing/ceph-upgrade.rst b/doc/source/testing/ceph-upgrade.rst new file mode 100644 index 0000000000..97c4661265 --- /dev/null +++ b/doc/source/testing/ceph-upgrade.rst @@ -0,0 +1,593 @@ +============ +Ceph Upgrade +============ + +This guide documents steps showing Ceph version upgrade. The main goal of this +document is to demostrate Ceph chart update without downtime for OSH components. + +Test Scenario: +============== +Upgrade Ceph component version from ``12.2.4`` to ``12.2.5`` without downtime +to OSH components. + +Setup: +====== +- 3 Node (VM based) env. +- Followed OSH multinode guide steps to setup nodes and install k8 cluster +- Followed OSH multinode guide steps upto Ceph install + +Plan: +===== +1) Install Ceph charts (12.2.4) by updating Docker images in overrides. +2) Install OSH components as per OSH multinode guide. +3) Upgrade Ceph charts to version 12.2.5 by updating docker images in overrides. + + +Docker Images: +============== +1) Ceph Luminous point release images for Ceph components + +.. code-block:: console + + Ceph 12.2.4: ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64 + Ceph 12.2.5: ceph/daemon:master-a8d20ed-luminous-ubuntu-16.04-x86_64 + +2) Ceph RBD provisioner docker images. + +.. code-block:: console + + quay.io/external_storage/rbd-provisioner:v0.1.0 + quay.io/external_storage/rbd-provisioner:v0.1.1 + +3) Ceph Cephfs provisioner docker images. + +.. code-block:: console + + quay.io/external_storage/cephfs-provisioner:v0.1.1 + quay.io/external_storage/cephfs-provisioner:v0.1.2 + +Steps: +====== + +.. note:: + Follow all steps from OSH multinode guide with below changes. + +1) Install Ceph charts (version 12.2.4) + + + Update ceph install script ``./tools/deployment/multinode/030-ceph.sh`` + to add ``images:`` section in overrides as shown below. + +.. note:: + OSD count is set to 3 based on env setup. + +.. note:: + Following is a partial part from script to show changes. + +.. code-block:: yaml + + tee /tmp/ceph.yaml << EOF + ... + network: + public: ${CEPH_PUBLIC_NETWORK} + cluster: ${CEPH_CLUSTER_NETWORK} + images: + tags: + ceph_bootstrap: 'docker.io/ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64' + ceph_config_helper: 'docker.io/port/ceph-config-helper:v1.10.3' + ceph_rbd_pool: 'docker.io/port/ceph-config-helper:v1.10.3' + ceph_mon_check: 'docker.io/port/ceph-config-helper:v1.10.3' + ceph_mon: 'docker.io/ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64' + ceph_osd: 'docker.io/ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64' + ceph_mds: 'docker.io/ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64' + ceph_mgr: 'docker.io/ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64' + ceph_rgw: 'docker.io/ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64' + ceph_cephfs_provisioner: 'quay.io/external_storage/cephfs-provisioner:v0.1.1' + ceph_rbd_provisioner: 'quay.io/external_storage/rbd-provisioner:v0.1.0' + conf: + ceph: + global: + fsid: ${CEPH_FS_ID} + rgw_ks: + enabled: true + pool: + crush: + tunables: ${CRUSH_TUNABLES} + target: + # NOTE(portdirect): 5 nodes, with one osd per node + osd: 5 + pg_per_osd: 100 + ... + EOF + +.. note:: + ``ceph_bootstrap``, ``ceph-config_helper`` and ``ceph_rbs_pool`` images + are used for jobs. ``ceph_mon_check`` has one script that is stable so no + need to upgrade. + +2) Deploy and Validate Ceph + +.. code-block:: console + + + kubectl exec -n ceph ceph-mon-4c8xs -- ceph -s + cluster: + id: 39061799-d25e-4f3b-8c1a-a350e4c6d06c + health: HEALTH_OK + + services: + mon: 3 daemons, quorum mnode1,mnode2,mnode3 + mgr: mnode2(active), standbys: mnode3 + mds: cephfs-1/1/1 up {0=mds-ceph-mds-745576757f-4vdn4=up:active}, 1 up:standby + osd: 3 osds: 3 up, 3 in + rgw: 2 daemons active + + data: + pools: 18 pools, 93 pgs + objects: 208 objects, 3359 bytes + usage: 72175 MB used, 75739 MB / 144 GB avail + pgs: 93 active+clean + +3) Check Ceph Pods + +.. code-block:: console + + ubuntu@mnode1:/opt/openstack-helm$ kubectl get pods -n ceph + NAME READY STATUS RESTARTS AGE + ceph-bootstrap-s4jkx 0/1 Completed 0 10m + ceph-cephfs-client-key-generator-6bmzz 0/1 Completed 0 3m + ceph-cephfs-provisioner-784c6f9d59-2z6ww 1/1 Running 0 3m + ceph-cephfs-provisioner-784c6f9d59-sg8wv 1/1 Running 0 3m + ceph-mds-745576757f-4vdn4 1/1 Running 0 6m + ceph-mds-745576757f-bxdcs 1/1 Running 0 6m + ceph-mds-keyring-generator-f5lxf 0/1 Completed 0 10m + ceph-mgr-86bdc7c64b-7ptr4 1/1 Running 0 6m + ceph-mgr-86bdc7c64b-xgplj 1/1 Running 0 6m + ceph-mgr-keyring-generator-w7nxq 0/1 Completed 0 10m + ceph-mon-4c8xs 1/1 Running 0 10m + ceph-mon-check-d85994946-zzwb4 1/1 Running 0 10m + ceph-mon-keyring-generator-jdgfw 0/1 Completed 0 10m + ceph-mon-kht8d 1/1 Running 0 10m + ceph-mon-mkpmm 1/1 Running 0 10m + ceph-osd-default-83945928-7jz4s 1/1 Running 0 8m + ceph-osd-default-83945928-bh82j 1/1 Running 0 8m + ceph-osd-default-83945928-t9szk 1/1 Running 0 8m + ceph-osd-keyring-generator-6rg65 0/1 Completed 0 10m + ceph-rbd-pool-z8vlc 0/1 Completed 0 6m + ceph-rbd-provisioner-84665cb84f-6s55r 1/1 Running 0 3m + ceph-rbd-provisioner-84665cb84f-chwhd 1/1 Running 0 3m + ceph-rgw-74559877-h56xs 1/1 Running 0 6m + ceph-rgw-74559877-pfjr5 1/1 Running 0 6m + ceph-rgw-keyring-generator-6rwct 0/1 Completed 0 10m + ceph-storage-keys-generator-bgj2t 0/1 Completed 0 10m + ingress-796d8cf8d6-nzrd2 1/1 Running 0 11m + ingress-796d8cf8d6-qqvq9 1/1 Running 0 11m + ingress-error-pages-54454dc79b-d5r5w 1/1 Running 0 11m + ingress-error-pages-54454dc79b-gfpqv 1/1 Running 0 11m + + +4) Check version of each Ceph components. + +.. code-block:: console + + ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-4c8xs -- ceph -v + ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable) + + ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-osd-default-83945928-7jz4s -- ceph -v + ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable) + + ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-mgr-86bdc7c64b-7ptr4 -- ceph -v + ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable) + + ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-mds-745576757f-4vdn4 -- ceph -v + ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable) + + ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-rgw-74559877-h56xs -- ceph -v + ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable) + +5) Check which images Provisionors and Mon-Check PODs are using + +.. note:: + Showing partial output from kubectl describe command to show which image is Docker + container is using + +.. code-block:: console + + ubuntu@mnode1:~$ kubectl describe pod -n ceph ceph-cephfs-provisioner-784c6f9d59-2z6ww + + Containers: + ceph-cephfs-provisioner: + Container ID: docker://98ed65617f6c4b60fe60d94b8707e52e0dd4c87791e7d72789a0cb603fa80e2c + Image: quay.io/external_storage/cephfs-provisioner:v0.1.1 + +.. code-block:: console + + ubuntu@mnode1:~$ kubectl describe pod -n ceph ceph-rbd-provisioner-84665cb84f-6s55r + + Containers: + ceph-rbd-provisioner: + Container ID: docker://383be3d653cecf4cbf0c3c7509774d39dce54102309f1f0bdb07cdc2441e5e47 + Image: quay.io/external_storage/rbd-provisioner:v0.1.0 + +.. code-block:: console + + ubuntu@mnode1:~$ kubectl describe pod -n ceph ceph-mon-check-d85994946-zzwb4 + + Containers: + ceph-mon: + Container ID: docker://d5a3396f99704038ab8ef6bfe329013ed46472ebb8e26dddc140b621329f0f92 + Image: docker.io/port/ceph-config-helper:v1.10.3 + + +6) Install Openstack charts + +Continue with OSH multinode guide to install other Openstack charts. + +7) Capture Ceph pods statuses. + +.. code-block:: console + + NAME READY STATUS RESTARTS AGE + ceph-bootstrap-s4jkx 0/1 Completed 0 2h + ceph-cephfs-client-key-generator-6bmzz 0/1 Completed 0 2h + ceph-cephfs-provisioner-784c6f9d59-2z6ww 1/1 Running 0 2h + ceph-cephfs-provisioner-784c6f9d59-sg8wv 1/1 Running 0 2h + ceph-mds-745576757f-4vdn4 1/1 Running 0 2h + ceph-mds-745576757f-bxdcs 1/1 Running 0 2h + ceph-mds-keyring-generator-f5lxf 0/1 Completed 0 2h + ceph-mgr-86bdc7c64b-7ptr4 1/1 Running 0 2h + ceph-mgr-86bdc7c64b-xgplj 1/1 Running 0 2h + ceph-mgr-keyring-generator-w7nxq 0/1 Completed 0 2h + ceph-mon-4c8xs 1/1 Running 0 2h + ceph-mon-check-d85994946-zzwb4 1/1 Running 0 2h + ceph-mon-keyring-generator-jdgfw 0/1 Completed 0 2h + ceph-mon-kht8d 1/1 Running 0 2h + ceph-mon-mkpmm 1/1 Running 0 2h + ceph-osd-default-83945928-7jz4s 1/1 Running 0 2h + ceph-osd-default-83945928-bh82j 1/1 Running 0 2h + ceph-osd-default-83945928-t9szk 1/1 Running 0 2h + ceph-osd-keyring-generator-6rg65 0/1 Completed 0 2h + ceph-rbd-pool-z8vlc 0/1 Completed 0 2h + ceph-rbd-provisioner-84665cb84f-6s55r 1/1 Running 0 2h + ceph-rbd-provisioner-84665cb84f-chwhd 1/1 Running 0 2h + ceph-rgw-74559877-h56xs 1/1 Running 0 2h + ceph-rgw-74559877-pfjr5 1/1 Running 0 2h + ceph-rgw-keyring-generator-6rwct 0/1 Completed 0 2h + ceph-storage-keys-generator-bgj2t 0/1 Completed 0 2h + ingress-796d8cf8d6-nzrd2 1/1 Running 0 2h + ingress-796d8cf8d6-qqvq9 1/1 Running 0 2h + ingress-error-pages-54454dc79b-d5r5w 1/1 Running 0 2h + ingress-error-pages-54454dc79b-gfpqv 1/1 Running 0 2h + +8) Capture Openstack pods statuses. + +.. code-block:: console + + NAME READY STATUS RESTARTS AGE + cinder-api-67495cdffc-24fhs 1/1 Running 0 51m + cinder-api-67495cdffc-kz5fn 1/1 Running 0 51m + cinder-backup-65b7bd9b79-8n9pb 1/1 Running 0 51m + cinder-scheduler-9ddbb7878-rbt4l 1/1 Running 0 51m + cinder-volume-75bf4cc9bd-6298x 1/1 Running 0 51m + glance-api-68f6df4d5d-q84hs 1/1 Running 0 1h + glance-api-68f6df4d5d-qbfwb 1/1 Running 0 1h + glance-registry-5957979dc5-h54tc 1/1 Running 0 1h + glance-registry-5957979dc5-sglg7 1/1 Running 0 1h + ingress-7b4bc84cdd-84dtj 1/1 Running 0 2h + ingress-7b4bc84cdd-ws45r 1/1 Running 0 2h + ingress-error-pages-586c7f86d6-dlpm2 1/1 Running 0 2h + ingress-error-pages-586c7f86d6-w7cj2 1/1 Running 0 2h + keystone-api-7d9759db58-dz6kt 1/1 Running 0 1h + keystone-api-7d9759db58-pvsc2 1/1 Running 0 1h + libvirt-f7ngc 1/1 Running 0 24m + libvirt-gtjc7 1/1 Running 0 24m + libvirt-qmwf5 1/1 Running 0 24m + mariadb-ingress-84894687fd-m8fkr 1/1 Running 0 1h + mariadb-ingress-error-pages-78fb865f84-c6th5 1/1 Running 0 1h + mariadb-server-0 1/1 Running 0 1h + memcached-memcached-5db74ddfd5-qjgvz 1/1 Running 0 1h + neutron-dhcp-agent-default-9bpxc 1/1 Running 0 16m + neutron-l3-agent-default-47n7k 1/1 Running 0 16m + neutron-metadata-agent-default-hp46c 1/1 Running 0 16m + neutron-ovs-agent-default-6sbtg 1/1 Running 0 16m + neutron-ovs-agent-default-nl8fr 1/1 Running 0 16m + neutron-ovs-agent-default-tvmc4 1/1 Running 0 16m + neutron-server-775c765d9f-cx2gk 1/1 Running 0 16m + neutron-server-775c765d9f-ll5ml 1/1 Running 0 16m + nova-api-metadata-557c68cb46-8f8d5 1/1 Running 1 16m + nova-api-osapi-7658bfd554-7fbtx 1/1 Running 0 16m + nova-api-osapi-7658bfd554-v7qcr 1/1 Running 0 16m + nova-compute-default-g2jbd 1/1 Running 0 16m + nova-compute-default-ljcbc 1/1 Running 0 16m + nova-compute-default-mr24c 1/1 Running 0 16m + nova-conductor-64457cf995-lbv65 1/1 Running 0 16m + nova-conductor-64457cf995-zts48 1/1 Running 0 16m + nova-consoleauth-c595f68bc-2269j 1/1 Running 0 16m + nova-consoleauth-c595f68bc-tbmcn 1/1 Running 0 16m + nova-novncproxy-54467b9c66-vp49j 1/1 Running 0 16m + nova-placement-api-655cd9fcf6-5p5sc 1/1 Running 0 16m + nova-placement-api-655cd9fcf6-r46dk 1/1 Running 0 16m + nova-scheduler-59647c6d9f-vm78p 1/1 Running 0 16m + openvswitch-db-cv47r 1/1 Running 0 41m + openvswitch-db-dq7rc 1/1 Running 0 41m + openvswitch-db-znp6l 1/1 Running 0 41m + openvswitch-vswitchd-8p2j5 1/1 Running 0 41m + openvswitch-vswitchd-v9rrp 1/1 Running 0 41m + openvswitch-vswitchd-wlgkx 1/1 Running 0 41m + rabbitmq-rabbitmq-0 1/1 Running 0 1h + rabbitmq-rabbitmq-1 1/1 Running 0 1h + rabbitmq-rabbitmq-2 1/1 Running 0 1h + + +9) Upgrade Ceph charts to update version + +Use Ceph override file ``ceph.yaml`` that was generated previously and update +images section as below + +``cp /tmp/ceph.yaml ceph-update.yaml`` + +Update, image section in new overrides ``ceph-update.yaml`` as shown below + +.. code-block:: yaml + + images: + tags: + ceph_bootstrap: 'docker.io/ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64' + ceph_config_helper: 'docker.io/port/ceph-config-helper:v1.10.3' + ceph_rbd_pool: 'docker.io/port/ceph-config-helper:v1.10.3' + ceph_mon_check: 'docker.io/port/ceph-config-helper:v1.10.3' + ceph_mon: 'docker.io/ceph/daemon:master-a8d20ed-luminous-ubuntu-16.04-x86_64' + ceph_osd: 'docker.io/ceph/daemon:master-a8d20ed-luminous-ubuntu-16.04-x86_64' + ceph_mds: 'docker.io/ceph/daemon:master-a8d20ed-luminous-ubuntu-16.04-x86_64' + ceph_mgr: 'docker.io/ceph/daemon:master-a8d20ed-luminous-ubuntu-16.04-x86_64' + ceph_rgw: 'docker.io/ceph/daemon:master-a8d20ed-luminous-ubuntu-16.04-x86_64' + ceph_cephfs_provisioner: 'quay.io/external_storage/cephfs-provisioner:v0.1.2' + ceph_rbd_provisioner: 'quay.io/external_storage/rbd-provisioner:v0.1.1' + + +10) Update Ceph Mon chart with new overrides + + +``helm upgrade ceph-mon ./ceph-mon --values=ceph-update.yaml`` + +``series of console outputs:`` + +.. code-block:: console + + ceph-mon-4c8xs 0/1 Terminating 0 2h + ceph-mon-check-d85994946-zzwb4 1/1 Running 0 2h + ceph-mon-keyring-generator-jdgfw 0/1 Completed 0 2h + ceph-mon-kht8d 1/1 Running 0 2h + ceph-mon-mkpmm 1/1 Running 0 2h + +.. code-block:: console + + ceph-mon-7zxjs 1/1 Running 1 4m + ceph-mon-84xt2 1/1 Running 1 2m + ceph-mon-check-d85994946-zzwb4 1/1 Running 0 2h + ceph-mon-fsrv4 1/1 Running 1 6m + ceph-mon-keyring-generator-jdgfw 0/1 Completed 0 2h + + +``Results:`` Mon pods got updated one by one (rolling updates). Each Mon pod +got respawn and was in 1/1 running state before next Mon pod got updated. +Each Mon pod got restarted. Other ceph pods were not affected with this update. +No interruption to OSH pods. + + +11) Update Ceph OSD chart with new overrides: + +``helm upgrade ceph-osd ./ceph-osd --values=ceph-update.yaml`` + +``series of console outputs:`` + +.. code-block:: console + + ceph-osd-default-83945928-7jz4s 0/1 Terminating 0 2h + ceph-osd-default-83945928-bh82j 1/1 Running 0 2h + ceph-osd-default-83945928-t9szk 1/1 Running 0 2h + ceph-osd-keyring-generator-6rg65 0/1 Completed 0 2h + +.. code-block:: console + + ceph-osd-default-83945928-l84tl 1/1 Running 0 9m + ceph-osd-default-83945928-twzmk 1/1 Running 0 6m + ceph-osd-default-83945928-wxpmh 1/1 Running 0 11m + ceph-osd-keyring-generator-6rg65 0/1 Completed 0 2h + +``Results:`` Rolling updates (one pod at a time). Other ceph pods are running. +No interruption to OSH pods. + + +12) Update Ceph Client chart with new overrides: + +``helm upgrade ceph-client ./ceph-client --values=ceph-update.yaml`` + +.. code-block:: console + + ceph-mds-5fdcb5c64c-t9nmb 0/1 Init:0/2 0 11s + ceph-mds-745576757f-4vdn4 1/1 Running 0 2h + ceph-mds-745576757f-bxdcs 1/1 Running 0 2h + ceph-mgr-86bdc7c64b-7ptr4 1/1 Terminating 0 2h + ceph-mgr-86bdc7c64b-xgplj 0/1 Terminating 0 2h + ceph-rgw-57c68b7cd5-vxcc5 0/1 Init:1/3 0 11s + ceph-rgw-74559877-h56xs 1/1 Running 0 2h + ceph-rgw-74559877-pfjr5 1/1 Running 0 2h + +.. code-block:: console + + ceph-mds-5fdcb5c64c-c52xq 1/1 Running 0 2m + ceph-mds-5fdcb5c64c-t9nmb 1/1 Running 0 2m + ceph-mgr-654f97cbfd-9kcvb 1/1 Running 0 1m + ceph-mgr-654f97cbfd-gzb7k 1/1 Running 0 1m + ceph-rgw-57c68b7cd5-vxcc5 1/1 Running 0 2m + ceph-rgw-57c68b7cd5-zmdqb 1/1 Running 0 2m + +``Results:`` Rolling updates (one pod at a time). Other ceph pods are running. +No interruption to OSH pods. + +13) Update Ceph Provisioners chart with new overrides: + +``helm upgrade ceph-provisioners ./ceph-provisioners --values=ceph-update.yaml`` + +.. code-block:: console + + ceph-cephfs-provisioner-784c6f9d59-2z6ww 0/1 Terminating 0 2h + ceph-cephfs-provisioner-784c6f9d59-sg8wv 0/1 Terminating 0 2h + ceph-rbd-provisioner-84665cb84f-6s55r 0/1 Terminating 0 2h + ceph-rbd-provisioner-84665cb84f-chwhd 0/1 Terminating 0 2h + + +.. code-block:: console + + ceph-cephfs-provisioner-65ffbd47c4-cl4hj 1/1 Running 0 1m + ceph-cephfs-provisioner-65ffbd47c4-qrtg2 1/1 Running 0 1m + ceph-rbd-provisioner-5bfb577ffd-b7tkx 1/1 Running 0 1m + ceph-rbd-provisioner-5bfb577ffd-m6gg6 1/1 Running 0 1m + +``Results:`` All provisioner pods got terminated at once (same time). Other ceph +pods are running. No interruption to OSH pods. + +14) Capture final Ceph pod statuses: + +.. code-block:: console + + ceph-bootstrap-s4jkx 0/1 Completed 0 2h + ceph-cephfs-client-key-generator-6bmzz 0/1 Completed 0 2h + ceph-cephfs-provisioner-65ffbd47c4-cl4hj 1/1 Running 0 2m + ceph-cephfs-provisioner-65ffbd47c4-qrtg2 1/1 Running 0 2m + ceph-mds-5fdcb5c64c-c52xq 1/1 Running 0 8m + ceph-mds-5fdcb5c64c-t9nmb 1/1 Running 0 8m + ceph-mds-keyring-generator-f5lxf 0/1 Completed 0 2h + ceph-mgr-654f97cbfd-9kcvb 1/1 Running 0 8m + ceph-mgr-654f97cbfd-gzb7k 1/1 Running 0 8m + ceph-mgr-keyring-generator-w7nxq 0/1 Completed 0 2h + ceph-mon-7zxjs 1/1 Running 1 27m + ceph-mon-84xt2 1/1 Running 1 24m + ceph-mon-check-d85994946-zzwb4 1/1 Running 0 2h + ceph-mon-fsrv4 1/1 Running 1 29m + ceph-mon-keyring-generator-jdgfw 0/1 Completed 0 2h + ceph-osd-default-83945928-l84tl 1/1 Running 0 19m + ceph-osd-default-83945928-twzmk 1/1 Running 0 16m + ceph-osd-default-83945928-wxpmh 1/1 Running 0 21m + ceph-osd-keyring-generator-6rg65 0/1 Completed 0 2h + ceph-rbd-pool-z8vlc 0/1 Completed 0 2h + ceph-rbd-provisioner-5bfb577ffd-b7tkx 1/1 Running 0 2m + ceph-rbd-provisioner-5bfb577ffd-m6gg6 1/1 Running 0 2m + ceph-rgw-57c68b7cd5-vxcc5 1/1 Running 0 8m + ceph-rgw-57c68b7cd5-zmdqb 1/1 Running 0 8m + ceph-rgw-keyring-generator-6rwct 0/1 Completed 0 2h + ceph-storage-keys-generator-bgj2t 0/1 Completed 0 2h + ingress-796d8cf8d6-nzrd2 1/1 Running 0 2h + ingress-796d8cf8d6-qqvq9 1/1 Running 0 2h + ingress-error-pages-54454dc79b-d5r5w 1/1 Running 0 2h + ingress-error-pages-54454dc79b-gfpqv 1/1 Running 0 2h + +15) Capture final Openstack pod statuses: + +.. code-block:: console + + cinder-api-67495cdffc-24fhs 1/1 Running 0 1h + cinder-api-67495cdffc-kz5fn 1/1 Running 0 1h + cinder-backup-65b7bd9b79-8n9pb 1/1 Running 0 1h + cinder-scheduler-9ddbb7878-rbt4l 1/1 Running 0 1h + cinder-volume-75bf4cc9bd-6298x 1/1 Running 0 1h + glance-api-68f6df4d5d-q84hs 1/1 Running 0 2h + glance-api-68f6df4d5d-qbfwb 1/1 Running 0 2h + glance-registry-5957979dc5-h54tc 1/1 Running 0 2h + glance-registry-5957979dc5-sglg7 1/1 Running 0 2h + ingress-7b4bc84cdd-84dtj 1/1 Running 0 2h + ingress-7b4bc84cdd-ws45r 1/1 Running 0 2h + ingress-error-pages-586c7f86d6-dlpm2 1/1 Running 0 2h + ingress-error-pages-586c7f86d6-w7cj2 1/1 Running 0 2h + keystone-api-7d9759db58-dz6kt 1/1 Running 0 2h + keystone-api-7d9759db58-pvsc2 1/1 Running 0 2h + libvirt-f7ngc 1/1 Running 0 1h + libvirt-gtjc7 1/1 Running 0 1h + libvirt-qmwf5 1/1 Running 0 1h + mariadb-ingress-84894687fd-m8fkr 1/1 Running 0 2h + mariadb-ingress-error-pages-78fb865f84-c6th5 1/1 Running 0 2h + mariadb-server-0 1/1 Running 0 2h + memcached-memcached-5db74ddfd5-qjgvz 1/1 Running 0 2h + neutron-dhcp-agent-default-9bpxc 1/1 Running 0 52m + neutron-l3-agent-default-47n7k 1/1 Running 0 52m + neutron-metadata-agent-default-hp46c 1/1 Running 0 52m + neutron-ovs-agent-default-6sbtg 1/1 Running 0 52m + neutron-ovs-agent-default-nl8fr 1/1 Running 0 52m + neutron-ovs-agent-default-tvmc4 1/1 Running 0 52m + neutron-server-775c765d9f-cx2gk 1/1 Running 0 52m + neutron-server-775c765d9f-ll5ml 1/1 Running 0 52m + nova-api-metadata-557c68cb46-8f8d5 1/1 Running 1 52m + nova-api-osapi-7658bfd554-7fbtx 1/1 Running 0 52m + nova-api-osapi-7658bfd554-v7qcr 1/1 Running 0 52m + nova-compute-default-g2jbd 1/1 Running 0 52m + nova-compute-default-ljcbc 1/1 Running 0 52m + nova-compute-default-mr24c 1/1 Running 0 52m + nova-conductor-64457cf995-lbv65 1/1 Running 0 52m + nova-conductor-64457cf995-zts48 1/1 Running 0 52m + nova-consoleauth-c595f68bc-2269j 1/1 Running 0 52m + nova-consoleauth-c595f68bc-tbmcn 1/1 Running 0 52m + nova-novncproxy-54467b9c66-vp49j 1/1 Running 0 52m + nova-placement-api-655cd9fcf6-5p5sc 1/1 Running 0 52m + nova-placement-api-655cd9fcf6-r46dk 1/1 Running 0 52m + nova-scheduler-59647c6d9f-vm78p 1/1 Running 0 52m + openvswitch-db-cv47r 1/1 Running 0 1h + openvswitch-db-dq7rc 1/1 Running 0 1h + openvswitch-db-znp6l 1/1 Running 0 1h + openvswitch-vswitchd-8p2j5 1/1 Running 0 1h + openvswitch-vswitchd-v9rrp 1/1 Running 0 1h + openvswitch-vswitchd-wlgkx 1/1 Running 0 1h + rabbitmq-rabbitmq-0 1/1 Running 0 2h + rabbitmq-rabbitmq-1 1/1 Running 0 2h + rabbitmq-rabbitmq-2 1/1 Running 0 2h + +16) Confirm Ceph component's version. + +.. code-block:: console + + ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-fsrv4 -- ceph -v + ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable) + + ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-osd-default-83945928-l84tl -- ceph -v + ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable) + + ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-rgw-57c68b7cd5-vxcc5 -- ceph -v + ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable) + + ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-mgr-654f97cbfd-gzb7k -- ceph -v + ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable) + + ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-mds-5fdcb5c64c-c52xq -- ceph -v + ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable) + +17) Check which images Provisionors and Mon-Check PODs are using + +.. code-block:: console + + ubuntu@mnode1:/opt/openstack-helm$ kubectl describe pod -n ceph ceph-cephfs-provisioner-65ffbd47c4-cl4hj + + Containers: + ceph-cephfs-provisioner: + Container ID: docker://079f148c1fb9ba13ed6caa0ca9d1e63b455373020a565a212b5bd261cbaedb43 + Image: quay.io/external_storage/cephfs-provisioner:v0.1.2 + +.. code-block:: console + + ubuntu@mnode1:/opt/openstack-helm$ kubectl describe pod -n ceph ceph-rbd-provisioner-5bfb577ffd-b7tkx + + Containers: + ceph-rbd-provisioner: + Container ID: docker://55b18b3400e8753f49f1343ee918a308ed1760816a1ce9797281dbfe3c5f9671 + Image: quay.io/external_storage/rbd-provisioner:v0.1.1 + +.. code-block:: console + + ubuntu@mnode1:/opt/openstack-helm$ kubectl describe pod -n ceph ceph-mon-check-d85994946-zzwb4 + + Containers: + ceph-mon: + Container ID: docker://d5a3396f99704038ab8ef6bfe329013ed46472ebb8e26dddc140b621329f0f92 + Image: docker.io/port/ceph-config-helper:v1.10.3 + +Conclusion: +=========== +Ceph can be upgreade without downtime for Openstack components in a multinoe env. diff --git a/doc/source/testing/index.rst b/doc/source/testing/index.rst index db9f155726..79e3f575ac 100644 --- a/doc/source/testing/index.rst +++ b/doc/source/testing/index.rst @@ -7,3 +7,4 @@ Testing helm-tests ceph-resiliency/index + ceph-upgrade