Merge "Document Ceph upgrade test scenario"
This commit is contained in:
Zuul
Gerrit Code Review
commit
a3270762cc
593
doc/source/testing/ceph-upgrade.rst
Normal file
593
doc/source/testing/ceph-upgrade.rst
Normal file
@ -0,0 +1,593 @@
|
|||||||
|
============
|
||||||
|
Ceph Upgrade
|
||||||
|
============
|
||||||
|
|
||||||
|
This guide documents steps showing Ceph version upgrade. The main goal of this
|
||||||
|
document is to demostrate Ceph chart update without downtime for OSH components.
|
||||||
|
|
||||||
|
Test Scenario:
|
||||||
|
==============
|
||||||
|
Upgrade Ceph component version from ``12.2.4`` to ``12.2.5`` without downtime
|
||||||
|
to OSH components.
|
||||||
|
|
||||||
|
Setup:
|
||||||
|
======
|
||||||
|
- 3 Node (VM based) env.
|
||||||
|
- Followed OSH multinode guide steps to setup nodes and install k8 cluster
|
||||||
|
- Followed OSH multinode guide steps upto Ceph install
|
||||||
|
|
||||||
|
Plan:
|
||||||
|
=====
|
||||||
|
1) Install Ceph charts (12.2.4) by updating Docker images in overrides.
|
||||||
|
2) Install OSH components as per OSH multinode guide.
|
||||||
|
3) Upgrade Ceph charts to version 12.2.5 by updating docker images in overrides.
|
||||||
|
|
||||||
|
|
||||||
|
Docker Images:
|
||||||
|
==============
|
||||||
|
1) Ceph Luminous point release images for Ceph components
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
Ceph 12.2.4: ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64
|
||||||
|
Ceph 12.2.5: ceph/daemon:master-a8d20ed-luminous-ubuntu-16.04-x86_64
|
||||||
|
|
||||||
|
2) Ceph RBD provisioner docker images.
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
quay.io/external_storage/rbd-provisioner:v0.1.0
|
||||||
|
quay.io/external_storage/rbd-provisioner:v0.1.1
|
||||||
|
|
||||||
|
3) Ceph Cephfs provisioner docker images.
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
quay.io/external_storage/cephfs-provisioner:v0.1.1
|
||||||
|
quay.io/external_storage/cephfs-provisioner:v0.1.2
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
======
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
Follow all steps from OSH multinode guide with below changes.
|
||||||
|
|
||||||
|
1) Install Ceph charts (version 12.2.4)
|
||||||
|
|
||||||
|
|
||||||
|
Update ceph install script ``./tools/deployment/multinode/030-ceph.sh``
|
||||||
|
to add ``images:`` section in overrides as shown below.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
OSD count is set to 3 based on env setup.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
Following is a partial part from script to show changes.
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
tee /tmp/ceph.yaml << EOF
|
||||||
|
...
|
||||||
|
network:
|
||||||
|
public: ${CEPH_PUBLIC_NETWORK}
|
||||||
|
cluster: ${CEPH_CLUSTER_NETWORK}
|
||||||
|
images:
|
||||||
|
tags:
|
||||||
|
ceph_bootstrap: 'docker.io/ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64'
|
||||||
|
ceph_config_helper: 'docker.io/port/ceph-config-helper:v1.10.3'
|
||||||
|
ceph_rbd_pool: 'docker.io/port/ceph-config-helper:v1.10.3'
|
||||||
|
ceph_mon_check: 'docker.io/port/ceph-config-helper:v1.10.3'
|
||||||
|
ceph_mon: 'docker.io/ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64'
|
||||||
|
ceph_osd: 'docker.io/ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64'
|
||||||
|
ceph_mds: 'docker.io/ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64'
|
||||||
|
ceph_mgr: 'docker.io/ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64'
|
||||||
|
ceph_rgw: 'docker.io/ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64'
|
||||||
|
ceph_cephfs_provisioner: 'quay.io/external_storage/cephfs-provisioner:v0.1.1'
|
||||||
|
ceph_rbd_provisioner: 'quay.io/external_storage/rbd-provisioner:v0.1.0'
|
||||||
|
conf:
|
||||||
|
ceph:
|
||||||
|
global:
|
||||||
|
fsid: ${CEPH_FS_ID}
|
||||||
|
rgw_ks:
|
||||||
|
enabled: true
|
||||||
|
pool:
|
||||||
|
crush:
|
||||||
|
tunables: ${CRUSH_TUNABLES}
|
||||||
|
target:
|
||||||
|
# NOTE(portdirect): 5 nodes, with one osd per node
|
||||||
|
osd: 5
|
||||||
|
pg_per_osd: 100
|
||||||
|
...
|
||||||
|
EOF
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
``ceph_bootstrap``, ``ceph-config_helper`` and ``ceph_rbs_pool`` images
|
||||||
|
are used for jobs. ``ceph_mon_check`` has one script that is stable so no
|
||||||
|
need to upgrade.
|
||||||
|
|
||||||
|
2) Deploy and Validate Ceph
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
+ kubectl exec -n ceph ceph-mon-4c8xs -- ceph -s
|
||||||
|
cluster:
|
||||||
|
id: 39061799-d25e-4f3b-8c1a-a350e4c6d06c
|
||||||
|
health: HEALTH_OK
|
||||||
|
|
||||||
|
services:
|
||||||
|
mon: 3 daemons, quorum mnode1,mnode2,mnode3
|
||||||
|
mgr: mnode2(active), standbys: mnode3
|
||||||
|
mds: cephfs-1/1/1 up {0=mds-ceph-mds-745576757f-4vdn4=up:active}, 1 up:standby
|
||||||
|
osd: 3 osds: 3 up, 3 in
|
||||||
|
rgw: 2 daemons active
|
||||||
|
|
||||||
|
data:
|
||||||
|
pools: 18 pools, 93 pgs
|
||||||
|
objects: 208 objects, 3359 bytes
|
||||||
|
usage: 72175 MB used, 75739 MB / 144 GB avail
|
||||||
|
pgs: 93 active+clean
|
||||||
|
|
||||||
|
3) Check Ceph Pods
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ubuntu@mnode1:/opt/openstack-helm$ kubectl get pods -n ceph
|
||||||
|
NAME READY STATUS RESTARTS AGE
|
||||||
|
ceph-bootstrap-s4jkx 0/1 Completed 0 10m
|
||||||
|
ceph-cephfs-client-key-generator-6bmzz 0/1 Completed 0 3m
|
||||||
|
ceph-cephfs-provisioner-784c6f9d59-2z6ww 1/1 Running 0 3m
|
||||||
|
ceph-cephfs-provisioner-784c6f9d59-sg8wv 1/1 Running 0 3m
|
||||||
|
ceph-mds-745576757f-4vdn4 1/1 Running 0 6m
|
||||||
|
ceph-mds-745576757f-bxdcs 1/1 Running 0 6m
|
||||||
|
ceph-mds-keyring-generator-f5lxf 0/1 Completed 0 10m
|
||||||
|
ceph-mgr-86bdc7c64b-7ptr4 1/1 Running 0 6m
|
||||||
|
ceph-mgr-86bdc7c64b-xgplj 1/1 Running 0 6m
|
||||||
|
ceph-mgr-keyring-generator-w7nxq 0/1 Completed 0 10m
|
||||||
|
ceph-mon-4c8xs 1/1 Running 0 10m
|
||||||
|
ceph-mon-check-d85994946-zzwb4 1/1 Running 0 10m
|
||||||
|
ceph-mon-keyring-generator-jdgfw 0/1 Completed 0 10m
|
||||||
|
ceph-mon-kht8d 1/1 Running 0 10m
|
||||||
|
ceph-mon-mkpmm 1/1 Running 0 10m
|
||||||
|
ceph-osd-default-83945928-7jz4s 1/1 Running 0 8m
|
||||||
|
ceph-osd-default-83945928-bh82j 1/1 Running 0 8m
|
||||||
|
ceph-osd-default-83945928-t9szk 1/1 Running 0 8m
|
||||||
|
ceph-osd-keyring-generator-6rg65 0/1 Completed 0 10m
|
||||||
|
ceph-rbd-pool-z8vlc 0/1 Completed 0 6m
|
||||||
|
ceph-rbd-provisioner-84665cb84f-6s55r 1/1 Running 0 3m
|
||||||
|
ceph-rbd-provisioner-84665cb84f-chwhd 1/1 Running 0 3m
|
||||||
|
ceph-rgw-74559877-h56xs 1/1 Running 0 6m
|
||||||
|
ceph-rgw-74559877-pfjr5 1/1 Running 0 6m
|
||||||
|
ceph-rgw-keyring-generator-6rwct 0/1 Completed 0 10m
|
||||||
|
ceph-storage-keys-generator-bgj2t 0/1 Completed 0 10m
|
||||||
|
ingress-796d8cf8d6-nzrd2 1/1 Running 0 11m
|
||||||
|
ingress-796d8cf8d6-qqvq9 1/1 Running 0 11m
|
||||||
|
ingress-error-pages-54454dc79b-d5r5w 1/1 Running 0 11m
|
||||||
|
ingress-error-pages-54454dc79b-gfpqv 1/1 Running 0 11m
|
||||||
|
|
||||||
|
|
||||||
|
4) Check version of each Ceph components.
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-4c8xs -- ceph -v
|
||||||
|
ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
|
||||||
|
|
||||||
|
ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-osd-default-83945928-7jz4s -- ceph -v
|
||||||
|
ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
|
||||||
|
|
||||||
|
ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-mgr-86bdc7c64b-7ptr4 -- ceph -v
|
||||||
|
ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
|
||||||
|
|
||||||
|
ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-mds-745576757f-4vdn4 -- ceph -v
|
||||||
|
ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
|
||||||
|
|
||||||
|
ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-rgw-74559877-h56xs -- ceph -v
|
||||||
|
ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
|
||||||
|
|
||||||
|
5) Check which images Provisionors and Mon-Check PODs are using
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
Showing partial output from kubectl describe command to show which image is Docker
|
||||||
|
container is using
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ubuntu@mnode1:~$ kubectl describe pod -n ceph ceph-cephfs-provisioner-784c6f9d59-2z6ww
|
||||||
|
|
||||||
|
Containers:
|
||||||
|
ceph-cephfs-provisioner:
|
||||||
|
Container ID: docker://98ed65617f6c4b60fe60d94b8707e52e0dd4c87791e7d72789a0cb603fa80e2c
|
||||||
|
Image: quay.io/external_storage/cephfs-provisioner:v0.1.1
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ubuntu@mnode1:~$ kubectl describe pod -n ceph ceph-rbd-provisioner-84665cb84f-6s55r
|
||||||
|
|
||||||
|
Containers:
|
||||||
|
ceph-rbd-provisioner:
|
||||||
|
Container ID: docker://383be3d653cecf4cbf0c3c7509774d39dce54102309f1f0bdb07cdc2441e5e47
|
||||||
|
Image: quay.io/external_storage/rbd-provisioner:v0.1.0
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ubuntu@mnode1:~$ kubectl describe pod -n ceph ceph-mon-check-d85994946-zzwb4
|
||||||
|
|
||||||
|
Containers:
|
||||||
|
ceph-mon:
|
||||||
|
Container ID: docker://d5a3396f99704038ab8ef6bfe329013ed46472ebb8e26dddc140b621329f0f92
|
||||||
|
Image: docker.io/port/ceph-config-helper:v1.10.3
|
||||||
|
|
||||||
|
|
||||||
|
6) Install Openstack charts
|
||||||
|
|
||||||
|
Continue with OSH multinode guide to install other Openstack charts.
|
||||||
|
|
||||||
|
7) Capture Ceph pods statuses.
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
NAME READY STATUS RESTARTS AGE
|
||||||
|
ceph-bootstrap-s4jkx 0/1 Completed 0 2h
|
||||||
|
ceph-cephfs-client-key-generator-6bmzz 0/1 Completed 0 2h
|
||||||
|
ceph-cephfs-provisioner-784c6f9d59-2z6ww 1/1 Running 0 2h
|
||||||
|
ceph-cephfs-provisioner-784c6f9d59-sg8wv 1/1 Running 0 2h
|
||||||
|
ceph-mds-745576757f-4vdn4 1/1 Running 0 2h
|
||||||
|
ceph-mds-745576757f-bxdcs 1/1 Running 0 2h
|
||||||
|
ceph-mds-keyring-generator-f5lxf 0/1 Completed 0 2h
|
||||||
|
ceph-mgr-86bdc7c64b-7ptr4 1/1 Running 0 2h
|
||||||
|
ceph-mgr-86bdc7c64b-xgplj 1/1 Running 0 2h
|
||||||
|
ceph-mgr-keyring-generator-w7nxq 0/1 Completed 0 2h
|
||||||
|
ceph-mon-4c8xs 1/1 Running 0 2h
|
||||||
|
ceph-mon-check-d85994946-zzwb4 1/1 Running 0 2h
|
||||||
|
ceph-mon-keyring-generator-jdgfw 0/1 Completed 0 2h
|
||||||
|
ceph-mon-kht8d 1/1 Running 0 2h
|
||||||
|
ceph-mon-mkpmm 1/1 Running 0 2h
|
||||||
|
ceph-osd-default-83945928-7jz4s 1/1 Running 0 2h
|
||||||
|
ceph-osd-default-83945928-bh82j 1/1 Running 0 2h
|
||||||
|
ceph-osd-default-83945928-t9szk 1/1 Running 0 2h
|
||||||
|
ceph-osd-keyring-generator-6rg65 0/1 Completed 0 2h
|
||||||
|
ceph-rbd-pool-z8vlc 0/1 Completed 0 2h
|
||||||
|
ceph-rbd-provisioner-84665cb84f-6s55r 1/1 Running 0 2h
|
||||||
|
ceph-rbd-provisioner-84665cb84f-chwhd 1/1 Running 0 2h
|
||||||
|
ceph-rgw-74559877-h56xs 1/1 Running 0 2h
|
||||||
|
ceph-rgw-74559877-pfjr5 1/1 Running 0 2h
|
||||||
|
ceph-rgw-keyring-generator-6rwct 0/1 Completed 0 2h
|
||||||
|
ceph-storage-keys-generator-bgj2t 0/1 Completed 0 2h
|
||||||
|
ingress-796d8cf8d6-nzrd2 1/1 Running 0 2h
|
||||||
|
ingress-796d8cf8d6-qqvq9 1/1 Running 0 2h
|
||||||
|
ingress-error-pages-54454dc79b-d5r5w 1/1 Running 0 2h
|
||||||
|
ingress-error-pages-54454dc79b-gfpqv 1/1 Running 0 2h
|
||||||
|
|
||||||
|
8) Capture Openstack pods statuses.
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
NAME READY STATUS RESTARTS AGE
|
||||||
|
cinder-api-67495cdffc-24fhs 1/1 Running 0 51m
|
||||||
|
cinder-api-67495cdffc-kz5fn 1/1 Running 0 51m
|
||||||
|
cinder-backup-65b7bd9b79-8n9pb 1/1 Running 0 51m
|
||||||
|
cinder-scheduler-9ddbb7878-rbt4l 1/1 Running 0 51m
|
||||||
|
cinder-volume-75bf4cc9bd-6298x 1/1 Running 0 51m
|
||||||
|
glance-api-68f6df4d5d-q84hs 1/1 Running 0 1h
|
||||||
|
glance-api-68f6df4d5d-qbfwb 1/1 Running 0 1h
|
||||||
|
glance-registry-5957979dc5-h54tc 1/1 Running 0 1h
|
||||||
|
glance-registry-5957979dc5-sglg7 1/1 Running 0 1h
|
||||||
|
ingress-7b4bc84cdd-84dtj 1/1 Running 0 2h
|
||||||
|
ingress-7b4bc84cdd-ws45r 1/1 Running 0 2h
|
||||||
|
ingress-error-pages-586c7f86d6-dlpm2 1/1 Running 0 2h
|
||||||
|
ingress-error-pages-586c7f86d6-w7cj2 1/1 Running 0 2h
|
||||||
|
keystone-api-7d9759db58-dz6kt 1/1 Running 0 1h
|
||||||
|
keystone-api-7d9759db58-pvsc2 1/1 Running 0 1h
|
||||||
|
libvirt-f7ngc 1/1 Running 0 24m
|
||||||
|
libvirt-gtjc7 1/1 Running 0 24m
|
||||||
|
libvirt-qmwf5 1/1 Running 0 24m
|
||||||
|
mariadb-ingress-84894687fd-m8fkr 1/1 Running 0 1h
|
||||||
|
mariadb-ingress-error-pages-78fb865f84-c6th5 1/1 Running 0 1h
|
||||||
|
mariadb-server-0 1/1 Running 0 1h
|
||||||
|
memcached-memcached-5db74ddfd5-qjgvz 1/1 Running 0 1h
|
||||||
|
neutron-dhcp-agent-default-9bpxc 1/1 Running 0 16m
|
||||||
|
neutron-l3-agent-default-47n7k 1/1 Running 0 16m
|
||||||
|
neutron-metadata-agent-default-hp46c 1/1 Running 0 16m
|
||||||
|
neutron-ovs-agent-default-6sbtg 1/1 Running 0 16m
|
||||||
|
neutron-ovs-agent-default-nl8fr 1/1 Running 0 16m
|
||||||
|
neutron-ovs-agent-default-tvmc4 1/1 Running 0 16m
|
||||||
|
neutron-server-775c765d9f-cx2gk 1/1 Running 0 16m
|
||||||
|
neutron-server-775c765d9f-ll5ml 1/1 Running 0 16m
|
||||||
|
nova-api-metadata-557c68cb46-8f8d5 1/1 Running 1 16m
|
||||||
|
nova-api-osapi-7658bfd554-7fbtx 1/1 Running 0 16m
|
||||||
|
nova-api-osapi-7658bfd554-v7qcr 1/1 Running 0 16m
|
||||||
|
nova-compute-default-g2jbd 1/1 Running 0 16m
|
||||||
|
nova-compute-default-ljcbc 1/1 Running 0 16m
|
||||||
|
nova-compute-default-mr24c 1/1 Running 0 16m
|
||||||
|
nova-conductor-64457cf995-lbv65 1/1 Running 0 16m
|
||||||
|
nova-conductor-64457cf995-zts48 1/1 Running 0 16m
|
||||||
|
nova-consoleauth-c595f68bc-2269j 1/1 Running 0 16m
|
||||||
|
nova-consoleauth-c595f68bc-tbmcn 1/1 Running 0 16m
|
||||||
|
nova-novncproxy-54467b9c66-vp49j 1/1 Running 0 16m
|
||||||
|
nova-placement-api-655cd9fcf6-5p5sc 1/1 Running 0 16m
|
||||||
|
nova-placement-api-655cd9fcf6-r46dk 1/1 Running 0 16m
|
||||||
|
nova-scheduler-59647c6d9f-vm78p 1/1 Running 0 16m
|
||||||
|
openvswitch-db-cv47r 1/1 Running 0 41m
|
||||||
|
openvswitch-db-dq7rc 1/1 Running 0 41m
|
||||||
|
openvswitch-db-znp6l 1/1 Running 0 41m
|
||||||
|
openvswitch-vswitchd-8p2j5 1/1 Running 0 41m
|
||||||
|
openvswitch-vswitchd-v9rrp 1/1 Running 0 41m
|
||||||
|
openvswitch-vswitchd-wlgkx 1/1 Running 0 41m
|
||||||
|
rabbitmq-rabbitmq-0 1/1 Running 0 1h
|
||||||
|
rabbitmq-rabbitmq-1 1/1 Running 0 1h
|
||||||
|
rabbitmq-rabbitmq-2 1/1 Running 0 1h
|
||||||
|
|
||||||
|
|
||||||
|
9) Upgrade Ceph charts to update version
|
||||||
|
|
||||||
|
Use Ceph override file ``ceph.yaml`` that was generated previously and update
|
||||||
|
images section as below
|
||||||
|
|
||||||
|
``cp /tmp/ceph.yaml ceph-update.yaml``
|
||||||
|
|
||||||
|
Update, image section in new overrides ``ceph-update.yaml`` as shown below
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
images:
|
||||||
|
tags:
|
||||||
|
ceph_bootstrap: 'docker.io/ceph/daemon:master-0351083-luminous-ubuntu-16.04-x86_64'
|
||||||
|
ceph_config_helper: 'docker.io/port/ceph-config-helper:v1.10.3'
|
||||||
|
ceph_rbd_pool: 'docker.io/port/ceph-config-helper:v1.10.3'
|
||||||
|
ceph_mon_check: 'docker.io/port/ceph-config-helper:v1.10.3'
|
||||||
|
ceph_mon: 'docker.io/ceph/daemon:master-a8d20ed-luminous-ubuntu-16.04-x86_64'
|
||||||
|
ceph_osd: 'docker.io/ceph/daemon:master-a8d20ed-luminous-ubuntu-16.04-x86_64'
|
||||||
|
ceph_mds: 'docker.io/ceph/daemon:master-a8d20ed-luminous-ubuntu-16.04-x86_64'
|
||||||
|
ceph_mgr: 'docker.io/ceph/daemon:master-a8d20ed-luminous-ubuntu-16.04-x86_64'
|
||||||
|
ceph_rgw: 'docker.io/ceph/daemon:master-a8d20ed-luminous-ubuntu-16.04-x86_64'
|
||||||
|
ceph_cephfs_provisioner: 'quay.io/external_storage/cephfs-provisioner:v0.1.2'
|
||||||
|
ceph_rbd_provisioner: 'quay.io/external_storage/rbd-provisioner:v0.1.1'
|
||||||
|
|
||||||
|
|
||||||
|
10) Update Ceph Mon chart with new overrides
|
||||||
|
|
||||||
|
|
||||||
|
``helm upgrade ceph-mon ./ceph-mon --values=ceph-update.yaml``
|
||||||
|
|
||||||
|
``series of console outputs:``
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ceph-mon-4c8xs 0/1 Terminating 0 2h
|
||||||
|
ceph-mon-check-d85994946-zzwb4 1/1 Running 0 2h
|
||||||
|
ceph-mon-keyring-generator-jdgfw 0/1 Completed 0 2h
|
||||||
|
ceph-mon-kht8d 1/1 Running 0 2h
|
||||||
|
ceph-mon-mkpmm 1/1 Running 0 2h
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ceph-mon-7zxjs 1/1 Running 1 4m
|
||||||
|
ceph-mon-84xt2 1/1 Running 1 2m
|
||||||
|
ceph-mon-check-d85994946-zzwb4 1/1 Running 0 2h
|
||||||
|
ceph-mon-fsrv4 1/1 Running 1 6m
|
||||||
|
ceph-mon-keyring-generator-jdgfw 0/1 Completed 0 2h
|
||||||
|
|
||||||
|
|
||||||
|
``Results:`` Mon pods got updated one by one (rolling updates). Each Mon pod
|
||||||
|
got respawn and was in 1/1 running state before next Mon pod got updated.
|
||||||
|
Each Mon pod got restarted. Other ceph pods were not affected with this update.
|
||||||
|
No interruption to OSH pods.
|
||||||
|
|
||||||
|
|
||||||
|
11) Update Ceph OSD chart with new overrides:
|
||||||
|
|
||||||
|
``helm upgrade ceph-osd ./ceph-osd --values=ceph-update.yaml``
|
||||||
|
|
||||||
|
``series of console outputs:``
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ceph-osd-default-83945928-7jz4s 0/1 Terminating 0 2h
|
||||||
|
ceph-osd-default-83945928-bh82j 1/1 Running 0 2h
|
||||||
|
ceph-osd-default-83945928-t9szk 1/1 Running 0 2h
|
||||||
|
ceph-osd-keyring-generator-6rg65 0/1 Completed 0 2h
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ceph-osd-default-83945928-l84tl 1/1 Running 0 9m
|
||||||
|
ceph-osd-default-83945928-twzmk 1/1 Running 0 6m
|
||||||
|
ceph-osd-default-83945928-wxpmh 1/1 Running 0 11m
|
||||||
|
ceph-osd-keyring-generator-6rg65 0/1 Completed 0 2h
|
||||||
|
|
||||||
|
``Results:`` Rolling updates (one pod at a time). Other ceph pods are running.
|
||||||
|
No interruption to OSH pods.
|
||||||
|
|
||||||
|
|
||||||
|
12) Update Ceph Client chart with new overrides:
|
||||||
|
|
||||||
|
``helm upgrade ceph-client ./ceph-client --values=ceph-update.yaml``
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ceph-mds-5fdcb5c64c-t9nmb 0/1 Init:0/2 0 11s
|
||||||
|
ceph-mds-745576757f-4vdn4 1/1 Running 0 2h
|
||||||
|
ceph-mds-745576757f-bxdcs 1/1 Running 0 2h
|
||||||
|
ceph-mgr-86bdc7c64b-7ptr4 1/1 Terminating 0 2h
|
||||||
|
ceph-mgr-86bdc7c64b-xgplj 0/1 Terminating 0 2h
|
||||||
|
ceph-rgw-57c68b7cd5-vxcc5 0/1 Init:1/3 0 11s
|
||||||
|
ceph-rgw-74559877-h56xs 1/1 Running 0 2h
|
||||||
|
ceph-rgw-74559877-pfjr5 1/1 Running 0 2h
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ceph-mds-5fdcb5c64c-c52xq 1/1 Running 0 2m
|
||||||
|
ceph-mds-5fdcb5c64c-t9nmb 1/1 Running 0 2m
|
||||||
|
ceph-mgr-654f97cbfd-9kcvb 1/1 Running 0 1m
|
||||||
|
ceph-mgr-654f97cbfd-gzb7k 1/1 Running 0 1m
|
||||||
|
ceph-rgw-57c68b7cd5-vxcc5 1/1 Running 0 2m
|
||||||
|
ceph-rgw-57c68b7cd5-zmdqb 1/1 Running 0 2m
|
||||||
|
|
||||||
|
``Results:`` Rolling updates (one pod at a time). Other ceph pods are running.
|
||||||
|
No interruption to OSH pods.
|
||||||
|
|
||||||
|
13) Update Ceph Provisioners chart with new overrides:
|
||||||
|
|
||||||
|
``helm upgrade ceph-provisioners ./ceph-provisioners --values=ceph-update.yaml``
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ceph-cephfs-provisioner-784c6f9d59-2z6ww 0/1 Terminating 0 2h
|
||||||
|
ceph-cephfs-provisioner-784c6f9d59-sg8wv 0/1 Terminating 0 2h
|
||||||
|
ceph-rbd-provisioner-84665cb84f-6s55r 0/1 Terminating 0 2h
|
||||||
|
ceph-rbd-provisioner-84665cb84f-chwhd 0/1 Terminating 0 2h
|
||||||
|
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ceph-cephfs-provisioner-65ffbd47c4-cl4hj 1/1 Running 0 1m
|
||||||
|
ceph-cephfs-provisioner-65ffbd47c4-qrtg2 1/1 Running 0 1m
|
||||||
|
ceph-rbd-provisioner-5bfb577ffd-b7tkx 1/1 Running 0 1m
|
||||||
|
ceph-rbd-provisioner-5bfb577ffd-m6gg6 1/1 Running 0 1m
|
||||||
|
|
||||||
|
``Results:`` All provisioner pods got terminated at once (same time). Other ceph
|
||||||
|
pods are running. No interruption to OSH pods.
|
||||||
|
|
||||||
|
14) Capture final Ceph pod statuses:
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ceph-bootstrap-s4jkx 0/1 Completed 0 2h
|
||||||
|
ceph-cephfs-client-key-generator-6bmzz 0/1 Completed 0 2h
|
||||||
|
ceph-cephfs-provisioner-65ffbd47c4-cl4hj 1/1 Running 0 2m
|
||||||
|
ceph-cephfs-provisioner-65ffbd47c4-qrtg2 1/1 Running 0 2m
|
||||||
|
ceph-mds-5fdcb5c64c-c52xq 1/1 Running 0 8m
|
||||||
|
ceph-mds-5fdcb5c64c-t9nmb 1/1 Running 0 8m
|
||||||
|
ceph-mds-keyring-generator-f5lxf 0/1 Completed 0 2h
|
||||||
|
ceph-mgr-654f97cbfd-9kcvb 1/1 Running 0 8m
|
||||||
|
ceph-mgr-654f97cbfd-gzb7k 1/1 Running 0 8m
|
||||||
|
ceph-mgr-keyring-generator-w7nxq 0/1 Completed 0 2h
|
||||||
|
ceph-mon-7zxjs 1/1 Running 1 27m
|
||||||
|
ceph-mon-84xt2 1/1 Running 1 24m
|
||||||
|
ceph-mon-check-d85994946-zzwb4 1/1 Running 0 2h
|
||||||
|
ceph-mon-fsrv4 1/1 Running 1 29m
|
||||||
|
ceph-mon-keyring-generator-jdgfw 0/1 Completed 0 2h
|
||||||
|
ceph-osd-default-83945928-l84tl 1/1 Running 0 19m
|
||||||
|
ceph-osd-default-83945928-twzmk 1/1 Running 0 16m
|
||||||
|
ceph-osd-default-83945928-wxpmh 1/1 Running 0 21m
|
||||||
|
ceph-osd-keyring-generator-6rg65 0/1 Completed 0 2h
|
||||||
|
ceph-rbd-pool-z8vlc 0/1 Completed 0 2h
|
||||||
|
ceph-rbd-provisioner-5bfb577ffd-b7tkx 1/1 Running 0 2m
|
||||||
|
ceph-rbd-provisioner-5bfb577ffd-m6gg6 1/1 Running 0 2m
|
||||||
|
ceph-rgw-57c68b7cd5-vxcc5 1/1 Running 0 8m
|
||||||
|
ceph-rgw-57c68b7cd5-zmdqb 1/1 Running 0 8m
|
||||||
|
ceph-rgw-keyring-generator-6rwct 0/1 Completed 0 2h
|
||||||
|
ceph-storage-keys-generator-bgj2t 0/1 Completed 0 2h
|
||||||
|
ingress-796d8cf8d6-nzrd2 1/1 Running 0 2h
|
||||||
|
ingress-796d8cf8d6-qqvq9 1/1 Running 0 2h
|
||||||
|
ingress-error-pages-54454dc79b-d5r5w 1/1 Running 0 2h
|
||||||
|
ingress-error-pages-54454dc79b-gfpqv 1/1 Running 0 2h
|
||||||
|
|
||||||
|
15) Capture final Openstack pod statuses:
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
cinder-api-67495cdffc-24fhs 1/1 Running 0 1h
|
||||||
|
cinder-api-67495cdffc-kz5fn 1/1 Running 0 1h
|
||||||
|
cinder-backup-65b7bd9b79-8n9pb 1/1 Running 0 1h
|
||||||
|
cinder-scheduler-9ddbb7878-rbt4l 1/1 Running 0 1h
|
||||||
|
cinder-volume-75bf4cc9bd-6298x 1/1 Running 0 1h
|
||||||
|
glance-api-68f6df4d5d-q84hs 1/1 Running 0 2h
|
||||||
|
glance-api-68f6df4d5d-qbfwb 1/1 Running 0 2h
|
||||||
|
glance-registry-5957979dc5-h54tc 1/1 Running 0 2h
|
||||||
|
glance-registry-5957979dc5-sglg7 1/1 Running 0 2h
|
||||||
|
ingress-7b4bc84cdd-84dtj 1/1 Running 0 2h
|
||||||
|
ingress-7b4bc84cdd-ws45r 1/1 Running 0 2h
|
||||||
|
ingress-error-pages-586c7f86d6-dlpm2 1/1 Running 0 2h
|
||||||
|
ingress-error-pages-586c7f86d6-w7cj2 1/1 Running 0 2h
|
||||||
|
keystone-api-7d9759db58-dz6kt 1/1 Running 0 2h
|
||||||
|
keystone-api-7d9759db58-pvsc2 1/1 Running 0 2h
|
||||||
|
libvirt-f7ngc 1/1 Running 0 1h
|
||||||
|
libvirt-gtjc7 1/1 Running 0 1h
|
||||||
|
libvirt-qmwf5 1/1 Running 0 1h
|
||||||
|
mariadb-ingress-84894687fd-m8fkr 1/1 Running 0 2h
|
||||||
|
mariadb-ingress-error-pages-78fb865f84-c6th5 1/1 Running 0 2h
|
||||||
|
mariadb-server-0 1/1 Running 0 2h
|
||||||
|
memcached-memcached-5db74ddfd5-qjgvz 1/1 Running 0 2h
|
||||||
|
neutron-dhcp-agent-default-9bpxc 1/1 Running 0 52m
|
||||||
|
neutron-l3-agent-default-47n7k 1/1 Running 0 52m
|
||||||
|
neutron-metadata-agent-default-hp46c 1/1 Running 0 52m
|
||||||
|
neutron-ovs-agent-default-6sbtg 1/1 Running 0 52m
|
||||||
|
neutron-ovs-agent-default-nl8fr 1/1 Running 0 52m
|
||||||
|
neutron-ovs-agent-default-tvmc4 1/1 Running 0 52m
|
||||||
|
neutron-server-775c765d9f-cx2gk 1/1 Running 0 52m
|
||||||
|
neutron-server-775c765d9f-ll5ml 1/1 Running 0 52m
|
||||||
|
nova-api-metadata-557c68cb46-8f8d5 1/1 Running 1 52m
|
||||||
|
nova-api-osapi-7658bfd554-7fbtx 1/1 Running 0 52m
|
||||||
|
nova-api-osapi-7658bfd554-v7qcr 1/1 Running 0 52m
|
||||||
|
nova-compute-default-g2jbd 1/1 Running 0 52m
|
||||||
|
nova-compute-default-ljcbc 1/1 Running 0 52m
|
||||||
|
nova-compute-default-mr24c 1/1 Running 0 52m
|
||||||
|
nova-conductor-64457cf995-lbv65 1/1 Running 0 52m
|
||||||
|
nova-conductor-64457cf995-zts48 1/1 Running 0 52m
|
||||||
|
nova-consoleauth-c595f68bc-2269j 1/1 Running 0 52m
|
||||||
|
nova-consoleauth-c595f68bc-tbmcn 1/1 Running 0 52m
|
||||||
|
nova-novncproxy-54467b9c66-vp49j 1/1 Running 0 52m
|
||||||
|
nova-placement-api-655cd9fcf6-5p5sc 1/1 Running 0 52m
|
||||||
|
nova-placement-api-655cd9fcf6-r46dk 1/1 Running 0 52m
|
||||||
|
nova-scheduler-59647c6d9f-vm78p 1/1 Running 0 52m
|
||||||
|
openvswitch-db-cv47r 1/1 Running 0 1h
|
||||||
|
openvswitch-db-dq7rc 1/1 Running 0 1h
|
||||||
|
openvswitch-db-znp6l 1/1 Running 0 1h
|
||||||
|
openvswitch-vswitchd-8p2j5 1/1 Running 0 1h
|
||||||
|
openvswitch-vswitchd-v9rrp 1/1 Running 0 1h
|
||||||
|
openvswitch-vswitchd-wlgkx 1/1 Running 0 1h
|
||||||
|
rabbitmq-rabbitmq-0 1/1 Running 0 2h
|
||||||
|
rabbitmq-rabbitmq-1 1/1 Running 0 2h
|
||||||
|
rabbitmq-rabbitmq-2 1/1 Running 0 2h
|
||||||
|
|
||||||
|
16) Confirm Ceph component's version.
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-fsrv4 -- ceph -v
|
||||||
|
ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
|
||||||
|
|
||||||
|
ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-osd-default-83945928-l84tl -- ceph -v
|
||||||
|
ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
|
||||||
|
|
||||||
|
ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-rgw-57c68b7cd5-vxcc5 -- ceph -v
|
||||||
|
ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
|
||||||
|
|
||||||
|
ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-mgr-654f97cbfd-gzb7k -- ceph -v
|
||||||
|
ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
|
||||||
|
|
||||||
|
ubuntu@mnode1:/opt/openstack-helm$ kubectl exec -n ceph ceph-mds-5fdcb5c64c-c52xq -- ceph -v
|
||||||
|
ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
|
||||||
|
|
||||||
|
17) Check which images Provisionors and Mon-Check PODs are using
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ubuntu@mnode1:/opt/openstack-helm$ kubectl describe pod -n ceph ceph-cephfs-provisioner-65ffbd47c4-cl4hj
|
||||||
|
|
||||||
|
Containers:
|
||||||
|
ceph-cephfs-provisioner:
|
||||||
|
Container ID: docker://079f148c1fb9ba13ed6caa0ca9d1e63b455373020a565a212b5bd261cbaedb43
|
||||||
|
Image: quay.io/external_storage/cephfs-provisioner:v0.1.2
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ubuntu@mnode1:/opt/openstack-helm$ kubectl describe pod -n ceph ceph-rbd-provisioner-5bfb577ffd-b7tkx
|
||||||
|
|
||||||
|
Containers:
|
||||||
|
ceph-rbd-provisioner:
|
||||||
|
Container ID: docker://55b18b3400e8753f49f1343ee918a308ed1760816a1ce9797281dbfe3c5f9671
|
||||||
|
Image: quay.io/external_storage/rbd-provisioner:v0.1.1
|
||||||
|
|
||||||
|
.. code-block:: console
|
||||||
|
|
||||||
|
ubuntu@mnode1:/opt/openstack-helm$ kubectl describe pod -n ceph ceph-mon-check-d85994946-zzwb4
|
||||||
|
|
||||||
|
Containers:
|
||||||
|
ceph-mon:
|
||||||
|
Container ID: docker://d5a3396f99704038ab8ef6bfe329013ed46472ebb8e26dddc140b621329f0f92
|
||||||
|
Image: docker.io/port/ceph-config-helper:v1.10.3
|
||||||
|
|
||||||
|
Conclusion:
|
||||||
|
===========
|
||||||
|
Ceph can be upgreade without downtime for Openstack components in a multinoe env.
|
||||||
@ -7,3 +7,4 @@ Testing
|
|||||||
|
|
||||||
helm-tests
|
helm-tests
|
||||||
ceph-resiliency/index
|
ceph-resiliency/index
|
||||||
|
ceph-upgrade
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user