openstack
/
openstack-helm
Code Issues Proposed changes

Cleanup troubleshooting docs 

This PS deletes superfluous documentation and empty files from
the troubleshooting section, and also updates some of the docs
regarding persistent storage troubleshooting.

Change-Id: I60876e40ffe1a1f88ac6c761e8bf10beee92c2a0
This commit is contained in:
Larry Rensing 2017-07-03 15:47:43 -05:00
parent c0ab4757e0
commit 10c9e8a13a
6 changed files with 53 additions and 216 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
doc/source/install/multinode.rst
View File

@ -390,6 +390,12 @@ created, using the following commands:
kubectl exec -n ceph ceph-mon-0 -- ceph osd pool create images 8
kubectl exec -n ceph ceph-mon-0 -- ceph osd pool create vms 8
The number of placement groups can be altered by replacing the 8
to meet your needs. It is important to note that using too large
of a number for your placement groups may result in Ceph
becoming unhealthy. For more information on this topic, reference
Ceph's documentation `here <http://docs.ceph.com/docs/master/rados/operations/placement-groups/#choosing-the-number-of-placement-groups>`_.
MariaDB Installation and Verification
-------------------------------------

57
doc/source/troubleshooting/database.rst
View File

@ -8,34 +8,6 @@ deploying Charts in this repository.
Galera Cluster
==============
**CHART:** openstack-helm/mariadb (when ``developer-mode: false``)
MariaDB is a ``StatefulSet`` (``PetSets`` have been retired in
Kubernetes v1.5.0). As such, it initiates a 'seed' which is used to
deploy MariaDB members via `affinity/anti-affinity
<https://kubernetes.io/docs/user-guide/node-selection/>`__
features. Ceph uses this as well. So what you will notice is the
following behavior:
::
openstack mariadb-0 0/1 Running 0 28s 10.25.49.199 kubenode05
openstack mariadb-seed-0ckf4 1/1 Running 0 48s 10.25.162.197 kubenode01
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
openstack mariadb-0 1/1 Running 0 1m 10.25.49.199 kubenode05
openstack mariadb-1 0/1 Pending 0 0s <none>
openstack mariadb-1 0/1 Pending 0 0s <none> kubenode04
openstack mariadb-1 0/1 ContainerCreating 0 0s <none> kubenode04
openstack mariadb-1 0/1 Running 0 3s 10.25.178.74 kubenode04
What you're seeing is the output of
``kubectl get pods -o wide --all-namespaces``, which is used to monitor
the seed host preparing each of the MariaDB/Galera members in order:
mariadb-0, then mariadb-1, then mariadb-2. This process can take up to a
few minutes, so be patient.
To test MariaDB, do the following:
::
@ -50,32 +22,3 @@ To test MariaDB, do the following:
| performance_schema |
+--------------------+
admin@kubenode01:~/projects/openstack-helm$
Now you can see that MariaDB is loaded, with databases intact! If you're
at this point, the rest of the installation is easy. You can run the
following to check on Galera:
::
admin@kubenode01:~/projects/openstack-helm$ kubectl describe po/mariadb-0 -n openstack
Name: mariadb-0
Namespace: openstack
Node: kubenode05/192.168.3.25
Start Time: Fri, 23 Dec 2016 16:15:49 -0500
Labels: app=mariadb
galera=enabled
Status: Running
IP: 10.25.49.199
Controllers: StatefulSet/mariadb
...
...
...
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
5s 5s 1 {default-scheduler } Normal Scheduled Successfully assigned mariadb-0 to kubenode05
3s 3s 1 {kubelet kubenode05} spec.containers{mariadb} Normal Pulling pulling image "quay.io/stackanetes/stackanetes-mariadb:newton"
2s 2s 1 {kubelet kubenode05} spec.containers{mariadb} Normal Pulled Successfully pulled image "quay.io/stackanetes/stackanetes-mariadb:newton"
2s 2s 1 {kubelet kubenode05} spec.containers{mariadb} Normal Created Created container with docker id f702bd7c11ef; Security:[seccomp=unconfined]
2s 2s 1 {kubelet kubenode05} spec.containers{mariadb} Normal Started Started container with docker id f702bd7c11ef
So you can see that galera is enabled.

64
doc/source/troubleshooting/development.rst
View File

@ -1,64 +0,0 @@
Diagnosing the problem
----------------------
In order to protect your general sanity, we've included a curated list
of verification and troubleshooting steps that may help you avoid some
potential issues while developing Openstack-Helm.
MariaDB
~~~~~~~
To verify the state of MariaDB, use the following command:
::
$ kubectl exec mariadb-0 -it -n openstack -- mysql -u root -p password -e 'show databases;'
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
+--------------------+
$
Helm Server/Repository
~~~~~~~~~~~~~~~~~~~~~~
Sometimes you will run into Helm server or repository issues. For our
purposes, it's mostly safe to whack these. If you are developing
charts for other projects, use at your own risk (you most likely know
how to resolve these issues already).
To check for a running instance of Helm Server:
::
$ ps -a | grep "helm serve"
29452 ttys004 0:00.23 helm serve .
35721 ttys004 0:00.00 grep --color=auto helm serve
Kill the "helm serve" running process:
::
$ kill 29452
To clear out previous Helm repositories, and reinstall a local
repository:
::
$ helm repo list
NAME URL
stable https://kubernetes-charts.storage.googleapis.com/
local http://localhost:8879/charts
$
$ helm repo remove local
This allows you to read your local repository, if you ever need to do
these steps:
::
$ helm repo add local http://localhost:8879/charts

2
doc/source/troubleshooting/index.rst
View File

@ -8,8 +8,6 @@ Sometimes things go wrong. These guides will help you solve many common issues w
:maxdepth: 2
database
development
networking
persistent-storage
Getting help

9
doc/source/troubleshooting/networking.rst
View File

@ -1,9 +0,0 @@
==========
Networking
==========
This guide is to help users debug any networking issues when deploying
Charts in this repository.
Diagnosing the problem
======================

131
doc/source/troubleshooting/persistent-storage.rst
View File

@ -3,81 +3,15 @@ Persistent Storage
==================
This guide is to help users debug any general storage issues when
deploying Charts in this repository.
deploying charts in this repository.
Ceph
====
**CHART:** openstack-helm/ceph
Ceph Deployment Status
~~~~~~~~~~~~~~~~~~~~~~
Ceph Validating PVC
~~~~~~~~~~~~~~~~~~~
To validate persistent volume claim (PVC) creation, we've placed a test
manifest in the ``./test/`` directory. Deploy this pvc and explore the
deployment:
::
admin@kubenode01:~$ kubectl get pvc -o wide --all-namespaces -w
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
ceph pvc-test Bound pvc-bc768dea-c93e-11e6-817f-001fc69c26d1 1Gi RWO 9h
admin@kubenode01:~$
The output above indicates that the PVC is 'bound' correctly. Now
digging deeper:
::
admin@kubenode01:~/projects/openstack-helm$ kubectl describe pvc pvc-test -n ceph
Name: pvc-test
Namespace: ceph
StorageClass: general
Status: Bound
Volume: pvc-bc768dea-c93e-11e6-817f-001fc69c26d1
Labels: <none>
Capacity: 1Gi
Access Modes: RWO
No events.
admin@kubenode01:~/projects/openstack-helm$
We can see that we have a VolumeID, and the 'capacity' is 1GB. It is a
'general' storage class. It is just a simple test. You can safely delete
this test by issuing the following:
::
admin@kubenode01:~/projects/openstack-helm$ kubectl delete pvc pvc-test -n ceph
persistentvolumeclaim "pvc-test" deleted
admin@kubenode01:~/projects/openstack-helm$
Ceph Validating StorageClass
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Next we can look at the storage class, to make sure that it was created
correctly:
::
admin@kubenode01:~$ kubectl describe storageclass/general
Name: general
IsDefaultClass: No
Annotations: <none>
Provisioner: kubernetes.io/rbd
Parameters: adminId=admin,adminSecretName=pvc-ceph-conf-combined-storageclass,adminSecretNamespace=ceph,monitors=ceph-mon.ceph:6789,pool=rbd,userId=admin,userSecretName=pvc-ceph-client-key
No events.
admin@kubenode01:~$
The parameters is what we're looking for here. If we see parameters
passed to the StorageClass correctly, we will see the
``ceph-mon.ceph:6789`` hostname/port, things like ``userid``, and
appropriate secrets used for volume claims. This all looks great, and it
time to Ceph itself.
Ceph Validation
~~~~~~~~~~~~~~~
Most commonly, we want to validate that Ceph is working correctly. This
First, we want to validate that Ceph is working correctly. This
can be done with the following ceph command:
::
@ -96,26 +30,55 @@ can be done with the following ceph command:
admin@kubenode01:~$
Use one of your Ceph Monitors to check the status of the cluster. A
couple of things to note above; our health is 'HEALTH\_OK', we have 3
mons, we've established a quorum, and we can see that our active mds is
'ceph-mds-2810413505-gtjgv'. We have a healthy environment.
couple of things to note above; our health is `HEALTH\_OK`, we have 3
mons, we've established a quorum, and we can see that all of our OSDs
are up and in the OSD map.
For Glance and Cinder to operate, you will need to create some storage
pools for these systems. Additionally, Nova can be configured to use a
pool as well, but this is off by default.
PVC Preliminary Validation
~~~~~~~~~~~~~~~~~~~~~~~~~~
Before proceeding, it is important to ensure that you have deployed a
client key in the namespace you wish to fulfill ``PersistentVolumeClaims``.
To verify that your deployment namespace has a client key:
::
kubectl exec -n ceph -it ceph-mon-0 ceph osd pool create volumes 128
kubectl exec -n ceph -it ceph-mon-0 ceph osd pool create images 128
admin@kubenode01: $ kubectl get secret -n openstack
NAME TYPE DATA AGE
default-token-nvl10 kubernetes.io/service-account-token 3 7d
pvc-ceph-client-key kubernetes.io/rbd 1 6m
Nova storage would be added like this:
Without this, your RBD-backed PVCs will never reach the ``Bound`` state. For
more information, see how to `activate namespace for ceph <../install/multinode.html#activating-control-plane-namespace-for-ceph>`_.
Note: This step is not relevant for PVCs within the same namespace Ceph
was deployed.
Ceph Validating PVC Operation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To validate persistent volume claim (PVC) creation, we've placed a test
manifest `here <https://raw.githubusercontent.com/openstack/openstack-helm/master/tests/pvc-test.yaml>`_.
Deploy this manifest and verify the job completes successfully.
Ceph Validating StorageClass
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Next we can look at the storage class, to make sure that it was created
correctly:
::
kubectl exec -n ceph -it ceph-mon-0 ceph osd pool create vms 128
admin@kubenode01:~$ kubectl describe storageclass/general
Name: general
IsDefaultClass: No
Annotations: <none>
Provisioner: kubernetes.io/rbd
Parameters: adminId=admin,adminSecretName=pvc-ceph-conf-combined-storageclass,adminSecretNamespace=ceph,monitors=ceph-mon.ceph:6789,pool=rbd,userId=admin,userSecretName=pvc-ceph-client-key
No events.
admin@kubenode01:~$
The choosing the amount of storage is up to you and can be changed by
replacing the 128 to meet your needs.
We are now ready to install our next chart, MariaDB.
The parameters are what we're looking for here. If we see parameters
passed to the StorageClass correctly, we will see the
``ceph-mon.ceph:6789`` hostname/port, things like ``userid``, and
appropriate secrets used for volume claims.