diff --git a/.gitignore b/.gitignore index ee3c0fba8..13e9d592a 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,13 @@ peggles/ # Unit test / coverage reports .tox/ config-ssh + +# Sphinx Build Files +_build + +# Various user specific files +.DS_Store +.idea/ +.vimrc +*.swp +.vscode/ \ No newline at end of file diff --git a/doc/source/troubleshooting_guide.rst b/doc/source/troubleshooting_guide.rst index 072a4897d..77c8f5f62 100644 --- a/doc/source/troubleshooting_guide.rst +++ b/doc/source/troubleshooting_guide.rst @@ -1,3 +1,4 @@ +===================== Troubleshooting Guide ===================== @@ -10,9 +11,117 @@ root cause of the problem. For additional support you can contact the Airship team via `IRC or mailing list `__, -use `Airship bug tracker `__ +use `Airship bug tracker `__ to search and create issues. +.. contents:: Table of Contents + :depth: 3 + +--------------------- +Perform Health Checks +--------------------- + +The first step in troubleshooting an Airship deployment is to identify unhealthy +services by performing health checks. + +Verify Peering is established +----------------------------- + +:: + + sudo /opt/cni/bin/calicoctl node status + + Calico process is running. + IPv4 BGP status + +--------------+-----------+-------+------------+-------------+ + | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | + +--------------+-----------+-------+------------+-------------+ + | 172.29.0.2 | global | up | 2018-05-22 | Established | + | 172.29.0.3 | global | up | 2018-05-22 | Established | + +--------------+-----------+-------+------------+-------------+ + IPv6 BGP status No IPv6 peers found. + +Verify that **STATE** is ``up`` and **INFO** is ``Established``. However, if +**STATE** is ``start`` and **INFO** is ``Connect``, peering has failed. + +For more information on Calico troubleshooting, visit the +`Calico Documentation `__ + +Verify the Health of Kubernetes +------------------------------- + +:: + + # Verify that for all nodes, STATE is Ready. + # + # Note: After a reboot, it may take as long as 30 minutes for + # a node to stabilize and reach a Ready condition. + kubectl get nodes + + # Verify that liveness probes for all pods are working. + # This command exposes pods whose liveness probe is failing. + kubectl get pods --all-namespaces | grep Running | grep 0/ + + # Verify that all pods are in the Running or Completed state. + # This command exposes pods that are not running or completed. + kubectl get pods --all-namespaces | grep -v Running | Completed + + # Look for crashed pods. + kubectl get pods --all-namespaces -o wide | grep Crash + + # Check the health of core services. + kubectl get pods --all-namespaces -o wide | grep core + kubectl get services --all-namespaces | grep core + + # Check the health of proxy services. + kubectl get pods --all-namespaces -o wide | grep proxy + + # Get all pod details. + kubectl get pods --all-namespaces -o wide -w + + # Look for failed jobs. + kubectl get jobs – --all-namespaces -o wide | grep -v "1 1" + +Verify the Health of OpenStack +------------------------------ + +Check OpenStack's health by issuing the following commands at the terminal, +in order to do so you must have a set an OpenStack RC file, details +`here `__ + +:: + + # Verify Keystone by requesting a token. + openstack token issue + + # Verify networks. + openstack network list + + # Verify subnets. + openstack subnet list + + # Verify VMs. + openstack server list + + # Verify compute hypervisors. + openstack hypervisor list + + # Verify Images + openstack image list + +Check for kube-proxy iptables NAT Issues +---------------------------------------- + +:: + + # Check the iptables and make sure the IP addresses are the same: + % iptables -n -t nat -L | grep coredns + % kubectl -n kube-system get -o wide pod | grep coredns + +----------------------- Configuring Airship CLI ----------------------- @@ -32,6 +141,7 @@ how to get it configured on your environment. # Run it without arguments to get a help message. sudo ./treasuremap/tools/airship +--------------------- Manifests Preparation --------------------- @@ -62,6 +172,7 @@ Example: sudo ./treasuremap/tools/airship pegleg site -r treasuremap/ \ render -o rendered.txt ${SITE} +------------------ Deployment Failure ------------------ @@ -122,6 +233,7 @@ by Kubernetes to satisfy replication factor. # Restart Armada API service. kubectl delete pod -n ucp armada-api-d5f757d5-6z6nv +---- Ceph ---- @@ -132,8 +244,6 @@ For more information on Ceph debugging follow an official Although Ceph tolerates failures of multiple OSDs, it is important to make sure that your Ceph cluster is healthy. -Example: - :: # Get a name of Ceph Monitor pod. @@ -175,3 +285,9 @@ There are a few other commands that may be useful during the debugging: # List all Ceph block devices mounted on a specific host. mount | grep rbd + + # Exec into the Monitor pod + MON_POD=$(sudo kubectl get --no-headers pods -n=ceph \ + l="application=ceph,component=mon" | awk '{ print $1; exit }') + echo $MON_POD + sudo kubectl exec -n ceph ${MON_POD} -- ceph -s