From 8f031c6b03b496182e9bfa3ec5774821d7d41d42 Mon Sep 17 00:00:00 2001 From: Paul Bourke Date: Mon, 28 Nov 2016 13:06:27 +0000 Subject: [PATCH] Update Kibana documentation Simplify some of the Kibana docs, and add some more detailed information on searching logs. * Add a summary of what Kibana is * Make it easier to see how to access Kibana (e.g. say port 5601 instead of ) * Add note how to enable Kibana via centralised logging * Expand the search section with an example. Change-Id: Ia01e2b52fa758eb45fa417da7c6622150b2d5156 --- doc/kibana-guide.rst | 116 +++++++++++++++++++++++++++++++------------ 1 file changed, 85 insertions(+), 31 deletions(-) diff --git a/doc/kibana-guide.rst b/doc/kibana-guide.rst index 17712cdedf..dc0441fbd0 100644 --- a/doc/kibana-guide.rst +++ b/doc/kibana-guide.rst @@ -4,33 +4,36 @@ Kibana in Kolla =============== -Default index pattern -===================== +An OpenStack deployment generates vast amounts of log data. In order to +successfully monitor this and use it to diagnose problems, the standard "ssh +and grep" solution quickly becomes unmanageable. -After successful Kibana deployment, it can be accessed on -: -or : in any web -browser after authenticating with ```` and ````. +Kolla can deploy Kibana as part of the E*K stack in order to allow operators to +search and visualise logs in a centralised manner. -The values ````, ````, -```` and ```` can be found in -``/kolla/ansible/group_vars/all.yml`` or if the default -values are overridden, in ``/etc/kolla/globals.yml``. The value -```` can be found in ``/etc/kolla/passwords.yml``. -By default kibana_server_port is set to 5601. +Preparation and deployment +========================== + +Modify the configuration file ``/etc/kolla/globals.yml`` and change +the following: + +:: + + enable_central_logging: "yes" + +After successful deployment, Kibana can be accessed using a browser on +``:5601``. + +The default username is ``kibana``, the password can be located under +```` in ``/etc/kolla/passwords.yml``. When Kibana is opened for the first time, it requires creating a default index -pattern. To view, analyse and search logs, at least one index pattern has to -be created. To match indices stored in ElasticSearch, we suggest to use -following configuration: +pattern. To view, analyse and search logs, at least one index pattern has to be +created. To match indices stored in ElasticSearch, we suggest setting the +"Index name or pattern" field to ``log-*``. The rest of the fields can be left +as is. -#. Index contains time-based events - check -#. Use event times to create index names [DEPRECATED] - not checked -#. Index name or pattern - log-* -#. Do not expand index pattern when searching (Not recommended) - not checked -#. Time-field name - Timestamp - -After setting parameters, one can create an index with *Create* button. +After setting parameters, create an index by clicking the ``Create`` button. .. note:: This step is necessary until the default Kibana dashboard is implemented in Kolla. @@ -38,15 +41,66 @@ After setting parameters, one can create an index with *Create* button. Search logs - Discover tab ========================== -Logs search is available under Discover tab. In the menu on the left side, -one can choose any field that will be included in a new search. To do this, -add button has to be pressed. This button appears after pointing any field -from available ones. After adding a specific field, it is marked as selected -field in the menu on the left. Search panel is updated automatically. To -remove field from a current search, remove button has to be pressed. This -button appears after pointing any field from selected ones. -Current search can be saved by using 'Save search' option in the menu on the -right. +Operators can create and store searches based on various fields from logs, for +example, "show all logs marked with ERROR on nova-compute". + +To do this, click the ``Discover`` tab. Fields from the logs can be filtered by +hovering over entries from the left hand side, and clicking ``add`` or +``remove``. Add the following fields: + +* Hostname +* Payload +* severity_label +* programname + +This yields an easy to read list of all log events from each node in the +deployment within the last 15 minutes. A "tail like" functionality can be +achieved by clicking the clock icon in the top right hand corner of the screen, +and selecting ``Auto-refresh``. + +Logs can also be filtered down further. To use the above example, type +``programname:nova-compute`` in the search bar. Click the drop-down arrow from +one of the results, then the small magnifying glass icon from beside the +programname field. This should now show a list of all events from nova-compute +services across the cluster. + +The current search can also be saved by clicking the ``Save Search`` icon +available from the menu on the right hand side. + +Example: using Kibana to diagnose a common failure +-------------------------------------------------- + +The following example demonstrates how Kibana can be used to diagnose a common +OpenStack problem, where an instance fails to launch with the error 'No valid +host was found'. + +First, re-run the server creation with ``--debug``: + +:: + + openstack --debug server create --image cirros --flavor m1.tiny \ + --key-name mykey --nic net-id=00af016f-dffe-4e3c-a9b8-ec52ccd8ea65 \ + demo1 + +In this output, look for the key ``X-Compute-Request-Id``. This is a unique +identifier that can be used to track the request through the system. An +example ID looks like this: + +:: + + X-Compute-Request-Id: req-c076b50a-6a22-48bf-8810-b9f41176a6d5 + +Taking the value of ``X-Compute-Request-Id``, enter the value into the Kibana +search bar, minus the leading ``req-``. Assuming some basic filters have been +added as shown in the previous section, Kibana should now show the path this +request made through the OpenStack deployment, starting at a ``nova-api`` on +a control node, through the ``nova-scheduler``, ``nova-conductor``, and finally +``nova-compute``. Inspecting the ``Payload`` of the entries marked ``ERROR`` +should quickly lead to the source of the problem. + +While some knowledge is still required of how Nova works in this instance, it +can still be seen how Kibana helps in tracing this data, particularly in a +large scale deployment scenario. Visualize data - Visualize tab ==============================