From 6d6005dd2801037256045bdc45cbac01b7546bed Mon Sep 17 00:00:00 2001
From: Joshua Hesketh <josh@nitrotech.org>
Date: Tue, 24 Mar 2015 20:32:10 +1100
Subject: [PATCH] Move tempest log documentation into devstack-gate

Copy the tempest documentation that is appended to apache footers
into this project. This will help allow jobs to push up the
documentation along with the logs.

Change-Id: I8b26cc58c2ad6ca029726d227cf1d38d72465627
---
 help/README                |   2 +
 help/tempest-logs.html     | 158 +++++++++++++++++++++++++++++++++++++
 help/tempest-overview.html |  56 +++++++++++++
 3 files changed, 216 insertions(+)
 create mode 100644 help/README
 create mode 100644 help/tempest-logs.html
 create mode 100644 help/tempest-overview.html

diff --git a/help/README b/help/README
new file mode 100644
index 00000000..3353fdfc
--- /dev/null
+++ b/help/README
@@ -0,0 +1,2 @@
+This directory contains help files on how to read the logs generated
+by devstack-gate. These are usually uploaded along-side the logs.
diff --git a/help/tempest-logs.html b/help/tempest-logs.html
new file mode 100644
index 00000000..e669d224
--- /dev/null
+++ b/help/tempest-logs.html
@@ -0,0 +1,158 @@
+<h1>Guide to Devstack Gate Logs</h1>
+<p>
+  Above is a collection of log files from
+  the <a href="../console.html">current tempest run</a>. Within them
+  should be everything you need to get to the bottom of a test
+  failure. The screen-* logs will be your most valuable tools in this
+  process. Use the timestamp of the failed test in
+  the <a href="../console.html">current tempest run</a>.
+</p>
+<h2>Types of logs </h2>
+<p>
+<ul>
+  <li> <b>cinder</b>
+  <ul>
+    <li><a href="screen-c-api.txt.gz">screen-c-api.txt.gz</a>: cinder-api
+    <li><a href="screen-c-bak.txt.gz">screen-c-bak.txt.gz</a>: cinder-backup
+    <li><a href="screen-c-sch.txt.gz">screen-c-sch.txt.gz</a>: cinder-scheduler
+    <li><a href="screen-c-vol.txt.gz">screen-c-vol.txt.gz</a>: cinder-volume
+  </ul>
+  <li> <b>ceilometer</b>
+  <ul>
+    <li><a href="screen-ceilometer-acentral.txt.gz">screen-ceilometer-acentral.txt.gz</a>: ceilometer-agent-central
+    <li><a href="screen-ceilometer-acompute.txt.gz">screen-ceilometer-acompute.txt.gz</a>: ceilometer-agent-compute
+    <li><a href="screen-ceilometer-alarm-evaluator.txt.gz">screen-ceilometer-alarm-evaluator.txt.gz</a>: ceilometer-alarm-evaluator
+    <li><a href="screen-ceilometer-alarm-notifier.txt.gz">screen-ceilometer-alarm-notifier.txt.gz</a>: ceilometer-alarm-notifier
+    <li><a href="screen-ceilometer-anotification.txt.gz">screen-ceilometer-anotification.txt.gz</a>: ceilometer-agent-notifier
+    <li><a href="screen-ceilometer-api.txt.gz">screen-ceilometer-api.txt.gz</a>: ceilometer-api
+    <li><a href="screen-ceilometer-collector.txt.gz">screen-ceilometer-collector.txt.gz</a>: ceilometer-collector
+  </ul>
+  <li> <b>glance</b>
+  <ul>
+    <li><a href="screen-g-api.txt.gz">screen-g-api.txt.gz</a>: glance-api
+    <li><a href="screen-g-reg.txt.gz">screen-g-reg.txt.gz</a>: glance-registry
+  </ul>
+  <li><b>heat</b>
+  <ul>
+    <li><a href="screen-h-api-cfn.txt.gz">screen-h-api-cfn.txt.gz</a>: heat-api-cfn
+    <li><a href="screen-h-api-cw.txt.gz">screen-h-api-cw.txt.gz</a>: heat-api-cloudwatch
+    <li><a href="screen-h-api.txt.gz">screen-h-api.txt.gz</a>: heat-api
+    <li><a href="screen-h-eng.txt.gz">screen-h-eng.txt.gz</a>: heat-engine
+  </ul>
+  <li> <b>horizon</b>
+  <ul>
+    <li><a href="horizon_error.txt.gz">horizon_error.txt.gz</a>: horizon logs
+  </ul>
+  <li> <b>ironic</b>
+  <ul>
+    <li><a href="ironic-bm-logs/">ironic-bm-logs/</a>: output from the last successful boot of an ironic "bare metal" VM
+    <li><a href="screen-ir-api.txt.gz">screen-ir-api.txt.gz</a>: ironic-api
+    <li><a href="screen-ir-cond.txt.gz">screen-ir-cond.txt.gz</a>: ironic-conductor
+  </ul>
+  <li> <b>keystone</b>
+  <ul>
+    <li><a href="apache/keystone.txt.gz">keystone.txt.gz</a>: keystone log (Apache Httpd)
+    <li><a href="apache/keystone_access.txt.gz">keystone_access.txt.gz</a>: keystone access log (Apache Httpd)
+    <li><a href="screen-key.txt.gz">screen-key.txt.gz</a>: keystone log (eventlet)
+  </ul>
+  <li> <b>nova</b>
+  <ul>
+    <li><a href="screen-n-api.txt.gz">screen-n-api.txt.gz</a>: nova-api
+    <li><a href="screen-n-cond.txt.gz">screen-n-cond.txt.gz</a>: nova-conductor
+    <li><a href="screen-n-cpu.txt.gz">screen-n-cpu.txt.gz</a>: nova-compute
+    <li><a href="screen-n-crt.txt.gz">screen-n-crt.txt.gz</a>: nova-cert
+    <li><a href="screen-n-net.txt.gz">screen-n-net.txt.gz</a>: nova-network
+    <li><a href="screen-n-obj.txt.gz">screen-n-obj.txt.gz</a>: nova-objectstore
+    <li><a href="screen-n-sch.txt.gz">screen-n-sch.txt.gz</a>: nova-scheduler
+  </ul>
+  <li> <b>neutron</b>
+  <ul>
+    <li><a href="screen-q-agt.txt.gz">screen-q-agt.txt.gz</a>: neutron-openvswitch-agent
+    <li><a href="screen-q-dhcp.txt.gz">screen-q-dhcp.txt.gz</a>: neutron-dhcp-agent
+    <li><a href="screen-q-lbaas.txt.gz">screen-q-lbaas.txt.gz</a>: neutron-lbaas-agent
+    <li><a href="screen-q-meta.txt.gz">screen-q-meta.txt.gz</a>: neutron-metadata-agent
+    <li><a href="screen-q-metering.txt.gz">screen-q-metering.txt.gz</a>: neutron-metering-agent
+    <li><a href="screen-q-svc.txt.gz">screen-q-svc.txt.gz</a>: neutron-server
+    <li><a href="screen-q-vpn.txt.gz">screen-q-vpn.txt.gz</a>: neutron-vpn-agent
+  </ul>
+  <li> <b>swift</b>
+  <ul>
+    <li><a href="screen-s-account.txt.gz">screen-s-account.txt.gz</a>: swift-account-server
+    <li><a href="screen-s-container.txt.gz">screen-s-container.txt.gz</a>: swift-container-server
+    <li><a href="screen-s-object.txt.gz">screen-s-object.txt.gz</a>: swift-object-server
+    <li><a href="screen-s-proxy.txt.gz">screen-s-proxt.txt.gz</a>: swift-proxy-server
+  </ul>
+  <li> <b>system</b>
+  <ul>
+    <li><a href="pip-freeze.txt.gz">pip-freeze.txt.gz</a>: List of pip installed python packages. Output of 'pip freeze'
+    <li><a href="dpkg-l.txt.gz">dpkg-l.txt.gz</a>: List of apt-get installed packages. Output of 'dpkg -l'
+    <li><a href="df.txt.gz">df.txt.gz</a>:
+    <li><a href="syslog.txt.gz">syslog.txt.gz</a>: syslog for the test slave
+    <li><a href="screen-dstat.txt.gz">screen-dstat.txt.gz</a>: dstat output during the test job
+    <li><a href="sudoers.txt.gz">sudoers.txt.gz</a>: sudoers file
+  </ul>
+  <li> <b>trove</b>
+  <ul>
+    <li><a href="screen-tr-api.txt.gz">screen-tr-api.txt.gz</a>: trove-api
+    <li><a href="screen-tr-cond.txt.gz">screen-tr-cond.txt.gz</a>: trove-conductor
+    <li><a href="screen-tr-tmgr.txt.gz">screen-tr-tmgr.txt.gz</a>: trove-taskmanager
+  </ul>
+  <li> <b>tempest</b>
+  <ul>
+    <li><a href="tempest.txt.gz">tempest.txt.gz</a>: Tempest log file
+    <li><a href="tempest_conf.txt.gz">tempest_conf.txt.gz</a>: Tempest config file
+    <li><a href="subunit_log.txt.gz">subunit_log.txt.gz</a>: Subunit v1 stream from tempest run
+    <li><a href="testr_results.html.gz">testr_results.html.gz</a>: html formatted output of test results
+  </ul>
+  <li> <b>devstack</b>
+  <ul>
+      <li><a href=devstacklog.txt.gz>devstacklog.txt.gz</a>: Devstack log
+  </ul>
+</ul>
+</p>
+<h2>Nova Compute Fails</h2>
+<p>
+  If there is a compute test failure, especially a server not getting
+  created correctly, or being in an unexpected state, the following is
+  typically the most fruitful order to look at things:
+<ul>
+  <li><a href="screen-n-api.txt.gz">screen-n-api.txt.gz</a> - the nova
+    api log, which will show top level failures. Make sure the request
+    that was being sent in actually succeeded.
+  <li><a href="screen-n-cpu.txt.gz">screen-n-cpu.txt.gz</a> - the nova
+    compute log. If a libvirt or qemu issue happened during guest
+    creation it will be here.
+  <li><a href="screen-n-sch.txt.gz">screen-n-sch.txt.gz</a> - the nova
+    scheduler. Some times there are races in allocating resources, and
+    the scheduler will throw a WARNING if it couldn't allocate the
+    requested resources.
+  <li>all other nova logs
+</ul>
+<h2>Cinder Volume Fails</h2>
+<p>
+  If there is a volume failure in the test, the following is typically
+  the most fruitful order to look at things:
+  <ul>
+    <li><a href="screen-c-api.txt.gz">screen-c-api.txt.gz</a> - the
+      cinder api log, which will show top level failures. Make sure
+      the request that was being sent in actually succeeded.
+    <li><a href="screen-c-vol.txt.gz">screen-c-vol.txt.gz</a> - the
+      cinder agent log. If there was a local allocation error it will be
+      here.
+    <li><a href="screen-c-sch.txt.gz">screen-c-sch.txt.gz</a> - the
+      cinder scheduler. Some times there are races in allocating
+      resources, and the scheduler will throw a WARNING if it couldn't
+      allocate the requested resources.
+  </ul>
+</p>
+<h2>About this Help</h2>
+<p>
+  This help file is part of the
+  <a href="https://git.openstack.org/cgit/openstack-infra/devstack-gate">
+      openstack-infra/devstack-gate</a>
+  project, and can be found at
+  <a href="https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/help/tempest-logs.html">
+      help/tempest-logs.html
+  </a>.
+  The file can be updated via the standard OpenStack Gerrit Review process.
+</p>
diff --git a/help/tempest-overview.html b/help/tempest-overview.html
new file mode 100644
index 00000000..c1dc4623
--- /dev/null
+++ b/help/tempest-overview.html
@@ -0,0 +1,56 @@
+<h1>Guide to Tempest Results Runs</h1>
+<p>
+  You are looking at the full test results of a devstack setup and
+  tempest run for the OpenStack gate, as well as all the logs of all the
+  relevant services that were running during that tempest test run. From
+  them you should have enough information to debug.
+</p>
+<h2>How To Debug - Quickstart</h2>
+<p>
+  <ol>
+    <li>scroll to the end of console.html
+    <li>work your way backwards until the first failing test
+    <li>copy the timestamp of that failing test
+    <li>go into logs directory and look at that time stamp in related
+    service logs for traces, failures, or other oddities
+  </ol>
+</p>
+<h2>File Overview</h2>
+<h3>console.html</h3>
+This file contains the stdout/stderr console of the devstack-gate
+job. The basic flow of the file goes as follows:
+<ul>
+  <li>Boot guest from base Linux image
+  <li>Use devstack to install all required packages (debs and pips)
+  <li>Use devstack to setup all the services from upstream master except
+    for the particular project being tested, which is pulled from
+    gerrit.
+  <li>Run devstack exercises (very basic sanity checking)
+  <li>Run tempest tests
+</ul>
+<p>
+  All the devstack setup is done under bash tracing, so
+  is <b>extremely</b> verbose. This is to ensure enough data is captured
+  so that you can debug failures in the gate with the information provided.
+</p>
+<p>
+  The tempest tests are the last 1% of the console.html. When looking
+  at failures it is typical best to start at the end of the file and
+  work backwards.
+</p>
+<h3>logs</h3>
+<p>
+  The <a href="logs">logs</a> directory contains all the screen logs
+  from all the services during the devstack-gate run.
+</p>
+<h2>About this Help</h2>
+<p>
+  This help file is part of the
+  <a href="https://git.openstack.org/cgit/openstack-infra/devstack-gate">
+      openstack-infra/devstack-gate</a>
+  project, and can be found at
+  <a href="https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/help/tempest-overview.html">
+      help/tempest-overview.html
+  </a>.
+  The file can be updated via the standard OpenStack Gerrit Review process.
+</p>