53b7635a19
A Linux instance has to be configured to send log output to the serial console whenever the Compute service collects log output from there. Otherwise, the log output will be empty. Added this as a troubleshooting tip to Cloud Admin Guide. Change-Id: I5adb28c2d73c41792c1d67f51e11edfc7474ed99 Closes-Bug: #1284962
200 lines
11 KiB
XML
200 lines
11 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<section xmlns="http://docbook.org/ns/docbook"
|
|
xmlns:xi="http://www.w3.org/2001/XInclude"
|
|
xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0"
|
|
xml:id="section_compute-troubleshooting">
|
|
<title>Troubleshoot Compute</title>
|
|
<para>Common problems for Compute typically involve misconfigured
|
|
networking or credentials that are not sourced properly in the
|
|
environment. Also, most flat networking configurations do not
|
|
enable <command>ping</command> or <command>ssh</command> from
|
|
a compute node to the instances that run on that node. Another
|
|
common problem is trying to run 32-bit images on a 64-bit
|
|
compute node. This section shows you how to troubleshoot
|
|
Compute.</para>
|
|
<section xml:id="log-files-for-openstack-compute">
|
|
<title>Compute service logging</title>
|
|
<para>Compute stores a log file for each service in
|
|
<filename>/var/log/nova</filename>. For example,
|
|
<filename>nova-compute.log</filename> is the log for
|
|
the <systemitem class="service">nova-compute</systemitem>
|
|
service. You can set the following options to format log
|
|
strings for the nova.log module in the
|
|
<filename>nova.conf</filename> file:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para><literal>logging_context_format_string</literal></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><literal>logging_default_format_string</literal></para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>If the log level is set to <literal>debug</literal>, you
|
|
can also specify
|
|
<literal>logging_debug_format_suffix</literal> to
|
|
append extra formatting. For information about what
|
|
variables are available for the formatter see: <link
|
|
xlink:href="http://docs.python.org/library/logging.html#formatter"
|
|
>http://docs.python.org/library/logging.html#formatter</link>.</para>
|
|
<para>You have two options for logging for OpenStack Compute
|
|
based on configuration settings. In
|
|
<filename>nova.conf</filename>, include the
|
|
<literal>logfile</literal> option to enable logging.
|
|
Alternatively you can set <literal>use_syslog=1</literal>
|
|
so that the nova daemon logs to syslog.</para>
|
|
</section>
|
|
<section xml:id="section_compute-common-errors-and-fixes">
|
|
<title>Common errors and fixes for Compute</title>
|
|
<para>The <link xlink:href="ask.openstack.org"
|
|
>ask.openstack.org</link> site offers a place to ask
|
|
and answer questions, and you can also mark questions as
|
|
frequently asked questions. This section describes some
|
|
errors people have posted previously. Bugs are constantly
|
|
being fixed, so online resources are a great way to get
|
|
the most up-to-date errors and fixes.</para>
|
|
<section xml:id="section_credential-errors">
|
|
<title>Credential errors, 401, and 403 forbidden
|
|
errors</title>
|
|
<para>Missing credentials cause a
|
|
<errorcode>403</errorcode>
|
|
<errortext>forbidden</errortext> error. To resolve
|
|
this issue, use one of these methods:<orderedlist>
|
|
<listitem>
|
|
<para><emphasis role="bold">Manual
|
|
method</emphasis>. Get get the
|
|
<filename>novarc</filename> file from
|
|
the project ZIP file, save existing
|
|
credentials in case of override. and
|
|
manually source the
|
|
<filename>novarc</filename>
|
|
file.</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><emphasis role="bold">Script
|
|
method</emphasis>. Generates
|
|
<filename>novarc</filename> from the
|
|
project ZIP file and sources it for
|
|
you.</para>
|
|
</listitem>
|
|
</orderedlist></para>
|
|
<para>When you run <systemitem class="service"
|
|
>nova-api</systemitem> the first time, it
|
|
generates the certificate authority information,
|
|
including <filename>openssl.cnf</filename>. If you
|
|
start the CA services before this, you might not be
|
|
able to create your ZIP file. Restart the services.
|
|
When your CA information is available, create your ZIP
|
|
file.</para>
|
|
<para>Also, check your HTTP proxy settings to see whether
|
|
they cause problems with <filename>novarc</filename>
|
|
creation.</para>
|
|
</section>
|
|
<section xml:id="section_instance-errors">
|
|
<title>Instance errors</title>
|
|
<para>Sometimes a particular instance shows
|
|
<literal>pending</literal> or you cannot SSH to
|
|
it. Sometimes the image itself is the problem. For
|
|
example, when you use flat manager networking, you do
|
|
not have a DHCP server and certain images do not
|
|
support interface injection; you cannot connect to
|
|
them. The fix for this problem is to use an image that
|
|
does support this method, such as Ubuntu, which
|
|
obtains an IP address correctly with FlatManager
|
|
network settings.</para>
|
|
<para>To troubleshoot other possible problems with an
|
|
instance, such as an instance that stays in a spawning
|
|
state, check the directory for the particular instance
|
|
under <filename>/var/lib/nova/instances</filename> on
|
|
the <systemitem class="service"
|
|
>nova-compute</systemitem> host and make sure that
|
|
these files are present:</para>
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para><filename>libvirt.xml</filename></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><filename>disk</filename></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><filename>disk-raw</filename></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><filename>kernel</filename></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para><filename>ramdisk</filename></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>After the instance starts,
|
|
<filename>console.log</filename></para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
<para>If any files are missing, empty, or very small, the
|
|
<systemitem class="service"
|
|
>nova-compute</systemitem> service did not
|
|
successfully download the images from the Image
|
|
Service.</para>
|
|
<para>Also check <filename>nova-compute.log</filename> for
|
|
exceptions. Sometimes they do not appear in the
|
|
console output.</para>
|
|
<para>Next, check the log file for the instance in the
|
|
<filename>/var/log/libvirt/qemu</filename>
|
|
directory to see if it exists and has any useful error
|
|
messages in it.</para>
|
|
<para>Finally, from the
|
|
<filename>/var/lib/nova/instances</filename>
|
|
directory for the instance, see if this command
|
|
returns an error:</para>
|
|
<screen><prompt>#</prompt> <userinput>virsh create libvirt.xml</userinput></screen>
|
|
</section>
|
|
<section xml:id="section_compute-empty-log-output">
|
|
<title>Empty log output for Linux instances</title>
|
|
<para>You can view the log output of running instances
|
|
from either the <guilabel>Log</guilabel> tab of the
|
|
dashboard or the output of
|
|
<command>nova console-log</command>. In some cases, the
|
|
log output of a running Linux instance will be empty or
|
|
only display a single character (for example,
|
|
the <keycap>?</keycap> character).</para>
|
|
<para>This occurs when the Compute service attempts to
|
|
retrieve the log output of the instance via a serial
|
|
console while the instance itself is not configured to
|
|
send output to the console. To rectify this, append the
|
|
following parameters to kernel arguments specified in the
|
|
instance's boot loader:</para>
|
|
<screen>console=tty0 console=ttyS0,115200n8</screen>
|
|
<para>Upon rebooting, the instance will be configured to
|
|
send output to the Compute service.</para>
|
|
</section>
|
|
</section>
|
|
<section xml:id="reset-state">
|
|
<title>Reset the state of an instance</title>
|
|
<para>If an instance remains in an intermediate state, such as
|
|
<literal>deleting</literal>, you can use the
|
|
<command>nova reset-state</command> command to
|
|
manually reset the state of an instance to an error state.
|
|
You can then delete the instance. For example:</para>
|
|
<screen><prompt>$</prompt> <userinput>nova reset-state c6bbbf26-b40a-47e7-8d5c-eb17bf65c485</userinput>
|
|
<prompt>$</prompt> <userinput>nova delete c6bbbf26-b40a-47e7-8d5c-eb17bf65c485</userinput></screen>
|
|
<para>You can also use the <parameter>--active</parameter>
|
|
parameter to force the instance back to an active state
|
|
instead of an error state. For example:</para>
|
|
<screen><prompt>$</prompt> <userinput>nova reset-state --active c6bbbf26-b40a-47e7-8d5c-eb17bf65c485</userinput></screen>
|
|
</section>
|
|
<section xml:id="problems-with-injection">
|
|
<title>Injection problems</title>
|
|
<para>If instances do not boot or boot slowly, investigate
|
|
file injection as a cause.</para>
|
|
<para>To disable injection in libvirt, set
|
|
<option>libvirt_inject_partition</option> to
|
|
<literal>-2</literal>.</para>
|
|
<note>
|
|
<para>If you have not enabled the configuration drive and
|
|
you want to make user-specified files available from
|
|
the metadata server for to improve performance and
|
|
avoid boot failure if injection fails, you must
|
|
disable injection.</para>
|
|
</note>
|
|
</section>
|
|
</section>
|