openstack-manuals/doc/common/section_support-compute.xml

<?xml version="1.0" encoding="UTF-8"?>
<section xmlns="http://docbook.org/ns/docbook"
  xmlns:xi="http://www.w3.org/2001/XInclude"
  xmlns:xlink="http://www.w3.org/1999/xlink"
  version="5.0"
  xml:id="section_compute-troubleshooting">
    <title>Troubleshoot Compute</title>
    <para>Common problems for Compute typically involve misconfigured
        networking or credentials that are not sourced properly in the
        environment. Also, most flat networking configurations do not
        enable <command>ping</command> or <command>ssh</command> from
        a compute node to the instances that run on that node. Another
        common problem is trying to run 32-bit images on a 64-bit
        compute node. This section shows you how to troubleshoot
        Compute.</para>
    <section xml:id="log-files-for-openstack-compute">
        <title>Compute service logging</title>
        <para>Compute stores a log file for each service in
                <filename>/var/log/nova</filename>. For example,
                <filename>nova-compute.log</filename> is the log for
            the <systemitem class="service">nova-compute</systemitem>
            service. You can set the following options to format log
            strings for the nova.log module in the
                <filename>nova.conf</filename> file:</para>
        <itemizedlist>
            <listitem>
                <para><literal>logging_context_format_string</literal></para>
            </listitem>
            <listitem>
                <para><literal>logging_default_format_string</literal></para>
            </listitem>
        </itemizedlist>
        <para>If the log level is set to <literal>debug</literal>, you
            can also specify
                <literal>logging_debug_format_suffix</literal> to
            append extra formatting. For information about what
            variables are available for the formatter see: <link
                xlink:href="http://docs.python.org/library/logging.html#formatter-objects"
                >http://docs.python.org/library/logging.html#formatter-objects</link>.</para>
        <para>You have two options for logging for OpenStack Compute
            based on configuration settings. In
                <filename>nova.conf</filename>, include the
                <literal>logfile</literal> option to enable logging.
            Alternatively you can set <literal>use_syslog = 1</literal>
            so that the nova daemon logs to syslog.</para>
    </section>
    <section xml:id="section_compute-GuruMed-reports">
        <title>Guru Meditation reports</title>
        <para>A Guru Meditation report is sent by the Compute Service upon receipt of the
                <systemitem>SIGUSR1</systemitem> signal. This report is a general-purpose error
            report, including a complete report of the service's current state, and is sent to
                <systemitem>stderr</systemitem>.</para>
        <para>For example, if you redirect error output to <filename>nova-api-err.log</filename>
            using <command>nova-api 2&gt;/var/log/nova/nova-api-err.log</command>, resulting in the
            process ID 8675, you can then run:</para>
        <screen><prompt>#</prompt> <userinput>kill -USR1 8675</userinput></screen>
        <para>This command triggers the Guru Meditation report to be printed to
            <filename>/var/log/nova/nova-api-err.log</filename>.</para>
        <para>The report has the following sections:
            <itemizedlist>
                <listitem>
                  <para>
                    Package: Displays information about the package to
                    which the process belongs, including version
                    information.
                </para>
                </listitem>
                <listitem>
                  <para>
                    Threads: Displays stack traces and thread IDs for
                    each of the threads within the process.</para>
                </listitem>
                <listitem>
                  <para>
                    Green Threads: Displays stack traces for each of
                    the green threads within the process (green
                    threads do not have thread IDs).
                </para>
                </listitem>
                <listitem>
                  <para>
                    Configuration: Lists all configuration options
                    currently accessible through the CONF object for
                    the current process.
                </para>
                </listitem>
             </itemizedlist></para>
        <para>For more information, see <link
            xlink:href="http://docs.openstack.org/developer/nova/devref/gmr.html"
            >Guru Meditation Reports</link>.</para>
    </section>
    <section xml:id="section_compute-common-errors-and-fixes">
        <title>Common errors and fixes for Compute</title>
        <para>The <link xlink:href="http://ask.openstack.org"
                >ask.openstack.org</link> site offers a place to ask
            and answer questions, and you can also mark questions as
            frequently asked questions. This section describes some
            errors people have posted previously. Bugs are constantly
            being fixed, so online resources are a great way to get
            the most up-to-date errors and fixes.</para>
        <section xml:id="section_credential-errors">
            <title>Credential errors, 401, and 403 forbidden
                errors</title>
            <para>Missing credentials cause a
                    <errorcode>403</errorcode>
                <errortext>forbidden</errortext> error. To resolve
                this issue, use one of these methods:<orderedlist>
                    <listitem>
                        <para><emphasis role="bold">Manual
                                method</emphasis>. Gets the
                                <filename>novarc</filename> file from
                            the project ZIP file, saves existing
                            credentials in case of override, and
                            manually sources the
                                <filename>novarc</filename>
                            file.</para>
                    </listitem>
                    <listitem>
                        <para><emphasis role="bold">Script
                                method</emphasis>. Generates
                                <filename>novarc</filename> from the
                            project ZIP file and sources it for
                            you.</para>
                    </listitem>
                </orderedlist></para>
            <para>When you run <systemitem class="service"
                    >nova-api</systemitem> the first time, it
                generates the certificate authority information,
                including <filename>openssl.cnf</filename>. If you
                start the CA services before this, you might not be
                able to create your ZIP file. Restart the services.
                When your CA information is available, create your ZIP
                file.</para>
            <para>Also, check your HTTP proxy settings to see whether
                they cause problems with <filename>novarc</filename>
                creation.</para>
        </section>
        <section xml:id="section_instance-errors">
            <title>Instance errors</title>
            <para>Sometimes a particular instance shows
                    <literal>pending</literal> or you cannot SSH to
                it. Sometimes the image itself is the problem. For
                example, when you use flat manager networking, you do
                not have a DHCP server and certain images do not
                support interface injection; you cannot connect to
                them. The fix for this problem is to use an image that
                does support this method, such as Ubuntu, which
                obtains an IP address correctly with FlatManager
                network settings.</para>
            <para>To troubleshoot other possible problems with an
                instance, such as an instance that stays in a spawning
                state, check the directory for the particular instance
                under <filename>/var/lib/nova/instances</filename> on
                the <systemitem class="service"
                    >nova-compute</systemitem> host and make sure that
                these files are present:</para>
            <itemizedlist>
                <listitem>
                    <para><filename>libvirt.xml</filename></para>
                </listitem>
                <listitem>
                    <para><filename>disk</filename></para>
                </listitem>
                <listitem>
                    <para><filename>disk-raw</filename></para>
                </listitem>
                <listitem>
                    <para><filename>kernel</filename></para>
                </listitem>
                <listitem>
                    <para><filename>ramdisk</filename></para>
                </listitem>
                <listitem>
                    <para>After the instance starts,
                            <filename>console.log</filename></para>
                </listitem>
            </itemizedlist>
            <para>If any files are missing, empty, or very small, the
                    <systemitem class="service"
                    >nova-compute</systemitem> service did not
                successfully download the images from the Image
                Service.</para>
            <para>Also check <filename>nova-compute.log</filename> for
                exceptions. Sometimes they do not appear in the
                console output.</para>
            <para>Next, check the log file for the instance in the
                    <filename>/var/log/libvirt/qemu</filename>
                directory to see if it exists and has any useful error
                messages in it.</para>
            <para>Finally, from the
                    <filename>/var/lib/nova/instances</filename>
                directory for the instance, see if this command
                returns an error:</para>
            <screen><prompt>#</prompt> <userinput>virsh create libvirt.xml</userinput></screen>
        </section>
        <section xml:id="section_compute-empty-log-output">
            <title>Empty log output for Linux instances</title>
            <para>You can view the log output of running instances
            from either the <guilabel>Log</guilabel> tab of the
            dashboard or the output of
            <command>nova console-log</command>. In some cases, the
            log output of a running Linux instance will be empty or
            only display a single character (for example,
            the <keycap>?</keycap> character).</para>
            <para>This occurs when the Compute service attempts to
            retrieve the log output of the instance via a serial
            console while the instance itself is not configured to
            send output to the console. To rectify this, append the
            following parameters to kernel arguments specified in the
            instance's boot loader:</para>
            <screen>console=tty0 console=ttyS0,115200n8</screen>
            <para>Upon rebooting, the instance will be configured to
            send output to the Compute service.</para>
        </section>
        </section>
    <section xml:id="reset-state">
        <title>Reset the state of an instance</title>
        <para>If an instance remains in an intermediate state, such as
                <literal>deleting</literal>, you can use the
                <command>nova reset-state</command> command to
            manually reset the state of an instance to an error state.
            You can then delete the instance. For example:</para>
        <screen><prompt>$</prompt> <userinput>nova reset-state c6bbbf26-b40a-47e7-8d5c-eb17bf65c485</userinput>
<prompt>$</prompt> <userinput>nova delete c6bbbf26-b40a-47e7-8d5c-eb17bf65c485</userinput></screen>
        <para>You can also use the <parameter>--active</parameter>
            parameter to force the instance back to an active state
            instead of an error state. For example:</para>
        <screen><prompt>$</prompt> <userinput>nova reset-state --active c6bbbf26-b40a-47e7-8d5c-eb17bf65c485</userinput></screen>
    </section>
    <section xml:id="problems-with-injection">
        <title>Injection problems</title>
        <para>If instances do not boot or boot slowly, investigate
            file injection as a cause.</para>
        <para>To disable injection in libvirt, set the following in
            <filename>nova.conf</filename>:</para>
        <programlisting language="ini">[libvirt]
inject_partition = -2</programlisting>
        <note>
            <para>If you have not enabled the configuration drive and
                you want to make user-specified files available from
                the metadata server for to improve performance and
                avoid boot failure if injection fails, you must
                disable injection.</para>
        </note>
    </section>
<section xml:id="live-snapshotting-fail">
        <title>Disable live snapshotting</title>
        <para>If you use libvirt version <literal>1.2.2</literal>,
              you may experience problems with live snapshots creation.
              Occasionally, libvirt of the specified version
              fails to perform the live snapshotting under load
              that presupposes a concurrent creation of
              multiple snapshots.</para>

        <para>To effectively disable the libvirt live snapshotting, until the problem
              is resolved, configure the <option>disable_libvirt_livesnapshot</option>
              option. You can turn off the live snapshotting mechanism by setting
              up its value to <literal>True</literal> in the
              <literal>[workarounds]</literal> section
              of the <filename>nova.conf</filename> configuration file:</para>
              <programlisting language="ini">[workarounds]
disable_libvirt_livesnapshot = True</programlisting>
</section>
</section>