Add block store troubleshoot docs

This patch adds block storage troubleshooting documentation,
which superseeds the old documentation.

It's actually https://review.openstack.org/#/c/49972 - but that
patch became a rebase hell, hence the new one.

Change-Id: I37621e1bede368b57a5af5e33b2d131c7e28d488
This commit is contained in:
Scott Hamilton 2013-10-11 11:11:10 +11:00 committed by Tom Fifield
parent 13dee3e734
commit c641abaa62
6 changed files with 257 additions and 84 deletions

View File

@ -112,9 +112,17 @@
Guide</citetitle></link>.</para> Guide</citetitle></link>.</para>
</section> </section>
<?hard-pagebreak?> <?hard-pagebreak?>
<xi:include href="section_troubleshoot-cinder.xml"/>
<xi:include href="section_multi_backend.xml"/> <xi:include href="section_multi_backend.xml"/>
<xi:include href="section_backup-block-storage-disks.xml"/> <xi:include href="section_backup-block-storage-disks.xml"/>
<xi:include href="section_volume-migration.xml"/> <xi:include href="section_volume-migration.xml"/>
</section> </section>
<section xml:id="troubleshooting-cinder-install">
<title>Troubleshooting your installation</title>
<para>This section contains a few useful tips for troubleshoting
your Block Storage installation</para>
<xi:include href="section_block_storage_troubleshooting.xml"/>
<xi:include href="section_ts_cinder_config.xml"/>
<xi:include href="section_ts_multipath_warn.xml"/>
<xi:include href="section_ts_vol_attach_miss_sg_scan.xml"/>
</section>
</chapter> </chapter>

View File

@ -0,0 +1,90 @@
<?xml version="1.0" encoding="UTF-8"?>
<section xml:id="section_block_storage_troubleshooting" xmlns="http://docbook.org/ns/docbook"
xmlns:xi="http://www.w3.org/2001/XInclude" xmlns:xlink="http://www.w3.org/1999/xlink"
version="5.0">
<title xml:id="block_storage_troubleshooting">Troubleshooting OpenStack Block Storage</title>
<section xml:id="ts-general-issues">
<title>General Troubleshooting Guidelines</title>
<para>There are two log files that are especially helpful for solving volume creation
failures, the <systemitem class="service">cinder-api</systemitem> log and the
<systemitem class="service">cinder-volume</systemitem> log. The <systemitem
class="service">cinder-api</systemitem> log is useful for determining if you have
endpoint or connectivity issues. If you send a request to create a volume and it fails,
it's a good idea to look in the <systemitem class="service">cinder-api</systemitem> log
first and see if the request even made it to the Cinder service. If the request is
logged and there are no errors or trace-backs, then you can check the <systemitem
class="service">cinder-volume</systemitem> log for errors or trace-backs.</para>
<para>The following entries in the <filename>cinder.openstack.common.log</filename> file can
be used to assist in troubleshooting your block storage configuration.</para>
<para>
<programlisting language="ini">
# Print debugging output (set logging level to DEBUG instead
# of default WARNING level). (boolean value)
#debug=false
# Print more verbose output (set logging level to INFO instead
# of default WARNING level). (boolean value)
#verbose=false
# Log output to standard error (boolean value)
#use_stderr=true
# Default file mode used when creating log files (string
# value)
#logfile_mode=0644
# format string to use for log messages with context (string
# value)
#logging_context_format_string=%(asctime)s.%(msecs)03d %(levelname)s %(name)s [%(request_id)s %(user)s %(tenant)s] %(instance)s%(message)s
# format string to use for log mes #logging_default_format_string=%(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [-] %(instance)s%(message)s
# data to append to log format when level is DEBUG (string
# value)
#logging_debug_format_suffix=%(funcName)s %(pathname)s:%(lineno)d
# prefix each line of exception output with this format
# (string value)
#logging_exception_prefix=%(asctime)s.%(msecs)03d %(process)d TRACE %(name)s %(instance)s
# list of logger=LEVEL pairs (list value)
#default_log_levels=amqplib=WARN,sqlalchemy=WARN,boto=WARN,suds=INFO,keystone=INFO,eventlet.wsgi.server=WARNsages without context
# (string value)
# If an instance is passed with the log message, format it
# like this (string value)
#instance_format="[instance: %(uuid)s]"
# If an instance UUID is passed with the log message, format
# it like this (string value)
# A logging.Formatter log message format string which may use
# any of the available logging.LogRecord attributes. Default:
# %(default)s (string value)
#log_format=%(asctime)s %(levelname)8s [%(name)s] %(message)s
# Format string for %%(asctime)s in log records. Default:
# %(default)s (string value)
#log_date_format=%Y-%m-%d %H:%M:%S
# (Optional) Name of log file to output to. If not set,
# logging will go to stdout. (string value)
#log_file=&lt;None>
# (Optional) The directory to keep log files in (will be
# prepended to --log-file) (string value)
#log_dir=&lt;None>
#instance_uuid_format="[instance: %(uuid)s]"
# If this option is specified, the logging configuration file
# specified is used and overrides any other logging options
# specified. Please see the Python logging module
# documentation for details on logging configuration files.
# (string value) # Use syslog for logging. (boolean value)
#use_syslog=false
# syslog facility to receive log lines (string value)
#syslog_log_facility=LOG_USER
#log_config=&lt;None></programlisting>
</para>
</section>
</section>

View File

@ -1,83 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<section xml:id="troubleshoot-cinder" xmlns="http://docbook.org/ns/docbook"
xmlns:xi="http://www.w3.org/2001/XInclude" xmlns:xlink="http://www.w3.org/1999/xlink"
version="1.0">
<title>Troubleshoot your cinder installation</title>
<para>This section is intended to help solve some basic and common
errors that are encountered during set up and configuration of
Cinder. The focus here is on failed creation of volumes. The
most important thing to know is where to look in case of a
failure. Two log files are especially helpful when volume
creation fails: <systemitem class="service"
>cinder-api</systemitem> log and <systemitem
class="service">cinder-volume</systemitem> log.</para>
<para>The <systemitem class="service">cinder-api</systemitem> log is useful in determining if you have
endpoint or connectivity issues. If you send a request to
create a volume and it fails, it's a good idea to look here
first and see if the request even made it to the Cinder
service. If the request seems to be logged, and there are no
errors or trace-backs then you can move to the <systemitem class="service">cinder-volume</systemitem>
log and look for errors or trace-backs there.</para>
<para>There are some common issues to look out for. The following
describes some common configuration issues with suggested
solutions.</para>
<para><emphasis role="bold"><emphasis role="underline">Create commands are in <systemitem class="service">cinder-api</systemitem> log
with no error</emphasis></emphasis></para>
<para>
<itemizedlist>
<listitem>
<para><emphasis role="bold"><literal>state_path</literal> and <literal>volumes_dir</literal> settings</emphasis></para>
<para>Cinder uses <command>tgtd</command>
as the default iscsi helper and implements persistent targets.
This means that in the case of a tgt restart or
even a node reboot your existing volumes on that
node will be restored automatically with their
original IQN.</para>
<para>In order to make this possible the iSCSI target information needs to be stored
in a file on creation that can be queried in case of restart of the tgt daemon.
By default, Cinder uses a <literal>state_path</literal> variable, which if installing via Yum or
APT should be set to <filename>/var/lib/cinder/</filename>. The next part is the <literal>volumes_dir</literal>
variable, by default this just simply appends a "<literal>volumes</literal>" directory to the
<literal>state_path</literal>. The result is a file-tree <filename>/var/lib/cinder/volumes/</filename>.</para>
<para>While this should all be handled for you by you installer, it can go wrong. If
you're having trouble creating volumes and this directory does not exist you
should see an error message in the <systemitem class="service">cinder-volume</systemitem> log indicating that the
<literal>volumes_dir</literal> doesn't exist, and it should give you information to specify what
path exactly it was looking for.</para>
</listitem>
<listitem>
<para><emphasis role="bold">persistent tgt include file</emphasis></para>
<para>Along with the <literal>volumes_dir</literal> mentioned above, the iSCSI target driver also needs
to be configured to look in the correct place for the persist files. This is a
simple entry in <filename>/etc/tgt/conf.d</filename>, and you should have created this when you went
through the install guide. If you haven't or you're running into issues, verify
that you have a file <filename>/etc/tgt/conf.d/cinder.conf</filename>.</para>
<para>If the file is not there, create it, as follows:</para><screen><prompt>$</prompt> <userinput>sudo sh -c "echo 'include /var/lib/cinder/volumes/*' >> /etc/tgt/conf.d/cinder.conf"</userinput></screen>
</listitem>
</itemizedlist>
</para>
<para><emphasis role="bold"><emphasis role="underline">No sign of attach call in the <systemitem class="service">cinder-api</systemitem>
log</emphasis></emphasis></para>
<para>This is most likely going to be a minor adjustment to your
<filename>nova.conf</filename> file. Make sure that your
<filename>nova.conf</filename> has the following
entry:<programlisting>volume_api_class=nova.volume.cinder.API</programlisting></para>
<para>Make certain that you explicitly set
<option>enabled_apis</option> because the default includes
<option>osapi_volume</option>:<programlisting>enabled_apis=ec2,osapi_compute,metadata</programlisting></para>
<para><emphasis role="bold">Failed to create iscsi target error in the <filename>cinder-volume.log</filename></emphasis></para>
<programlisting language="bash">2013-03-12 01:35:43 1248 TRACE cinder.openstack.common.rpc.amqp ISCSITargetCreateFailed: Failed to create iscsi target for volume volume-137641b2-af72-4a2f-b243-65fdccd38780.</programlisting>
<para>You might see this error in
<filename>cinder-volume.log</filename> after trying to
create a volume that is 1 GB.</para>
<para>To fix this issue, change the content of the
<filename>/etc/tgt/targets.conf</filename> from
<literal>include /etc/tgt/conf.d/*.conf</literal> to
<literal>include
/etc/tgt/conf.d/cinder_tgt.conf</literal>, as follows:</para>
<programlisting language="bash">include /etc/tgt/conf.d/cinder_tgt.conf
include /etc/tgt/conf.d/cinder.conf
default-driver iscsi</programlisting>
<para>Then restart tgt and <literal>cinder-*</literal> services so they pick up the new configuration.</para>
</section>

View File

@ -0,0 +1,100 @@
<?xml version="1.0" encoding="UTF-8"?>
<section xml:id="section_ts_cinder_config" xmlns="http://docbook.org/ns/docbook"
xmlns:xi="http://www.w3.org/2001/XInclude" xmlns:xlink="http://www.w3.org/1999/xlink"
version="1.0" linkend="section_ts_cinder_config">
<title xml:id="ts_block_config">Troubleshooting your Block Storage Configuration</title>
<para>This section is intended to help solve some basic and common errors that are encountered
during setup and configuration of the Cinder block storage service. The focus here is on
failed creation of volumes. The most important thing to know is where to look in case of a
failure.</para>
<para>There are two log files that are especially helpful for solving volume creation failures,
the <systemitem class="service">cinder-api</systemitem> log and the <systemitem
class="service">cinder-volume</systemitem> log. The <systemitem class="service"
>cinder-api</systemitem> log is useful for determining if you have endpoint or
connectivity issues. If you send a request to create a volume and it fails, it's a good idea
to look in the <systemitem class="service">cinder-api</systemitem> log first and see if the
request even made it to the Cinder service. If the request is logged and there are no errors
or trace-backs, then you can check the <systemitem class="service"
>cinder-volume</systemitem> log for errors or trace-backs.</para>
<para>
<note>
<para>Create commands are listed in the <systemitem class="service"
>cinder-api</systemitem> log.</para>
</note>
</para>
<para>Here are some common issues discovered during configuration, and some suggested solutions
.</para>
<para>
<itemizedlist>
<listitem>
<para>Isusues with <literal>state_path</literal> and <literal>volumes_dir</literal>
settings.</para>
<para>Cinder uses <command>tgtd</command> as the default iscsi helper and implements
persistent targets. This means that in the case of a tgt restart or even a node
reboot your existing volumes on that node will be restored automatically with
their original IQN.</para>
<para>In order to make this possible the iSCSI target information needs to be stored
in a file on creation that can be queried in case of restart of the tgt daemon.
By default, Cinder uses a <literal>state_path</literal> variable, which if
installing with Yum or APT should be set to
<filename>/var/lib/cinder/</filename>. The next part is the
<literal>volumes_dir</literal> variable, by default this just simply appends
a "<literal>volumes</literal>" directory to the <literal>state_path</literal>.
The result is a file-tree <filename>/var/lib/cinder/volumes/</filename>.</para>
<para>While this should all be handled by the installer, it can go wrong. If you are
having trouble creating volumes and this directory does not exist you should see
an error message in the <systemitem class="service">cinder-volume</systemitem>
log indicating that the <literal>volumes_dir</literal> doesn't exist, and it
should give you information to specify what path exactly it was looking
for.</para>
</listitem>
<listitem>
<para>The persistent tgt include file.</para>
<para>Along with the <literal>volumes_dir</literal> mentioned above, the iSCSI
target driver also needs to be configured to look in the correct place for the
persist files. This is a simple entry in <filename>/etc/tgt/conf.d</filename>,
and you should have created this when you went through the install guide. If you
haven't or you're running into issues, verify that you have a file
<filename>/etc/tgt/conf.d/cinder.conf</filename>.</para>
<para>If the file is not there, you can create with the following
command:<programlisting>
sudo sh -c "echo 'include /var/lib/cinder/volumes/*' >> /etc/tgt/conf.d/cinder.conf"
</programlisting></para>
</listitem>
<listitem>
<para>No sign of attach call in the <systemitem class="service"
>cinder-api</systemitem> log.</para>
<para>This is most likely going to be a minor adjustment to your
<filename>nova.conf</filename> file. Make sure that your
<filename>nova.conf</filename> has the following
entry:<programlisting>
volume_api_class=nova.volume.cinder.API
</programlisting></para>
<caution>
<para>Make certain that you explicitly set <filename>enabled_apis</filename>
because the default will include
<filename>osapi_volume</filename>:<programlisting>
enabled_apis=ec2,osapi_compute,metadata
</programlisting>
</para>
</caution>
</listitem>
<listitem>
<para>Failed to create iscsi target error in the
<filename>cinder-volume.log</filename> file.</para>
<programlisting language="bash">2013-03-12 01:35:43 1248 TRACE cinder.openstack.common.rpc.amqp ISCSITargetCreateFailed: Failed to create iscsi target for volume volume-137641b2-af72-4a2f-b243-65fdccd38780.
</programlisting>
<para>You may see this error in <filename>cinder-volume.log</filename> after trying
to create a volume that is 1 GB. To fix this issue:</para>
<para>Change content of the <filename>/etc/tgt/targets.conf</filename> from "include
/etc/tgt/conf.d/*.conf" to: include /etc/tgt/conf.d/cinder_tgt.conf:</para>
<programlisting language="bash">
include /etc/tgt/conf.d/cinder_tgt.conf
include /etc/tgt/conf.d/cinder.conf
default-driver iscsi</programlisting>
<para>Then restart tgt and <literal>cinder-*</literal> services so they pick up the
new configuration.</para>
</listitem>
</itemizedlist>
</para>
</section>

View File

@ -0,0 +1,28 @@
<?xml version="1.0" encoding="UTF-8"?>
<?xml-model href="http://docbook.org/xml/5.0/rng/docbook.rng" schematypens="http://relaxng.org/ns/structure/1.0"?>
<?xml-model href="http://docbook.org/xml/5.0/rng/docbook.rng" type="application/xml" schematypens="http://purl.oclc.org/dsdl/schematron"?>
<section xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink"
version="5.0" xml:id="section_ts_multipath_warn" linkend="section_ts_multipath_warn">
<title>Multipath Call Failed Exit</title>
<section xml:id="section_ts_multipath_warn_problem">
<title>Problem</title>
<para>Multipath call failed exit. This warning occurs in the Nova log if you do not have the
optional <filename>multipath-tools</filename> package installed on the Compute node.
This is an optional package and the volume attachment does work without the multipath
tools installed. If the <filename>multipath-tools</filename> package is installed on the
Compute node, it is used to perform the volume attachment. The IDs in your message are
unique to your system.</para>
<programlisting>WARNING nova.storage.linuxscsi [req-cac861e3-8b29-4143-8f1b-705d0084e571 admin
admin|req-cac861e3-8b29-4143-8f1b-705d0084e571 admin admin] Multipath call failed exit
(96)</programlisting>
</section>
<section xml:id="section_ts_multipath_warn_solution">
<title>Solution</title>
<para>Run the following command on the Compute node to install the
<filename>multipath-tools</filename> packages.</para>
<para>
<screen><prompt>$</prompt> <userinput>sudo apt-get install multipath-tools</userinput></screen>
</para>
</section>
</section>

View File

@ -0,0 +1,30 @@
<?xml version="1.0" encoding="UTF-8"?>
<?xml-model href="http://docbook.org/xml/5.0/rng/docbook.rng" schematypens="http://relaxng.org/ns/structure/1.0"?>
<?xml-model href="http://docbook.org/xml/5.0/rng/docbook.rng" type="application/xml" schematypens="http://purl.oclc.org/dsdl/schematron"?>
<section xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink"
version="5.0" xml:id="section_ts_vol_attach_miss_sg_scan"
linkend="section_ts_vol_attach_miss_sg_scan">
<title>Failed to Attach Volume, Missing sg_scan</title>
<section xml:id="section_ts_vol_attach_miss_sg_scan_problem">
<title>Problem</title>
<para>Failed to attach volume to an instance, <filename>sg_scan</filename> file not found.
This warning and error occurs when the <filename>sg3-utils</filename> package is not
installed on the Compute node. The IDs in your message are unique to your system.</para>
<programlisting>ERROR nova.compute.manager [req-cf2679fd-dd9e-4909-807f-48fe9bda3642 admin admin|req-cf2679fd-dd9e-4909-807f-48fe9bda3642 admin admin] [instance:
7d7c92e0-49fa-4a8e-87c7-73f22a9585d5|instance: 7d7c92e0-49fa-4a8e-87c7-73f22a9585d5]
Failed to attach volume 4cc104c4-ac92-4bd6-9b95-c6686746414a at
/dev/vdcTRACE
nova.compute.manager [instance: 7d7c92e0-49fa-4a8e-87c7-73f22a9585d5|instance:
7d7c92e0-49fa-4a8e-87c7-73f22a9585d5] Stdout: '/usr/local/bin/nova-rootwrap: Executable
not found: /usr/bin/sg_scan</programlisting>
</section>
<section xml:id="section_ts_vol_attach_miss_sg_scan_solution">
<title>Solution</title>
<para>Run the following command on the Compute node to install the
<filename>sg3-utils</filename> packages.</para>
<para>
<programlisting>$sudo apt-get install sg3-utils</programlisting>
</para>
</section>
</section>