83a7ce8ce0
xrange is not defined in python3. Rename xrange() to range(). Change-Id: Ifb1c9cfd863ce6dfe3cced3eca7ea8e539d8a5e9
209 lines
7.2 KiB
ReStructuredText
209 lines
7.2 KiB
ReStructuredText
===========================
|
|
Troubleshoot Object Storage
|
|
===========================
|
|
|
|
For Object Storage, everything is logged in ``/var/log/syslog`` (or
|
|
``messages`` on some distros). Several settings enable further
|
|
customization of logging, such as ``log_name``, ``log_facility``, and
|
|
``log_level``, within the object server configuration files.
|
|
|
|
Drive failure
|
|
~~~~~~~~~~~~~
|
|
|
|
Problem
|
|
-------
|
|
|
|
Drive failure can prevent Object Storage performing replication.
|
|
|
|
Solution
|
|
--------
|
|
|
|
In the event that a drive has failed, the first step is to make sure the
|
|
drive is unmounted. This will make it easier for Object Storage to work
|
|
around the failure until it has been resolved. If the drive is going to
|
|
be replaced immediately, then it is just best to replace the drive,
|
|
format it, remount it, and let replication fill it up.
|
|
|
|
If you cannot replace the drive immediately, then it is best to leave it
|
|
unmounted, and remove the drive from the ring. This will allow all the
|
|
replicas that were on that drive to be replicated elsewhere until the
|
|
drive is replaced. Once the drive is replaced, it can be re-added to the
|
|
ring.
|
|
|
|
You can look at error messages in the ``/var/log/kern.log`` file for
|
|
hints of drive failure.
|
|
|
|
Server failure
|
|
~~~~~~~~~~~~~~
|
|
|
|
Problem
|
|
-------
|
|
|
|
The server is potentially offline, and may have failed, or require a
|
|
reboot.
|
|
|
|
Solution
|
|
--------
|
|
|
|
If a server is having hardware issues, it is a good idea to make sure
|
|
the Object Storage services are not running. This will allow Object
|
|
Storage to work around the failure while you troubleshoot.
|
|
|
|
If the server just needs a reboot, or a small amount of work that should
|
|
only last a couple of hours, then it is probably best to let Object
|
|
Storage work around the failure and get the machine fixed and back
|
|
online. When the machine comes back online, replication will make sure
|
|
that anything that is missing during the downtime will get updated.
|
|
|
|
If the server has more serious issues, then it is probably best to
|
|
remove all of the server's devices from the ring. Once the server has
|
|
been repaired and is back online, the server's devices can be added back
|
|
into the ring. It is important that the devices are reformatted before
|
|
putting them back into the ring as it is likely to be responsible for a
|
|
different set of partitions than before.
|
|
|
|
Detect failed drives
|
|
~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Problem
|
|
-------
|
|
|
|
When drives fail, it can be difficult to detect that a drive has failed,
|
|
and the details of the failure.
|
|
|
|
Solution
|
|
--------
|
|
|
|
It has been our experience that when a drive is about to fail, error
|
|
messages appear in the ``/var/log/kern.log`` file. There is a script called
|
|
``swift-drive-audit`` that can be run via cron to watch for bad drives. If
|
|
errors are detected, it will unmount the bad drive, so that Object
|
|
Storage can work around it. The script takes a configuration file with
|
|
the following settings:
|
|
|
|
.. list-table:: **Description of configuration options for [drive-audit] in drive-audit.conf**
|
|
:header-rows: 1
|
|
|
|
* - Configuration option = Default value
|
|
- Description
|
|
* - ``device_dir = /srv/node``
|
|
- Directory devices are mounted under
|
|
* - ``error_limit = 1``
|
|
- Number of errors to find before a device is unmounted
|
|
* - ``log_address = /dev/log``
|
|
- Location where syslog sends the logs to
|
|
* - ``log_facility = LOG_LOCAL0``
|
|
- Syslog log facility
|
|
* - ``log_file_pattern = /var/log/kern.*[!.][!g][!z]``
|
|
- Location of the log file with globbing pattern to check against device
|
|
errors locate device blocks with errors in the log file
|
|
* - ``log_level = INFO``
|
|
- Logging level
|
|
* - ``log_max_line_length = 0``
|
|
- Caps the length of log lines to the value given; no limit if set to 0,
|
|
the default.
|
|
* - ``log_to_console = False``
|
|
- No help text available for this option.
|
|
* - ``minutes = 60``
|
|
- Number of minutes to look back in ``/var/log/kern.log``
|
|
* - ``recon_cache_path = /var/cache/swift``
|
|
- Directory where stats for a few items will be stored
|
|
* - ``regex_pattern_1 = \berror\b.*\b(dm-[0-9]{1,2}\d?)\b``
|
|
- No help text available for this option.
|
|
* - ``unmount_failed_device = True``
|
|
- No help text available for this option.
|
|
|
|
.. warning::
|
|
|
|
This script has only been tested on Ubuntu 10.04; use with caution on
|
|
other operating systems in production.
|
|
|
|
Emergency recovery of ring builder files
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Problem
|
|
-------
|
|
|
|
An emergency might prevent a successful backup from restoring the
|
|
cluster to operational status.
|
|
|
|
Solution
|
|
--------
|
|
|
|
You should always keep a backup of swift ring builder files. However, if
|
|
an emergency occurs, this procedure may assist in returning your cluster
|
|
to an operational state.
|
|
|
|
Using existing swift tools, there is no way to recover a builder file
|
|
from a ``ring.gz`` file. However, if you have a knowledge of Python, it
|
|
is possible to construct a builder file that is pretty close to the one
|
|
you have lost.
|
|
|
|
.. warning::
|
|
|
|
This procedure is a last-resort for emergency circumstances. It
|
|
requires knowledge of the swift python code and may not succeed.
|
|
|
|
#. Load the ring and a new ringbuilder object in a Python REPL:
|
|
|
|
.. code-block:: python
|
|
|
|
>>> from swift.common.ring import RingData, RingBuilder
|
|
>>> ring = RingData.load('/path/to/account.ring.gz')
|
|
|
|
#. Start copying the data we have in the ring into the builder:
|
|
|
|
.. code-block:: python
|
|
|
|
>>> import math
|
|
>>> partitions = len(ring._replica2part2dev_id[0])
|
|
>>> replicas = len(ring._replica2part2dev_id)
|
|
|
|
>>> builder = RingBuilder(int(math.log(partitions, 2)), replicas, 1)
|
|
>>> builder.devs = ring.devs
|
|
>>> builder._replica2part2dev = ring._replica2part2dev_id
|
|
>>> builder._last_part_moves_epoch = 0
|
|
>>> from array import array
|
|
>>> builder._last_part_moves = array('B', (0 for _ in range(partitions)))
|
|
>>> builder._set_parts_wanted()
|
|
>>> for d in builder._iter_devs():
|
|
d['parts'] = 0
|
|
>>> for p2d in builder._replica2part2dev:
|
|
for dev_id in p2d:
|
|
builder.devs[dev_id]['parts'] += 1
|
|
|
|
This is the extent of the recoverable fields.
|
|
|
|
#. For ``min_part_hours`` you either have to remember what the value you
|
|
used was, or just make up a new one:
|
|
|
|
.. code-block:: python
|
|
|
|
>>> builder.change_min_part_hours(24) # or whatever you want it to be
|
|
|
|
#. Validate the builder. If this raises an exception, check your
|
|
previous code:
|
|
|
|
.. code-block:: python
|
|
|
|
>>> builder.validate()
|
|
|
|
#. After it validates, save the builder and create a new ``account.builder``:
|
|
|
|
.. code-block:: python
|
|
|
|
>>> import pickle
|
|
>>> pickle.dump(builder.to_dict(), open('account.builder', 'wb'), protocol=2)
|
|
>>> exit ()
|
|
|
|
#. You should now have a file called ``account.builder`` in the current
|
|
working directory. Run
|
|
:command:`swift-ring-builder account.builder write_ring` and compare the new
|
|
``account.ring.gz`` to the ``account.ring.gz`` that you started
|
|
from. They probably are not byte-for-byte identical, but if you load them
|
|
in a REPL and their ``_replica2part2dev_id`` and ``devs`` attributes are
|
|
the same (or nearly so), then you are in good shape.
|
|
|
|
#. Repeat the procedure for ``container.ring.gz`` and
|
|
``object.ring.gz``, and you might get usable builder files.
|