openstack-manuals/doc/admin-guide/source/cli-analyzing-log-files-with-swift.rst

=================
Analyze log files
=================

Use the swift command-line client for Object Storage to analyze log files.

The swift client is simple to use, scalable, and flexible.

Use the swift client :option:`-o` or :option:`-output` option to get
short answers to questions about logs.

You can use the :option:`-o` or :option:`--output` option with a single object
download to redirect the command output to a specific file or to STDOUT
(``-``). The ability to redirect the output to STDOUT enables you to
pipe (``|``) data without saving it to disk first.

Upload and analyze log files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

#. This example assumes that ``logtest`` directory contains the
   following log files.

   .. code-block:: console

      2010-11-16-21_access.log
      2010-11-16-22_access.log
      2010-11-15-21_access.log
      2010-11-15-22_access.log


   Each file uses the following line format.

   .. code-block:: console

      Nov 15 21:53:52 lucid64 proxy-server - 127.0.0.1 15/Nov/2010/22/53/52 DELETE /v1/AUTH_cd4f57824deb4248a533f2c28bf156d3/2eefc05599d44df38a7f18b0b42ffedd HTTP/1.0 204 - \
       - test%3Atester%2CAUTH_tkcdab3c6296e249d7b7e2454ee57266ff - - - txaba5984c-aac7-460e-b04b-afc43f0c6571 - 0.0432


#. Change into the ``logtest`` directory:

   .. code-block:: console

      $ cd logtest

#. Upload the log files into the ``logtest`` container:

   .. code-block:: console

      $ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing upload logtest *.log

   .. code-block:: console

      2010-11-16-21_access.log
      2010-11-16-22_access.log
      2010-11-15-21_access.log
      2010-11-15-22_access.log

#. Get statistics for the account:

   .. code-block:: console

      $ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \
      -q stat

   .. code-block:: console

      Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
      Containers: 1
      Objects: 4
      Bytes: 5888268

#. Get statistics for the ``logtest`` container:

   .. code-block:: console

      $ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \
      stat logtest

   .. code-block:: console

      Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
      Container: logtest
      Objects: 4
      Bytes: 5864468
      Read ACL:
      Write ACL:

#. List all objects in the logtest container:

   .. code-block:: console

      $ swift -A http:///swift-auth.com:11000/v1.0 -U test:tester -K testing \
      list logtest

   .. code-block:: console

      2010-11-15-21_access.log
      2010-11-15-22_access.log
      2010-11-16-21_access.log
      2010-11-16-22_access.log

Download and analyze an object
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This example uses the :option:`-o` option and a hyphen (``-``) to get
information about an object.

Use the :command:`swift download` command to download the object. On this
command, stream the output to ``awk`` to break down requests by return
code and the date ``2200 on November 16th, 2010``.

Using the log line format, find the request type in column 9 and the
return code in column 12.

After ``awk`` processes the output, it pipes it to ``sort`` and ``uniq
-c`` to sum up the number of occurrences for each request type and
return code combination.

#. Download an object:

   .. code-block:: console

      $ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \
           download -o - logtest 2010-11-16-22_access.log | awk '{ print \
           $9"-"$12}' | sort | uniq -c

   .. code-block:: console

       805 DELETE-204
       12 DELETE-404
       2 DELETE-409
       723 GET-200
       142 GET-204
       74 GET-206
       80 GET-304
       34 GET-401
       5 GET-403
       18 GET-404
       166 GET-412
       2 GET-416
       50 HEAD-200
       17 HEAD-204
       20 HEAD-401
       8 HEAD-404
       30 POST-202
       25 POST-204
       22 POST-400
       6 POST-404
       842 PUT-201
       2 PUT-202
       32 PUT-400
       4 PUT-403
       4 PUT-404
       2 PUT-411
       6 PUT-412
       6 PUT-413
       2 PUT-422
       8 PUT-499

#. Discover how many PUT requests are in each log file.

   Use a bash for loop with awk and swift with the :option:`-o` or
   :option:`--output` option and a hyphen (``-``) to discover how many
   PUT requests are in each log file.

   Run the :command:`swift list` command to list objects in the logtest
   container. Then, for each item in the list, run the
   :command:`swift download -o -` command. Pipe the output into grep to
   filter the PUT requests. Finally, pipe into ``wc -l`` to count the lines.

   .. code-block:: console

       $ for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester \
        -K testing list logtest` ; \
               do  echo -ne "PUTS - " ; swift -A \
               http://swift-auth.com:11000/v1.0 -U test:tester \
               -K testing download -o -  logtest $f | grep PUT | wc -l ; \
           done

   .. code-block:: console

       2010-11-15-21_access.log - PUTS - 402
       2010-11-15-22_access.log - PUTS - 1091
       2010-11-16-21_access.log - PUTS - 892
       2010-11-16-22_access.log - PUTS - 910

#. List the object names that begin with a specified string.

#. Run the :command:`swift list -p 2010-11-15` command to list objects
   in the logtest container that begin with the ``2010-11-15`` string.

#. For each item in the list, run the :command:`swift download -o -` command.

#. Pipe the output to :command:`grep` and :command:`wc`.
   Use the :command:`echo` command to display the object name.

   .. code-block:: console

       $ for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester \
        -K testing list -p 2010-11-15 logtest` ; \
               do  echo -ne "$f - PUTS - " ; swift -A \
               http://127.0.0.1:11000/v1.0 -U test:tester \
               -K testing download -o - logtest $f | grep PUT | wc -l ; \
             done

   .. code-block:: console

      2010-11-15-21_access.log - PUTS - 402
      2010-11-15-22_access.log - PUTS - 910