================= Analyze log files ================= Use the swift command-line client for Object Storage to analyze log files. The swift client is simple to use, scalable, and flexible. Use the swift client :option:`-o` or :option:`-output` option to get short answers to questions about logs. You can use the :option:`-o` or :option:`--output` option with a single object download to redirect the command output to a specific file or to STDOUT (``-``). The ability to redirect the output to STDOUT enables you to pipe (``|``) data without saving it to disk first. Upload and analyze log files ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #. This example assumes that ``logtest`` directory contains the following log files. .. code-block:: console 2010-11-16-21_access.log 2010-11-16-22_access.log 2010-11-15-21_access.log 2010-11-15-22_access.log Each file uses the following line format. .. code-block:: console Nov 15 21:53:52 lucid64 proxy-server - 127.0.0.1 15/Nov/2010/22/53/52 DELETE /v1/AUTH_cd4f57824deb4248a533f2c28bf156d3/2eefc05599d44df38a7f18b0b42ffedd HTTP/1.0 204 - \ - test%3Atester%2CAUTH_tkcdab3c6296e249d7b7e2454ee57266ff - - - txaba5984c-aac7-460e-b04b-afc43f0c6571 - 0.0432 #. Change into the ``logtest`` directory: .. code-block:: console $ cd logtest #. Upload the log files into the ``logtest`` container: .. code-block:: console $ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing upload logtest *.log .. code-block:: console 2010-11-16-21_access.log 2010-11-16-22_access.log 2010-11-15-21_access.log 2010-11-15-22_access.log #. Get statistics for the account: .. code-block:: console $ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \ -q stat .. code-block:: console Account: AUTH_cd4f57824deb4248a533f2c28bf156d3 Containers: 1 Objects: 4 Bytes: 5888268 #. Get statistics for the ``logtest`` container: .. code-block:: console $ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \ stat logtest .. code-block:: console Account: AUTH_cd4f57824deb4248a533f2c28bf156d3 Container: logtest Objects: 4 Bytes: 5864468 Read ACL: Write ACL: #. List all objects in the logtest container: .. code-block:: console $ swift -A http:///swift-auth.com:11000/v1.0 -U test:tester -K testing \ list logtest .. code-block:: console 2010-11-15-21_access.log 2010-11-15-22_access.log 2010-11-16-21_access.log 2010-11-16-22_access.log Download and analyze an object ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This example uses the :option:`-o` option and a hyphen (``-``) to get information about an object. Use the :command:`swift download` command to download the object. On this command, stream the output to ``awk`` to break down requests by return code and the date ``2200 on November 16th, 2010``. Using the log line format, find the request type in column 9 and the return code in column 12. After ``awk`` processes the output, it pipes it to ``sort`` and ``uniq -c`` to sum up the number of occurrences for each request type and return code combination. #. Download an object: .. code-block:: console $ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \ download -o - logtest 2010-11-16-22_access.log | awk '{ print \ $9"-"$12}' | sort | uniq -c .. code-block:: console 805 DELETE-204 12 DELETE-404 2 DELETE-409 723 GET-200 142 GET-204 74 GET-206 80 GET-304 34 GET-401 5 GET-403 18 GET-404 166 GET-412 2 GET-416 50 HEAD-200 17 HEAD-204 20 HEAD-401 8 HEAD-404 30 POST-202 25 POST-204 22 POST-400 6 POST-404 842 PUT-201 2 PUT-202 32 PUT-400 4 PUT-403 4 PUT-404 2 PUT-411 6 PUT-412 6 PUT-413 2 PUT-422 8 PUT-499 #. Discover how many PUT requests are in each log file. Use a bash for loop with awk and swift with the :option:`-o` or :option:`--output` option and a hyphen (``-``) to discover how many PUT requests are in each log file. Run the :command:`swift list` command to list objects in the logtest container. Then, for each item in the list, run the :command:`swift download -o -` command. Pipe the output into grep to filter the PUT requests. Finally, pipe into ``wc -l`` to count the lines. .. code-block:: console $ for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester \ -K testing list logtest` ; \ do echo -ne "PUTS - " ; swift -A \ http://swift-auth.com:11000/v1.0 -U test:tester \ -K testing download -o - logtest $f | grep PUT | wc -l ; \ done .. code-block:: console 2010-11-15-21_access.log - PUTS - 402 2010-11-15-22_access.log - PUTS - 1091 2010-11-16-21_access.log - PUTS - 892 2010-11-16-22_access.log - PUTS - 910 #. List the object names that begin with a specified string. #. Run the :command:`swift list -p 2010-11-15` command to list objects in the logtest container that begin with the ``2010-11-15`` string. #. For each item in the list, run the :command:`swift download -o -` command. #. Pipe the output to :command:`grep` and :command:`wc`. Use the :command:`echo` command to display the object name. .. code-block:: console $ for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester \ -K testing list -p 2010-11-15 logtest` ; \ do echo -ne "$f - PUTS - " ; swift -A \ http://127.0.0.1:11000/v1.0 -U test:tester \ -K testing download -o - logtest $f | grep PUT | wc -l ; \ done .. code-block:: console 2010-11-15-21_access.log - PUTS - 402 2010-11-15-22_access.log - PUTS - 910