2ce5b11b1a
This patch changes the name of the Admin-Guide from the Cloud
Admin Guide to the Administrator guide. This affects the
filename in the repository, and references to cloud administrators
within the document texts.
1.) Changing instances of 'cloud administrator'
to 'administrator'.
2.) Change links from '/admin-guide-cloud/' to
'/admin-guide/' within the Admin Guide.
3.) Adjust .htaccess file.
Change-Id: I7f21a710e922981aa295afc0616de36fd819b523
Implements: blueprint user-guides-reorganised
211 lines
5.9 KiB
ReStructuredText
211 lines
5.9 KiB
ReStructuredText
=================
|
|
Analyze log files
|
|
=================
|
|
|
|
Use the swift command-line client for Object Storage to analyze log files.
|
|
|
|
The swift client is simple to use, scalable, and flexible.
|
|
|
|
Use the swift client :option:`-o` or :option:`-output` option to get
|
|
short answers to questions about logs.
|
|
|
|
You can use the :option:`-o` or :option:`--output` option with a single object
|
|
download to redirect the command output to a specific file or to STDOUT
|
|
(``-``). The ability to redirect the output to STDOUT enables you to
|
|
pipe (``|``) data without saving it to disk first.
|
|
|
|
Upload and analyze log files
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
#. This example assumes that ``logtest`` directory contains the
|
|
following log files.
|
|
|
|
.. code-block:: console
|
|
|
|
2010-11-16-21_access.log
|
|
2010-11-16-22_access.log
|
|
2010-11-15-21_access.log
|
|
2010-11-15-22_access.log
|
|
|
|
|
|
Each file uses the following line format.
|
|
|
|
.. code-block:: console
|
|
|
|
Nov 15 21:53:52 lucid64 proxy-server - 127.0.0.1 15/Nov/2010/22/53/52 DELETE /v1/AUTH_cd4f57824deb4248a533f2c28bf156d3/2eefc05599d44df38a7f18b0b42ffedd HTTP/1.0 204 - \
|
|
- test%3Atester%2CAUTH_tkcdab3c6296e249d7b7e2454ee57266ff - - - txaba5984c-aac7-460e-b04b-afc43f0c6571 - 0.0432
|
|
|
|
|
|
#. Change into the ``logtest`` directory:
|
|
|
|
.. code-block:: console
|
|
|
|
$ cd logtest
|
|
|
|
#. Upload the log files into the ``logtest`` container:
|
|
|
|
.. code-block:: console
|
|
|
|
$ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing upload logtest *.log
|
|
|
|
.. code-block:: console
|
|
|
|
2010-11-16-21_access.log
|
|
2010-11-16-22_access.log
|
|
2010-11-15-21_access.log
|
|
2010-11-15-22_access.log
|
|
|
|
#. Get statistics for the account:
|
|
|
|
.. code-block:: console
|
|
|
|
$ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \
|
|
-q stat
|
|
|
|
.. code-block:: console
|
|
|
|
Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
|
|
Containers: 1
|
|
Objects: 4
|
|
Bytes: 5888268
|
|
|
|
#. Get statistics for the ``logtest`` container:
|
|
|
|
.. code-block:: console
|
|
|
|
$ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \
|
|
stat logtest
|
|
|
|
.. code-block:: console
|
|
|
|
Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
|
|
Container: logtest
|
|
Objects: 4
|
|
Bytes: 5864468
|
|
Read ACL:
|
|
Write ACL:
|
|
|
|
#. List all objects in the logtest container:
|
|
|
|
.. code-block:: console
|
|
|
|
$ swift -A http:///swift-auth.com:11000/v1.0 -U test:tester -K testing \
|
|
list logtest
|
|
|
|
.. code-block:: console
|
|
|
|
2010-11-15-21_access.log
|
|
2010-11-15-22_access.log
|
|
2010-11-16-21_access.log
|
|
2010-11-16-22_access.log
|
|
|
|
Download and analyze an object
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
This example uses the :option:`-o` option and a hyphen (``-``) to get
|
|
information about an object.
|
|
|
|
Use the :command:`swift download` command to download the object. On this
|
|
command, stream the output to ``awk`` to break down requests by return
|
|
code and the date ``2200 on November 16th, 2010``.
|
|
|
|
Using the log line format, find the request type in column 9 and the
|
|
return code in column 12.
|
|
|
|
After ``awk`` processes the output, it pipes it to ``sort`` and ``uniq
|
|
-c`` to sum up the number of occurrences for each request type and
|
|
return code combination.
|
|
|
|
#. Download an object:
|
|
|
|
.. code-block:: console
|
|
|
|
$ swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \
|
|
download -o - logtest 2010-11-16-22_access.log | awk '{ print \
|
|
$9"-"$12}' | sort | uniq -c
|
|
|
|
.. code-block:: console
|
|
|
|
805 DELETE-204
|
|
12 DELETE-404
|
|
2 DELETE-409
|
|
723 GET-200
|
|
142 GET-204
|
|
74 GET-206
|
|
80 GET-304
|
|
34 GET-401
|
|
5 GET-403
|
|
18 GET-404
|
|
166 GET-412
|
|
2 GET-416
|
|
50 HEAD-200
|
|
17 HEAD-204
|
|
20 HEAD-401
|
|
8 HEAD-404
|
|
30 POST-202
|
|
25 POST-204
|
|
22 POST-400
|
|
6 POST-404
|
|
842 PUT-201
|
|
2 PUT-202
|
|
32 PUT-400
|
|
4 PUT-403
|
|
4 PUT-404
|
|
2 PUT-411
|
|
6 PUT-412
|
|
6 PUT-413
|
|
2 PUT-422
|
|
8 PUT-499
|
|
|
|
#. Discover how many PUT requests are in each log file.
|
|
|
|
Use a bash for loop with awk and swift with the :option:`-o` or
|
|
:option:`--output` option and a hyphen (``-``) to discover how many
|
|
PUT requests are in each log file.
|
|
|
|
Run the :command:`swift list` command to list objects in the logtest
|
|
container. Then, for each item in the list, run the
|
|
:command:`swift download -o -` command. Pipe the output into grep to
|
|
filter the PUT requests. Finally, pipe into ``wc -l`` to count the lines.
|
|
|
|
.. code-block:: console
|
|
|
|
$ for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester \
|
|
-K testing list logtest` ; \
|
|
do echo -ne "PUTS - " ; swift -A \
|
|
http://swift-auth.com:11000/v1.0 -U test:tester \
|
|
-K testing download -o - logtest $f | grep PUT | wc -l ; \
|
|
done
|
|
|
|
.. code-block:: console
|
|
|
|
2010-11-15-21_access.log - PUTS - 402
|
|
2010-11-15-22_access.log - PUTS - 1091
|
|
2010-11-16-21_access.log - PUTS - 892
|
|
2010-11-16-22_access.log - PUTS - 910
|
|
|
|
#. List the object names that begin with a specified string.
|
|
|
|
#. Run the :command:`swift list -p 2010-11-15` command to list objects
|
|
in the logtest container that begin with the ``2010-11-15`` string.
|
|
|
|
#. For each item in the list, run the :command:`swift download -o -` command.
|
|
|
|
#. Pipe the output to :command:`grep` and :command:`wc`.
|
|
Use the :command:`echo` command to display the object name.
|
|
|
|
.. code-block:: console
|
|
|
|
$ for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester \
|
|
-K testing list -p 2010-11-15 logtest` ; \
|
|
do echo -ne "$f - PUTS - " ; swift -A \
|
|
http://127.0.0.1:11000/v1.0 -U test:tester \
|
|
-K testing download -o - logtest $f | grep PUT | wc -l ; \
|
|
done
|
|
|
|
.. code-block:: console
|
|
|
|
2010-11-15-21_access.log - PUTS - 402
|
|
2010-11-15-22_access.log - PUTS - 910
|
|
|