155 lines
7.9 KiB
XML
155 lines
7.9 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<section xmlns="http://docbook.org/ns/docbook"
|
|
xmlns:xi="http://www.w3.org/2001/XInclude"
|
|
xmlns:xlink="http://www.w3.org/1999/xlink"
|
|
version="5.0"
|
|
xml:id="analyzing-log-files-with-swift-cli">
|
|
<title>Analyze log files</title>
|
|
<para>Use the swift command-line client to analyze log files.</para>
|
|
<para>The swift client is simple to use, scalable, and flexible.</para><para>Use the swift client <option>-o</option> or
|
|
<option>-output</option> option to get short answers to
|
|
questions about logs,</para>
|
|
<para>You can use the <parameter>-o</parameter> or <parameter>--output</parameter> option with a
|
|
single object download to redirect the command output to a
|
|
specific file or to STDOUT (<literal>-</literal>). The ability
|
|
to redirect the output to STDOUT enables you to pipe
|
|
(<literal>|</literal>) data without saving it to disk
|
|
first.</para>
|
|
<section xml:id="swift_file_analysis"><title>Upload and analyze log files</title><procedure>
|
|
<step>
|
|
<para>This example assumes that
|
|
<filename>logtest</filename> directory
|
|
contains the following log files:</para>
|
|
<example><title>Example files</title> <screen>2010-11-16-21_access.log
|
|
2010-11-16-22_access.log
|
|
2010-11-15-21_access.log
|
|
2010-11-15-22_access.log</screen></example>
|
|
<para>Each file uses the following line format:</para>
|
|
<example><title>Log line format</title><screen><computeroutput>Nov 15 21:53:52 lucid64 proxy-server - 127.0.0.1 15/Nov/2010/22/53/52 DELETE /v1/AUTH_cd4f57824deb4248a533f2c28bf156d3/2eefc05599d44df38a7f18b0b42ffedd HTTP/1.0 204 - \
|
|
- test%3Atester%2CAUTH_tkcdab3c6296e249d7b7e2454ee57266ff - - - txaba5984c-aac7-460e-b04b-afc43f0c6571 - 0.0432</computeroutput></screen></example>
|
|
</step><step>
|
|
<para>Change into the <filename class="directory"
|
|
>logtest</filename> directory:</para>
|
|
<screen><prompt>$</prompt> <userinput>cd logtest</userinput></screen></step>
|
|
<step>
|
|
<para>Upload the log files into the <filename>logtest</filename>
|
|
container:</para>
|
|
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing upload logtest *.log</userinput></screen>
|
|
<screen><computeroutput>2010-11-16-21_access.log
|
|
2010-11-16-22_access.log
|
|
2010-11-15-21_access.log
|
|
2010-11-15-22_access.log</computeroutput></screen></step><step>
|
|
<para>Get statistics for the account:</para>
|
|
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing -q stat</userinput></screen>
|
|
<screen><computeroutput>Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
|
|
Containers: 1
|
|
Objects: 4
|
|
Bytes: 5888268</computeroutput></screen></step>
|
|
<step><para>Get statistics for the logtest container:</para>
|
|
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing stat logtest</userinput> </screen>
|
|
<screen><computeroutput>Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
|
|
Container: logtest
|
|
Objects: 4
|
|
Bytes: 5864468
|
|
Read ACL:
|
|
Write ACL:</computeroutput></screen></step>
|
|
<step><para>List all objects in the logtest container:</para>
|
|
<screen><prompt>$</prompt> <userinput>swift -A http:///swift-auth.com:11000/v1.0 -U test:tester -K testing list logtest</userinput></screen>
|
|
<screen><computeroutput>2010-11-15-21_access.log
|
|
2010-11-15-22_access.log
|
|
2010-11-16-21_access.log
|
|
2010-11-16-22_access.log</computeroutput></screen></step></procedure></section>
|
|
<section xml:id="swift_upload_files"><title>Download and analyze an object</title>
|
|
<procedure>
|
|
<para>This example uses the <option>-o</option>
|
|
option and a hyphen (<literal>-</literal>) to get
|
|
information about an object.</para>
|
|
<para>Use the swift <command>download</command> command to
|
|
download the object. On this command, stream the
|
|
output to <command>awk</command> to break down
|
|
requests by return code and the date <literal>2200 on
|
|
November 16th, 2010</literal>.</para>
|
|
<para>Using the log line format, find the request type in
|
|
column 9 and the return code in column 12.</para>
|
|
<para>After <command>awk</command> processes the output,
|
|
it pipes it to <command>sort</command> and
|
|
<command>uniq -c</command> to sum up the number of
|
|
occurrences for each request type and return code
|
|
combination.</para>
|
|
<step>
|
|
<para>Download an object:</para>
|
|
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \
|
|
download -o - logtest 2010-11-16-22_access.log | awk '{ print $9"-"$12}' | sort | uniq -c</userinput> </screen>
|
|
<screen><computeroutput>805 DELETE-204
|
|
12 DELETE-404
|
|
2 DELETE-409
|
|
723 GET-200
|
|
142 GET-204
|
|
74 GET-206
|
|
80 GET-304
|
|
34 GET-401
|
|
5 GET-403
|
|
18 GET-404
|
|
166 GET-412
|
|
2 GET-416
|
|
50 HEAD-200
|
|
17 HEAD-204
|
|
20 HEAD-401
|
|
8 HEAD-404
|
|
30 POST-202
|
|
25 POST-204
|
|
22 POST-400
|
|
6 POST-404
|
|
842 PUT-201
|
|
2 PUT-202
|
|
32 PUT-400
|
|
4 PUT-403
|
|
4 PUT-404
|
|
2 PUT-411
|
|
6 PUT-412
|
|
6 PUT-413
|
|
2 PUT-422
|
|
8 PUT-499</computeroutput></screen></step>
|
|
<step><para>Discover how many PUT requests are in each log file.</para>
|
|
<para>Use a bash for loop with awk and swift with
|
|
the <parameter>-o</parameter> or <parameter>--output</parameter> option and a
|
|
hyphen (<literal>-</literal>) to discover how many
|
|
PUT requests are in each log file.</para>
|
|
|
|
<para>Run the swift <command>list</command> command to
|
|
list objects in the logtest container. Then, for
|
|
each item in the list, run the swift
|
|
<command>download -o -</command> command. Pipe
|
|
the output into grep to filter the PUT requests.
|
|
Finally, pipe into <command>wc -l</command> to
|
|
count the lines.</para>
|
|
<screen><prompt>$</prompt> <userinput>for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing list logtest` ; \
|
|
do echo -ne "PUTS - " ; swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing download -o - logtest $f | grep PUT | wc -l ; \
|
|
done</userinput></screen>
|
|
<screen><computeroutput>2010-11-15-21_access.log - PUTS - 402
|
|
2010-11-15-22_access.log - PUTS - 1091
|
|
2010-11-16-21_access.log - PUTS - 892
|
|
2010-11-16-22_access.log - PUTS - 910</computeroutput></screen>
|
|
</step><step>
|
|
<para>List the object names that begin with a specified
|
|
string.</para>
|
|
<para>Run the swift <command>list -p 2010-11-15</command>
|
|
command to list objects in the logtest container
|
|
that begin with the <literal>2010-11-15</literal>
|
|
string.</para>
|
|
<para>For each item in the list, run the swift
|
|
<command>download -o -</command> command.</para>
|
|
<para>Pipe the output to <command>grep</command> and
|
|
<command>wc</command>. Use the
|
|
<command>echo</command> command to display the
|
|
object name.</para>
|
|
|
|
<screen><prompt>$</prompt> <userinput>for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing list -p 2010-11-15 logtest` ; \
|
|
do echo -ne "$f - PUTS - " ; swift -A http://127.0.0.1:11000/v1.0 -U test:tester \
|
|
-K testing download -o - logtest $f | grep PUT | wc -l ; \
|
|
done</userinput></screen>
|
|
<screen><computeroutput>2010-11-15-21_access.log - PUTS - 402
|
|
2010-11-15-22_access.log - PUTS - 910</computeroutput></screen></step></procedure>
|
|
</section>
|
|
</section>
|