155 lines
7.9 KiB
XML
155 lines
7.9 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
||
<section
|
||
xmlns="http://docbook.org/ns/docbook"
|
||
xmlns:xi="http://www.w3.org/2001/XInclude"
|
||
xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0"
|
||
xml:id="analyzing-log-files-with-swift-cli">
|
||
<title>Analyze log files</title>
|
||
<para>Use the swift command-line client to analyze log files.</para>
|
||
<para>The swift client is simple to use, scalable, and flexible.</para><para>Use the swift client <option>-o</option> or
|
||
<option>-output</option> option to get short answers to
|
||
questions about logs,</para>
|
||
<para>You can use the <option>-o —output</option> option with a
|
||
single object download to redirect the command output to a
|
||
specific file or to STDOUT (<literal>-</literal>). The ability
|
||
to redirect the output to STDOUT enables you to pipe
|
||
(<literal>|</literal>) data without saving it to disk
|
||
first.</para>
|
||
<section xml:id="swift_file_analysis"><title>Upload and analyze log files</title><procedure>
|
||
<step>
|
||
<para>This example assumes that
|
||
<filename>logtest</filename> directory
|
||
contains the following log files:</para>
|
||
<example><title>Example files</title> <screen>2010-11-16-21_access.log
|
||
2010-11-16-22_access.log
|
||
2010-11-15-21_access.log
|
||
2010-11-15-22_access.log</screen></example>
|
||
<para>Each file uses the following line format:</para>
|
||
<example><title>Log line format</title><screen><computeroutput>Nov 15 21:53:52 lucid64 proxy-server - 127.0.0.1 15/Nov/2010/22/53/52 DELETE /v1/AUTH_cd4f57824deb4248a533f2c28bf156d3/2eefc05599d44df38a7f18b0b42ffedd HTTP/1.0 204 - \
|
||
- test%3Atester%2CAUTH_tkcdab3c6296e249d7b7e2454ee57266ff - - - txaba5984c-aac7-460e-b04b-afc43f0c6571 - 0.0432</computeroutput></screen></example>
|
||
</step><step>
|
||
<para>Change into the <filename class="directory"
|
||
>logtest</filename> directory:</para>
|
||
<screen><prompt>$</prompt> <userinput>cd logtest</userinput></screen></step>
|
||
<step>
|
||
<para>Upload the log files into the <filename>logtest</filename>
|
||
container:</para>
|
||
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing upload logtest *.log</userinput></screen>
|
||
<screen><computeroutput>2010-11-16-21_access.log
|
||
2010-11-16-22_access.log
|
||
2010-11-15-21_access.log
|
||
2010-11-15-22_access.log</computeroutput></screen></step><step>
|
||
<para>Get statistics for the account:</para>
|
||
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing -q stat</userinput></screen>
|
||
<screen><computeroutput>Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
|
||
Containers: 1
|
||
Objects: 4
|
||
Bytes: 5888268</computeroutput></screen></step>
|
||
<step><para>Get statistics for the logtest container:</para>
|
||
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing stat logtest</userinput> </screen>
|
||
<screen><computeroutput>Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
|
||
Container: logtest
|
||
Objects: 4
|
||
Bytes: 5864468
|
||
Read ACL:
|
||
Write ACL:</computeroutput></screen></step>
|
||
<step><para>List all objects in the logtest container:</para>
|
||
<screen><prompt>$</prompt> <userinput>swift -A http:///swift-auth.com:11000/v1.0 -U test:tester -K testing list logtest</userinput></screen>
|
||
<screen><computeroutput>2010-11-15-21_access.log
|
||
2010-11-15-22_access.log
|
||
2010-11-16-21_access.log
|
||
2010-11-16-22_access.log</computeroutput></screen></step></procedure></section>
|
||
<section xml:id="swift_upload_files"><title>Download and analyze an object</title>
|
||
<procedure>
|
||
<para>This example uses the <option>-o —output</option>
|
||
option and a hyphen (<literal>-</literal>) to get
|
||
information about an object.</para>
|
||
<para>Use the swift <command>download</command> command to
|
||
download the object. On this command, stream the
|
||
output to <command>awk</command> to break down
|
||
requests by return code and the date <literal>2200 on
|
||
November 16th, 2010</literal>.</para>
|
||
<para>Using the log line format, find the request type in
|
||
column 9 and the return code in column 12.</para>
|
||
<para>After <command>awk</command> processes the output,
|
||
it pipes it to <command>sort</command> and
|
||
<command>uniq -c</command> to sum up the number of
|
||
occurrences for each request type and return code
|
||
combination.</para>
|
||
<step>
|
||
<para>Download an object:</para>
|
||
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \
|
||
download -o - logtest 2010-11-16-22_access.log | awk ‘{ print $9”-“$12}’ | sort | uniq -c</userinput> </screen>
|
||
<screen><computeroutput>805 DELETE-204
|
||
12 DELETE-404
|
||
2 DELETE-409
|
||
723 GET-200
|
||
142 GET-204
|
||
74 GET-206
|
||
80 GET-304
|
||
34 GET-401
|
||
5 GET-403
|
||
18 GET-404
|
||
166 GET-412
|
||
2 GET-416
|
||
50 HEAD-200
|
||
17 HEAD-204
|
||
20 HEAD-401
|
||
8 HEAD-404
|
||
30 POST-202
|
||
25 POST-204
|
||
22 POST-400
|
||
6 POST-404
|
||
842 PUT-201
|
||
2 PUT-202
|
||
32 PUT-400
|
||
4 PUT-403
|
||
4 PUT-404
|
||
2 PUT-411
|
||
6 PUT-412
|
||
6 PUT-413
|
||
2 PUT-422
|
||
8 PUT-499</computeroutput></screen></step>
|
||
<step><para>Discover how many PUT requests are in each log file.</para>
|
||
<para>Use a bash for loop with awk and swift with
|
||
the <option>-o —output</option> option and a
|
||
hyphen (<literal>-</literal>) to discover how many
|
||
PUT requests are in each log file.</para>
|
||
|
||
<para>Run the swift <command>list</command> command to
|
||
list objects in the logtest container. Then, for
|
||
each item in the list, run the swift
|
||
<command>download -o -</command> command. Pipe
|
||
the output into grep to filter the PUT requests.
|
||
Finally, pipe into <command>wc -l</command> to
|
||
count the lines.</para>
|
||
<screen><prompt>$</prompt> <userinput>for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing list logtest` ; \
|
||
do echo -ne “PUTS - ” ; swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing download -o - logtest $f | grep PUT | wc -l ; \
|
||
done</userinput></screen>
|
||
<screen><computeroutput>2010-11-15-21_access.log - PUTS - 402
|
||
2010-11-15-22_access.log - PUTS - 1091
|
||
2010-11-16-21_access.log - PUTS - 892
|
||
2010-11-16-22_access.log - PUTS - 910</computeroutput></screen>
|
||
</step><step>
|
||
<para>List the object names that begin with a specified
|
||
string.</para>
|
||
<para>Run the swift <command>list -p 2010-11-15</command>
|
||
command to list objects in the logtest container
|
||
that begin with the <literal>2010-11-15</literal>
|
||
string.</para>
|
||
<para>For each item in the list, run the swift
|
||
<command>download -o -</command> command.</para>
|
||
<para>Pipe the output to <command>grep</command> and
|
||
<command>wc</command>. Use the
|
||
<command>echo</command> command to display the
|
||
object name.</para>
|
||
|
||
<screen><prompt>$</prompt> <userinput>for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing list -p 2010-11-15 logtest` ; \
|
||
do echo -ne “$f - PUTS - ” ; swift -A http://127.0.0.1:11000/v1.0 -U test:tester \
|
||
-K testing download -o - logtest $f | grep PUT | wc -l ; \
|
||
done</userinput></screen>
|
||
<screen><computeroutput>2010-11-15-21_access.log - PUTS - 402
|
||
2010-11-15-22_access.log - PUTS - 910</computeroutput></screen></step></procedure>
|
||
</section>
|
||
</section>
|