Fix install error and edit the "analyze logs" chapter (swift)

bug: #1225111

Change-Id: I33b6a3ddf851e11535f9dcecd8acf1d56f7d36ac
author: diane fleming
This commit is contained in:
Diane Fleming 2013-09-13 15:46:27 -05:00
parent ba50e344c5
commit f343c0e357
2 changed files with 105 additions and 103 deletions

View File

@ -146,7 +146,7 @@
<para>You must install each client separately.</para>
<para>Run the following command to install or update a
client package:</para>
<screen><prompt>$</prompt> <userinput>sudo pip install [--update] python-<replaceable>PROJECT</replaceable>client</userinput></screen>
<screen><prompt>$</prompt> <userinput>sudo pip install [--upgrade] python-<replaceable>PROJECT</replaceable>client</userinput></screen>
<para>Where <replaceable>PROJECT</replaceable> is the
project name and has one of the following
values:</para>

View File

@ -4,79 +4,82 @@
xmlns:xi="http://www.w3.org/2001/XInclude"
xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0"
xml:id="analyzing-log-files-with-swift-cli">
<title>Analyzing Log Files with Swift CLI</title>
<para>When you want quick, command-line answers to
questions about logs, you can use swift with the -o or
-output option. The -o —output option can only be used
with a single object download to redirect the data
stream to either a different file name or to STDOUT
(-). The ability to redirect the output to STDOUT
allows you to pipe “|” data without saving it to disk
first. One common use case is being able to do some
quick log file analysis. First lets use swift to
setup some data for the examples. The “logtest”
directory contains four log files with the following
line format.</para>
<para>
<literallayout>Files:
2010-11-16-21_access.log
2010-11-16-22_access.log
2010-11-15-21_access.log
2010-11-15-22_access.log
Log lines:
Nov 15 21:53:52 lucid64 proxy-server - 127.0.0.1 15/Nov/2010/22/53/52 DELETE /v1/AUTH_cd4f57824deb4248a533f2c28bf156d3/2eefc05599d44df38a7f18b0b42ffedd HTTP/1.0 204 - \
- test%3Atester%2CAUTH_tkcdab3c6296e249d7b7e2454ee57266ff - - - txaba5984c-aac7-460e-b04b-afc43f0c6571 - 0.0432
</literallayout>
</para>
<para>The swift tool can easily upload the four log files
into a container named “logtest”:</para>
<para><screen>
<prompt>$</prompt> <userinput>cd logs</userinput>
<prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing upload logtest *.log</userinput>
</screen>
<programlisting>
2010-11-16-21_access.log
<title>Analyze log files</title>
<para>Use the swift command-line client to analyze log files. </para>
<para>The swift client is simple to use, scalable, and flexible. </para><para>Use the swift client <option>-o</option> or
<option>-output</option> option to get short answers to
questions about logs,</para>
<para>You can use the <option>-o —output</option> option with a
single object download to redirect the command output to a
specific file or to STDOUT (<literal>-</literal>). The ability
to redirect the output to STDOUT enables you to pipe
(<literal>|</literal>) data without saving it to disk
first. </para>
<section xml:id="swift_file_analysis"><title>Upload and analyze log files</title><procedure>
<step>
<para>This example assumes that
<filename>logtest</filename> directory
contains the following log files:</para>
<example><title>Example files</title> <screen>2010-11-16-21_access.log
2010-11-16-22_access.log
2010-11-15-21_access.log
2010-11-15-22_access.log
</programlisting>
Get statistics on the account: <screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing -q stat</userinput></screen>
<programlisting>
Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
2010-11-15-22_access.log</screen></example>
<para>Each file uses the following line format:</para>
<example><title>Log line format</title><screen><computeroutput>Nov 15 21:53:52 lucid64 proxy-server - 127.0.0.1 15/Nov/2010/22/53/52 DELETE /v1/AUTH_cd4f57824deb4248a533f2c28bf156d3/2eefc05599d44df38a7f18b0b42ffedd HTTP/1.0 204 - \
- test%3Atester%2CAUTH_tkcdab3c6296e249d7b7e2454ee57266ff - - - txaba5984c-aac7-460e-b04b-afc43f0c6571 - 0.0432</computeroutput></screen></example>
</step><step>
<para>Change into the <filename class="directory"
>logtest</filename> directory:</para>
<screen><prompt>$</prompt> <userinput>cd logtest</userinput></screen></step>
<step>
<para>Upload the log files into the <filename>logtest</filename>
container:</para>
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing upload logtest *.log</userinput></screen>
<screen><computeroutput>2010-11-16-21_access.log
2010-11-16-22_access.log
2010-11-15-21_access.log
2010-11-15-22_access.log</computeroutput></screen></step><step>
<para>Get statistics for the account:</para>
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing -q stat</userinput></screen>
<screen><computeroutput>Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
Containers: 1
Objects: 4
Bytes: 5888268
</programlisting>
Get statistics on the container:
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing stat logtest</userinput> </screen><programlisting>Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
Bytes: 5888268</computeroutput></screen></step>
<step><para>Get statistics for the logtest container:</para>
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing stat logtest</userinput> </screen>
<screen><computeroutput>Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
Container: logtest
Objects: 4
Bytes: 5864468
Read ACL:
Write ACL:
</programlisting>
List all the objects in the container:
<screen><prompt>$</prompt> <userinput>swift -A http:///swift-auth.com:11000/v1.0 -U test:tester -K testing list logtest</userinput></screen><programlisting>2010-11-15-21_access.log
Write ACL:</computeroutput></screen></step>
<step><para> List all objects in the logtest container:</para>
<screen><prompt>$</prompt> <userinput>swift -A http:///swift-auth.com:11000/v1.0 -U test:tester -K testing list logtest</userinput></screen>
<screen><computeroutput>2010-11-15-21_access.log
2010-11-15-22_access.log
2010-11-16-21_access.log
2010-11-16-22_access.log</programlisting>
</para>
<para>The following examples use the -o —output option
with (-) to help answer questions about the uploaded
log files. The swift command will download an object,
stream it to awk to determine the breakdown of
requests by return code for everything during 2200 on
November 16th, 2010. Based on the log line format
column 9 is the type of request and column 12 is the
return code. After awk processes the data stream it is
piped to sort and then uniq -c to sum up the number of
occurrences for each combination of request type and
return code.</para>
<para><screen>
<prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing download -o - logtest 2010-11-16-22_access.log | awk { print $9”-“$12} | sort | uniq -c</userinput>
</screen>
<programlisting>805 DELETE-204
2010-11-16-22_access.log</computeroutput></screen></step></procedure></section>
<section xml:id="swift_upload_files"><title>Download and analyze an object</title> <procedure>
<para>This example uses the <option>-o —output</option>
option and a hyphen (<literal>-</literal>) to get
information about an object. </para>
<para>Use the swift <command>download</command> command to
download the object. On this command, stream the
output to <command>awk</command> to break down
requests by return code and the date <literal>2200 on
November 16th, 2010</literal>. </para>
<para>Using the log line format, find the request type in
column 9 and the return code in column 12. </para>
<para>After <command>awk</command> processes the output,
it pipes it to <command>sort</command> and
<command>uniq -c</command> to sum up the number of
occurrences for each request type and return code
combination.</para>
<step>
<para>Download an object:</para>
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \
download -o - logtest 2010-11-16-22_access.log | awk { print $9”-“$12} | sort | uniq -c</userinput> </screen>
<screen><computeroutput>805 DELETE-204
12 DELETE-404
2 DELETE-409
723 GET-200
@ -105,47 +108,46 @@ Write ACL:
6 PUT-412
6 PUT-413
2 PUT-422
8 PUT-499</programlisting>
</para>
<para>This example uses a bash for loop with awk, swift
with its -o —output option with a hyphen (-) to find
out how many PUT requests are in each log file. First
create a list of objects by running swift with the
list command on the “logtest” container; then for each
item in the list run swift with download -o - then
pipe the output into grep to filter the put requests
and finally into wc -l to count the lines.</para>
<para>
<screen><prompt>$</prompt> <userinput>for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing list logtest` ; \
8 PUT-499</computeroutput></screen></step>
<step><para>Discover how many PUT requests are in each log file.</para>
<para>Use a bash for loop with awk and swift with
the <option>-o —output</option> option and a
hyphen (<literal>-</literal>) to discover how many
PUT requests are in each log file. </para>
<para>Run the swift <command>list</command> command to
list objects in the logtest container. Then, for
each item in the list, run the swift
<command>download -o -</command> command. Pipe
the output into grep to filter the PUT requests.
Finally, pipe into <command>wc -l</command> to
count the lines.</para>
<screen><prompt>$</prompt> <userinput>for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing list logtest` ; \
do echo -ne “PUTS - ” ; swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing download -o - logtest $f | grep PUT | wc -l ; \
done</userinput></screen>
<programlisting>
2010-11-15-21_access.log - PUTS - 402
<screen><computeroutput>2010-11-15-21_access.log - PUTS - 402
2010-11-15-22_access.log - PUTS - 1091
2010-11-16-21_access.log - PUTS - 892
2010-11-16-22_access.log - PUTS - 910
</programlisting>
</para>
<para>By adding the -p —prefix option a prefix query is
performed on the list to return only the object names
that begin with a specific string. Lets determine out
how many PUT requests are in each object with a name
beginning with “2010-11-15”. First create a list of
objects by running swift with the list command on the
“logtest” container with the prefix option -p
2010-11-15. Then on each of item(s) returned run swift
with the download -o - then pipe the output to grep
and wc as in the previous example. The echo command is
added to display the object name.</para>
<para>
<screen><prompt>$</prompt> <userinput>for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing list -p 2010-11-15 logtest` ; \
do echo -ne “$f - PUTS - ” ; swift -A http://127.0.0.1:11000/v1.0 -U test:tester -K testing download -o - logtest $f | grep PUT | wc -l ; \
2010-11-16-22_access.log - PUTS - 910</computeroutput></screen>
</step><step>
<para>List the object names that begin with a specified
string.</para>
<para>Run the swift <command>list -p 2010-11-15</command>
command to list objects in the logtest container
that begin with the <literal>2010-11-15</literal>
string. </para>
<para>For each item in the list, run the swift
<command>download -o -</command> command. </para>
<para>Pipe the output to <command>grep</command> and
<command>wc</command>. Use the
<command>echo</command> command to display the
object name.</para>
<screen><prompt>$</prompt> <userinput>for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing list -p 2010-11-15 logtest` ; \
do echo -ne “$f - PUTS - ” ; swift -A http://127.0.0.1:11000/v1.0 -U test:tester \
-K testing download -o - logtest $f | grep PUT | wc -l ; \
done</userinput></screen>
<programlisting>2010-11-15-21_access.log - PUTS - 402
2010-11-15-22_access.log - PUTS - 910</programlisting>
</para>
<para>The swift utility is simple, scalable, flexible and
provides useful solutions all of which are core
principles of cloud computing; with the -o output
option being just one of its many features.</para>
</section>
<screen><computeroutput>2010-11-15-21_access.log - PUTS - 402
2010-11-15-22_access.log - PUTS - 910</computeroutput></screen></step></procedure>
</section>
</section>