Fix install error and edit the "analyze logs" chapter (swift)
bug: #1225111 Change-Id: I33b6a3ddf851e11535f9dcecd8acf1d56f7d36ac author: diane fleming
This commit is contained in:
Diane Fleming
parent
ba50e344c5
commit
f343c0e357
@ -146,7 +146,7 @@
|
||||
<para>You must install each client separately.</para>
|
||||
<para>Run the following command to install or update a
|
||||
client package:</para>
|
||||
<screen><prompt>$</prompt> <userinput>sudo pip install [--update] python-<replaceable>PROJECT</replaceable>client</userinput></screen>
|
||||
<screen><prompt>$</prompt> <userinput>sudo pip install [--upgrade] python-<replaceable>PROJECT</replaceable>client</userinput></screen>
|
||||
<para>Where <replaceable>PROJECT</replaceable> is the
|
||||
project name and has one of the following
|
||||
values:</para>
|
||||
|
||||
@ -4,79 +4,82 @@
|
||||
xmlns:xi="http://www.w3.org/2001/XInclude"
|
||||
xmlns:xlink="http://www.w3.org/1999/xlink" version="5.0"
|
||||
xml:id="analyzing-log-files-with-swift-cli">
|
||||
<title>Analyzing Log Files with Swift CLI</title>
|
||||
<para>When you want quick, command-line answers to
|
||||
questions about logs, you can use swift with the -o or
|
||||
-output option. The -o —output option can only be used
|
||||
with a single object download to redirect the data
|
||||
stream to either a different file name or to STDOUT
|
||||
(-). The ability to redirect the output to STDOUT
|
||||
allows you to pipe “|” data without saving it to disk
|
||||
first. One common use case is being able to do some
|
||||
quick log file analysis. First let’s use swift to
|
||||
setup some data for the examples. The “logtest”
|
||||
directory contains four log files with the following
|
||||
line format.</para>
|
||||
<para>
|
||||
<literallayout>Files:
|
||||
2010-11-16-21_access.log
|
||||
2010-11-16-22_access.log
|
||||
2010-11-15-21_access.log
|
||||
2010-11-15-22_access.log
|
||||
|
||||
Log lines:
|
||||
Nov 15 21:53:52 lucid64 proxy-server - 127.0.0.1 15/Nov/2010/22/53/52 DELETE /v1/AUTH_cd4f57824deb4248a533f2c28bf156d3/2eefc05599d44df38a7f18b0b42ffedd HTTP/1.0 204 - \
|
||||
- test%3Atester%2CAUTH_tkcdab3c6296e249d7b7e2454ee57266ff - - - txaba5984c-aac7-460e-b04b-afc43f0c6571 - 0.0432
|
||||
</literallayout>
|
||||
</para>
|
||||
<para>The swift tool can easily upload the four log files
|
||||
into a container named “logtest”:</para>
|
||||
<para><screen>
|
||||
<prompt>$</prompt> <userinput>cd logs</userinput>
|
||||
<prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing upload logtest *.log</userinput>
|
||||
</screen>
|
||||
<programlisting>
|
||||
2010-11-16-21_access.log
|
||||
<title>Analyze log files</title>
|
||||
<para>Use the swift command-line client to analyze log files. </para>
|
||||
<para>The swift client is simple to use, scalable, and flexible. </para><para>Use the swift client <option>-o</option> or
|
||||
<option>-output</option> option to get short answers to
|
||||
questions about logs,</para>
|
||||
<para>You can use the <option>-o —output</option> option with a
|
||||
single object download to redirect the command output to a
|
||||
specific file or to STDOUT (<literal>-</literal>). The ability
|
||||
to redirect the output to STDOUT enables you to pipe
|
||||
(<literal>|</literal>) data without saving it to disk
|
||||
first. </para>
|
||||
<section xml:id="swift_file_analysis"><title>Upload and analyze log files</title><procedure>
|
||||
<step>
|
||||
<para>This example assumes that
|
||||
<filename>logtest</filename> directory
|
||||
contains the following log files:</para>
|
||||
<example><title>Example files</title> <screen>2010-11-16-21_access.log
|
||||
2010-11-16-22_access.log
|
||||
2010-11-15-21_access.log
|
||||
2010-11-15-22_access.log
|
||||
</programlisting>
|
||||
Get statistics on the account: <screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing -q stat</userinput></screen>
|
||||
<programlisting>
|
||||
Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
|
||||
2010-11-15-22_access.log</screen></example>
|
||||
<para>Each file uses the following line format:</para>
|
||||
<example><title>Log line format</title><screen><computeroutput>Nov 15 21:53:52 lucid64 proxy-server - 127.0.0.1 15/Nov/2010/22/53/52 DELETE /v1/AUTH_cd4f57824deb4248a533f2c28bf156d3/2eefc05599d44df38a7f18b0b42ffedd HTTP/1.0 204 - \
|
||||
- test%3Atester%2CAUTH_tkcdab3c6296e249d7b7e2454ee57266ff - - - txaba5984c-aac7-460e-b04b-afc43f0c6571 - 0.0432</computeroutput></screen></example>
|
||||
</step><step>
|
||||
<para>Change into the <filename class="directory"
|
||||
>logtest</filename> directory:</para>
|
||||
<screen><prompt>$</prompt> <userinput>cd logtest</userinput></screen></step>
|
||||
<step>
|
||||
<para>Upload the log files into the <filename>logtest</filename>
|
||||
container:</para>
|
||||
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing upload logtest *.log</userinput></screen>
|
||||
<screen><computeroutput>2010-11-16-21_access.log
|
||||
2010-11-16-22_access.log
|
||||
2010-11-15-21_access.log
|
||||
2010-11-15-22_access.log</computeroutput></screen></step><step>
|
||||
<para>Get statistics for the account:</para>
|
||||
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing -q stat</userinput></screen>
|
||||
<screen><computeroutput>Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
|
||||
Containers: 1
|
||||
Objects: 4
|
||||
Bytes: 5888268
|
||||
</programlisting>
|
||||
Get statistics on the container:
|
||||
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing stat logtest</userinput> </screen><programlisting>Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
|
||||
Bytes: 5888268</computeroutput></screen></step>
|
||||
<step><para>Get statistics for the logtest container:</para>
|
||||
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing stat logtest</userinput> </screen>
|
||||
<screen><computeroutput>Account: AUTH_cd4f57824deb4248a533f2c28bf156d3
|
||||
Container: logtest
|
||||
Objects: 4
|
||||
Bytes: 5864468
|
||||
Read ACL:
|
||||
Write ACL:
|
||||
</programlisting>
|
||||
List all the objects in the container:
|
||||
<screen><prompt>$</prompt> <userinput>swift -A http:///swift-auth.com:11000/v1.0 -U test:tester -K testing list logtest</userinput></screen><programlisting>2010-11-15-21_access.log
|
||||
Write ACL:</computeroutput></screen></step>
|
||||
<step><para> List all objects in the logtest container:</para>
|
||||
<screen><prompt>$</prompt> <userinput>swift -A http:///swift-auth.com:11000/v1.0 -U test:tester -K testing list logtest</userinput></screen>
|
||||
<screen><computeroutput>2010-11-15-21_access.log
|
||||
2010-11-15-22_access.log
|
||||
2010-11-16-21_access.log
|
||||
2010-11-16-22_access.log</programlisting>
|
||||
</para>
|
||||
<para>The following examples use the -o —output option
|
||||
with (-) to help answer questions about the uploaded
|
||||
log files. The swift command will download an object,
|
||||
stream it to awk to determine the breakdown of
|
||||
requests by return code for everything during 2200 on
|
||||
November 16th, 2010. Based on the log line format
|
||||
column 9 is the type of request and column 12 is the
|
||||
return code. After awk processes the data stream it is
|
||||
piped to sort and then uniq -c to sum up the number of
|
||||
occurrences for each combination of request type and
|
||||
return code.</para>
|
||||
<para><screen>
|
||||
<prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing download -o - logtest 2010-11-16-22_access.log | awk ‘{ print $9”-“$12}’ | sort | uniq -c</userinput>
|
||||
</screen>
|
||||
<programlisting>805 DELETE-204
|
||||
2010-11-16-22_access.log</computeroutput></screen></step></procedure></section>
|
||||
<section xml:id="swift_upload_files"><title>Download and analyze an object</title> <procedure>
|
||||
<para>This example uses the <option>-o —output</option>
|
||||
option and a hyphen (<literal>-</literal>) to get
|
||||
information about an object. </para>
|
||||
<para>Use the swift <command>download</command> command to
|
||||
download the object. On this command, stream the
|
||||
output to <command>awk</command> to break down
|
||||
requests by return code and the date <literal>2200 on
|
||||
November 16th, 2010</literal>. </para>
|
||||
<para>Using the log line format, find the request type in
|
||||
column 9 and the return code in column 12. </para>
|
||||
<para>After <command>awk</command> processes the output,
|
||||
it pipes it to <command>sort</command> and
|
||||
<command>uniq -c</command> to sum up the number of
|
||||
occurrences for each request type and return code
|
||||
combination.</para>
|
||||
<step>
|
||||
<para>Download an object:</para>
|
||||
<screen><prompt>$</prompt> <userinput>swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing \
|
||||
download -o - logtest 2010-11-16-22_access.log | awk ‘{ print $9”-“$12}’ | sort | uniq -c</userinput> </screen>
|
||||
<screen><computeroutput>805 DELETE-204
|
||||
12 DELETE-404
|
||||
2 DELETE-409
|
||||
723 GET-200
|
||||
@ -105,47 +108,46 @@ Write ACL:
|
||||
6 PUT-412
|
||||
6 PUT-413
|
||||
2 PUT-422
|
||||
8 PUT-499</programlisting>
|
||||
</para>
|
||||
<para>This example uses a bash for loop with awk, swift
|
||||
with its -o —output option with a hyphen (-) to find
|
||||
out how many PUT requests are in each log file. First
|
||||
create a list of objects by running swift with the
|
||||
list command on the “logtest” container; then for each
|
||||
item in the list run swift with download -o - then
|
||||
pipe the output into grep to filter the put requests
|
||||
and finally into wc -l to count the lines.</para>
|
||||
<para>
|
||||
<screen><prompt>$</prompt> <userinput>for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing list logtest` ; \
|
||||
8 PUT-499</computeroutput></screen></step>
|
||||
<step><para>Discover how many PUT requests are in each log file.</para>
|
||||
<para>Use a bash for loop with awk and swift with
|
||||
the <option>-o —output</option> option and a
|
||||
hyphen (<literal>-</literal>) to discover how many
|
||||
PUT requests are in each log file. </para>
|
||||
|
||||
<para>Run the swift <command>list</command> command to
|
||||
list objects in the logtest container. Then, for
|
||||
each item in the list, run the swift
|
||||
<command>download -o -</command> command. Pipe
|
||||
the output into grep to filter the PUT requests.
|
||||
Finally, pipe into <command>wc -l</command> to
|
||||
count the lines.</para>
|
||||
<screen><prompt>$</prompt> <userinput>for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing list logtest` ; \
|
||||
do echo -ne “PUTS - ” ; swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing download -o - logtest $f | grep PUT | wc -l ; \
|
||||
done</userinput></screen>
|
||||
<programlisting>
|
||||
2010-11-15-21_access.log - PUTS - 402
|
||||
<screen><computeroutput>2010-11-15-21_access.log - PUTS - 402
|
||||
2010-11-15-22_access.log - PUTS - 1091
|
||||
2010-11-16-21_access.log - PUTS - 892
|
||||
2010-11-16-22_access.log - PUTS - 910
|
||||
</programlisting>
|
||||
</para>
|
||||
<para>By adding the -p —prefix option a prefix query is
|
||||
performed on the list to return only the object names
|
||||
that begin with a specific string. Let’s determine out
|
||||
how many PUT requests are in each object with a name
|
||||
beginning with “2010-11-15”. First create a list of
|
||||
objects by running swift with the list command on the
|
||||
“logtest” container with the prefix option -p
|
||||
2010-11-15. Then on each of item(s) returned run swift
|
||||
with the download -o - then pipe the output to grep
|
||||
and wc as in the previous example. The echo command is
|
||||
added to display the object name.</para>
|
||||
<para>
|
||||
<screen><prompt>$</prompt> <userinput>for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing list -p 2010-11-15 logtest` ; \
|
||||
do echo -ne “$f - PUTS - ” ; swift -A http://127.0.0.1:11000/v1.0 -U test:tester -K testing download -o - logtest $f | grep PUT | wc -l ; \
|
||||
2010-11-16-22_access.log - PUTS - 910</computeroutput></screen>
|
||||
</step><step>
|
||||
<para>List the object names that begin with a specified
|
||||
string.</para>
|
||||
<para>Run the swift <command>list -p 2010-11-15</command>
|
||||
command to list objects in the logtest container
|
||||
that begin with the <literal>2010-11-15</literal>
|
||||
string. </para>
|
||||
<para>For each item in the list, run the swift
|
||||
<command>download -o -</command> command. </para>
|
||||
<para>Pipe the output to <command>grep</command> and
|
||||
<command>wc</command>. Use the
|
||||
<command>echo</command> command to display the
|
||||
object name.</para>
|
||||
|
||||
<screen><prompt>$</prompt> <userinput>for f in `swift -A http://swift-auth.com:11000/v1.0 -U test:tester -K testing list -p 2010-11-15 logtest` ; \
|
||||
do echo -ne “$f - PUTS - ” ; swift -A http://127.0.0.1:11000/v1.0 -U test:tester \
|
||||
-K testing download -o - logtest $f | grep PUT | wc -l ; \
|
||||
done</userinput></screen>
|
||||
<programlisting>2010-11-15-21_access.log - PUTS - 402
|
||||
2010-11-15-22_access.log - PUTS - 910</programlisting>
|
||||
</para>
|
||||
<para>The swift utility is simple, scalable, flexible and
|
||||
provides useful solutions all of which are core
|
||||
principles of cloud computing; with the -o output
|
||||
option being just one of its many features.</para>
|
||||
</section>
|
||||
<screen><computeroutput>2010-11-15-21_access.log - PUTS - 402
|
||||
2010-11-15-22_access.log - PUTS - 910</computeroutput></screen></step></procedure>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user