x/poppy
Code Issues Proposed changes

feat: add log delivery pig script

Browse Source 
- The hadoop script will allow split up the provider's logs that are
  piped into it, based on those domains that have log delivery enabled.

- README.rst contains instructions on how the script is meant to be
  used.

Implements: blueprint log-delivery

Change-Id: I4434175bead26e9b78a3115038af55b25a62163c
This commit is contained in:
Sriram Madapusi Vasudevan 2015-06-11 14:42:35 -04:00
parent 68b8d96a9c
commit 92577d2272
2 changed files with 33 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
hadoop/README.rst Normal file
View File

@ -0,0 +1,22 @@
Log Delivery
============
The pig script needs to be run in a hadoop cluster, after piping all the required logs from a provider with whom services are set up with.
NOTE:
* All the domains that need to have logs delivered need to copied into the Hadoop Cluster, under the name `domains_log.tsv`
* The corresponding Provider URL needs to be also set
How to run a Pig Script
=======================
$ pig -p INPUT=~/log_source -p OUTPUT=~/logs_output -p PROVIDER_URL_EXT=mycdn
Output
======
There should be directories created under OUTPUT, with each directory corresponding to a domain that had log delivered enabled, and log files underneath each of those directories pertaining to that domain.
$ logs_output/mydomain/mydomain-0000.gz
$ logs_output/yourdomain/yourdomain-0000.gz

11
hadoop/log_delivery.pig Normal file
View File

@ -0,0 +1,11 @@
REGISTER /usr/lib/pig/piggybank.jar;
logs = LOAD '$INPUT/*.gz' USING PigStorage('\t') AS (date, time, ip, method, uri, status, bytes:long, time_taken, referer, user_agent, cookie, country);
log_domains = LOAD '$INPUT/domains_log.tsv' USING PigStorage('\n') AS domains;
formatted_logs = FOREACH logs GENERATE ip, '-', '-', org.apache.pig.builtin.StringConcat('[',date,':',time, ' +0000',']') , org.apache.pig.builtin.StringConcat('"', method,' ', uri,' ','HTTP/1.1', '"'), status, bytes, referer, user_agent, REGEX_EXTRACT(uri, '/([^/]*).$PROVIDER_URL_EXT(/.*)', 1) AS domain;
delivery_enabled_formamatted_logs = JOIN log_domains BY domains, formatted_logs BY domain;
STORE delivery_enabled_formamatted_logs INTO '$OUTPUT' USING org.apache.pig.piggybank.storage.MultiStorage('$OUTPUT', 10, 'gz', '\\t');