openstack/swift
Code Issues Proposed changes

Make log format for requests configurable

Browse Source 
Add the log_msg_template option in proxy-server.conf and log_format in
a/c/o-server.conf. It is a string parsable by Python's format()
function. Some fields containing user data might be anonymized by using
log_anonymization_method and log_anonymization_salt.

Change-Id: I29e30ef45fe3f8a026e7897127ffae08a6a80cd9
This commit is contained in:
Gilles Biannic
2018-03-01 11:31:12 +01:00
committed by Romain LE DISEZ
parent 89eced960c
commit a4cc353375
19 changed files with 582 additions and 125 deletions
Download Patch File Download Diff File Expand all files Collapse all files

46
doc/manpages/proxy-server.conf.5
View File

@@ -730,6 +730,52 @@ Note: reveal_sensitive_prefix will not affect the value logged with access_log_h
What HTTP methods are allowed for StatsD logging (comma-sep); request methods
not in this list will have "BAD_METHOD" for the <verb> portion of the metric.
Default is "GET,HEAD,POST,PUT,DELETE,COPY,OPTIONS".
.IP \fBlog_anonymization_method\fR
Hashing algorithm for anonymization. Must be one of algorithms supported by Python's hashlib. Default is MD5.
.IP \fBlog_anonymization_salt\fR
Salt added as prefix before hashing the value to anonymize. Default is empty (no salt).
.IP "\fBlog_msg_template\fR"
Template used to format access logs. All words surrounded by curly brackets will be substituted with the appropriate values.
.RE
.PD 0
.RS 10
.IP "Some keywords map to timestamps and can be converted to standard dates formats using the matching transformers: 'datetime', 'asctime' or 'iso8601'."
.IP "Other transformers for timestamps are 's', 'ms', 'us' and 'ns' for seconds, milliseconds, microseconds and nanoseconds."
.IP "Python's strftime directives can also be used as tranformers (a, A, b, B, c, d, H, I, j, m, M, p, S, U, w, W, x, X, y, Y, Z)."
.IP "Some keywords map to user data that could be anonymized by using the transformer 'anonymized'."
.IP "Keywords availables are:"
.PD 0
.RS 7
.IP "client_ip (anonymizable)"
.IP "remote_addr (anonymizable)"
.IP "method (request method)"
.IP "path (anonymizable)"
.IP "protocol"
.IP "status_int"
.IP "referer (anonymizable)"
.IP "user_agent (anonymizable)"
.IP "auth_token"
.IP "bytes_recvd (number of bytes received)"
.IP "bytes_sent (number of bytes sent)"
.IP "client_etag (anonymizable)"
.IP "transaction_id"
.IP "headers (anonymizable)"
.IP "request_time (difference between start and end timestamps)
.IP "source"
.IP "log_info"
.IP "start_time (timestamp at the receiving, timestamp)"
.IP "end_time (timestamp at the end of the treatment, timestamp)"
.IP "policy_index"
.IP "account (account name, anonymizable)"
.IP "container (container name, anonymizable)"
.IP "object (object name, anonymizable)"
.PD
.RE
.IP "Example: '{client_ip.anonymized} {remote_addr.anonymized} {start_time.iso8601} {end_time.H}:{end_time.M} {method} acc:{account} cnt:{container} obj:{object.anonymized}'"
.IP "Default: '{client_ip} {remote_addr} {end_time.datetime} {method} {path} {protocol} {status_int} {referer} {user_agent} {auth_token} {bytes_recvd} {bytes_sent} {client_etag} {transaction_id} {headers} {request_time} {source} {log_info} {start_time} {end_time} {policy_index}'"
.IP "Warning: A bad log message template will raise an error in initialization."
.RE
.PD

58
doc/source/logs.rst
View File

@@ -26,47 +26,71 @@ Proxy Logs
The proxy logs contain the record of all external API requests made to the
proxy server. Swift's proxy servers log requests using a custom format
designed to provide robust information and simple processing. The log format
is::
designed to provide robust information and simple processing. It is possible
to change this format with the ``log_msg_template`` config parameter.
The default log format is::
client_ip remote_addr datetime request_method request_path protocol
status_int referer user_agent auth_token bytes_recvd bytes_sent
client_etag transaction_id headers request_time source log_info
request_start_time request_end_time policy_index
{client_ip} {remote_addr} {end_time.datetime} {method} {path} {protocol}
{status_int} {referer} {user_agent} {auth_token} {bytes_recvd}
{bytes_sent} {client_etag} {transaction_id} {headers} {request_time}
{source} {log_info} {start_time} {end_time} {policy_index}
Some keywords, signaled by the (anonymizable) flag, can be anonymized by
using the transformer 'anonymized'. The data are applied the hashing method of
`log_anonymization_method` and an optional salt `log_anonymization_salt`.
Some keywords, signaled by the (timestamp) flag, can be converted to standard
dates formats using the matching transformers: 'datetime', 'asctime' or
'iso8601'. Other transformers for timestamps are 's', 'ms', 'us' and 'ns' for
seconds, milliseconds, microseconds and nanoseconds. Python's strftime
directives can also be used as tranformers (a, A, b, B, c, d, H, I, j, m, M, p,
S, U, w, W, x, X, y, Y, Z).
Example {client_ip.anonymized} {remote_addr.anonymized} {start_time.iso8601}
{end_time.H}:{end_time.M} {method} acc:{account} cnt:{container}
obj:{object.anonymized}
=================== ==========================================================
**Log Field** **Value**
------------------- ----------------------------------------------------------
client_ip Swift's guess at the end-client IP, taken from various
headers in the request.
headers in the request. (anonymizable)
remote_addr The IP address of the other end of the TCP connection.
datetime Timestamp of the request, in
day/month/year/hour/minute/second format.
request_method The HTTP verb in the request.
request_path The path portion of the request.
(anonymizable)
end_time Timestamp of the request. (timestamp)
method The HTTP verb in the request.
path The path portion of the request. (anonymizable)
protocol The transport protocol used (currently one of http or
https).
status_int The response code for the request.
referer The value of the HTTP Referer header.
user_agent The value of the HTTP User-Agent header.
referer The value of the HTTP Referer header. (anonymizable)
user_agent The value of the HTTP User-Agent header. (anonymizable)
auth_token The value of the auth token. This may be truncated or
otherwise obscured.
bytes_recvd The number of bytes read from the client for this request.
bytes_sent The number of bytes sent to the client in the body of the
response. This is how many bytes were yielded to the WSGI
server.
client_etag The etag header value given by the client.
client_etag The etag header value given by the client. (anonymizable)
transaction_id The transaction id of the request.
headers The headers given in the request.
headers The headers given in the request. (anonymizable)
request_time The duration of the request.
source The "source" of the request. This may be set for requests
that are generated in order to fulfill client requests,
e.g. bulk uploads.
log_info Various info that may be useful for diagnostics, e.g. the
value of any x-delete-at header.
request_start_time High-resolution timestamp from the start of the request.
request_end_time High-resolution timestamp from the end of the request.
start_time High-resolution timestamp from the start of the request.
(timestamp)
end_time High-resolution timestamp from the end of the request.
(timestamp)
policy_index The value of the storage policy index.
account The account part extracted from the path of the request.
(anonymizable)
container The container part extracted from the path of the request.
(anonymizable)
object The object part extracted from the path of the request.
(anonymizable)
=================== ==========================================================
In one log line, all of the above fields are space-separated and url-encoded.