38 Commits

Author SHA1 Message Date
Clark Boylan
d2e3cc232f Handle journald timestamps in multiline logstash
Journald uses syslog not iso8601 timestamps. Handle both in the
mutliline rules for oslofmt so that we properly append multiline events
together. As is we treat the entire document as one large multiline
event.

Change-Id: I8eb865c9f31f26eada3318d06373bc71aacda047
2017-05-03 08:34:04 -07:00
Sean Dague
50ecf988ac Support journald files
Change-Id: If87a627e3f746f5bd2fb01138efb976279b47d32
2017-05-01 13:46:52 -04:00
Clark Boylan
85d319d9e9 Use ruby 1.8 compat erb for logstash config
The old logstash config was not ruby 1.8 compatible and we got funny
results out of it. This version should work with ruby 1.8 and beyond.

Change-Id: Ibe824dda7c96e5b333329ce25f65a14d3ebdef9c
2016-03-10 15:16:04 -08:00
Jenkins
4978adf39e Merge "Logstash 2.0 compat ES output rule" 2016-03-10 22:22:20 +00:00
Clark Boylan
1fa20ffef6 Logstash 2.0 compat ES output rule
Logstash 2.0 defaults to HTTP elasticsearch output which means that the
elasticsearch output (no _http) does HTTP and a new elasticsearch_java
output exists if you want to continue doing the native api output. We
had been doing HTTP so just need to update the output name. The host
parameter is also deprecated and you must pass an array to the hosts
parameter instead so update that as well.

Note that this switches from using a local ES daemon to talking to the
cluster itself directly because new logstash is able to load balance
over http. This reduces the overhead necessary to have resilient ES
connections.

Note this is not compatbile with Logstash 1.3.3 which is what we are
currently running so this change should only go in as part of an upgrade
to Logstash 2.0 and beyond.

Change-Id: I788ecb936f9fa5a006332ed626f90c33a255d9bf
2016-03-10 09:58:53 -08:00
K Jonathan Harker
0d576e8191 Add more logstash timestamp formats
Logstash is failing to index some events because it can't parse the
timestamp.

{:timestamp=>"2015-11-24T06:45:06.308000+0000", :message=>"Failed parsing date from field",
:field=>"logdate", :value=>"2015-11-24 06:41:40.274488",
:exception=>java.lang.IllegalArgumentException: Invalid format: "2015-11-24 06:41:40.274488",
:level=>:warn}

Add another string match for these more precise timestamps, and also add
ISO8601 as a fallback pattern.

Change-Id: Ic821bbc353be2883099ac84a2f33e363864dcd94
2015-12-14 13:52:51 -08:00
Sean Dague
686886c8d6 fix error multiline folding for indexing
The previous folding rules for TRACE log level had the effect that all
TRACES were folded into the ERROR that was emitted before them. This
was fine and appropriate. When we dropped TRACE tag for these and used
ERROR, this was extended to ERROR.

This is incorrect.

While ERROR tags are used in stack traces in a multiline way, they
also just emit some times. We don't want to *always* fold them into
the previous line. Doing so means that logstash treats all our ERROR
log messages as what came before. Typically INFO.

One additional indication of a stacktrace is the log message always
has a process id in it. We can minimally modify the grok rule to
require %{NUMBER}, which should do the right thing here.

This should bring back ERROR log lines in logstash being listed under
loglevel:ERROR.

Change-Id: I20372686212d080d4bc5c5578c418546005260fd
2015-11-09 11:32:37 -05:00
Matt Riedemann
23e70213f2 Handle 'ERROR' for multiline oslofmt
oslo.log commit 27f7fe52fefcb81c5fe01736841d7a20149caf02 changed the
"logging_exception_prefix" config option to use the ERROR rather than
TRACE level which breaks multi-line queries in logstash.o.o, which is
kind of a big deal for elastic-recheck to not have super generic
queries.

This makes the log level an OR for TRACE or ERROR for versions of
oslo.log used in stable branches and ERROR for master.

This should behave OK otherwise because of the %{NUMBER} in the pattern
where NUMBER corresponds to the pid in logging_exception_prefix:

  %(asctime)s.%(msecs)03d %(process)d ERROR %(name)s %(instance)s

Closes-Bug: #1474545

Change-Id: Iddd7f51b6a57deb6d56fc9f702992e7b14cf5cde
2015-07-14 14:03:46 -07:00
Jenkins
ed90813411 Merge "Move keystone logs over to oslofrmt" 2014-09-07 20:43:24 +00:00
pcrews
3ad20ce40a Altering indexer config to properly match on libvirtd files
The tags associated with libvirtd.txt in logstash are libvirtd.txt
and libvirt, the index config looked for libvirtd as the current
setup does not appear to match - we only have {%logmessage%} for message
content now. Setting search tag to 'libvirt'.

Change-Id: I27ae3b034581571285f42dab674ed85ff820cb6f
Partial-bug: 1350121
2014-08-01 16:13:49 -07:00
Joe Gordon
ecb27653d8 Move keystone logs over to oslofrmt
After the successful implementation of
https://blueprints.launchpad.net/keystone/+spec/unified-logging-in-keystone
keystone uses oslo logging instead of its own. Update logstash settings
to reflect this.

We currently don't have any keystone logs in logstash, this should fix
that.

Change-Id: I7acfa35bfd112a15d2ec3aa7338fb28b37a827cb
2014-08-01 15:28:09 +10:00
Clark Boylan
9f42006bd1 Use local ES balancers on logstash workers
Set up small non master non data elasticsearch daemons on logstash
workers to act as local load balancers for the elasticsearch http
protocol.

Change-Id: Ie3729f851ebef3331a6b69f718e57d663209bfc2
2014-07-07 17:01:57 -07:00
Clark Boylan
a339be3b8c Convert logstash ES output to HTTP.
The logstash elasticsearch output seems to degrade over time and slow
down. Restarting the logstash daemon temporarily corrects this problem.
Switch to the elasticsearch HTTP output to see if that corrects the
problem as well.

Note the logstash watchdog is disabled by this change as logstash
daemons using elasticsearch HTTP output will not join the elasticsearch
cluster which will force the watchdog to always trip. To avoid this
issue disable the watchdog.

Change-Id: I77044b26fa10fb1fc3690a0464d79d55bed2fe00
2014-07-07 13:14:01 -07:00
Sean Dague
ae96d3bf6c add horizon_error to the indexed logs
this adds horizon_error to the indexed log files, which is very
useful in determining how horizon fails, as we get very specific
stack traces.

Change-Id: Ifb323e327dbc2931100a4552d029d91209c4bbba
2014-03-17 12:37:39 -07:00
Clark Boylan
b2ef46c5c7 Better logstash field data.
We are currently using a lot of wildcard searches in elasticsearch which
are slow. Provide better field data so that we can replace those
wildcard searches with filters. In particular add a short uuid field and
make the filename tag field the basename of the filepath so that grenade
and non grenade files all end up with the same tags.

Change-Id: If558017fceae96bcf197e611ab5cac1cfe7ae9bf
2014-03-13 14:42:58 -07:00
Clark Boylan
5a8351635d Upgrade to logstash 1.3.3 and ElasticSearch 0.90.9
Upgrade to logstash 1.3.3 and ElasticSearch 0.90.9. Logstash 1.3.3 comes
with many performance improvements and bug fixes and needs ElasticSearch
0.90.9 to use the ElasticSearch output so upgrade ES as well. These
upgrades come with config changes for logstash as well to use new
features and disable others that are not wanted.

Change-Id: Ia180c5edfa755214a801f2cad06f091125814b0e
2014-02-11 17:07:34 -08:00
Clark Boylan
1ce035b2d2 Remove deprecated logstash max_inflight_requests
The logstash ElasticSearch output max_inflight_requests option has been
deprecated and should not be used. Remove it.

Change-Id: I3644523f8605f17be7d9ed6f20f2020290f0fb28
2014-02-11 15:28:46 -08:00
Clark Boylan
5824bc7e24 Parse timestamps in libvirt logs properly.
The libvirt logs use a slightly different timestamp than the other log
files (iso8601 including the timezone offset) which makes logstash
unhappy when it receives these timestamps. Parse them properly to keep
logstash logs under control and hopefully happy.

Change-Id: I18b7f263d083d9babbdf1404298cb7def0b8501a
2014-02-11 11:31:19 -08:00
Joe Gordon
bdd4fc245c Add logstash rules to parse libvirtd.txt
Change-Id: I09185dae08d46821f804e3ad43205205aab9345c
2014-01-31 15:27:54 -08:00
Clark Boylan
1f07114346 Upgrade Logstash to 1.2.1.
* modules/logstash/manifests/init.pp: Download and install Logstash
1.2.1.

* modules/openstack_project/files/logstash/log-gearman-client.py:
Logstash 1.2.1 comes with a new schema. Update the job data sent to log
push workers to better accomodate the new schema.

* modules/openstack_project/files/logstash/log-gearman-worker.py: Push
Logstash 1.2.1 schema compliant JSON to the Logstash TCP input.

* modules/openstack_project/templates/logstash/indexer.conf.erb:
Logstash 1.2.1 comes with a new schema and many input and filter
changes. Use the newly supported features like conditionals to keep the
config up to date.

* modules/kibana/templates/config.rb.erb: Change the default field for
kibana to 'message'. It was @message which is deprecated in the new
logstash schema.

Change-Id: Id19fc05bcce8d42c5c0cf33df3da7e95f5794107
2013-10-21 10:17:48 -07:00
Clark Boylan
219cef8d06 Don't index logs with DEBUG log level.
ElasticSearch has a hard time performing queries on large amounts of
data. It must load the fields it is searching on into memory and that
can cause the node to use all of the memory allocated to it which in
turn causes a lot of garbage collection essentially taking the node
offline. Filter out DEBUG log messages in the devstack logs to reduce
the amount of data that must be loaded into memory when performing
searches.

Change-Id: Icfe9c8c17ccef4f9379c774eef791f43463dcf6d
2013-08-06 11:52:26 -07:00
Clark Boylan
33367c88de Add support for elasticsearch cluster.
We need to expand our elasticsearch install base. Update puppet to make
this possible.

Change-Id: Id0dae839b12ebf47715cf40a363832e0f661a94f
Reviewed-on: https://review.openstack.org/33910
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-07-01 17:48:16 +00:00
Clark Boylan
ffcf19a546 Handle mutliline oslofmt traceback in logstash.
* modules/openstack_project/templates/logstash/indexer.conf.erb: Add a
mutliline filter that matches oslo format TRACE level log lines. This
will collect all lines related to the traceback in a single event.

Change-Id: I201e407b6bd6fbb5b2db94fa6757f184709f2eaa
Reviewed-on: https://review.openstack.org/32962
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-06-18 19:03:57 +00:00
Clark Boylan
6405ae32f9 Add swift logs to Logstash.
* modules/openstack_project/files/logstash/logstash-worker1/jenkins-log-pusher.yaml:
Add the swift log files to the list of files to be processed by
logstash-worker1. This does not include the swift proxy log file as it
will require extra parsing.

* modules/openstack_project/templates/logstash/indexer.conf.erb: Add
Logstash filters for apache combined log format files.

Change-Id: I7545ace8f7601bdca453e0d2ac1b2233823878ce
Reviewed-on: https://review.openstack.org/31103
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-30 20:44:48 +00:00
Clark Boylan
bd685327e5 Add syslog logs to Logstash.
* modules/openstack_project/files/logstash/logstash-worker1/jenkins-log-pusher.yaml:
Add the syslog log file to the list of files to be processed by
logstash-worker1.

* modules/openstack_project/templates/logstash/indexer.conf.erb: Add
Logstash filters for syslog format files.

Change-Id: I0f8f58ab484949eb0506842bdb98385767a50333
Reviewed-on: https://review.openstack.org/31097
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-05-30 20:44:42 +00:00
Clark Boylan
f8e624d1c2 Add keystone logs to Logstash.
* modules/openstack_project/files/logstash/logstash-worker1/jenkins-log-pusher.yaml:
Add the keystone log file to the list of files to be processed by
logstash-worker1.

* modules/openstack_project/templates/logstash/indexer.conf.erb: Add
Logstash filters for keystone format files.

Change-Id: I5a72fc17ed1f37b816581faabe44f26f8cc36db2
Reviewed-on: https://review.openstack.org/31096
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-05-30 20:44:10 +00:00
Clark Boylan
c9d484e797 Handle multiline Logstash events per file.
* modules/openstack_project/files/logstash/log-pusher.py: Add a filename
field to logstash events that can be used to associate multiline events
in files to their appropriate parents in the same file.

* modules/openstack_project/templates/logstash/indexer.conf.erb: Add
stream_identities to the mutliline filters that use the source host and
file name to determine relationships between mutliline events.

Change-Id: Ia325c0e1257131ab1b721c4df8f70f6bea1d0b99
Reviewed-on: https://review.openstack.org/30953
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-05-30 20:44:08 +00:00
Clark Boylan
963328f7ce Set ElasticSearch output node name to hostname.
* modules/openstack_project/templates/logstash/indexer.conf.erb: Set the
node name parameter in the ElasticSearch output template to be the
hostname of the host running logstash.

Change-Id: I8ade373403ca59d6d5768df175405167c9c8b4be
Reviewed-on: https://review.openstack.org/30925
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Elizabeth Krumbach Joseph <lyz@princessleia.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-30 17:17:24 +00:00
Clark Boylan
d637726079 Increase Logstash ES output max_inflight_requests.
The default value for the Logstash ElasticSearch max_inflight_requests
parameters is 50. Bump that up to 512, to have a bigger firehose to
ElasticSearch. ElasticSearch is the piece of the Logstash puzzle that
doesn't seem to have trouble with bigger firehoses.

Change-Id: I3eeaa036c68b00a8daa0896a4256b30043f04aaa
Reviewed-on: https://review.openstack.org/30769
Reviewed-by: Khai Do <zaro0508@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-29 00:33:30 +00:00
Clark Boylan
5dd697f3c4 Fix elasticsearch connectivity on logstash.o.o.
The new elasticsearch server needs to be able to connect back to
logstash over the elasticsearch ports. Also make kibana talk to the new
elasticsearch server.

Change-Id: I88af40c7885b263fcef5ff16210f073cd0a696a5
Reviewed-on: https://review.openstack.org/30354
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-23 23:46:30 +00:00
Clark Boylan
538793a051 Add cinder and quantum screen logs to logstash.
* modules/openstack_project/files/logstash/jenkins-log-pusher.yaml:
Add the cinder and quantum screen log files to the log pusher. Tag them
with screen and oslofmt (screen because they are screen logs and oslofmt
because they use the oslo log message format).

* modules/openstack_project/templates/logstash/indexer.conf.erb:
Add a grep filter to remove the screen log header lines. Add a multiline
filter to handle oslo log format multi line events. Add a grok filter to
parse the oslo format logs. Handle timestamps without millisecond
precision. Remove event_message field if that message was properly
parsed.

Change-Id: Icd18e252a512416e0cce5ee0e27942b072a25e09
Reviewed-on: https://review.openstack.org/29985
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-23 22:10:42 +00:00
Clark Boylan
e43e208a99 Switch to dedicated elasticsearch node.
Switch to a large dedicated elasticsearch node as sharing resources
between logstash, kibana, jenkins-log-pusher, and elasticsearch results
in a constrained environment.

Change-Id: I39e6210f2c577429be2cb38aca09111a0f56f9be
Reviewed-on: https://review.openstack.org/30344
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-23 21:37:01 +00:00
Clark Boylan
b137954246 Make logstash GREEDYDATA accept newlines.
* modules/openstack_project/templates/logstash/indexer.conf.erb: The
mutliline filter can add newlines to the @message field of events. Make
the grok filter consume the newlines.

Change-Id: I8353a482baf83ec1e728e4852c59c37b5e078e8a
Reviewed-on: https://review.openstack.org/29984
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-22 22:33:58 +00:00
Clark Boylan
104c5ef5a8 Use log-pusher injected tags during logstash grok
* modules/openstack_project/templates/logstash/indexer.conf.erb: The
log-pusher.py script tags events with the filename of the log generating
the event. Use these values instead of different types to differentiate
Jenkins console logs from nova logs and so on.

Note that filters must match all of the values in the tags array (they
are ANDed together). This may mean that as the logstash filter rulesets
grow we will need a set of initial mutates to convert
'logs/screen-n-api.txt' and 'logs/screen-n-cpu.txt' tags to 'nova' to
allow for common grok and parsing filters.

Change-Id: I2769bc05a2e9cc7e8dbc46849e052146b9fee75e
Reviewed-on: https://review.openstack.org/28119
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-06 19:32:10 +00:00
Clark Boylan
aae5ffc65e Make the Jenkins log pusher a service.
* modules/openstack_project/manifests/logstash.pp: Run the Jenkins log
pusher script as a service. This is the first step in making Logstash
use the TCP inputs instead of pipe inputs.

* modules/openstack_project/files/logstash/jenkins-log-pusher.init: Add
a simple init script for the Jenkins log pusher.

* modules/openstack_project/templates/logstash/indexer.conf.erb: Switch
to TCP input instead of pipe input as the new Jenkins log pusher service
will push log events over TCP.

Change-Id: Id80c710abd5facd71d18afb2b250b2d7d92dec2d
Reviewed-on: https://review.openstack.org/28074
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-06 19:32:05 +00:00
Clark Boylan
9863c147dc Turn on log-pusher.py debug logging.
* modules/openstack_project/templates/logstash/indexer.conf.erb: Enable
log-pusher.py debug logging to help determine why logstash is restarting
the log-pusher and to see why the logs stop flowing at times.

Change-Id: I250a6400c2fd29201865e9f8c5761dc2f4fbf5c3
Reviewed-on: https://review.openstack.org/27871
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-01 00:28:05 +00:00
Clark Boylan
7086b8f459 Run single node logstash without redis.
logstash.o.o is running into memory contention. The three logstash
processes (agent, indexer, web), redis, and elasticsearch are all memory
hogs. Remove redis which allows us to consolidate the work of the agent
and indexer into a single process.

If we decide to add remote logstash agents we will want to add redis to
the mix again. This is probably best done on a second host dedicated to
redis.

Change-Id: I5ba96b4a27d91cb73c9380ed47d0495b9005effa
Reviewed-on: https://review.openstack.org/27452
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-25 17:44:22 +00:00
Clark Boylan
457a9d8764 Add skeleton logstash module.
This new logstash module adds classes to install logstash agents and
indexers as well as redis and elasticsearch. The configuration for each
of these services is rudimentary but it shouldn't be difficult to expand
the configs and make them useful.

Also, add a logstash.openstack.org node that will have an agent,
indexer, web frontend, redis, and elasticsearch installed on it.

Change-Id: I25b635f088f99d45cfaa70ed122c6433d3784937
Reviewed-on: https://review.openstack.org/19871
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-01-29 20:44:09 +00:00