We are going to cleanup firehose.openstack.org as it never really ended
up being used for significant things and we would need to rewrite the
puppet into ansible at this point. Before we cleanup the server ensure
that things are not talking to it.
The only thing I can find that externally talks to it is the subunit
workers. germqtt and lpmqtt run on firehose so will be cleaned out when
firehose goes away.
Change-Id: I5b657aad1a276a18e58d09f5b2108940d0bd8ac2
This created confusion when updating configs to handle journald. Remove
the unused files and update docs to point at the proper config location.
Change-Id: Ifd8d8868b124b72a86cf7b5acb30480e72b903ed
By default the elasticsearch output plugin for logstash has no timeout.
I think this is the cause of logstash daemons hanging when we have
elasticsearch node outages. Goal here is to have the timeout force the
request to be retried again (ideally against a different server entirely
but even if it is simply retried to the same server we should operate
properly again once the server that went down is back up).
Change-Id: I0511f32f8ee3f0c40987cab3e2528a85595785b2
This reverts commit a000d47ed1.
We have updated the crm processing to collapse indentical log files with
different paths or suffixes into a single file name for processing. This
should dramatically reduce disk consumption for crm114 learning making
it safe to use again.
Change-Id: I50aa5fa434de51c689e5c2e51070db0ee1a76bae
The logstash worker nodes are running out of disk because crm114 has
enough unique files to process that at 13MB per file we are still
filling up many gigabytes of data and filling out disks.
We will need to work on collapsing files (perhaps not use their full
path and just use their basename?) and work to eliminate log files that
are unique like those that have the hostname in them (as this means we
create a new file for every job run).
Change-Id: Ie729a3d2628f0ded65574c267e31c14f6b795b29
The mqtt username and password variables in the logstash work config
template were incorrect and didn't match what was being set in
logstash_worker.pp. This commit fixes it so everything matches.
Change-Id: I959fbd0cf1d7535ff75b42c228e5f74ada8fe5d3
This commit enables the mqtt notifications in the gearman worker. Most
of the setup was already there from our previous failed attempts to
leverage the logstash plugin directly. This just pivots that to use the
gearman worker instead.
Change-Id: I6becec12604c61fe50d3e6b9c7ed9f9e9be311ae
Depends-On: I0a17444cce18dd4b63f1f924e393483f6d8fe8eb
Depends-On: I43be3562780c61591ebede61f3a8929e8217f199
Journald uses syslog not iso8601 timestamps. Handle both in the
mutliline rules for oslofmt so that we properly append multiline events
together. As is we treat the entire document as one large multiline
event.
Change-Id: I8eb865c9f31f26eada3318d06373bc71aacda047
This patch fixes a mistake on my part, previously the location of
the ca certs on archlinux, but on ubuntu this isn't where they
live. So the mqtt libs can't properly encrypt the traffic and
things are getting stuck in a loop and not working. This commit fixes
this by writing the cert from instantssl/comodo to disk and then
pointing the worker at that. This way it should always work.
Change-Id: I21b1a64b457545115ff862e3c3388c5892c5497b
This commit enables the mqtt support in the subunit gearman worker so
the workers will report over mqtt when they process a subunit file.
Change-Id: Ifff2f57740f160e328c0548254e16b04e6ab6c4e
Depends-On: Ibd13b737eccf52863a69d20843cb7d50242f7bb9
This commit will enable the mqtt output plugin on logstash workers. So
now we'll be emitting logstash events to the firehose for anyone to
listen to.
Change-Id: I570a461ee13f5dfa5494554df1cc321fc6cdbf6c
Depends-On: I7255f6c256ab3e3ca33caf69b71bf4ffab02c1bb
This uses new logstash conf.d features to properly link in the
configuration from the logstash-filters repo. This should make
configuring logstash far more flexible and reconsumable.
Change-Id: Ia304eb7e73c82ca5ce85967fbf442bb4bc5f8b7a
Depends-On: Icbca7a6ba0c5a94a273ef158f707311b588483fd
The old logstash config was not ruby 1.8 compatible and we got funny
results out of it. This version should work with ruby 1.8 and beyond.
Change-Id: Ibe824dda7c96e5b333329ce25f65a14d3ebdef9c
Logstash 2.0 defaults to HTTP elasticsearch output which means that the
elasticsearch output (no _http) does HTTP and a new elasticsearch_java
output exists if you want to continue doing the native api output. We
had been doing HTTP so just need to update the output name. The host
parameter is also deprecated and you must pass an array to the hosts
parameter instead so update that as well.
Note that this switches from using a local ES daemon to talking to the
cluster itself directly because new logstash is able to load balance
over http. This reduces the overhead necessary to have resilient ES
connections.
Note this is not compatbile with Logstash 1.3.3 which is what we are
currently running so this change should only go in as part of an upgrade
to Logstash 2.0 and beyond.
Change-Id: I788ecb936f9fa5a006332ed626f90c33a255d9bf
Logstash is failing to index some events because it can't parse the
timestamp.
{:timestamp=>"2015-11-24T06:45:06.308000+0000", :message=>"Failed parsing date from field",
:field=>"logdate", :value=>"2015-11-24 06:41:40.274488",
:exception=>java.lang.IllegalArgumentException: Invalid format: "2015-11-24 06:41:40.274488",
:level=>:warn}
Add another string match for these more precise timestamps, and also add
ISO8601 as a fallback pattern.
Change-Id: Ic821bbc353be2883099ac84a2f33e363864dcd94
The previous folding rules for TRACE log level had the effect that all
TRACES were folded into the ERROR that was emitted before them. This
was fine and appropriate. When we dropped TRACE tag for these and used
ERROR, this was extended to ERROR.
This is incorrect.
While ERROR tags are used in stack traces in a multiline way, they
also just emit some times. We don't want to *always* fold them into
the previous line. Doing so means that logstash treats all our ERROR
log messages as what came before. Typically INFO.
One additional indication of a stacktrace is the log message always
has a process id in it. We can minimally modify the grok rule to
require %{NUMBER}, which should do the right thing here.
This should bring back ERROR log lines in logstash being listed under
loglevel:ERROR.
Change-Id: I20372686212d080d4bc5c5578c418546005260fd
oslo.log commit 27f7fe52fefcb81c5fe01736841d7a20149caf02 changed the
"logging_exception_prefix" config option to use the ERROR rather than
TRACE level which breaks multi-line queries in logstash.o.o, which is
kind of a big deal for elastic-recheck to not have super generic
queries.
This makes the log level an OR for TRACE or ERROR for versions of
oslo.log used in stable branches and ERROR for master.
This should behave OK otherwise because of the %{NUMBER} in the pattern
where NUMBER corresponds to the pid in logging_exception_prefix:
%(asctime)s.%(msecs)03d %(process)d ERROR %(name)s %(instance)s
Closes-Bug: #1474545
Change-Id: Iddd7f51b6a57deb6d56fc9f702992e7b14cf5cde
The tags associated with libvirtd.txt in logstash are libvirtd.txt
and libvirt, the index config looked for libvirtd as the current
setup does not appear to match - we only have {%logmessage%} for message
content now. Setting search tag to 'libvirt'.
Change-Id: I27ae3b034581571285f42dab674ed85ff820cb6f
Partial-bug: 1350121
After the successful implementation of
https://blueprints.launchpad.net/keystone/+spec/unified-logging-in-keystone
keystone uses oslo logging instead of its own. Update logstash settings
to reflect this.
We currently don't have any keystone logs in logstash, this should fix
that.
Change-Id: I7acfa35bfd112a15d2ec3aa7338fb28b37a827cb
Set up small non master non data elasticsearch daemons on logstash
workers to act as local load balancers for the elasticsearch http
protocol.
Change-Id: Ie3729f851ebef3331a6b69f718e57d663209bfc2
The logstash elasticsearch output seems to degrade over time and slow
down. Restarting the logstash daemon temporarily corrects this problem.
Switch to the elasticsearch HTTP output to see if that corrects the
problem as well.
Note the logstash watchdog is disabled by this change as logstash
daemons using elasticsearch HTTP output will not join the elasticsearch
cluster which will force the watchdog to always trip. To avoid this
issue disable the watchdog.
Change-Id: I77044b26fa10fb1fc3690a0464d79d55bed2fe00
this adds horizon_error to the indexed log files, which is very
useful in determining how horizon fails, as we get very specific
stack traces.
Change-Id: Ifb323e327dbc2931100a4552d029d91209c4bbba
We are currently using a lot of wildcard searches in elasticsearch which
are slow. Provide better field data so that we can replace those
wildcard searches with filters. In particular add a short uuid field and
make the filename tag field the basename of the filepath so that grenade
and non grenade files all end up with the same tags.
Change-Id: If558017fceae96bcf197e611ab5cac1cfe7ae9bf
Upgrade to logstash 1.3.3 and ElasticSearch 0.90.9. Logstash 1.3.3 comes
with many performance improvements and bug fixes and needs ElasticSearch
0.90.9 to use the ElasticSearch output so upgrade ES as well. These
upgrades come with config changes for logstash as well to use new
features and disable others that are not wanted.
Change-Id: Ia180c5edfa755214a801f2cad06f091125814b0e
The logstash ElasticSearch output max_inflight_requests option has been
deprecated and should not be used. Remove it.
Change-Id: I3644523f8605f17be7d9ed6f20f2020290f0fb28
The libvirt logs use a slightly different timestamp than the other log
files (iso8601 including the timezone offset) which makes logstash
unhappy when it receives these timestamps. Parse them properly to keep
logstash logs under control and hopefully happy.
Change-Id: I18b7f263d083d9babbdf1404298cb7def0b8501a
* modules/logstash/manifests/init.pp: Download and install Logstash
1.2.1.
* modules/openstack_project/files/logstash/log-gearman-client.py:
Logstash 1.2.1 comes with a new schema. Update the job data sent to log
push workers to better accomodate the new schema.
* modules/openstack_project/files/logstash/log-gearman-worker.py: Push
Logstash 1.2.1 schema compliant JSON to the Logstash TCP input.
* modules/openstack_project/templates/logstash/indexer.conf.erb:
Logstash 1.2.1 comes with a new schema and many input and filter
changes. Use the newly supported features like conditionals to keep the
config up to date.
* modules/kibana/templates/config.rb.erb: Change the default field for
kibana to 'message'. It was @message which is deprecated in the new
logstash schema.
Change-Id: Id19fc05bcce8d42c5c0cf33df3da7e95f5794107
ElasticSearch has a hard time performing queries on large amounts of
data. It must load the fields it is searching on into memory and that
can cause the node to use all of the memory allocated to it which in
turn causes a lot of garbage collection essentially taking the node
offline. Filter out DEBUG log messages in the devstack logs to reduce
the amount of data that must be loaded into memory when performing
searches.
Change-Id: Icfe9c8c17ccef4f9379c774eef791f43463dcf6d
We need to expand our elasticsearch install base. Update puppet to make
this possible.
Change-Id: Id0dae839b12ebf47715cf40a363832e0f661a94f
Reviewed-on: https://review.openstack.org/33910
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
* modules/openstack_project/templates/logstash/indexer.conf.erb: Add a
mutliline filter that matches oslo format TRACE level log lines. This
will collect all lines related to the traceback in a single event.
Change-Id: I201e407b6bd6fbb5b2db94fa6757f184709f2eaa
Reviewed-on: https://review.openstack.org/32962
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
* modules/openstack_project/files/logstash/logstash-worker1/jenkins-log-pusher.yaml:
Add the swift log files to the list of files to be processed by
logstash-worker1. This does not include the swift proxy log file as it
will require extra parsing.
* modules/openstack_project/templates/logstash/indexer.conf.erb: Add
Logstash filters for apache combined log format files.
Change-Id: I7545ace8f7601bdca453e0d2ac1b2233823878ce
Reviewed-on: https://review.openstack.org/31103
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
* modules/openstack_project/files/logstash/logstash-worker1/jenkins-log-pusher.yaml:
Add the syslog log file to the list of files to be processed by
logstash-worker1.
* modules/openstack_project/templates/logstash/indexer.conf.erb: Add
Logstash filters for syslog format files.
Change-Id: I0f8f58ab484949eb0506842bdb98385767a50333
Reviewed-on: https://review.openstack.org/31097
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
* modules/openstack_project/files/logstash/logstash-worker1/jenkins-log-pusher.yaml:
Add the keystone log file to the list of files to be processed by
logstash-worker1.
* modules/openstack_project/templates/logstash/indexer.conf.erb: Add
Logstash filters for keystone format files.
Change-Id: I5a72fc17ed1f37b816581faabe44f26f8cc36db2
Reviewed-on: https://review.openstack.org/31096
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
* modules/openstack_project/files/logstash/log-pusher.py: Add a filename
field to logstash events that can be used to associate multiline events
in files to their appropriate parents in the same file.
* modules/openstack_project/templates/logstash/indexer.conf.erb: Add
stream_identities to the mutliline filters that use the source host and
file name to determine relationships between mutliline events.
Change-Id: Ia325c0e1257131ab1b721c4df8f70f6bea1d0b99
Reviewed-on: https://review.openstack.org/30953
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
* modules/openstack_project/templates/logstash/indexer.conf.erb: Set the
node name parameter in the ElasticSearch output template to be the
hostname of the host running logstash.
Change-Id: I8ade373403ca59d6d5768df175405167c9c8b4be
Reviewed-on: https://review.openstack.org/30925
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Elizabeth Krumbach Joseph <lyz@princessleia.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
The default value for the Logstash ElasticSearch max_inflight_requests
parameters is 50. Bump that up to 512, to have a bigger firehose to
ElasticSearch. ElasticSearch is the piece of the Logstash puzzle that
doesn't seem to have trouble with bigger firehoses.
Change-Id: I3eeaa036c68b00a8daa0896a4256b30043f04aaa
Reviewed-on: https://review.openstack.org/30769
Reviewed-by: Khai Do <zaro0508@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
The new elasticsearch server needs to be able to connect back to
logstash over the elasticsearch ports. Also make kibana talk to the new
elasticsearch server.
Change-Id: I88af40c7885b263fcef5ff16210f073cd0a696a5
Reviewed-on: https://review.openstack.org/30354
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
* modules/openstack_project/files/logstash/jenkins-log-pusher.yaml:
Add the cinder and quantum screen log files to the log pusher. Tag them
with screen and oslofmt (screen because they are screen logs and oslofmt
because they use the oslo log message format).
* modules/openstack_project/templates/logstash/indexer.conf.erb:
Add a grep filter to remove the screen log header lines. Add a multiline
filter to handle oslo log format multi line events. Add a grok filter to
parse the oslo format logs. Handle timestamps without millisecond
precision. Remove event_message field if that message was properly
parsed.
Change-Id: Icd18e252a512416e0cce5ee0e27942b072a25e09
Reviewed-on: https://review.openstack.org/29985
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Switch to a large dedicated elasticsearch node as sharing resources
between logstash, kibana, jenkins-log-pusher, and elasticsearch results
in a constrained environment.
Change-Id: I39e6210f2c577429be2cb38aca09111a0f56f9be
Reviewed-on: https://review.openstack.org/30344
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
* modules/openstack_project/templates/logstash/indexer.conf.erb: The
mutliline filter can add newlines to the @message field of events. Make
the grok filter consume the newlines.
Change-Id: I8353a482baf83ec1e728e4852c59c37b5e078e8a
Reviewed-on: https://review.openstack.org/29984
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
* modules/openstack_project/templates/logstash/indexer.conf.erb: The
log-pusher.py script tags events with the filename of the log generating
the event. Use these values instead of different types to differentiate
Jenkins console logs from nova logs and so on.
Note that filters must match all of the values in the tags array (they
are ANDed together). This may mean that as the logstash filter rulesets
grow we will need a set of initial mutates to convert
'logs/screen-n-api.txt' and 'logs/screen-n-cpu.txt' tags to 'nova' to
allow for common grok and parsing filters.
Change-Id: I2769bc05a2e9cc7e8dbc46849e052146b9fee75e
Reviewed-on: https://review.openstack.org/28119
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
* modules/openstack_project/manifests/logstash.pp: Run the Jenkins log
pusher script as a service. This is the first step in making Logstash
use the TCP inputs instead of pipe inputs.
* modules/openstack_project/files/logstash/jenkins-log-pusher.init: Add
a simple init script for the Jenkins log pusher.
* modules/openstack_project/templates/logstash/indexer.conf.erb: Switch
to TCP input instead of pipe input as the new Jenkins log pusher service
will push log events over TCP.
Change-Id: Id80c710abd5facd71d18afb2b250b2d7d92dec2d
Reviewed-on: https://review.openstack.org/28074
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Clark Boylan