The devstack summary logs are being picked up by logstash workers but
they are a proper subset of the full devstack log which we also index.
Avoid duplicative indexing by excluding the summary file.
This also addresses an issue with crm114 indexing these files with each
of them ending up with its own data dir due to timestamp data in the
filenames.
Change-Id: Ia397a3ceef0bf9f2198b4b6f9956d5b3dc9abcef
The libvirt log files can be quite large depending on the job run (into
the million line range). This means they take significant amounts of
time to index which backs up all the indexing. Lets remove these files
for now (mriedem, nova ptl, has said this won't be a great loss for
them) and if someone needs them indexed again they can help sort out how
to do it more quickly.
Change-Id: I6b567eaf8a59634bc2486423bd87553605bb9657
The screen-karaf.txt logs come in at a ~270MB, are not in oslo log
format, and don't filter by log level. Attempting to index this file has
OOM'd at least one logstash worker process. We didn't attempt to index
these files prior to the zuulv3 switch so lets just stop indexing them.
Change-Id: Iaab5883c9af9c5d0b323fc351c92869bde7edb9d
In tripleo we grep for errors in logs and put them in one file.
This patch is for indexing only errors file in logstash server.
Change-Id: I95021ca826c75d7750bc2e56243ecc638bc1a502
These two log files are fairly large, don't support log levels and were
matching the new screen-.* logfile regex. This meant they were getting
indexed when they weren't before the zuulv3 switch. Lets stop indexing
them while we get the logostash and elasticsearch stuff under control
then consider adding them back in more slowly and intentionally once
there.
Change-Id: If0a1e4a2967b35f10a467c9397697053caa23472
This reverts commit 1f4d096c346368d15869265e5a08ad93289e20dc.
We are reverting this as it appears to significantly add to the total
number of files being indexed and we are very behind. Also there appear
to be some bugs with this like indexing all of these log files as
console logs when many of them are openstack service logs.
Additionally this seems to have caused crm114 to fill up its disks as it
creates a new dir with learning data on disk for each unique file path.
We will need to update crm114 to better collapse files into things that
are less unique.
Reverting this allows us to take a step back and stabilize then we can
try to move forward with this more carefully ensuring that crm114 is
ready and that we know what we are indexing and apply the correct tags
to it.
Change-Id: Ie75d13f4275d74c7035e6b8f89bab47d1b053d03
Right now, we only index logstash.txt, which is fine now. The file is
~100K.
But this file doesn't context OpenStack services logs, and other logs
we want to index, so we can use elastic-recheck more powerfully.
So this patch index both undercloud & subnodes (overcloud) .log.txt
files present in /var/log/*.
Change-Id: Id5b9be6130e4e4506418a254a5e57e105fa4d03c
Because zuulv3 will never produce a console.html we can update the
zuulv3 logstash job submission rules to use job_output.txt in place of
console.html. We tag job_output.txt with console and console.html so
that old queries that possibly using the implicit console.html tag will
continue to work against the new file name.
Change-Id: I3b1e6af07806cad57b392bd5145ad2782cf6feb4
These roles replace the jenkins-log-client portion of the
logstash/subunit queue processing system. Much of the code is
derived from the jenkins-log-client.
Rather than subscribing to ZMQ events emitted from jenkins (and
later zuul v2.5), these roles are intended to be run in a late base
post playbook (right after log uploading). They examine the logs
directory which contains the files that were just uploaded and any
matching files have jobs submitted to the gearman queue. Currently
the module attempts to maintain compatability as much as possible,
but eventually we should represent some of the data in a more v3-like
manner, as well as consider adding additional data. Some suggestions
are noted in comments.
Change-Id: I10ea613d9278465b90d891371d2626b9e99f8f31