12 Commits

Author SHA1 Message Date
Clark Boylan
809b3e700a Don't index the devstack summary logs
The devstack summary logs are being picked up by logstash workers but
they are a proper subset of the full devstack log which we also index.
Avoid duplicative indexing by excluding the summary file.

This also addresses an issue with crm114 indexing these files with each
of them ending up with its own data dir due to timestamp data in the
filenames.

Change-Id: Ia397a3ceef0bf9f2198b4b6f9956d5b3dc9abcef
2017-12-04 13:05:44 -08:00
Zuul
a48e55cceb Merge "Add errors file to be indexed by logstash" 2017-11-29 18:16:14 +00:00
Clark Boylan
3d3e628709 Stop indexing libvirt logs
The libvirt log files can be quite large depending on the job run (into
the million line range). This means they take significant amounts of
time to index which backs up all the indexing. Lets remove these files
for now (mriedem, nova ptl, has said this won't be a great loss for
them) and if someone needs them indexed again they can help sort out how
to do it more quickly.

Change-Id: I6b567eaf8a59634bc2486423bd87553605bb9657
2017-11-20 15:27:29 -08:00
Clark Boylan
c150e17996 Don't index karaf logs
The screen-karaf.txt logs come in at a ~270MB, are not in oslo log
format, and don't filter by log level. Attempting to index this file has
OOM'd at least one logstash worker process. We didn't attempt to index
these files prior to the zuulv3 switch so lets just stop indexing them.

Change-Id: Iaab5883c9af9c5d0b323fc351c92869bde7edb9d
2017-11-16 15:50:35 -08:00
Sagi Shnaidman
05e97ad67b Add errors file to be indexed by logstash
In tripleo we grep for errors in logs and put them in one file.
This patch is for indexing only errors file in logstash server.

Change-Id: I95021ca826c75d7750bc2e56243ecc638bc1a502
2017-11-15 20:32:08 +02:00
Clark Boylan
3271b4c0d5 Stop indexing peakmem and dstat logs
These two log files are fairly large, don't support log levels and were
matching the new screen-.* logfile regex. This meant they were getting
indexed when they weren't before the zuulv3 switch. Lets stop indexing
them while we get the logostash and elasticsearch stuff under control
then consider adding them back in more slowly and intentionally once
there.

Change-Id: If0a1e4a2967b35f10a467c9397697053caa23472
2017-11-04 19:58:13 -07:00
Clark Boylan
98aaf14ec4 Revert "tripleo: index /var/log/*.log.txt files"
This reverts commit 1f4d096c346368d15869265e5a08ad93289e20dc.

We are reverting this as it appears to significantly add to the total
number of files being indexed and we are very behind. Also there appear
to be some bugs with this like indexing all of these log files as
console logs when many of them are openstack service logs.

Additionally this seems to have caused crm114 to fill up its disks as it
creates a new dir with learning data on disk for each unique file path.
We will need to update crm114 to better collapse files into things that
are less unique.

Reverting this allows us to take a step back and stabilize then we can
try to move forward with this more carefully ensuring that crm114 is
ready and that we know what we are indexing and apply the correct tags
to it.

Change-Id: Ie75d13f4275d74c7035e6b8f89bab47d1b053d03
2017-11-01 10:21:28 -07:00
Emilien Macchi
1f4d096c34 tripleo: index /var/log/*.log.txt files
Right now, we only index logstash.txt, which is fine now. The file is
~100K.
But this file doesn't context OpenStack services logs, and other logs
we want to index, so we can use elastic-recheck more powerfully.

So this patch index both undercloud & subnodes (overcloud) .log.txt
files present in /var/log/*.

Change-Id: Id5b9be6130e4e4506418a254a5e57e105fa4d03c
2017-10-19 11:26:01 -07:00
James E. Blair
e780f330ec Fix typo in logstash job submit config
Change-Id: I3ed6662066dba4a03a299814e910175fe83c5ec8
2017-09-14 14:36:56 -06:00
Clark Boylan
252132d11a Replace console.html with job_output.txt in zuulv3
Because zuulv3 will never produce a console.html we can update the
zuulv3 logstash job submission rules to use job_output.txt in place of
console.html. We tag job_output.txt with console and console.html so
that old queries that possibly using the implicit console.html tag will
continue to work against the new file name.

Change-Id: I3b1e6af07806cad57b392bd5145ad2782cf6feb4
2017-09-13 16:45:30 -07:00
James E. Blair
fad3ecd79b Update success to zuul_success
Change-Id: I59eb5e4c76b6de76028f0303bd4ffd4143c0da18
2017-09-13 09:19:40 -06:00
James E. Blair
47ae93c061 Add log processing roles
These roles replace the jenkins-log-client portion of the
logstash/subunit queue processing system.  Much of the code is
derived from the jenkins-log-client.

Rather than subscribing to ZMQ events emitted from jenkins (and
later zuul v2.5), these roles are intended to be run in a late base
post playbook (right after log uploading).  They examine the logs
directory which contains the files that were just uploaded and any
matching files have jobs submitted to the gearman queue.  Currently
the module attempts to maintain compatability as much as possible,
but eventually we should represent some of the data in a more v3-like
manner, as well as consider adding additional data.  Some suggestions
are noted in comments.

Change-Id: I10ea613d9278465b90d891371d2626b9e99f8f31
2017-09-11 10:12:28 -06:00