73 Commits

Author SHA1 Message Date
Jenkins
0aff86040c Merge "Index the os-collect-config logs in tripleo jobs" 2014-07-02 08:55:19 +00:00
Sean Dague
918b838594 index grenade logs in elastic search
We need to start doing this before we change grenade to not default
to spitting out everything to the console.

Change-Id: I58d32c27e78d2d6eb791f2d29c3aa886a5218c68
2014-06-16 07:08:46 -04:00
Derek Higgins
eb11de00c1 Index the os-collect-config logs in tripleo jobs
Add the os-collect-config logs to logstash to be indexed. Depends on
another commit in tripleo-ci to remove the random string from the log
name Ie84492ab981b06421d486579fc269ef6b3ad1815.

Change-Id: I2fe3aa4ec3c469d59a6dee8dd54f08d0bbf7b792
2014-06-13 15:52:15 +01:00
Matt Riedemann
e271f398f3 Index logs/tempest.txt for logstash queries
We need to index the tempest log so we can fingerprint errors using
multiple parts of the same traceback which doesn't work with the
console.html log.

Closes-Bug: #1323713

Change-Id: I34b2d67e12199ef0145d7a8d25d5385f944c78ed
2014-05-27 08:20:54 -07:00
Joe Gordon
5ddec07e72 Add grenade neutron logs to ElasticSearch
Now that we have a grenade neutron job that we are trying to stabilize
and gate on, we should collect the neutron logs in grenade so we can use
logstash and elastic-recheck with grenade neutron jobs.

Change-Id: I54c4079b1be00d5201d4fb84dd975576551728fe
2014-05-06 14:35:25 -07:00
Sean Dague
7291918fb7 index n-sched in grenade runs
Change-Id: I0bcf55d83c99f6de9f414b2f723da6abfb209a99
2014-04-24 07:00:04 -04:00
Matt Riedemann
f579ec76e6 Index logs/screen-n-api-meta.txt for logstash queries
We are seeing a race failure in check-tempest-dsvm-postgres-full jobs
where the error message shows up in the screen-n-api-meta log but we can
only get at it from screen-n-api logs (or console). We should index on
the screen-n-api-meta log so we can better filter the query.

Related-Bug: #921858

Change-Id: I1701ac83b2643d819245a7cdbbfb56cc4af12f5b
2014-04-21 13:35:48 -07:00
Jenkins
a2f0f2cc61 Merge "Remove savanna from logstash/jenkins-log-client.yaml" 2014-03-26 14:20:31 +00:00
Sean Dague
9fbc6f14cf add missing neutron logs
There are 3 logs for neutron services we are regularly running in
the gate which we aren't indexing. Ensure they are all indexed so
we can actually use them in elastic recheck. They are all pretty
small so shouldn't impact ES load.

Change-Id: I082094574ceb6197e30708f9e05ba4b7fdc6f8af
2014-03-20 09:02:40 -04:00
Sean Dague
ae96d3bf6c add horizon_error to the indexed logs
this adds horizon_error to the indexed log files, which is very
useful in determining how horizon fails, as we get very specific
stack traces.

Change-Id: Ifb323e327dbc2931100a4552d029d91209c4bbba
2014-03-17 12:37:39 -07:00
Jenkins
f9da1187d5 Merge "Better logstash field data." 2014-03-13 22:09:14 +00:00
Clark Boylan
b2ef46c5c7 Better logstash field data.
We are currently using a lot of wildcard searches in elasticsearch which
are slow. Provide better field data so that we can replace those
wildcard searches with filters. In particular add a short uuid field and
make the filename tag field the basename of the filepath so that grenade
and non grenade files all end up with the same tags.

Change-Id: If558017fceae96bcf197e611ab5cac1cfe7ae9bf
2014-03-13 14:42:58 -07:00
Jenkins
2a481cdd76 Merge "First step of renaming Savanna to Sahara" 2014-03-12 12:49:25 +00:00
Sean Dague
6db50ad13e stop indexing libvirt logs
We need a much higher level of output on libvirt logs to get to the
bottom of Bug 1254872. However, at that level of output, we crush
elastic search. So turn off indexing this log in the gate until we
get to the bottom of the bug and can return it to a more reasonable
logging level.

Change-Id: I9fec939883e50b421bc0530205e8b0bd7eab2350
Related-Bug: #1254872
2014-03-11 07:40:12 -04:00
Sergey Lukjanov
94db4b5357 Remove savanna from logstash/jenkins-log-client.yaml
Change-Id: I5d1745c430d2e76d7ed41a1750216d0def305eac
2014-03-11 10:35:48 +04:00
Sergey Lukjanov
b0c3661e55 First step of renaming Savanna to Sahara
Change-Id: I766129df0ebb21b6657e49778ac6246dee4c51e9
2014-03-10 20:27:57 +04:00
Clark Boylan
4454aac98f Incrase heap size for logstash indexers.
The default java heap size is too small for our logstash indexers.
Double it to 2g. Do this by adding an /etc/default/logstash-indexer file
that the upstart configs source if it is present. This required some
tweaks to the upstart configuration to load the defaults properly.

Co-Authored-By: K Jonathan Harker <k.jonathan.harker@hp.com>
Change-Id: I63447f59f3fa6d466a7d275476121fe8339479dc
2014-02-27 10:30:28 -08:00
Sean Dague
cb10a09e08 index sublog files
index the sublog files so that we can use them in elastic search,
and so that we can start removing content from console.html.

Depends-On: Iede34b970d090f855c701b69c1f5167a08ab9c52

Which brings us time stamping to the sublogs

Change-Id: I4b4484065fda168f4d5efc73e95736226fb36ed0
2014-02-24 18:29:39 -05:00
Jenkins
cca158a05c Merge "Add Savanna screen file to logstash indexing" 2014-02-10 20:14:07 +00:00
Joe Gordon
bdd4fc245c Add logstash rules to parse libvirtd.txt
Change-Id: I09185dae08d46821f804e3ad43205205aab9345c
2014-01-31 15:27:54 -08:00
Clark Boylan
5f78a77de0 Add grenade logs to logstash.
Add grenade new/ and old/ logs to logstash. To do this without tripling
HTTP GETs for every finished job add a job filter to the log gearman
client that when present only attempts to grab files if the job name
matches the job filter.

Change-Id: Ia33722bf71d482f2fd6b655b28090a10bf46af54
2014-01-31 14:14:12 -08:00
Sergey Lukjanov
099ecf35da Add Savanna screen file to logstash indexing
Change-Id: Ib63235f64cfa7f28942f6272857b0d413b6a0c38
2014-01-28 17:50:02 +04:00
Jenkins
bd587449c6 Merge "Add ironic log files to logstash indexing" 2014-01-27 19:51:00 +00:00
Monty Taylor
2dfcf4fb71 Add three new jenkins servers
Change-Id: I1143c1c7446890c47be12d8b115560bddadc1919
2014-01-22 12:50:40 -05:00
James E. Blair
f6500357f0 Add jenkins03 and 04 to most places
Add them to all the places it's safe to do so for now.  Don't actually
spin up any nodes for them yet.

Change-Id: I59e97be7e5b094af3153bc7d5dce0cff57996f55
2014-01-07 09:58:17 +08:00
Devananda van der Veen
2fbf5b626f Add ironic log files to logstash indexing
Add ironic log files to logstash indexing

Change-Id: I284d5e8016544f40352f7f52e7e8af95a32dd3f7
2014-01-02 11:54:44 -08:00
James E. Blair
4e0739fd36 Increase logstash workers/host to 4
Initial load testing suggests we should be able to handle at least
4 processes per host.

Also re-enable crm114.

Change-Id: Ia0158ad5f7f524c4fa0a80d479e2b74d28f0d1a6
2013-12-20 09:58:28 -08:00
Jenkins
da10edae57 Merge "Refactor logstash_worker into log_processor module" 2013-12-18 18:26:59 +00:00
James E. Blair
920fc0c42b Refactor logstash_worker into log_processor module
Separate the jenkins log client and worker bits into a new module
called log_processor with ::client and ::worker classes.

Instantiate two workers on each logstash worker node.

Change-Id: I7cfec410983c25633e6b555f22a85e9435884cfb
2013-12-18 00:40:49 +00:00
James E. Blair
5e99683bea Disable the crm114 filter
Change-Id: I50a80f6ec1403f1b895e2520a6d6b1494285e863
2013-12-17 08:55:34 -08:00
James E. Blair
6501547284 Process logs with CRM114
Try to identify the probability that each log line indicates an error.
Pass that information on to logstash.

Change-Id: I0b298c2e8c00d8fdf1c907215a7bbf27086bc80c
2013-12-13 13:57:23 -08:00
Monty Taylor
38f8d43ffc Fix pep8 errors in modules dir
There weren't that many of them - just fix them.

Change-Id: I6aa6d056746c4a6804146b89c12e6313f9d22d9a
2013-11-12 11:30:34 -05:00
Clark Boylan
1f07114346 Upgrade Logstash to 1.2.1.
* modules/logstash/manifests/init.pp: Download and install Logstash
1.2.1.

* modules/openstack_project/files/logstash/log-gearman-client.py:
Logstash 1.2.1 comes with a new schema. Update the job data sent to log
push workers to better accomodate the new schema.

* modules/openstack_project/files/logstash/log-gearman-worker.py: Push
Logstash 1.2.1 schema compliant JSON to the Logstash TCP input.

* modules/openstack_project/templates/logstash/indexer.conf.erb:
Logstash 1.2.1 comes with a new schema and many input and filter
changes. Use the newly supported features like conditionals to keep the
config up to date.

* modules/kibana/templates/config.rb.erb: Change the default field for
kibana to 'message'. It was @message which is deprecated in the new
logstash schema.

Change-Id: Id19fc05bcce8d42c5c0cf33df3da7e95f5794107
2013-10-21 10:17:48 -07:00
Clark Boylan
837801709a Increase log worker retry timeout length.
* modules/openstack_project/files/logstash/log-gearman-worker.py:
Increase the log worker retry timeout length to 255 seconds and do an
exponential backoff instead of retrying every second when incomplete
files are received. This should reduce load on the file server and make
console.html indexing more reliable.

Change-Id: I42d8f99b3ba2495bb1cac94c4b44e3598f7b9cb6
2013-10-02 17:34:04 -07:00
Clark Boylan
6fc3a1d583 Retrieve logs > DEBUG level.
* .../logstash/log-gearman-worker.py: Only GET log lines greater than
the DEBUG log level. This will cut down the amount of processing that
logstash needs to perform and should speed up logstash log processing.
This changes the order of the $file then $file.gz GETs. Now get $file.gz
first as the log filtering on the apache side only runs on the .gz
files.

Change-Id: I8d88787ddde541f5aec2aee59c27cb1b48e5e4e5
2013-09-26 12:22:23 -07:00
Clark Boylan
c09d6695d8 Put build UUID, log_url and project in log events.
Change-Id: I8d6ee94eb94e8fcdb082cad52654fcf0f7d9b3de
2013-09-13 17:04:34 -07:00
Jenkins
0e9795841e Merge "Handle log paths in logstash gearman client better" 2013-08-01 19:33:51 +00:00
Jenkins
738acae92c Merge "Handle html log annotations." 2013-08-01 19:32:44 +00:00
Clark Boylan
1678360e3b Handle log paths in logstash gearman client better
* modules/openstack_project/files/logstash/log-gearman-client.py:
Use os.path.join to join the base log path provided by Jenkins and the
log file provided in the gearman client config. This avoid needing to
worry about trailing slashes in the Jenkins provided path.

Change-Id: I973237dae6f0b7947d322489108a05a99a7cc0be
2013-07-31 15:32:54 -07:00
Jenkins
fb99b4fa74 Merge "Switch Zuul/JJB to use LOG_PATH" 2013-07-31 21:30:06 +00:00
Clark Boylan
a20c139f3d Handle html log annotations.
* modules/openstack_project/files/logstash/log-gearman-worker.py:
The annotated logs served by logs-dev and soon to be served to logs.o.o
return .txt files gzipped. log-gearman-worker.py needs to check the
Content-Type in the reponse headers to see if the txt files were gzipped
in order to properly handle this.

partial-bug: #1207047
Change-Id: I5981cde145a572a6e3d20e8369e407df151143ff
2013-07-31 11:35:00 -07:00
James E. Blair
0ccf564937 Switch Zuul/JJB to use LOG_PATH
This calculates the full LOG_PATH in the Zuul config and pases it
to Jenkins.  The new path is similar to the old but with the
substitution of a short ZUUL_UUID at the end instead of the Jenkins
build number in order to avoid collisions from multiple Jenkins
masters.

Periodic jobs add a node name to their log path to avoid collisions
from multiple masters.  Unfortunately, that value is not accessible
to the logstash worker.  This can be solved by having Zuul trigger
periodic jobs.

Add the ZUUL_REF to logstash as "build_ref".

Requires https://review.openstack.org/#/c/39130/

Change-Id: I40bad59e3ad8ed6b4706762ed8b833fd15c13b0d
2013-07-29 17:47:08 -07:00
James E. Blair
49b62c5d3a Add new jenkins masters.
Add jenkins01 and jenkins02, both of which will have unit test and
devstack slaves.  Leave jenkins.o.o alone; over time it will be
reduced so that it alone has special jobs and privileged slaves
attached to it.

Note that currently all of the jobs will be defined on all nodes,
including jobs on timers.  I think the long-term fix for that is to
have zuul schedule timed jobs.

Change-Id: I10bbd5555e5194b1031700975d5b3ae6b458b8b3
2013-07-29 09:24:39 -07:00
Clark Boylan
db2e5732b2 Increase Logstash Gearman worker queue length.
Logstash seems to perform better when it is not running out of buffer
space. Increase the buffer size to reduce contention and give the
gearman workers more room to work when things are queueing up. This
should increase throughput.

Change-Id: Ie4b5cc7a3ae4be517012c20559f59071b7b15dd5
2013-07-22 17:22:46 -07:00
Clark Boylan
0e66fcdce1 Properly root log path in logstash gearman client.
* modules/openstack_project/files/logstash/log-gearman-client.py:
BASE_LOG_PATH is a relative dir and is not rooted. Provide the root '/'
when constructing the source URL for the gearman workers.

Change-Id: I8fcfa19b048019398ffd370d74c271b9656a8688
2013-07-19 11:38:26 -07:00
James E. Blair
f7f2760fe0 Set the Zuul/Jenkins base log URL in a function
And use that when constructing log paths and URLs.  Use a substr
of the change id or commit sha when constructing URLs so the log
directories are deeper.

Make copying the test results a macro (it's used in several places).

Update the gearman log client to take advantage of the new parameter.

Requires https://review.openstack.org/#/c/36304/

Change-Id: I64faa35eddc4105271efa3de4f83b608b77655c2
2013-07-19 10:36:31 -07:00
Clark Boylan
f36ada63eb Add heat logs to logstash.
Change-Id: I7ba628bb5d7f160f67327310048973483b78b05a
Reviewed-on: https://review.openstack.org/34156
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-07-01 17:49:16 +00:00
James E. Blair
fa7d1e5e68 Have logstash be less verbose on 404 errors.
Change-Id: Ie6905039d4752d578566861ffd340cf607ad270b
Reviewed-on: https://review.openstack.org/32819
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-06-21 22:28:40 +00:00
Clark Boylan
abe222fa9e Fix logstash gearman puppet.
* modules/openstack_project/manifests/logstash.pp: Concat is not
available in our version of puppetlabs stdlib. Use flatten instead
which is available. Remove dependency on non existant logstash::indexer
class. Fix requires orders.

* modules/openstack_project/manifests/logstash_worker.pp: Fix requires
orders.

* modules/openstack_project/files/logstash/jenkins-log-client.init
* modules/openstack_project/files/logstash/jenkins-log-worker.init:
Set pidfile argument when calling scripts.

* modules/openstack_project/files/logstash/log-gearman-worker.py:
Use python2 compatible gzip.GzipFile instead of gzip.decompress. Send
work exception instead of work fail when an exception happens. Log these
exceptions locally as well.

Change-Id: Idf0a873215acb72187e058a0306a21ccd928d464
Reviewed-on: https://review.openstack.org/32804
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-06-17 21:08:48 +00:00
Clark Boylan
4aba312807 Use gearman to distribute logstash log pushing.
This change reorgs the logstash log pushing so that there is a central
gearman server that listens to Jenkins ZMQ events which are then
converted to per log file gearman jobs which are processed by gearman
workers. The central gearman server will live on logstash.o.o and the
existing logstash-worker hosts will be converted to gearman log pusher
workers.

This commit includes relavent documentation changes.

Change-Id: I45f7185c2479c54b090d223408dff268e1e8d7db
Reviewed-on: https://review.openstack.org/32455
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-06-12 21:45:42 +00:00