10 Commits

Author SHA1 Message Date
Clark Boylan
448b7d7768 Index zuul executor
It would be useful to know what zuul executor ran a job so that we can
track potential executor specific issues back to that executor via
logstash. Add the zuul executors hostname to the log processing payload
which will result in us storing it in elasticsearch.

Change-Id: Ifbd7c7f91a006bd63e3df6053297ccc36bb3240b
2018-01-09 15:37:21 -08:00
Clark Boylan
83276a7cc3 Better comment in logstash job submission role
Be more explicit in the logstash job submissions role library code about
why we can get away with treating foo.txt and foo.txt.gz as the same
content.

Change-Id: I44a7310f7aa149919cc9aa38510329ce879c8f17
2017-10-31 14:02:22 -07:00
Clark Boylan
cba0f2f11f Logstash jobs treat gz and non gz files as identical
We compress log files early so that we minimize network traffic and disk
storage requirements. But logically foo.txt and foo.txt.gz are identical
for our purposes when hosting log files. You can ask apache for either
and should get the same content.

This creates problems when indexing logs because we had been treating
the non .gz suffix'ed files as the logical names but now we don't have
them in all cases in ES because we submit jobs for the .gz file
resulting in the name in ES having the .gz suffix. Additionally if we
have both foo.txt and foo.txt.gz on disk when we generate the log
indexing jobs we'll submit two jobs to index the same log file contents.

We can address both these problems if we early on treat the non .gz as
the "logical" name and only submit jobs for the no suffix names.

Change-Id: Id99d6bc58f94c197b567ec4080ab24715128d855
2017-10-31 08:05:49 -07:00
Andreas Jaeger
4adbecad99 Fix flake8
fix a "bug" in that the flake8 configuration in tox.ini was exclusively
selecting H231 as the only error it would report, so it was missing the
errors in the python modules (such as submit_log_processor_jobs). Due to
this being the case for a long time (since 2004) limit the more thorough
linting to the roles/ and playbooks/ directories where we'll be adding
ansible plugins/modules/etc. Also, lint in jenkins/script and nodepool.

Fix problems found.

We can lint everything with pep8 once the zuul v2 scripts are removed,
not worth patching them right now.

Change-Id: I479f010643cf3b67c183d763510f07a33400d38b
Co-Authored-By: Jesse Keating <omgjlk@us.ibm.com>
2017-10-21 18:37:10 +02:00
James E. Blair
af0adec254 Add 90s connect timeout for submit log jobs
If the logstash gearman server is down, we don't want jobs to
hang indefinitely.

Change-Id: I2ea16bdae4cc0181feae3db0532c88485a52b70b
2017-10-15 13:03:57 -07:00
James E. Blair
efa6e0a2ba Fix hostvars logstash job submission error
The log server may have just been added to the inventory; we're
really interested in the node the job ran on, so make sure that
the node we grab is a nodepool node.

Change-Id: I3f228f9a0c913476f946f546cdd5a2d56864c549
2017-09-14 17:00:22 -06:00
James E. Blair
5ef48fa65f Fix project field in logstash job submission
We were sending the whole project dict instead of the name.

Change-Id: I911b0edb52ae663f636cc57142de9959432312a8
2017-09-14 16:51:59 -06:00
James E. Blair
2b464b1782 Return gearman job handles on log job submission
For easier debugging.  Also, override the invocation kwarg so that
ansible doesn't automatically log the arguments to this module
(since they include all of the hostvars which include way too much
data).

Change-Id: I415da15dd1dc154e08e159453235448c90a5e5d2
2017-09-13 14:55:41 -06:00
James E. Blair
6ffab6f745 Fix import errors in subunit job module
Change-Id: I0da368c933e8fb6e3de7e6c4e81c934636b0317e
2017-09-13 10:12:30 -06:00
James E. Blair
47ae93c061 Add log processing roles
These roles replace the jenkins-log-client portion of the
logstash/subunit queue processing system.  Much of the code is
derived from the jenkins-log-client.

Rather than subscribing to ZMQ events emitted from jenkins (and
later zuul v2.5), these roles are intended to be run in a late base
post playbook (right after log uploading).  They examine the logs
directory which contains the files that were just uploaded and any
matching files have jobs submitted to the gearman queue.  Currently
the module attempts to maintain compatability as much as possible,
but eventually we should represent some of the data in a more v3-like
manner, as well as consider adding additional data.  Some suggestions
are noted in comments.

Change-Id: I10ea613d9278465b90d891371d2626b9e99f8f31
2017-09-11 10:12:28 -06:00