27 Commits

Author SHA1 Message Date
Clark Boylan
58890e3b7e Make e-r crons valid puppet
Puppet does not like cron list syntax to be made as a normal string.
Instead you must give puppet cron resources a puppet list that it will
convert to a cron list...

Change-Id: Ia0839be9b7d8ae772065f782adf61f831e75b92e
2014-04-07 11:35:03 -07:00
Clark Boylan
0e27b0178e Run e-r batch jobs twice an hour ten minutes apart
Decrease the time between e-r batch jobs to half an hour between
consecutive runs of the same job to reduce the latency on the updates of
these reports. Offset each of the three batch jobs from each other by
ten minutes. The run times look like this now:

  minute: 0 10 20 30 40 50
  job:    1  2  3  1  2  3

Change-Id: Ic4176b019e6da71e8c1e33ca89f53b6700309a91
2014-03-28 12:56:51 -07:00
James E. Blair
fd14fd6112 Log recheckwatchbot messages
The qualname was wrong for this logger.

Change-Id: Idc20113fcb96af8b979d1b6552155f2a1e8c5c66
2014-03-03 08:08:18 -08:00
Clark Boylan
e24b23bc51 Run fewer es queries with elastic_recheck.
Currently elastic-recheck is hammering the ElasticSearch cluster every
15 minutes, basically bringing everything else to a halt while these
queries are processed. Spread elastic-recheck periodic queries out, run
each once an hour and 20 minutes after the previous set of queries.

Also, remove the auto triggering when files change, instead we wait for
the hour to roll over and give us new data.

Change-Id: Ibf5dad1071f65f8d676e8b9a0ec4331bada9943f
2014-02-10 16:25:24 -08:00
Joe Gordon
263aa363e0 Generate two elastic-recheck-graph json files all, and gate only
We want to display the hits for all queues and for just the gate queue.
A parameter was added to elastic-recheck-graph to support this, and a
second patch updates the web page to give the gate queue only option.
This patch updates the usage of elastic-recheck-gate.

Related elastic-recheck patchs:
 * I9217a2ceedf86ffe04851084df78238384fccd51
 * I23ed8d945c9517221622e1ca215c80539bb07bbd

Change-Id: Iac51aeaf152f50f8b9a064cd249b0920bef824df
2014-02-03 16:25:36 -08:00
Jenkins
15d8b80ecf Merge "Add projects section to elastic recheck bot yaml" 2014-01-27 20:05:21 +00:00
Sean Dague
45ea819a48 fix template directory for uncategorized page
your results can be dramatically improved by knowing what you are
doing.

Change-Id: Id9228173ea6270c82f98a75f9a0feeb8fb6a988f
2014-01-23 19:00:47 -05:00
Sean Dague
a2f54a86bc fix er_run commands
puppet exec doesn't run under a real shell, so running cd doesn't
work because it's not a command. Instead use cwd properly to
actually run these commands.

Also actually call the correct command to generate the uncategorized
jobs page.

Change-Id: I84ed19bbdc16d185316e3967efbd798e06a521d1
2014-01-23 16:36:38 -05:00
Matthew Treinish
62dcb59c9a Add projects section to elastic recheck bot yaml
The elastic-recheck change: Ic3dd76bad94213c7152c29a99c00ed23a2c01a31
will add a new section to the recheckwatchbot yaml file to specify
which bug targets get reported to which channels. This commit preempts
that change by adding the new section to the -qa channel to preserve
the existing functionality.

Change-Id: I452040883027b260c0480b9d7ac2e2ca476b72b3
2014-01-23 14:52:39 -05:00
Sean Dague
74c68501bc add in elastic-recheck-unclassified report
this adds the new elastic-recheck-unclassified report, which
generates an html page of all the unclassified gate failures.

it also runs this command, and the elastic-recheck-graph command
not only on cron, but also on updates to the repository. In order
to do this safely, we create a new er_safe_run.sh in /usr/local
that provides a flock front end to ensure we won't race on our
html writes.

Change-Id: I624f2629e2ac19a1394866f73f1aa9ba18e968ce
2014-01-21 18:29:30 -05:00
James E. Blair
2bb28339cb Move elastic-recheck bot to status.o.o
Also, normalize some parameters to the status manifest to make it
more readable (since they follow the same pattern).  And make the
ssh config for elastic-recheck bot match reviewday.

Change-Id: I2417f121e7b3685aab9540504cdd4c6db1754e67
2013-12-31 19:35:47 +00:00
Jeremy Stanley
e20f96f782 Correct elastic-recheck install subscription scope
* modules/elastic_recheck/manifests/bot.pp: The subscribe on the
install_elastic-recheck exec in ff1e7bc2 specified an explicit
scope, but exec resources are not scoped. Fix it.

Change-Id: I16ee575a24c9abe1db13f4c2164f8c49fa95980f
2013-12-12 19:30:47 +00:00
Clark Boylan
ff1e7bc208 Fix puppet subscription for elastic-recheck.
The installation of elastic-recheck is done on multiple machines but the
irc bot is only run on one machine. Therefore don't notify the bot
service from the install exec instead subscribe to the exec from the
service.

Change-Id: I11f3d32e40408a02d948d0533f5a4c45c983adae
2013-12-06 14:24:17 -08:00
Sean Dague
b3af619900 move to elastic-recheck web console in share
this provides the changes needed to handle the new elastic-recheck
web console installed from the elastic-recheck package in the
share tree.

factor out bot code to elastic_recheck::bot

this change moves elastic_recheck bot starting code out into
a separate module, which lets us include all the elastic_recheck
base module into the static site config.

Change-Id: If53523754494a746c152c5d1384db5426b096fc1
2013-12-06 15:52:40 -05:00
James E. Blair
72fba623fc Revert "move to elastic-recheck web console in share"
This reverts commit 4bf45a37e65b378b261685e27a81158dcc3e5c8f
2013-12-05 19:23:15 +00:00
Sean Dague
4bf45a37e6 move to elastic-recheck web console in share
this provides the changes needed to handle the new elastic-recheck
web console installed from the elastic-recheck package in the
share tree.

Change-Id: I8c8b76cb53b2f2eea026eb0f321e48903df71636
2013-12-02 13:54:28 -05:00
Sean Dague
6074bb47b9 use queries directory instead of file for elastic-recheck
moving elastic-recheck to a queries directory instead of a file
means we need to make the graphs support that. Also refactor out
the er_state_dir to make changing that easier when we pull in the
web console.

2Bupdate elastic-recheck bot config to use queries directory

Change-Id: Ifcc71511c066e8aee5cb74e013555c628f270e29
2013-12-02 13:54:03 -05:00
Sean Dague
ecdb2ced59 also link in /etc/elastic-recheck/queries
in I86317fcf6f1886ab5b6c0ee154b29e71865c52b7 we want to move to
a directory of queries instead of a single file. In order to
support this we need to support linking in that directory as well.
This needs to land prior to that change.

Change-Id: I3af42cff1050754c6844db994004fd203063bb3d
2013-12-01 13:14:13 -05:00
Monty Taylor
0222d3040e Use pip install instead of setup.py install
The pip vs. easy_install interactions continually get the systems
into a weird state. pip install $path_to_repo, on the other hand,
works like a charm.

Left out two graphite installs because they are doing path manipulation
and I want to do further testing before touching that.

Change-Id: I373b29aca03f1ebd20e533ebaaf5de9ef2db017e
2013-10-27 18:38:38 -04:00
Sean Dague
a1d70f868c auto restart elastic recheck on update
this ensures we notify the er daemon to restart after we've updated
the code, making it so we deploy instantly into the environment.

Change-Id: Ifa266c7c0248ce9d0431f44e76de87958503f401
2013-10-09 13:27:49 -04:00
James E. Blair
b2d83e9dba Increase verbosity in elastic-recheck debug log
Change-Id: I5c326d821cba2e4c66b7eaf0fdfeba43ae7cd238
2013-09-30 15:13:53 -07:00
James E. Blair
ace54c9adf Replace queries.json with queries.yaml for elastic-recheck
Change-Id: Id5d9b9c2729abbf8572c9c785c5026d0138b72c9
2013-09-30 15:05:43 -07:00
James E. Blair
9e8b831c47 Make elastic-recheck's shell bash
Change-Id: Ibbc24d172f9fd6eaf8ac0f3cf15df654f9979bbf
2013-09-30 15:02:25 -07:00
Jeremy Stanley
fd832de552 Move elastic-recheck shell account out of /var/run
* manifests/site.pp: Expect the SSH key in /home rather than in
/var/run.

* modules/elastic_recheck/manifests/init.pp: Change the recheck
shell account for elastic-recheck to use a homedir in /home.

Change-Id: I43f4a8cdcb803328388b7e92732e5cb9d6acb2a8
2013-09-30 21:44:03 +00:00
Jeremy Stanley
d1bd696b56 Turn on logging for elastic-recheck
* modules/elastic_recheck/files/elastic-recheck.init: Minor
whitespace cleanup for space-before-tab indent on a comment line.

* modules/elastic_recheck/files/logging.config: New logging
configuration file for elastic-recheck IRC bot.

* modules/elastic_recheck/manifests/init.pp: Install the new logging
configuration file.

* modules/elastic_recheck/templates/elastic-recheck.conf.erb: Tell
the IRC bot to use the new logging configuration file.

Change-Id: I47cabb01df209c9f020846bcb272b9a2f796b0e5
2013-09-30 14:27:21 -07:00
Jeremy Stanley
942e5f1dfb Correct elastic-recheck Puppet dependencies
* modules/elastic_recheck/files/elastic-recheck.init: Update the
daemon invocation now that it has an executable entrypoints wrapper.

* modules/elastic_recheck/manifests/init.pp: Correct the Puppet
dependency tree sufficiently to be able to apply the manifest
successfully.

Change-Id: Ib018564e66c29d73b30aaf50c9306dc6d67ba2b2
2013-09-30 20:22:51 +00:00
Clark Boylan
e0427cfbb7 Deploy elastic-recheck on logstash.openstack.org.
This elastic recheck thing that classifies failures based on logstash
queries is awesome. We should run it on the OpenStack developer
infrastructure. Deploy it on logstash.o.o to keep related services
together.

Change-Id: I16ae7a32c2c48474454df17931779d0f60c3ed85
2013-09-30 18:56:23 +00:00