323 Commits

Author SHA1 Message Date
David Shrewsbury
9c2fcbd8a4 Remove 'restart' for v3 executor
The proper way to shutdown the executor is to issue a 'stop' and then
wait for the executor to finish its current jobs before issuing a
'start'. Doing 'restart' doesn't ever seem to be the right thing to
do, so prevent folks from doing that.

Change-Id: I2630306a7290092c3e59f4be01a82950927bf60b
2018-02-07 10:13:50 -05:00
James E. Blair
5c759a128d Zuul v3: handle github paylods with zuul-web
Zuul-web now handles this, rather than the old webapp.

Change-Id: Ie6cf1892626097d1837870d9e4d1237d689674dc
2018-02-05 12:28:51 -08:00
James E. Blair
d44c025626 Revert "Zuulv3: Convert all proxy rules to zuul-web"
This reverts commit 1ccf68fc96cb5212c35a093339e7b61e8f34a77a.

Change-Id: I18770c65c6dc9c1e2d91e08e7e1336e06d13d974
2018-02-05 12:27:33 -08:00
James E. Blair
1ccf68fc96 Zuulv3: Convert all proxy rules to zuul-web
All Zuul v3 web services are handled by zuul-web now.

Change-Id: Id34949a364c80b86685aa5444750da0daabdf6f8
2018-02-03 07:53:16 -08:00
Jeremy Stanley
8683244caf Remove [webapp] section from v3 config
As of I8278d9ca81ed7b0a2a2189d42b8b69c5eea2bab5 the last remaining
use for the [webapp] config section (passing in the status_url) is
no longer required, so it is safe to remove in its entirety at this
point.

Change-Id: I9ceb695cf54397fa2ccbbd4e580843a3f37a997c
2018-02-03 15:44:09 +00:00
Jeremy Stanley
9753b1b889 Also include status_url in [web]
Starting with I8278d9ca81ed7b0a2a2189d42b8b69c5eea2bab5 the github
reporter looks for its status_url in the [web] section of the
configuration instead of [webapp], so put a copy there as well.

Change-Id: I55ec498e1e35832723d6ffeb62ffa06b9ee4c930
2018-02-03 01:27:24 +00:00
James E. Blair
970916b50e Add public key hosting to SSL site
It was only enabled on the non-ssl site.  Oh the irony.

Change-Id: Iaf5850e6a1c721fb93557d9c6f32e119493d9bfb
2018-01-25 13:04:15 -08:00
Monty Taylor
f9e69bc8f4
Re-enable gearman port listening check
postconditions was starting zuul, not zuul-scheduler. If we start
zuul-scheduler, then gearman should be running.

Change-Id: Icf228843495717fbb89b76b4ebad6b74d6e72b54
2018-01-21 10:19:39 -06:00
Clark Boylan
ab65041a1a Fix whitespace in ara install version checking
We have extra whitespace in the pip installed package listing that we
don't have in the upstream pypi package search so our string comparison
of the versions breaks. Fix this by trimming extra whitespace.

Change-Id: I04b28508ff9371397d42957f0d086aa9bb63a4f7
2018-01-20 18:23:33 +00:00
James E. Blair
241a72bbd6 Add TODO about fixing ara idempotency
Change-Id: If5ca46e09a59108ec6e5b68c8e23ed3a410d0fca
2018-01-20 18:23:29 +00:00
Monty Taylor
84d7483224
Add /connection to rewrite rules
We need to forward requests for /connection to zuul_status_url, which is
the scheduler port 8001.

This will need to get updated when we land the move-gh-to-zuul-web
patches.

Change-Id: I6d10e090b37218176e969128c337b0d431870d41
2018-01-20 12:22:53 -06:00
James E. Blair
737b2ff8ee
Update spec acceptance to zuulv3
This removes the test for the status page contents over HTTP because
zuul-web is not capable of providing that without a functional
configuration.

Change-Id: I53e30b053e6963e85247769ec6443439f9144582
2018-01-20 12:21:41 -06:00
Zuul
a7d59a62f0 Merge "Update fingergw init script for command socket" 2018-01-17 19:56:44 +00:00
James E. Blair
73ee7872cc Redirect curl stderr to /dev/null
It's chatty.

Change-Id: I06a167e07fa8e2ac78d7c0bd3a1c43cd56788009
2018-01-16 08:56:58 -08:00
Zuul
86034f28a6 Merge "Periodically retrieve and back up Zuul's status.json" 2018-01-16 15:25:32 +00:00
Zuul
8bf71446c4 Merge "Update merger / scheduler init script for zuulv3" 2018-01-16 15:18:29 +00:00
David Shrewsbury
2b3a4842ea Update fingergw init script for command socket
The zuul-fingergw in zuulv3 accepts the 'stop' command to stop
the daemon. This is the only valid way to stop it.

Change-Id: I86ac10c3a4d3af594fd32a5c0e316d626abf5a5e
2018-01-16 09:44:58 -05:00
David Moreau Simard
6df6a3ab81
Periodically retrieve and back up Zuul's status.json
Since the contents of the status.json is only available in-memory from
the Zuul scheduler, the contents are lost forever if the scheduler,
or the server it is hosted on, crash.

This implements two crons:
1) A curl to download a timestamped status.json, if it is available
2) File deletion so we keep no more than 180 timestamped status files
   at any time.

The files are stored at a location within the scope of the zuul-web
vhost in order to make it easily possible to dump queues with something
like:

    /opt/zuul/tools/zuul-changes.py http://zuulv3.openstack.org/backup/status_timestamp.json

Change-Id: Ia1923bfae0222559b8d2dcb5e0618848b3a618d0
2018-01-15 18:10:31 -05:00
David Shrewsbury
b74b0ace12 Start executor as 'zuul' user
No longer need to start as 'root'.

Depends-On: I6df685044c4ce81fd263043adba832609da100af
Change-Id: Ib36b6b45bb4ca147f6649a81edde2b9577d2604c
2018-01-10 13:34:43 -05:00
Clark Boylan
b513d9845e Don't assume pidfile location
When configuring the dir for the pidfiles in our init scripts don't
assume the path is /var/run/$NAME. The defaults are now /var/run/zuul
now and could be set to other values as well. Instead of assuming
/var/run/$NAME we instead take the dirname of $PIDFILE and configure
whatever that dir is instead.

Change-Id: Idfd5d19e5171278b9bb7005edb4ae927a01f3615
2018-01-03 09:26:43 -08:00
James E. Blair
22c76841cb Add support for zuul-fingergw
This is a new service which centrally proxies log streaming.

Change-Id: I1129be61c0a77dbd46e402204ea8c4da2889b016
2017-12-21 13:20:36 -08:00
James E. Blair
dcde52761b Fix typo in keys rewrite rule
Change-Id: I4bf13add2ae04df2e479258a78d822f4a61d3442
2017-12-21 09:15:30 -08:00
Zuul
40cc96cc73 Merge "Update zuulv3 rewrite rules for keys" 2017-12-19 21:56:41 +00:00
Zuul
cf62a0f430 Merge "Ensure ara is updated on executors" 2017-12-18 21:12:53 +00:00
James E. Blair
b44e05bc11 Ensure ara is updated on executors
Depends-On: Id4f6ae8cc4128083f0db72fb6e2dc81b9968e884
Change-Id: I0feb48b8b12d2346c328ab9c16aaa45dd36ae0bf
2017-12-18 19:31:32 +00:00
James E. Blair
04f80070d3 Update zuulv3 rewrite rules for keys
This is now tenant-scoped.

Change-Id: Ib85e25bc789ed31872413a2f4bc599012adb6893
2017-12-14 16:49:19 -08:00
James E. Blair
a405aafdf5 Update Zuulv3 url rewrites
This updates the rewrite rules to the current state of the zuulv3
dashboard, and removes the overlap with zuul v2 urls.

Change-Id: Idd84facca4a0e0170234bf7d4ee2b42811f52b67
2017-12-08 14:48:15 -08:00
Monty Taylor
18afb9dd6f Install angular.js and other static files for zuul web
Until we have the more generalized javascript tooling in place for zuul,
we need to add libraries directly here. The zuul web console requires
angular, so add it.

Some of the same dependencies are used in zuulv2 and zuulv3, but in a
different location.  Symlink them as appropriate

Change-Id: I75d5263ab2af3666d162b66394a537dfbe88010c
2017-12-08 14:17:38 -08:00
Paul Belanger
701acbfdbf
Update merger / scheduler init script for zuulv3
Zuulv3 now suport command socket to stop both merger and schedulers,
add logic to install new init scripts for them depending on zuulv2 or
zuulv3.

Change-Id: Iecc8c03896c145d29e42e6be6194d7db893c8515
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-12-07 15:31:05 -05:00
James E. Blair
0bfe6062a0 Zuul v3: add per-service default files
We're moving the default pidfile location in
https://review.openstack.org/517381

This change creates default files so that when running v3, the
new location is used.  The init scripts themselves are still
compatible with v2 and v3.

Change-Id: I57c486f4506114eb0447be2014baa42fc22cc8a2
2017-11-20 15:27:45 -08:00
Zuul
3fb2f3c381 Merge "Move statsd to correct zuul.conf file" 2017-10-15 23:09:56 +00:00
James E. Blair
22b49ea1e5 Move statsd to correct zuul.conf file
This was added to zuulv3, not zuulv2.

Change-Id: Ieb36a6122f9b4c30d547326e7fcd8377857e7d78
2017-10-15 15:17:37 -07:00
Jenkins
200b00c4e1 Merge "Add statsd to zuul.conf" 2017-10-13 20:40:55 +00:00
Monty Taylor
f410a47ae7
Split zuul and requirements install
We have some git-based requirements which pip installing zuul itself
doesn't get quite right.

Change-Id: I42a961d325bda9271d627c2ae4f3a6b61d8e601d
2017-10-12 17:44:24 -05:00
James E. Blair
42065b22ee Add statsd to zuul.conf
We're adding statsd support to zuul.conf; support it there and the
existing defaults file for compatability.

Change-Id: I08feb2dbb18146a1a962aa8b96e3d33e5f794459
2017-10-10 13:20:00 -07:00
James E. Blair
0ec00b15bc Add support for zookeeper timeout
Change-Id: I743bfa28d2faaa93bba6fc975d575696bd8a5a70
2017-09-28 10:40:14 -07:00
James E. Blair
6c306a2433 Add kazoo info logging
This provides information on connection state.

Change-Id: I03266e4e15697862bc2e6592b6662317ba39dcc8
2017-09-28 09:41:06 -07:00
James E. Blair
1f43ca60a9 Add gerrit_strip_branch_ref option
Change-Id: Id388274a7b628ad40a0e943087ec82596791c982
See: I012485065e0fd8692e4f2ef787dd0a62be303314
2017-09-19 07:51:34 -07:00
Monty Taylor
be57d371d8
Revert "Add debug logging for cachecontrol library"
This reverts commit 4e5378c2646d0b72ea12f9f1b8a301a007cabe65.

Change-Id: I1462a7cc9b7b2a6a1974ef13998d2310740ce2b8
2017-09-11 18:08:49 -06:00
Monty Taylor
4e5378c264
Add debug logging for cachecontrol library
It would be nice to see hits and misses along side our other things.

Change-Id: Idcd6261ca5e27d73eb9aff0d3cc8e60fbc722b1c
2017-08-30 13:14:54 -05:00
Jenkins
3ebcb481bb Merge "Depend on helper gem for spec_helper_acceptance" 2017-08-22 23:39:29 +00:00
Colleen Murphy
11ad996786 Depend on helper gem for spec_helper_acceptance
Instead of keeping a local copy of spec_helper_acceptance.rb and
requiring updates to all modules for any change, we can move it into the
common helper gem and require it from there. This will make it easier to
create and review changes that affect all puppet modules. Also change
the Gemfile to look for the gem in the local workspace if running in a
zuul environment.

Change-Id: Ie43d7fd4860f7c0f9593bdd7b5b4cd5b54bcadf4
2017-08-22 22:40:04 +02:00
Jenkins
e498f81f3f Merge "Remove zuul-executor pid file on stop" 2017-08-21 19:57:01 +00:00
Monty Taylor
d9cefce112
Add python-bpo-27945-backport PPA to executor
The python-bpo-27945-backport contains a backport of the patch that
fixes http://bugs.python.org/issue27945 and is a stop-gap while we wait
for the SRU
https://bugs.launchpad.net/ubuntu/+source/python3.5/+bug/1711724 to land
in Xenial.

Change-Id: Iaea633b8672110e49928a20d596b884c609d6345
2017-08-18 12:58:15 -05:00
James E. Blair
2216385a8d Allow executor to core dump
ansible-playbook is getting hit with sigsegv; allow the process
to core dump so we might obtain clues.

Change-Id: Ice0bbd4abea2997ee56a92544128fd136dcef048
2017-08-17 14:50:15 -07:00
Jenkins
43cf9a5990 Merge "Install ara on the executors" 2017-08-17 18:29:34 +00:00
Paul Belanger
416e683a43 Remove zuul-executor pid file on stop
Otherwise this is leaked and we are not able to start zuul-executor
again.

Change-Id: I3639a74af67bfa07ad1814b7d0bc5a995e4ef14a
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-08-17 13:58:31 -04:00
James E. Blair
a7ef39c2c0 Zuulv3: move the job dir under /var/lib/zuul
We want to ensure that our jobdir and the git repo cache are
on the same filesystem.  Configure the executors to use a dir
under /var/lib/zuul for the jobdir.  We will mount a new (large)
filesystem at /var/lib/zuul to accomodate both.

Change-Id: I91472fa609274aadcf4945aad44f25014c07b105
2017-08-16 10:10:28 -07:00
Monty Taylor
c93f2cba85
Install ara on the executors
zuul v3 supports optionally collecting data via ara if it's installed.
Install it so that we can produce ara html reports for all of our runs.

The current implementation is similar to the devstack-gate
implementation - it uses a sqlite db per job. The bubblewrap invocation
sets $HOME to the work_dir, so the sqlite db for each job goes in to the
root of the work_dir with no further config.

Change-Id: Iefeda37338c57466d671ea07e7d318fa92aafc35
Depends-On: I8facdf0b95b83d43c337058d70fe6bf71e17d570
2017-08-16 11:14:46 -05:00
Paul Belanger
0cc4c05cbd Move zuul site variables file into init.pp
Because we need to update zuul.conf with the path to the file, we
actually need to configure this variable within init.pp.

Change-Id: I734f2c338ceab2b1da2a3245423cb912b7dd8c00
Depends-On: I7983ff9cfaea1ee6d3b099824b5f31df98ce72a5
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2017-08-03 21:02:54 -04:00