Switch the launcher to Gearman.
Remove the Jenkins launcher and add a new Gearman launcher (designed to be compatible with Jenkins) in its place. See the documentation for how to set up the Gearman Plugin for Jenkins. Change-Id: Ie7224396271d7375f4ea42eebb57f883bc291738
This commit is contained in:
parent
bdafe495db
commit
1f4c2bb104
|
@ -10,8 +10,8 @@ Zuul is a program that is used to gate the source code repository of a
|
|||
project so that changes are only merged if they pass tests.
|
||||
|
||||
The main component of Zuul is the scheduler. It receives events
|
||||
related to proposed changes (currently from Gerrit), triggers tests
|
||||
based on those events (currently on Jenkins), and reports back.
|
||||
related to proposed changes, triggers tests based on those events, and
|
||||
reports back.
|
||||
|
||||
Contents:
|
||||
|
||||
|
|
|
@ -1,73 +1,74 @@
|
|||
:title: Launchers
|
||||
|
||||
.. _launchers:
|
||||
.. _Gearman: http://gearman.org/
|
||||
|
||||
.. _`Gearman Plugin`:
|
||||
https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin
|
||||
|
||||
.. _launchers:
|
||||
|
||||
Launchers
|
||||
=========
|
||||
|
||||
Zuul has a modular architecture for launching jobs. Currently only
|
||||
Jenkins is supported, but it should be fairly easy to add a module to
|
||||
support other systems. Zuul makes very few assumptions about the
|
||||
interface to a launcher -- if it can trigger jobs, cancel them, and
|
||||
receive success or failure reports, it should be able to be used with
|
||||
Zuul. Patches to this effect are welcome.
|
||||
Zuul has a modular architecture for launching jobs. Currently, the
|
||||
only supported module interfaces with Gearman_. This design allows
|
||||
any system to run jobs for Zuul simply by interfacing with a Gearman
|
||||
server. The recommended way of integrating a new job-runner with Zuul
|
||||
is via this method.
|
||||
|
||||
Jenkins
|
||||
If Gearman is unsuitable, Zuul may be extended with a new launcher
|
||||
module. Zuul makes very few assumptions about the interface to a
|
||||
launcher -- if it can trigger jobs, cancel them, and receive success
|
||||
or failure reports, it should be able to be used with Zuul. Patches
|
||||
to this effect are welcome.
|
||||
|
||||
Gearman
|
||||
-------
|
||||
|
||||
Zuul works with Jenkins using the Jenkins API and the notification
|
||||
module. It uses the Jenkins API to trigger jobs, passing in
|
||||
parameters indicating what should be tested. It recieves
|
||||
notifications on job completion via the notification API (so jobs must
|
||||
be conifigured to notify Zuul).
|
||||
Gearman_ is a general-purpose protocol for distributing jobs to any
|
||||
number of workers. Zuul works with Gearman by sending specific
|
||||
information with job requests to Gearman, and expects certain
|
||||
information to be returned on completion. This protocol is described
|
||||
in `Zuul-Gearman Protocol`_.
|
||||
|
||||
Jenkins Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul
|
||||
by providing an interface between Jenkins and Gearman. In this
|
||||
configuration, Zuul asks Gearman to run jobs, and Gearman can then
|
||||
distribute those jobs to any number of Jenkins systems (including
|
||||
multiple Jenkins masters).
|
||||
|
||||
Zuul will need access to a Jenkins user. Create a user in Jenkins,
|
||||
and then visit the configuration page for the user:
|
||||
In order for Zuul to run any jobs, you will need a running Gearman
|
||||
server. The latest version of gearmand from gearman.org is required
|
||||
in order to support canceling jobs while in the queue. The server is
|
||||
easy to set up -- just make sure that it allows connections from Zuul
|
||||
and any workers (e.g., Jenkins masters) on port 4730, and nowhere else
|
||||
(as the Gearman protocol does not include any provision for
|
||||
authentication.
|
||||
|
||||
https://jenkins.example.com/user/USERNAME/configure
|
||||
Gearman Jenkins Plugin
|
||||
----------------------
|
||||
|
||||
And click **Show API Token** to retrieve the API token for that user.
|
||||
You will need this later when configuring Zuul. Appropriate user
|
||||
permissions must be set under the Jenkins security matrix: under the
|
||||
``Global`` group of permissions, check ``Read``, then under the ``Job``
|
||||
group of permissions, check ``Read`` and ``Build``. Finally, under
|
||||
``Run`` check ``Update``. If using a per project matrix, make sure the
|
||||
user permissions are properly set for any jobs that you want Zuul to
|
||||
trigger.
|
||||
The `Gearman Plugin`_ can be installed in Jenkins in order to
|
||||
facilitate Jenkins running jobs for Zuul. Install the plugin and
|
||||
configure it with the hostname or IP address of your Gearman server
|
||||
and the port on which it is listening (4730 by default). It will
|
||||
automatically register all known Jenkins jobs as functions that Zuul
|
||||
can invoke via Gearman.
|
||||
|
||||
Make sure the notification plugin is installed. Visit the plugin
|
||||
manager on your jenkins:
|
||||
Any number of masters can be configured in this way, and Gearman will
|
||||
distribute jobs to all of them as appropriate.
|
||||
|
||||
https://jenkins.example.com/pluginManager/
|
||||
No special Jenkins job configuration is needed to support triggering
|
||||
by Zuul.
|
||||
|
||||
And install **Jenkins Notification plugin**. The homepage for the
|
||||
plugin is at:
|
||||
Zuul Parameters
|
||||
---------------
|
||||
|
||||
https://wiki.jenkins-ci.org/display/JENKINS/Notification+Plugin
|
||||
|
||||
Jenkins Job Configuration
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
For each job that you want Zuul to trigger, you will need to add a
|
||||
notification endpoint for the job on that job's configuration page.
|
||||
Click **Add Endpoint** and enter the following values:
|
||||
|
||||
**Protocol**
|
||||
``HTTP``
|
||||
**URL**
|
||||
``http://127.0.0.1:8001/jenkins_endpoint``
|
||||
|
||||
If you are running Zuul on a different server than Jenkins, enter the
|
||||
appropriate URL. Note that Zuul itself has no access controls, so
|
||||
ensure that only Jenkins is permitted to access that URL.
|
||||
|
||||
Zuul will pass some parameters to Jenkins for every job it launches.
|
||||
Check **This build is parameterized**, and add the following fields
|
||||
with the type **String Parameter**:
|
||||
Zuul will pass some parameters with every job it launches. The
|
||||
Gearman Plugin will ensure these are supplied as Jenkins build
|
||||
parameters, so they will be available for use in the job configuration
|
||||
as well as to the running job as environment variables. They are as
|
||||
follows:
|
||||
|
||||
**ZUUL_UUID**
|
||||
Zuul provided key to link builds with Gerrit events
|
||||
|
@ -75,27 +76,14 @@ with the type **String Parameter**:
|
|||
Zuul provided ref that includes commit(s) to build
|
||||
**ZUUL_COMMIT**
|
||||
The commit SHA1 at the head of ZUUL_REF
|
||||
|
||||
Those are the only required parameters. The ZUUL_UUID is needed for Zuul to
|
||||
keep track of the build, and the ZUUL_REF and ZUUL_COMMIT parameters are for
|
||||
use in preparing the git repo for the build.
|
||||
|
||||
.. note::
|
||||
The GERRIT_PROJECT and UUID parameters are deprecated respectively in
|
||||
favor of ZUUL_PROJECT and ZUUL_UUID.
|
||||
|
||||
The following parameters will be sent for all builds, but are not required so
|
||||
you do not need to configure Jenkins to accept them if you do not plan on using
|
||||
them:
|
||||
|
||||
**ZUUL_PROJECT**
|
||||
The project that triggered this build
|
||||
**ZUUL_PIPELINE**
|
||||
The Zuul pipeline that is building this job
|
||||
|
||||
The following parameters are optional and will only be provided for
|
||||
builds associated with changes (i.e., in response to patchset-created
|
||||
or comment-added events):
|
||||
The following additional parameters will only be provided for builds
|
||||
associated with changes (i.e., in response to patchset-created or
|
||||
comment-added events):
|
||||
|
||||
**ZUUL_BRANCH**
|
||||
The target branch for the change that triggered this build
|
||||
|
@ -107,7 +95,7 @@ or comment-added events):
|
|||
**ZUUL_PATCHSET**
|
||||
The Gerrit patchset number for the change that triggered this build
|
||||
|
||||
The following parameters are optional and will only be provided for
|
||||
The following additional parameters will only be provided for
|
||||
post-merge (ref-updated) builds:
|
||||
|
||||
**ZUUL_OLDREV**
|
||||
|
@ -139,7 +127,107 @@ That should be sufficient for a job that only builds a single project.
|
|||
If you have multiple interrelated projects (i.e., they share a Zuul
|
||||
Change Queue) that are built together, you may be able to configure
|
||||
the Git plugin to prepare them, or you may chose to use a shell script
|
||||
instead. The OpenStack project uses the following script to prepare
|
||||
the workspace for its integration testing:
|
||||
instead. As an example, the OpenStack project uses the following
|
||||
script to prepare the workspace for its integration testing:
|
||||
|
||||
https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh
|
||||
|
||||
|
||||
Zuul-Gearman Protocol
|
||||
---------------------
|
||||
|
||||
This section is only relevant if you intend to implement a new kind of
|
||||
worker that runs jobs for Zuul via Gearman. If you just want to use
|
||||
Jenkins, see `Gearman Jenkins Plugin`_ instead.
|
||||
|
||||
The Zuul protocol as used with Gearman is as follows:
|
||||
|
||||
Starting Builds
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
To start a build, Zuul invokes a Gearman function with the following
|
||||
format:
|
||||
|
||||
build:FUNCTION_NAME
|
||||
|
||||
where **FUNCTION_NAME** is the name of the job that should be run. If
|
||||
the job should run on a specific node (or class of node), Zuul will
|
||||
instead invoke:
|
||||
|
||||
build:FUNCTION_NAME:NODE_NAME
|
||||
|
||||
where **NODE_NAME** is the name or class of node on which the job
|
||||
should be run. This can be specified by setting the ZUUL_NODE
|
||||
parameter in a paremeter-function (see :ref:`zuulconf`).
|
||||
|
||||
Zuul sends the ZUUL_* parameters described in `Zuul Parameters`_
|
||||
encoded in JSON format as the argument included with the
|
||||
SUBMIT_JOB_UNIQ request to Gearman. A unique ID (equal to the
|
||||
ZUUL_UUID parameter) is also supplied to Gearman, and is accessible as
|
||||
an added Gearman parameter with GRAB_JOB_UNIQ.
|
||||
|
||||
When a Gearman worker starts running a job for Zuul, it should
|
||||
immediately send a WORK_DATA packet with the following information
|
||||
encoded in JSON format:
|
||||
|
||||
**full_url**
|
||||
The URL with the status or results of the build. Will be used in
|
||||
the status page and the final report.
|
||||
|
||||
**number**
|
||||
The build number (unique to this job).
|
||||
|
||||
**master**
|
||||
A unique identifier associated with the Gearman worker that can
|
||||
abort this build. See `Stopping Builds`_ for more information.
|
||||
|
||||
It should then immediately send a WORK_STATUS packet with a value of 0
|
||||
percent complete. It may then optionally send subsequent WORK_STATUS
|
||||
packets with updated completion values.
|
||||
|
||||
When the build is complete, it should send a final WORK_DATA packet
|
||||
with the following in JSON format:
|
||||
|
||||
**result**
|
||||
Either the string 'SUCCESS' if the job succeeded, or any other value
|
||||
that describes the result if the job failed.
|
||||
|
||||
Finally, it should send either a WORK_FAIL or WORK_COMPLETE packet as
|
||||
appropriate. A WORK_EXCEPTION packet will be interpreted as a
|
||||
WORK_FAIL, but the exception will be logged in Zuul's error log.
|
||||
|
||||
Stopping Builds
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
If Zuul needs to abort a build already in progress, it will invoke the
|
||||
following function through Gearman:
|
||||
|
||||
stop:MASTER_NAME
|
||||
|
||||
Where **MASTER_NAME** is the name of the master node supplied in the
|
||||
initial WORK_DATA packet when the job started. This is used to direct
|
||||
the stop: function invocation to the correct Gearman worker that is
|
||||
capable of stopping that particular job. The argument to the function
|
||||
will be the unique ID of the job that should be stopped.
|
||||
|
||||
The original job is expected to complete with a WORK_DATA and
|
||||
WORK_FAIL packet as described in `Starting Builds`_.
|
||||
|
||||
Build Descriptions
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In order to update the job running system with a description of the
|
||||
current state of all related builds, the job runner may optionally
|
||||
implement the following Gearman function:
|
||||
|
||||
set_description:MASTER_NAME
|
||||
|
||||
Where **MASTER_NAME** is used as described in `Stopping Builds`_. The
|
||||
argument to the function is the following encoded in JSON format:
|
||||
|
||||
**unique_id**
|
||||
The unique identifier of the build whose description should be
|
||||
updated.
|
||||
|
||||
**html_description**
|
||||
The description of the build in HTML format.
|
||||
|
|
|
@ -9,7 +9,8 @@ Configuration
|
|||
Zuul has three configuration files:
|
||||
|
||||
**zuul.conf**
|
||||
Credentials for Gerrit and Jenkins, locations of the other config files
|
||||
Connection information for Gerrit and Gearman, locations of the
|
||||
other config files
|
||||
**layout.yaml**
|
||||
Project and pipeline configuration -- what Zuul does
|
||||
**logging.conf**
|
||||
|
@ -27,30 +28,26 @@ Zuul will look for ``/etc/zuul/zuul.conf`` or ``~/zuul.conf`` to
|
|||
bootstrap its configuration. Alternately, you may specify ``-c
|
||||
/path/to/zuul.conf`` on the command line.
|
||||
|
||||
Gerrit and Jenkins credentials are each described in a section of
|
||||
zuul.conf. The location of the other two configuration files (as well
|
||||
as the location of the PID file when running Zuul as a server) are
|
||||
specified in a third section.
|
||||
Gerrit and Gearman connection information are each described in a
|
||||
section of zuul.conf. The location of the other two configuration
|
||||
files (as well as the location of the PID file when running Zuul as a
|
||||
server) are specified in a third section.
|
||||
|
||||
The three sections of this config and their options are documented below.
|
||||
You can also find an example zuul.conf file in the git
|
||||
`repository
|
||||
<https://github.com/openstack-infra/zuul/blob/master/etc/zuul.conf-sample>`_
|
||||
|
||||
jenkins
|
||||
gearman
|
||||
"""""""
|
||||
|
||||
**server**
|
||||
URL for the root of the Jenkins HTTP server.
|
||||
``server=https://jenkins.example.com``
|
||||
Hostname or IP address of the Gearman server.
|
||||
``server=gearman.example.com``
|
||||
|
||||
**user**
|
||||
User to authenticate against Jenkins with.
|
||||
``user=jenkins``
|
||||
|
||||
**apikey**
|
||||
Jenkins API Key credentials for the above user.
|
||||
``apikey=1234567890abcdef1234567890abcdef``
|
||||
**port**
|
||||
Port on which the Gearman server is listening
|
||||
``port=4730``
|
||||
|
||||
gerrit
|
||||
""""""
|
||||
|
@ -65,11 +62,11 @@ gerrit
|
|||
|
||||
**user**
|
||||
User name to use when logging into above server via ssh.
|
||||
``user=jenkins``
|
||||
``user=zuul``
|
||||
|
||||
**sshkey**
|
||||
Path to SSH key to use when logging into above server.
|
||||
``sshkey=/home/jenkins/.ssh/id_rsa``
|
||||
``sshkey=/home/zuul/.ssh/id_rsa``
|
||||
|
||||
zuul
|
||||
""""
|
||||
|
@ -107,13 +104,14 @@ zuul
|
|||
|
||||
**status_url**
|
||||
URL that will be posted in Zuul comments made to Gerrit changes when
|
||||
beginning Jenkins jobs for a change.
|
||||
``status_url=https://jenkins.example.com/zuul/status``
|
||||
starting jobs for a change.
|
||||
``status_url=https://zuul.example.com/status``
|
||||
|
||||
**url_pattern**
|
||||
If you are storing build logs external to Jenkins and wish to link to
|
||||
those logs when Zuul makes comments on Gerrit changes for completed
|
||||
jobs this setting configures what the URLs for those links should be.
|
||||
If you are storing build logs external to the system that originally
|
||||
ran jobs and wish to link to those logs when Zuul makes comments on
|
||||
Gerrit changes for completed jobs this setting configures what the
|
||||
URLs for those links should be.
|
||||
``http://logs.example.com/{change.number}/{change.patchset}/{pipeline.name}/{job.name}/{build.number}``
|
||||
|
||||
layout.yaml
|
||||
|
@ -410,13 +408,13 @@ each job as it builds a list from the project specification.
|
|||
|
||||
**failure-pattern (optional)**
|
||||
The URL that should be reported to Gerrit if the job fails.
|
||||
Defaults to the Jenkins build URL or the url_pattern configured in
|
||||
Defaults to the build URL or the url_pattern configured in
|
||||
zuul.conf. May be supplied as a string pattern with substitutions
|
||||
as described in url_pattern in :ref:`zuulconf`.
|
||||
|
||||
**success-pattern (optional)**
|
||||
The URL that should be reported to Gerrit if the job succeeds.
|
||||
Defaults to the Jenkins build URL or the url_pattern configured in
|
||||
Defaults to the build URL or the url_pattern configured in
|
||||
zuul.conf. May be supplied as a string pattern with substitutions
|
||||
as described in url_pattern in :ref:`zuulconf`.
|
||||
|
||||
|
@ -461,6 +459,10 @@ each job as it builds a list from the project specification.
|
|||
:param parameters: parameters to be passed to the job
|
||||
:type parameters: dict
|
||||
|
||||
If the parameter **ZUUL_NODE** is set by this function, then it will
|
||||
be used to specify on what node (or class of node) the job should be
|
||||
run.
|
||||
|
||||
Here is an example of setting the failure message for jobs that check
|
||||
whether a change merges cleanly::
|
||||
|
||||
|
|
|
@ -0,0 +1,2 @@
|
|||
def select_debian_node(change, params):
|
||||
params['ZUUL_NODE'] = 'debian'
|
|
@ -1,3 +1,6 @@
|
|||
includes:
|
||||
- python-file: custom_functions.py
|
||||
|
||||
pipelines:
|
||||
- name: check
|
||||
manager: IndependentPipelineManager
|
||||
|
@ -46,6 +49,8 @@ jobs:
|
|||
- name: project-testfile
|
||||
files:
|
||||
- '.*-requires'
|
||||
- name: node-project-test1
|
||||
parameter-function: select_debian_node
|
||||
|
||||
project-templates:
|
||||
- name: test-one-and-two
|
||||
|
@ -135,3 +140,9 @@ projects:
|
|||
template:
|
||||
- name: test-one-and-two
|
||||
projectname: project
|
||||
|
||||
- name: org/node-project
|
||||
gate:
|
||||
- node-project-merge:
|
||||
- node-project-test1
|
||||
- node-project-test2
|
||||
|
|
|
@ -1,7 +1,5 @@
|
|||
[jenkins]
|
||||
server=https://jenkins.example.com
|
||||
user=jenkins
|
||||
apikey=1234
|
||||
[gearman]
|
||||
server=127.0.0.1
|
||||
|
||||
[gerrit]
|
||||
server=review.example.com
|
||||
|
|
File diff suppressed because it is too large
Load Diff
|
@ -9,3 +9,4 @@ python-daemon
|
|||
extras
|
||||
statsd>=1.0.0,<3.0
|
||||
voluptuous>=0.6,<0.7
|
||||
http://tarballs.openstack.org/gear/gear-master.tar.gz#egg=gear
|
||||
|
|
4
tox.ini
4
tox.ini
|
@ -7,13 +7,14 @@ setenv = STATSD_HOST=localhost
|
|||
STATSD_PORT=8125
|
||||
deps = -r{toxinidir}/tools/pip-requires
|
||||
-r{toxinidir}/tools/test-requires
|
||||
commands = nosetests {posargs}
|
||||
commands = nosetests --logging-format="%(asctime)s %(name)-32s %(levelname)-8s %(message)s" {posargs}
|
||||
|
||||
[tox:jenkins]
|
||||
downloadcache = ~/cache/pip
|
||||
|
||||
[testenv:pep8]
|
||||
deps = pep8==1.3.3
|
||||
-r{toxinidir}/tools/pip-requires
|
||||
commands = pep8 --ignore=E123,E125,E128 --repeat --show-source --exclude=.venv,.tox,dist,doc,build .
|
||||
|
||||
[testenv:cover]
|
||||
|
@ -21,6 +22,7 @@ setenv = NOSE_WITH_COVERAGE=1
|
|||
|
||||
[testenv:pyflakes]
|
||||
deps = pyflakes
|
||||
-r{toxinidir}/tools/pip-requires
|
||||
commands = pyflakes zuul setup.py
|
||||
|
||||
[testenv:venv]
|
||||
|
|
|
@ -1,3 +1,4 @@
|
|||
#!/usr/bin/env python
|
||||
# Copyright 2012 Hewlett-Packard Development Company, L.P.
|
||||
# Copyright 2013 OpenStack Foundation
|
||||
#
|
||||
|
@ -88,7 +89,7 @@ class Server(object):
|
|||
def test_config(self):
|
||||
# See comment at top of file about zuul imports
|
||||
import zuul.scheduler
|
||||
import zuul.launcher.jenkins
|
||||
import zuul.launcher.gearman
|
||||
import zuul.trigger.gerrit
|
||||
|
||||
logging.basicConfig(level=logging.DEBUG)
|
||||
|
@ -98,20 +99,24 @@ class Server(object):
|
|||
def main(self):
|
||||
# See comment at top of file about zuul imports
|
||||
import zuul.scheduler
|
||||
import zuul.launcher.jenkins
|
||||
import zuul.launcher.gearman
|
||||
import zuul.trigger.gerrit
|
||||
import zuul.webapp
|
||||
|
||||
self.sched = zuul.scheduler.Scheduler()
|
||||
|
||||
jenkins = zuul.launcher.jenkins.Jenkins(self.config, self.sched)
|
||||
gearman = zuul.launcher.gearman.Gearman(self.config, self.sched)
|
||||
gerrit = zuul.trigger.gerrit.Gerrit(self.config, self.sched)
|
||||
webapp = zuul.webapp.WebApp(self.sched)
|
||||
|
||||
self.sched.setLauncher(jenkins)
|
||||
self.sched.setLauncher(gearman)
|
||||
self.sched.setTrigger(gerrit)
|
||||
|
||||
self.sched.start()
|
||||
self.sched.reconfigure(self.config)
|
||||
self.sched.resume()
|
||||
webapp.start()
|
||||
|
||||
signal.signal(signal.SIGHUP, self.reconfigure_handler)
|
||||
signal.signal(signal.SIGUSR1, self.exit_handler)
|
||||
while True:
|
||||
|
@ -168,3 +173,8 @@ def main():
|
|||
with daemon.DaemonContext(pidfile=pid):
|
||||
server.setup_logging()
|
||||
server.main()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.path.insert(0, '.')
|
||||
main()
|
||||
|
|
|
@ -0,0 +1,387 @@
|
|||
# Copyright 2012 Hewlett-Packard Development Company, L.P.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import gear
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
import threading
|
||||
from uuid import uuid4
|
||||
|
||||
from zuul.model import Build
|
||||
|
||||
|
||||
class GearmanCleanup(threading.Thread):
|
||||
""" A thread that checks to see if outstanding builds have
|
||||
completed without reporting back. """
|
||||
log = logging.getLogger("zuul.JenkinsCleanup")
|
||||
|
||||
def __init__(self, gearman):
|
||||
threading.Thread.__init__(self)
|
||||
self.gearman = gearman
|
||||
self.wake_event = threading.Event()
|
||||
self._stopped = False
|
||||
|
||||
def stop(self):
|
||||
self._stopped = True
|
||||
self.wake_event.set()
|
||||
|
||||
def run(self):
|
||||
while True:
|
||||
self.wake_event.wait(300)
|
||||
if self._stopped:
|
||||
return
|
||||
try:
|
||||
self.gearman.lookForLostBuilds()
|
||||
except:
|
||||
self.log.exception("Exception checking builds:")
|
||||
|
||||
|
||||
def getJobData(job):
|
||||
if not len(job.data):
|
||||
return {}
|
||||
d = job.data[-1]
|
||||
if not d:
|
||||
return {}
|
||||
return json.loads(d)
|
||||
|
||||
|
||||
class ZuulGearmanClient(gear.Client):
|
||||
def __init__(self, zuul_gearman):
|
||||
super(ZuulGearmanClient, self).__init__()
|
||||
self.__zuul_gearman = zuul_gearman
|
||||
|
||||
def handleWorkComplete(self, packet):
|
||||
job = super(ZuulGearmanClient, self).handleWorkComplete(packet)
|
||||
self.__zuul_gearman.onBuildCompleted(job)
|
||||
return job
|
||||
|
||||
def handleWorkFail(self, packet):
|
||||
job = super(ZuulGearmanClient, self).handleWorkFail(packet)
|
||||
self.__zuul_gearman.onBuildCompleted(job)
|
||||
return job
|
||||
|
||||
def handleWorkStatus(self, packet):
|
||||
job = super(ZuulGearmanClient, self).handleWorkStatus(packet)
|
||||
self.__zuul_gearman.onWorkStatus(job)
|
||||
return job
|
||||
|
||||
def handleWorkData(self, packet):
|
||||
job = super(ZuulGearmanClient, self).handleWorkData(packet)
|
||||
self.__zuul_gearman.onWorkStatus(job)
|
||||
return job
|
||||
|
||||
def handleDisconnect(self, job):
|
||||
job = super(ZuulGearmanClient, self).handleDisconnect(job)
|
||||
self.__zuul_gearman.onDisconnect(job)
|
||||
|
||||
def handleStatusRes(self, packet):
|
||||
try:
|
||||
job = super(ZuulGearmanClient, self).handleStatusRes(packet)
|
||||
except gear.UnknownJobError:
|
||||
handle = packet.getArgument(0)
|
||||
for build in self.__zuul_gearman.builds:
|
||||
if build.__gearman_job.handle == handle:
|
||||
self.__zuul_gearman.onUnknownJob(job)
|
||||
|
||||
|
||||
class Gearman(object):
|
||||
log = logging.getLogger("zuul.Gearman")
|
||||
negative_function_cache_ttl = 5
|
||||
|
||||
def __init__(self, config, sched):
|
||||
self.sched = sched
|
||||
self.builds = {}
|
||||
self.meta_jobs = {} # A list of meta-jobs like stop or describe
|
||||
server = config.get('gearman', 'server')
|
||||
if config.has_option('gearman', 'port'):
|
||||
port = config.get('gearman', 'port')
|
||||
else:
|
||||
port = 4730
|
||||
|
||||
self.gearman = ZuulGearmanClient(self)
|
||||
self.gearman.addServer(server, port)
|
||||
|
||||
self.cleanup_thread = GearmanCleanup(self)
|
||||
self.cleanup_thread.start()
|
||||
self.function_cache = set()
|
||||
self.function_cache_time = 0
|
||||
|
||||
def stop(self):
|
||||
self.log.debug("Stopping")
|
||||
self.cleanup_thread.stop()
|
||||
self.cleanup_thread.join()
|
||||
self.gearman.shutdown()
|
||||
self.log.debug("Stopped")
|
||||
|
||||
def isJobRegistered(self, name):
|
||||
if self.function_cache_time:
|
||||
for connection in self.gearman.active_connections:
|
||||
if connection.connect_time > self.function_cache_time:
|
||||
self.function_cache = set()
|
||||
self.function_cache_time = 0
|
||||
break
|
||||
if name in self.function_cache:
|
||||
self.log.debug("Function %s is registered" % name)
|
||||
return True
|
||||
if ((time.time() - self.function_cache_time) <
|
||||
self.negative_function_cache_ttl):
|
||||
self.log.debug("Function %s is not registered "
|
||||
"(negative ttl in effect)" % name)
|
||||
return False
|
||||
self.function_cache_time = time.time()
|
||||
for connection in self.gearman.active_connections:
|
||||
try:
|
||||
req = gear.StatusAdminRequest()
|
||||
connection.sendAdminRequest(req)
|
||||
req.waitForResponse()
|
||||
except Exception:
|
||||
self.log.exception("Exception while checking functions")
|
||||
continue
|
||||
for line in req.response.split('\n'):
|
||||
parts = [x.strip() for x in line.split()]
|
||||
if not parts or parts[0] == '.':
|
||||
continue
|
||||
self.function_cache.add(parts[0])
|
||||
if name in self.function_cache:
|
||||
self.log.debug("Function %s is registered" % name)
|
||||
return True
|
||||
self.log.debug("Function %s is not registered" % name)
|
||||
return False
|
||||
|
||||
def launch(self, job, change, pipeline, dependent_changes=[]):
|
||||
self.log.info("Launch job %s for change %s with dependent changes %s" %
|
||||
(job, change, dependent_changes))
|
||||
dependent_changes = dependent_changes[:]
|
||||
dependent_changes.reverse()
|
||||
uuid = str(uuid4().hex)
|
||||
params = dict(ZUUL_UUID=uuid,
|
||||
ZUUL_PROJECT=change.project.name)
|
||||
params['ZUUL_PIPELINE'] = pipeline.name
|
||||
if hasattr(change, 'refspec'):
|
||||
changes_str = '^'.join(
|
||||
['%s:%s:%s' % (c.project.name, c.branch, c.refspec)
|
||||
for c in dependent_changes + [change]])
|
||||
params['ZUUL_BRANCH'] = change.branch
|
||||
params['ZUUL_CHANGES'] = changes_str
|
||||
params['ZUUL_REF'] = ('refs/zuul/%s/%s' %
|
||||
(change.branch,
|
||||
change.current_build_set.ref))
|
||||
params['ZUUL_COMMIT'] = change.current_build_set.commit
|
||||
|
||||
zuul_changes = ' '.join(['%s,%s' % (c.number, c.patchset)
|
||||
for c in dependent_changes + [change]])
|
||||
params['ZUUL_CHANGE_IDS'] = zuul_changes
|
||||
params['ZUUL_CHANGE'] = str(change.number)
|
||||
params['ZUUL_PATCHSET'] = str(change.patchset)
|
||||
if hasattr(change, 'ref'):
|
||||
params['ZUUL_REFNAME'] = change.ref
|
||||
params['ZUUL_OLDREV'] = change.oldrev
|
||||
params['ZUUL_NEWREV'] = change.newrev
|
||||
params['ZUUL_SHORT_OLDREV'] = change.oldrev[:7]
|
||||
params['ZUUL_SHORT_NEWREV'] = change.newrev[:7]
|
||||
|
||||
params['ZUUL_REF'] = change.ref
|
||||
params['ZUUL_COMMIT'] = change.newrev
|
||||
|
||||
# This is what we should be heading toward for parameters:
|
||||
|
||||
# required:
|
||||
# ZUUL_UUID
|
||||
# ZUUL_REF (/refs/zuul/..., /refs/tags/foo, master)
|
||||
# ZUUL_COMMIT
|
||||
|
||||
# optional:
|
||||
# ZUUL_PROJECT
|
||||
# ZUUL_PIPELINE
|
||||
|
||||
# optional (changes only):
|
||||
# ZUUL_BRANCH
|
||||
# ZUUL_CHANGE
|
||||
# ZUUL_CHANGE_IDS
|
||||
# ZUUL_PATCHSET
|
||||
|
||||
# optional (ref updated only):
|
||||
# ZUUL_OLDREV
|
||||
# ZUUL_NEWREV
|
||||
# ZUUL_SHORT_NEWREV
|
||||
# ZUUL_SHORT_OLDREV
|
||||
|
||||
if callable(job.parameter_function):
|
||||
job.parameter_function(change, params)
|
||||
self.log.debug("Custom parameter function used for job %s, "
|
||||
"change: %s, params: %s" % (job, change, params))
|
||||
|
||||
if 'ZUUL_NODE' in params:
|
||||
name = "build:%s:%s" % (job.name, params['ZUUL_NODE'])
|
||||
else:
|
||||
name = "build:%s" % job.name
|
||||
build = Build(job, uuid)
|
||||
|
||||
gearman_job = gear.Job(name, json.dumps(params),
|
||||
unique=uuid)
|
||||
build.__gearman_job = gearman_job
|
||||
self.builds[uuid] = build
|
||||
|
||||
if not self.isJobRegistered(gearman_job.name):
|
||||
self.log.error("Job %s is not registered with Gearman" %
|
||||
gearman_job)
|
||||
self.onBuildCompleted(gearman_job, 'LOST')
|
||||
return build
|
||||
|
||||
try:
|
||||
self.gearman.submitJob(gearman_job)
|
||||
except Exception:
|
||||
self.log.exception("Unable to submit job to Gearman")
|
||||
self.onBuildCompleted(gearman_job, 'LOST')
|
||||
return build
|
||||
|
||||
gearman_job.waitForHandle(30)
|
||||
if not gearman_job.handle:
|
||||
self.log.error("No job handle was received for %s after 30 seconds"
|
||||
" marking as lost." %
|
||||
gearman_job)
|
||||
self.onBuildCompleted(gearman_job, 'LOST')
|
||||
|
||||
return build
|
||||
|
||||
def cancel(self, build):
|
||||
self.log.info("Cancel build %s for job %s" % (build, build.job))
|
||||
|
||||
if build.number:
|
||||
self.log.debug("Build %s has already started" % build)
|
||||
self.cancelRunningBuild(build)
|
||||
self.log.debug("Canceled running build %s" % build)
|
||||
return
|
||||
else:
|
||||
self.log.debug("Build %s has not started yet" % build)
|
||||
|
||||
self.log.debug("Looking for build %s in queue" % build)
|
||||
if self.cancelJobInQueue(build):
|
||||
self.log.debug("Removed build %s from queue" % build)
|
||||
return
|
||||
|
||||
self.log.debug("Still unable to find build %s to cancel" % build)
|
||||
if build.number:
|
||||
self.log.debug("Build %s has just started" % build)
|
||||
self.cancelRunningBuild(build)
|
||||
self.log.debug("Canceled just running build %s" % build)
|
||||
else:
|
||||
self.log.error("Build %s has not started but "
|
||||
"was not found in queue" % build)
|
||||
|
||||
def onBuildCompleted(self, job, result=None):
|
||||
if job.unique in self.meta_jobs:
|
||||
del self.meta_jobs[job.unique]
|
||||
return
|
||||
|
||||
build = self.builds.get(job.unique)
|
||||
if build:
|
||||
if result is None:
|
||||
data = getJobData(job)
|
||||
result = data.get('result')
|
||||
self.log.info("Build %s complete, result %s" %
|
||||
(job, result))
|
||||
build.result = result
|
||||
self.sched.onBuildCompleted(build)
|
||||
# The test suite expects the build to be removed from the
|
||||
# internal dict after it's added to the report queue.
|
||||
del self.builds[job.unique]
|
||||
else:
|
||||
if not job.name.startswith("stop:"):
|
||||
self.log.error("Unable to find build %s" % job.unique)
|
||||
|
||||
def onWorkStatus(self, job):
|
||||
data = getJobData(job)
|
||||
self.log.info("Build %s update" % job)
|
||||
build = self.builds.get(job.unique)
|
||||
if build:
|
||||
self.log.debug("Found build %s" % build)
|
||||
if not build.number:
|
||||
self.log.info("Build %s started" % job)
|
||||
build.url = data.get('full_url')
|
||||
build.number = data.get('number')
|
||||
build.__gearman_master = data.get('master')
|
||||
self.sched.onBuildStarted(build)
|
||||
build.fraction_complete = job.fraction_complete
|
||||
else:
|
||||
self.log.error("Unable to find build %s" % job.unique)
|
||||
|
||||
def onDisconnect(self, job):
|
||||
self.log.info("Gearman job %s lost due to disconnect" % job)
|
||||
self.onBuildCompleted(job, 'LOST')
|
||||
|
||||
def onUnknownJob(self, job):
|
||||
self.log.info("Gearman job %s lost due to unknown handle" % job)
|
||||
self.onBuildCompleted(job, 'LOST')
|
||||
|
||||
def cancelJobInQueue(self, build):
|
||||
job = build.__gearman_job
|
||||
|
||||
req = gear.CancelJobAdminRequest(job.handle)
|
||||
job.connection.sendAdminRequest(req)
|
||||
req.waitForResponse()
|
||||
self.log.debug("Response to cancel build %s request: %s" %
|
||||
(build, req.response.strip()))
|
||||
if req.response.startswith("OK"):
|
||||
try:
|
||||
del self.builds[job.unique]
|
||||
except:
|
||||
pass
|
||||
return True
|
||||
return False
|
||||
|
||||
def cancelRunningBuild(self, build):
|
||||
stop_uuid = str(uuid4().hex)
|
||||
stop_job = gear.Job("stop:%s" % build.__gearman_master,
|
||||
build.uuid,
|
||||
unique=stop_uuid)
|
||||
self.meta_jobs[stop_uuid] = stop_job
|
||||
self.log.debug("Submitting stop job: %s", stop_job)
|
||||
self.gearman.submitJob(stop_job)
|
||||
return True
|
||||
|
||||
def setBuildDescription(self, build, desc):
|
||||
try:
|
||||
name = "set_description:%s" % build.__gearman_master
|
||||
except AttributeError:
|
||||
# We haven't yet received the first data packet that tells
|
||||
# us where the job is running.
|
||||
return False
|
||||
|
||||
if not self.isJobRegistered(name):
|
||||
return False
|
||||
|
||||
desc_uuid = str(uuid4().hex)
|
||||
data = dict(unique_id=build.uuid,
|
||||
html_description=desc)
|
||||
desc_job = gear.Job(name, json.dumps(data), unique=desc_uuid)
|
||||
self.meta_jobs[desc_uuid] = desc_job
|
||||
self.log.debug("Submitting describe job: %s", desc_job)
|
||||
self.gearman.submitJob(desc_job)
|
||||
return True
|
||||
|
||||
def lookForLostBuilds(self):
|
||||
self.log.debug("Looking for lost builds")
|
||||
for build in self.builds.values():
|
||||
if build.result:
|
||||
# The build has finished, it will be removed
|
||||
continue
|
||||
job = build.__gearman_job
|
||||
if not job.handle:
|
||||
# The build hasn't been enqueued yet
|
||||
continue
|
||||
p = gear.Packet(gear.constants.REQ, gear.constants.GET_STATUS,
|
||||
job.handle)
|
||||
job.connection.sendPacket(p)
|
|
@ -1,499 +0,0 @@
|
|||
# Copyright 2012 Hewlett-Packard Development Company, L.P.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
# So we can name this module "jenkins" and still load the "jenkins"
|
||||
# system module
|
||||
from __future__ import absolute_import
|
||||
|
||||
import json
|
||||
import logging
|
||||
import pprint
|
||||
import threading
|
||||
import time
|
||||
import urllib # for extending jenkins lib
|
||||
import urllib2 # for extending jenkins lib
|
||||
from uuid import uuid4
|
||||
|
||||
import jenkins
|
||||
from paste import httpserver
|
||||
from webob import Request
|
||||
|
||||
from zuul.model import Build
|
||||
|
||||
# The amount of time we tolerate a change in build status without
|
||||
# receiving a notification
|
||||
JENKINS_GRACE_TIME = 60
|
||||
|
||||
|
||||
class JenkinsCallback(threading.Thread):
|
||||
log = logging.getLogger("zuul.JenkinsCallback")
|
||||
|
||||
def __init__(self, jenkins):
|
||||
threading.Thread.__init__(self)
|
||||
self.jenkins = jenkins
|
||||
|
||||
def run(self):
|
||||
httpserver.serve(self.app, host='0.0.0.0', port='8001')
|
||||
|
||||
def app(self, environ, start_response):
|
||||
request = Request(environ)
|
||||
if request.path == '/jenkins_endpoint':
|
||||
self.jenkins_endpoint(request)
|
||||
start_response('200 OK', [('content-type', 'text/html')])
|
||||
return ['Zuul good.']
|
||||
elif request.path == '/status':
|
||||
try:
|
||||
ret = self.jenkins.sched.formatStatusHTML()
|
||||
except:
|
||||
self.log.exception("Exception formatting status:")
|
||||
raise
|
||||
start_response('200 OK', [('content-type', 'text/html')])
|
||||
return [ret]
|
||||
elif request.path == '/status.json':
|
||||
try:
|
||||
ret = self.jenkins.sched.formatStatusJSON()
|
||||
except:
|
||||
self.log.exception("Exception formatting status:")
|
||||
raise
|
||||
start_response('200 OK', [('content-type', 'application/json'),
|
||||
('Access-Control-Allow-Origin', '*')])
|
||||
return [ret]
|
||||
else:
|
||||
start_response('200 OK', [('content-type', 'text/html')])
|
||||
return ['Zuul good.']
|
||||
|
||||
def jenkins_endpoint(self, request):
|
||||
try:
|
||||
data = json.loads(request.body)
|
||||
except:
|
||||
self.log.exception("Exception handling Jenkins notification:")
|
||||
raise # let wsgi handler process the issue
|
||||
if data:
|
||||
self.log.debug("Received data from Jenkins: \n%s" %
|
||||
(pprint.pformat(data)))
|
||||
build = data.get('build')
|
||||
if build:
|
||||
phase = build.get('phase')
|
||||
status = build.get('status')
|
||||
url = build.get('full_url')
|
||||
number = build.get('number')
|
||||
params = build.get('parameters')
|
||||
if params:
|
||||
# UUID is deprecated in favor of ZUUL_UUID
|
||||
uuid = params.get('ZUUL_UUID') or params.get('UUID')
|
||||
if (status and url and uuid and phase and
|
||||
phase == 'COMPLETED'):
|
||||
self.jenkins.onBuildCompleted(uuid,
|
||||
status,
|
||||
url,
|
||||
number)
|
||||
if (phase and phase == 'STARTED'):
|
||||
self.jenkins.onBuildStarted(uuid, url, number)
|
||||
|
||||
|
||||
class JenkinsCleanup(threading.Thread):
|
||||
""" A thread that checks to see if outstanding builds have
|
||||
completed without reporting back. """
|
||||
log = logging.getLogger("zuul.JenkinsCleanup")
|
||||
|
||||
def __init__(self, jenkins):
|
||||
threading.Thread.__init__(self)
|
||||
self.jenkins = jenkins
|
||||
self.wake_event = threading.Event()
|
||||
self._stopped = False
|
||||
|
||||
def stop(self):
|
||||
self._stopped = True
|
||||
self.wake_event.set()
|
||||
|
||||
def run(self):
|
||||
while True:
|
||||
self.wake_event.wait(180)
|
||||
if self._stopped:
|
||||
return
|
||||
try:
|
||||
self.jenkins.lookForLostBuilds()
|
||||
except:
|
||||
self.log.exception("Exception checking builds:")
|
||||
|
||||
|
||||
STOP_BUILD = 'job/%(name)s/%(number)s/stop'
|
||||
CANCEL_QUEUE = 'queue/item/%(number)s/cancelQueue'
|
||||
BUILD_INFO = 'job/%(name)s/%(number)s/api/json?depth=0'
|
||||
BUILD_DESCRIPTION = 'job/%(name)s/%(number)s/submitDescription'
|
||||
DEBUG = False
|
||||
|
||||
|
||||
class ExtendedJenkins(jenkins.Jenkins):
|
||||
def jenkins_open(self, req):
|
||||
'''
|
||||
Utility routine for opening an HTTP request to a Jenkins server.
|
||||
'''
|
||||
try:
|
||||
if self.auth:
|
||||
req.add_header('Authorization', self.auth)
|
||||
return urllib2.urlopen(req).read()
|
||||
except urllib2.HTTPError, e:
|
||||
if DEBUG:
|
||||
print e.msg
|
||||
print e.fp.read()
|
||||
raise
|
||||
|
||||
def stop_build(self, name, number):
|
||||
'''
|
||||
Stop a running Jenkins build.
|
||||
|
||||
@param name: Name of Jenkins job
|
||||
@type name: str
|
||||
@param number: Jenkins build number for the job
|
||||
@type number: int
|
||||
'''
|
||||
request = urllib2.Request(self.server + STOP_BUILD % locals())
|
||||
self.jenkins_open(request)
|
||||
|
||||
def cancel_queue(self, number):
|
||||
'''
|
||||
Cancel a queued build.
|
||||
|
||||
@param number: Jenkins queue number for the build
|
||||
@type number: int
|
||||
'''
|
||||
# Jenkins returns a 302 from this URL, unless Referer is not set,
|
||||
# then you get a 404.
|
||||
request = urllib2.Request(self.server + CANCEL_QUEUE % locals(),
|
||||
urllib.urlencode({}),
|
||||
headers={'Referer': self.server})
|
||||
self.jenkins_open(request)
|
||||
|
||||
def get_build_info(self, name, number):
|
||||
'''
|
||||
Get information for a build.
|
||||
|
||||
@param name: Name of Jenkins job
|
||||
@type name: str
|
||||
@param number: Jenkins build number for the job
|
||||
@type number: int
|
||||
@return: dictionary
|
||||
'''
|
||||
request = urllib2.Request(self.server + BUILD_INFO % locals())
|
||||
return json.loads(self.jenkins_open(request))
|
||||
|
||||
def set_build_description(self, name, number, description):
|
||||
'''
|
||||
Get information for a build.
|
||||
|
||||
@param name: Name of Jenkins job
|
||||
@type name: str
|
||||
@param number: Jenkins build number for the job
|
||||
@type number: int
|
||||
@param description: Bulid description to set
|
||||
@type description: str
|
||||
'''
|
||||
params = urllib.urlencode({'description': description})
|
||||
request = urllib2.Request(self.server + BUILD_DESCRIPTION % locals(),
|
||||
params)
|
||||
self.jenkins_open(request)
|
||||
|
||||
|
||||
class Jenkins(object):
|
||||
log = logging.getLogger("zuul.Jenkins")
|
||||
launch_retry_timeout = 5
|
||||
|
||||
def __init__(self, config, sched):
|
||||
self.sched = sched
|
||||
self.builds = {}
|
||||
server = config.get('jenkins', 'server')
|
||||
user = config.get('jenkins', 'user')
|
||||
apikey = config.get('jenkins', 'apikey')
|
||||
self.jenkins = ExtendedJenkins(server, user, apikey)
|
||||
self.callback_thread = JenkinsCallback(self)
|
||||
self.callback_thread.start()
|
||||
self.cleanup_thread = JenkinsCleanup(self)
|
||||
self.cleanup_thread.start()
|
||||
|
||||
def stop(self):
|
||||
self.cleanup_thread.stop()
|
||||
self.cleanup_thread.join()
|
||||
|
||||
#TODO: remove dependent_changes
|
||||
def launch(self, job, change, pipeline, dependent_changes=[]):
|
||||
self.log.info("Launch job %s for change %s with dependent changes %s" %
|
||||
(job, change, dependent_changes))
|
||||
dependent_changes = dependent_changes[:]
|
||||
dependent_changes.reverse()
|
||||
uuid = str(uuid4().hex)
|
||||
params = dict(UUID=uuid, # deprecated
|
||||
ZUUL_UUID=uuid,
|
||||
GERRIT_PROJECT=change.project.name, # deprecated
|
||||
ZUUL_PROJECT=change.project.name)
|
||||
params['ZUUL_PIPELINE'] = pipeline.name
|
||||
if hasattr(change, 'refspec'):
|
||||
changes_str = '^'.join(
|
||||
['%s:%s:%s' % (c.project.name, c.branch, c.refspec)
|
||||
for c in dependent_changes + [change]])
|
||||
params['GERRIT_BRANCH'] = change.branch # deprecated
|
||||
params['ZUUL_BRANCH'] = change.branch
|
||||
params['GERRIT_CHANGES'] = changes_str # deprecated
|
||||
params['ZUUL_CHANGES'] = changes_str
|
||||
params['ZUUL_REF'] = ('refs/zuul/%s/%s' %
|
||||
(change.branch,
|
||||
change.current_build_set.ref))
|
||||
params['ZUUL_COMMIT'] = change.current_build_set.commit
|
||||
|
||||
zuul_changes = ' '.join(['%s,%s' % (c.number, c.patchset)
|
||||
for c in dependent_changes + [change]])
|
||||
params['ZUUL_CHANGE_IDS'] = zuul_changes
|
||||
params['ZUUL_CHANGE'] = str(change.number)
|
||||
params['ZUUL_PATCHSET'] = str(change.patchset)
|
||||
if hasattr(change, 'ref'):
|
||||
params['GERRIT_REFNAME'] = change.ref # deprecated
|
||||
params['ZUUL_REFNAME'] = change.ref
|
||||
params['GERRIT_OLDREV'] = change.oldrev # deprecated
|
||||
params['ZUUL_OLDREV'] = change.oldrev
|
||||
params['GERRIT_NEWREV'] = change.newrev # deprecated
|
||||
params['ZUUL_NEWREV'] = change.newrev
|
||||
params['ZUUL_SHORT_OLDREV'] = change.oldrev[:7]
|
||||
params['ZUUL_SHORT_NEWREV'] = change.newrev[:7]
|
||||
|
||||
params['ZUUL_REF'] = change.ref
|
||||
params['ZUUL_COMMIT'] = change.newrev
|
||||
|
||||
# This is what we should be heading toward for parameters:
|
||||
|
||||
# required:
|
||||
# ZUUL_UUID
|
||||
# ZUUL_REF (/refs/zuul/..., /refs/tags/foo, master)
|
||||
# ZUUL_COMMIT
|
||||
|
||||
# optional:
|
||||
# ZUUL_PROJECT
|
||||
# ZUUL_PIPELINE
|
||||
|
||||
# optional (changes only):
|
||||
# ZUUL_BRANCH
|
||||
# ZUUL_CHANGE
|
||||
# ZUUL_CHANGE_IDS
|
||||
# ZUUL_PATCHSET
|
||||
|
||||
# optional (ref updated only):
|
||||
# ZUUL_OLDREV
|
||||
# ZUUL_NEWREV
|
||||
# ZUUL_SHORT_NEWREV
|
||||
# ZUUL_SHORT_OLDREV
|
||||
|
||||
if callable(job.parameter_function):
|
||||
job.parameter_function(change, params)
|
||||
self.log.debug("Custom parameter function used for job %s, "
|
||||
"change: %s, params: %s" % (job, change, params))
|
||||
|
||||
build = Build(job, uuid)
|
||||
# We can get the started notification on another thread before
|
||||
# this is done so we add the build even before we trigger the
|
||||
# job on Jenkins.
|
||||
self.builds[uuid] = build
|
||||
# Sometimes Jenkins may erroneously respond with a 404. Handle
|
||||
# that by retrying for 30 seconds.
|
||||
launched = False
|
||||
errored = False
|
||||
for count in range(6):
|
||||
try:
|
||||
self.jenkins.build_job(job.name, parameters=params)
|
||||
launched = True
|
||||
break
|
||||
except:
|
||||
errored = True
|
||||
self.log.exception("Exception launching build %s for "
|
||||
"job %s for change %s (will retry):" %
|
||||
(build, job, change))
|
||||
time.sleep(self.launch_retry_timeout)
|
||||
|
||||
if errored:
|
||||
if launched:
|
||||
self.log.error("Finally able to launch %s" % build)
|
||||
else:
|
||||
self.log.error("Unable to launch %s, even after retrying, "
|
||||
"declaring lost" % build)
|
||||
# To keep the queue moving, declare this as a lost build
|
||||
# so that the change will get dropped.
|
||||
self.onBuildCompleted(build.uuid, 'LOST', None, None)
|
||||
return build
|
||||
|
||||
def findBuildInQueue(self, build):
|
||||
for item in self.jenkins.get_queue_info():
|
||||
if 'actions' not in item:
|
||||
continue
|
||||
for action in item['actions']:
|
||||
if 'parameters' not in action:
|
||||
continue
|
||||
parameters = action['parameters']
|
||||
for param in parameters:
|
||||
# UUID is deprecated in favor of ZUUL_UUID
|
||||
if ((param['name'] in ['ZUUL_UUID', 'UUID'])
|
||||
and build.uuid == param['value']):
|
||||
return item
|
||||
return False
|
||||
|
||||
def cancel(self, build):
|
||||
self.log.info("Cancel build %s for job %s" % (build, build.job))
|
||||
if build.number:
|
||||
self.log.debug("Build %s has already started" % build)
|
||||
self.jenkins.stop_build(build.job.name, build.number)
|
||||
self.log.debug("Canceled running build %s" % build)
|
||||
return
|
||||
else:
|
||||
self.log.debug("Build %s has not started yet" % build)
|
||||
|
||||
self.log.debug("Looking for build %s in queue" % build)
|
||||
item = self.findBuildInQueue(build)
|
||||
if item:
|
||||
self.log.debug("Found queue item %s for build %s" %
|
||||
(item['id'], build))
|
||||
try:
|
||||
self.jenkins.cancel_queue(item['id'])
|
||||
self.log.debug("Canceled queue item %s for build %s" %
|
||||
(item['id'], build))
|
||||
return
|
||||
except:
|
||||
self.log.exception("Exception canceling queue item %s "
|
||||
"for build %s" % (item['id'], build))
|
||||
|
||||
self.log.debug("Still unable to find build %s to cancel" % build)
|
||||
if build.number:
|
||||
self.log.debug("Build %s has just started" % build)
|
||||
self.jenkins.stop_build(build.job.name, build.number)
|
||||
self.log.debug("Canceled just running build %s" % build)
|
||||
else:
|
||||
self.log.error("Build %s has not started but "
|
||||
"was not found in queue" % build)
|
||||
|
||||
def setBuildDescription(self, build, description):
|
||||
if not build.number:
|
||||
return
|
||||
try:
|
||||
self.jenkins.set_build_description(build.job.name,
|
||||
build.number,
|
||||
description)
|
||||
except:
|
||||
self.log.exception("Exception setting build description for %s" %
|
||||
build)
|
||||
|
||||
def onBuildCompleted(self, uuid, status, url, number):
|
||||
self.log.info("Build %s #%s complete, status %s" %
|
||||
(uuid, number, status))
|
||||
build = self.builds.get(uuid)
|
||||
if build:
|
||||
self.log.debug("Found build %s" % build)
|
||||
del self.builds[uuid]
|
||||
if url:
|
||||
build.url = url
|
||||
build.result = status
|
||||
build.number = number
|
||||
self.sched.onBuildCompleted(build)
|
||||
else:
|
||||
self.log.error("Unable to find build %s" % uuid)
|
||||
|
||||
def onBuildStarted(self, uuid, url, number):
|
||||
self.log.info("Build %s #%s started, url: %s" % (uuid, number, url))
|
||||
build = self.builds.get(uuid)
|
||||
if build:
|
||||
self.log.debug("Found build %s" % build)
|
||||
build.url = url
|
||||
build.number = number
|
||||
self.sched.onBuildStarted(build)
|
||||
else:
|
||||
self.log.error("Unable to find build %s" % uuid)
|
||||
|
||||
def lookForLostBuilds(self):
|
||||
self.log.debug("Looking for lost builds")
|
||||
lostbuilds = []
|
||||
for build in self.builds.values():
|
||||
if build.result:
|
||||
# The build has finished, it will be removed
|
||||
continue
|
||||
if build.number:
|
||||
# The build has started; see if it has finished
|
||||
try:
|
||||
info = self.jenkins.get_build_info(build.job.name,
|
||||
build.number)
|
||||
if hasattr(build, '_jenkins_missing_build_info'):
|
||||
del build._jenkins_missing_build_info
|
||||
except:
|
||||
self.log.exception("Exception getting info for %s" % build)
|
||||
# We can't look it up in jenkins. That could be transient.
|
||||
# If it keeps up, assume it's permanent.
|
||||
if hasattr(build, '_jenkins_missing_build_info'):
|
||||
missing_time = build._jenkins_missing_build_info
|
||||
if time.time() - missing_time > JENKINS_GRACE_TIME:
|
||||
self.log.debug("Lost build %s because "
|
||||
"it has started but "
|
||||
"the build URL is not working" %
|
||||
build)
|
||||
lostbuilds.append(build)
|
||||
else:
|
||||
build._jenkins_missing_build_info = time.time()
|
||||
continue
|
||||
|
||||
if not info:
|
||||
self.log.debug("Lost build %s because "
|
||||
"it started but "
|
||||
"info can not be retreived" % build)
|
||||
lostbuilds.append(build)
|
||||
continue
|
||||
if info['building']:
|
||||
# It has not finished.
|
||||
continue
|
||||
if info['duration'] == 0:
|
||||
# Possible jenkins bug -- not building, but no duration
|
||||
self.log.debug("Possible jenkins bug with build %s: "
|
||||
"not building, but no duration is set "
|
||||
"Build info %s:" % (build,
|
||||
pprint.pformat(info)))
|
||||
continue
|
||||
finish_time = (info['timestamp'] + info['duration']) / 1000
|
||||
if time.time() - finish_time > JENKINS_GRACE_TIME:
|
||||
self.log.debug("Lost build %s because "
|
||||
"it finished more than 5 minutes ago. "
|
||||
"Build info %s:" % (build,
|
||||
pprint.pformat(info)))
|
||||
lostbuilds.append(build)
|
||||
continue
|
||||
# Give it more time
|
||||
else:
|
||||
# The build has not started
|
||||
if time.time() - build.launch_time < JENKINS_GRACE_TIME:
|
||||
# It just started, give it a bit
|
||||
continue
|
||||
info = self.findBuildInQueue(build)
|
||||
if info:
|
||||
# It's in the queue. All good.
|
||||
continue
|
||||
if build.number:
|
||||
# We just got notified it started
|
||||
continue
|
||||
# It may have just started. If we keep ending up here,
|
||||
# assume the worst.
|
||||
if hasattr(build, '_jenkins_missing_from_queue'):
|
||||
missing_time = build._jenkins_missing_from_queue
|
||||
if time.time() - missing_time > JENKINS_GRACE_TIME:
|
||||
self.log.debug("Lost build %s because "
|
||||
"it has not started and "
|
||||
"is not in the queue" % build)
|
||||
lostbuilds.append(build)
|
||||
continue
|
||||
else:
|
||||
build._jenkins_missing_from_queue = time.time()
|
||||
|
||||
for build in lostbuilds:
|
||||
self.log.error("Declaring %s lost" % build)
|
||||
self.onBuildCompleted(build.uuid, 'LOST', None, None)
|
|
@ -457,6 +457,7 @@ class Build(object):
|
|||
self.launch_time = time.time()
|
||||
self.start_time = None
|
||||
self.end_time = None
|
||||
self.fraction_complete = None
|
||||
|
||||
def __repr__(self):
|
||||
return '<Build %s of %s>' % (self.uuid, self.job.name)
|
||||
|
|
|
@ -0,0 +1,50 @@
|
|||
# Copyright 2012 Hewlett-Packard Development Company, L.P.
|
||||
# Copyright 2013 OpenStack Foundation
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import logging
|
||||
import threading
|
||||
from paste import httpserver
|
||||
from webob import Request
|
||||
|
||||
|
||||
class WebApp(threading.Thread):
|
||||
log = logging.getLogger("zuul.WebApp")
|
||||
|
||||
def __init__(self, scheduler):
|
||||
threading.Thread.__init__(self)
|
||||
self.scheduler = scheduler
|
||||
|
||||
def run(self):
|
||||
self.server = httpserver.serve(self.app, host='0.0.0.0', port='8001',
|
||||
start_loop=False)
|
||||
self.server.serve_forever()
|
||||
|
||||
def stop(self):
|
||||
self.server.server_close()
|
||||
|
||||
def app(self, environ, start_response):
|
||||
request = Request(environ)
|
||||
if request.path == '/status.json':
|
||||
try:
|
||||
ret = self.scheduler.formatStatusJSON()
|
||||
except:
|
||||
self.log.exception("Exception formatting status:")
|
||||
raise
|
||||
start_response('200 OK', [('content-type', 'application/json'),
|
||||
('Access-Control-Allow-Origin', '*')])
|
||||
return [ret]
|
||||
else:
|
||||
start_response('404 Not Found', [('content-type', 'text/plain')])
|
||||
return ['Not found.']
|
Loading…
Reference in New Issue