zuul
/
zuul
Code Issues Proposed changes

Switch the launcher to Gearman. 

Remove the Jenkins launcher and add a new Gearman launcher (designed
to be compatible with Jenkins) in its place.

See the documentation for how to set up the Gearman Plugin for
Jenkins.

Change-Id: Ie7224396271d7375f4ea42eebb57f883bc291738
This commit is contained in:
James E. Blair 2013-04-26 08:40:46 -07:00
parent bdafe495db
commit 1f4c2bb104
14 changed files with 1419 additions and 1221 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
doc/source/index.rst
View File

@ -10,8 +10,8 @@ Zuul is a program that is used to gate the source code repository of a
project so that changes are only merged if they pass tests. project so that changes are only merged if they pass tests.
The main component of Zuul is the scheduler. It receives events The main component of Zuul is the scheduler. It receives events
related to proposed changes (currently from Gerrit), triggers tests related to proposed changes, triggers tests based on those events, and
based on those events (currently on Jenkins), and reports back. reports back.
Contents: Contents:

230
doc/source/launchers.rst
View File

@ -1,73 +1,74 @@
:title: Launchers :title: Launchers
.. _launchers: .. _Gearman: http://gearman.org/
.. _`Gearman Plugin`:
https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin
.. _launchers:
Launchers Launchers
========= =========
Zuul has a modular architecture for launching jobs. Currently only Zuul has a modular architecture for launching jobs. Currently, the
Jenkins is supported, but it should be fairly easy to add a module to only supported module interfaces with Gearman_. This design allows
support other systems. Zuul makes very few assumptions about the any system to run jobs for Zuul simply by interfacing with a Gearman
interface to a launcher -- if it can trigger jobs, cancel them, and server. The recommended way of integrating a new job-runner with Zuul
receive success or failure reports, it should be able to be used with is via this method.
Zuul. Patches to this effect are welcome.
Jenkins If Gearman is unsuitable, Zuul may be extended with a new launcher
module. Zuul makes very few assumptions about the interface to a
launcher -- if it can trigger jobs, cancel them, and receive success
or failure reports, it should be able to be used with Zuul. Patches
to this effect are welcome.
Gearman
------- -------
Zuul works with Jenkins using the Jenkins API and the notification Gearman_ is a general-purpose protocol for distributing jobs to any
module. It uses the Jenkins API to trigger jobs, passing in number of workers. Zuul works with Gearman by sending specific
parameters indicating what should be tested. It recieves information with job requests to Gearman, and expects certain
notifications on job completion via the notification API (so jobs must information to be returned on completion. This protocol is described
be conifigured to notify Zuul). in `Zuul-Gearman Protocol`_.
Jenkins Configuration The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul
~~~~~~~~~~~~~~~~~~~~~ by providing an interface between Jenkins and Gearman. In this
configuration, Zuul asks Gearman to run jobs, and Gearman can then
distribute those jobs to any number of Jenkins systems (including
multiple Jenkins masters).
Zuul will need access to a Jenkins user. Create a user in Jenkins, In order for Zuul to run any jobs, you will need a running Gearman
and then visit the configuration page for the user: server. The latest version of gearmand from gearman.org is required
in order to support canceling jobs while in the queue. The server is
easy to set up -- just make sure that it allows connections from Zuul
and any workers (e.g., Jenkins masters) on port 4730, and nowhere else
(as the Gearman protocol does not include any provision for
authentication.
https://jenkins.example.com/user/USERNAME/configure Gearman Jenkins Plugin
----------------------
And click **Show API Token** to retrieve the API token for that user. The `Gearman Plugin`_ can be installed in Jenkins in order to
You will need this later when configuring Zuul. Appropriate user facilitate Jenkins running jobs for Zuul. Install the plugin and
permissions must be set under the Jenkins security matrix: under the configure it with the hostname or IP address of your Gearman server
``Global`` group of permissions, check ``Read``, then under the ``Job`` and the port on which it is listening (4730 by default). It will
group of permissions, check ``Read`` and ``Build``. Finally, under automatically register all known Jenkins jobs as functions that Zuul
``Run`` check ``Update``. If using a per project matrix, make sure the can invoke via Gearman.
user permissions are properly set for any jobs that you want Zuul to
trigger.
Make sure the notification plugin is installed. Visit the plugin Any number of masters can be configured in this way, and Gearman will
manager on your jenkins: distribute jobs to all of them as appropriate.
https://jenkins.example.com/pluginManager/ No special Jenkins job configuration is needed to support triggering
by Zuul.
And install **Jenkins Notification plugin**. The homepage for the Zuul Parameters
plugin is at: ---------------
https://wiki.jenkins-ci.org/display/JENKINS/Notification+Plugin Zuul will pass some parameters with every job it launches. The
Gearman Plugin will ensure these are supplied as Jenkins build
Jenkins Job Configuration parameters, so they will be available for use in the job configuration
~~~~~~~~~~~~~~~~~~~~~~~~~ as well as to the running job as environment variables. They are as
follows:
For each job that you want Zuul to trigger, you will need to add a
notification endpoint for the job on that job's configuration page.
Click **Add Endpoint** and enter the following values:
**Protocol**
``HTTP``
**URL**
``http://127.0.0.1:8001/jenkins_endpoint``
If you are running Zuul on a different server than Jenkins, enter the
appropriate URL. Note that Zuul itself has no access controls, so
ensure that only Jenkins is permitted to access that URL.
Zuul will pass some parameters to Jenkins for every job it launches.
Check **This build is parameterized**, and add the following fields
with the type **String Parameter**:
**ZUUL_UUID** **ZUUL_UUID**
Zuul provided key to link builds with Gerrit events Zuul provided key to link builds with Gerrit events
@ -75,27 +76,14 @@ with the type **String Parameter**:
Zuul provided ref that includes commit(s) to build Zuul provided ref that includes commit(s) to build
**ZUUL_COMMIT** **ZUUL_COMMIT**
The commit SHA1 at the head of ZUUL_REF The commit SHA1 at the head of ZUUL_REF
Those are the only required parameters. The ZUUL_UUID is needed for Zuul to
keep track of the build, and the ZUUL_REF and ZUUL_COMMIT parameters are for
use in preparing the git repo for the build.
.. note::
The GERRIT_PROJECT and UUID parameters are deprecated respectively in
favor of ZUUL_PROJECT and ZUUL_UUID.
The following parameters will be sent for all builds, but are not required so
you do not need to configure Jenkins to accept them if you do not plan on using
them:
**ZUUL_PROJECT** **ZUUL_PROJECT**
The project that triggered this build The project that triggered this build
**ZUUL_PIPELINE** **ZUUL_PIPELINE**
The Zuul pipeline that is building this job The Zuul pipeline that is building this job
The following parameters are optional and will only be provided for The following additional parameters will only be provided for builds
builds associated with changes (i.e., in response to patchset-created associated with changes (i.e., in response to patchset-created or
or comment-added events): comment-added events):
**ZUUL_BRANCH** **ZUUL_BRANCH**
The target branch for the change that triggered this build The target branch for the change that triggered this build
@ -107,7 +95,7 @@ or comment-added events):
**ZUUL_PATCHSET** **ZUUL_PATCHSET**
The Gerrit patchset number for the change that triggered this build The Gerrit patchset number for the change that triggered this build
The following parameters are optional and will only be provided for The following additional parameters will only be provided for
post-merge (ref-updated) builds: post-merge (ref-updated) builds:
**ZUUL_OLDREV** **ZUUL_OLDREV**
@ -139,7 +127,107 @@ That should be sufficient for a job that only builds a single project.
If you have multiple interrelated projects (i.e., they share a Zuul If you have multiple interrelated projects (i.e., they share a Zuul
Change Queue) that are built together, you may be able to configure Change Queue) that are built together, you may be able to configure
the Git plugin to prepare them, or you may chose to use a shell script the Git plugin to prepare them, or you may chose to use a shell script
instead. The OpenStack project uses the following script to prepare instead. As an example, the OpenStack project uses the following
the workspace for its integration testing: script to prepare the workspace for its integration testing:
https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh
Zuul-Gearman Protocol
---------------------
This section is only relevant if you intend to implement a new kind of
worker that runs jobs for Zuul via Gearman. If you just want to use
Jenkins, see `Gearman Jenkins Plugin`_ instead.
The Zuul protocol as used with Gearman is as follows:
Starting Builds
~~~~~~~~~~~~~~~
To start a build, Zuul invokes a Gearman function with the following
format:
build:FUNCTION_NAME
where **FUNCTION_NAME** is the name of the job that should be run. If
the job should run on a specific node (or class of node), Zuul will
instead invoke:
build:FUNCTION_NAME:NODE_NAME
where **NODE_NAME** is the name or class of node on which the job
should be run. This can be specified by setting the ZUUL_NODE
parameter in a paremeter-function (see :ref:`zuulconf`).
Zuul sends the ZUUL_* parameters described in `Zuul Parameters`_
encoded in JSON format as the argument included with the
SUBMIT_JOB_UNIQ request to Gearman. A unique ID (equal to the
ZUUL_UUID parameter) is also supplied to Gearman, and is accessible as
an added Gearman parameter with GRAB_JOB_UNIQ.
When a Gearman worker starts running a job for Zuul, it should
immediately send a WORK_DATA packet with the following information
encoded in JSON format:
**full_url**
The URL with the status or results of the build. Will be used in
the status page and the final report.
**number**
The build number (unique to this job).
**master**
A unique identifier associated with the Gearman worker that can
abort this build. See `Stopping Builds`_ for more information.
It should then immediately send a WORK_STATUS packet with a value of 0
percent complete. It may then optionally send subsequent WORK_STATUS
packets with updated completion values.
When the build is complete, it should send a final WORK_DATA packet
with the following in JSON format:
**result**
Either the string 'SUCCESS' if the job succeeded, or any other value
that describes the result if the job failed.
Finally, it should send either a WORK_FAIL or WORK_COMPLETE packet as
appropriate. A WORK_EXCEPTION packet will be interpreted as a
WORK_FAIL, but the exception will be logged in Zuul's error log.
Stopping Builds
~~~~~~~~~~~~~~~
If Zuul needs to abort a build already in progress, it will invoke the
following function through Gearman:
stop:MASTER_NAME
Where **MASTER_NAME** is the name of the master node supplied in the
initial WORK_DATA packet when the job started. This is used to direct
the stop: function invocation to the correct Gearman worker that is
capable of stopping that particular job. The argument to the function
will be the unique ID of the job that should be stopped.
The original job is expected to complete with a WORK_DATA and
WORK_FAIL packet as described in `Starting Builds`_.
Build Descriptions
~~~~~~~~~~~~~~~~~~
In order to update the job running system with a description of the
current state of all related builds, the job runner may optionally
implement the following Gearman function:
set_description:MASTER_NAME
Where **MASTER_NAME** is used as described in `Stopping Builds`_. The
argument to the function is the following encoded in JSON format:
**unique_id**
The unique identifier of the build whose description should be
updated.
**html_description**
The description of the build in HTML format.

50
doc/source/zuul.rst
View File

@ -9,7 +9,8 @@ Configuration
Zuul has three configuration files: Zuul has three configuration files:
**zuul.conf** **zuul.conf**
Credentials for Gerrit and Jenkins, locations of the other config files Connection information for Gerrit and Gearman, locations of the
other config files
**layout.yaml** **layout.yaml**
Project and pipeline configuration -- what Zuul does Project and pipeline configuration -- what Zuul does
**logging.conf** **logging.conf**
@ -27,30 +28,26 @@ Zuul will look for ``/etc/zuul/zuul.conf`` or ``~/zuul.conf`` to
bootstrap its configuration. Alternately, you may specify ``-c bootstrap its configuration. Alternately, you may specify ``-c
/path/to/zuul.conf`` on the command line. /path/to/zuul.conf`` on the command line.
Gerrit and Jenkins credentials are each described in a section of Gerrit and Gearman connection information are each described in a
zuul.conf. The location of the other two configuration files (as well section of zuul.conf. The location of the other two configuration
as the location of the PID file when running Zuul as a server) are files (as well as the location of the PID file when running Zuul as a
specified in a third section. server) are specified in a third section.
The three sections of this config and their options are documented below. The three sections of this config and their options are documented below.
You can also find an example zuul.conf file in the git You can also find an example zuul.conf file in the git
`repository `repository
<https://github.com/openstack-infra/zuul/blob/master/etc/zuul.conf-sample>`_ <https://github.com/openstack-infra/zuul/blob/master/etc/zuul.conf-sample>`_
jenkins gearman
""""""" """""""
**server** **server**
URL for the root of the Jenkins HTTP server. Hostname or IP address of the Gearman server.
``server=https://jenkins.example.com`` ``server=gearman.example.com``
**user** **port**
User to authenticate against Jenkins with. Port on which the Gearman server is listening
``user=jenkins`` ``port=4730``
**apikey**
Jenkins API Key credentials for the above user.
``apikey=1234567890abcdef1234567890abcdef``
gerrit gerrit
"""""" """"""
@ -65,11 +62,11 @@ gerrit
**user** **user**
User name to use when logging into above server via ssh. User name to use when logging into above server via ssh.
``user=jenkins`` ``user=zuul``
**sshkey** **sshkey**
Path to SSH key to use when logging into above server. Path to SSH key to use when logging into above server.
``sshkey=/home/jenkins/.ssh/id_rsa`` ``sshkey=/home/zuul/.ssh/id_rsa``
zuul zuul
"""" """"
@ -107,13 +104,14 @@ zuul
**status_url** **status_url**
URL that will be posted in Zuul comments made to Gerrit changes when URL that will be posted in Zuul comments made to Gerrit changes when
beginning Jenkins jobs for a change. starting jobs for a change.
``status_url=https://jenkins.example.com/zuul/status`` ``status_url=https://zuul.example.com/status``
**url_pattern** **url_pattern**
If you are storing build logs external to Jenkins and wish to link to If you are storing build logs external to the system that originally
those logs when Zuul makes comments on Gerrit changes for completed ran jobs and wish to link to those logs when Zuul makes comments on
jobs this setting configures what the URLs for those links should be. Gerrit changes for completed jobs this setting configures what the
URLs for those links should be.
``http://logs.example.com/{change.number}/{change.patchset}/{pipeline.name}/{job.name}/{build.number}`` ``http://logs.example.com/{change.number}/{change.patchset}/{pipeline.name}/{job.name}/{build.number}``
layout.yaml layout.yaml
@ -410,13 +408,13 @@ each job as it builds a list from the project specification.
**failure-pattern (optional)** **failure-pattern (optional)**
The URL that should be reported to Gerrit if the job fails. The URL that should be reported to Gerrit if the job fails.
Defaults to the Jenkins build URL or the url_pattern configured in Defaults to the build URL or the url_pattern configured in
zuul.conf. May be supplied as a string pattern with substitutions zuul.conf. May be supplied as a string pattern with substitutions
as described in url_pattern in :ref:`zuulconf`. as described in url_pattern in :ref:`zuulconf`.
**success-pattern (optional)** **success-pattern (optional)**
The URL that should be reported to Gerrit if the job succeeds. The URL that should be reported to Gerrit if the job succeeds.
Defaults to the Jenkins build URL or the url_pattern configured in Defaults to the build URL or the url_pattern configured in
zuul.conf. May be supplied as a string pattern with substitutions zuul.conf. May be supplied as a string pattern with substitutions
as described in url_pattern in :ref:`zuulconf`. as described in url_pattern in :ref:`zuulconf`.
@ -461,6 +459,10 @@ each job as it builds a list from the project specification.
:param parameters: parameters to be passed to the job :param parameters: parameters to be passed to the job
:type parameters: dict :type parameters: dict
If the parameter **ZUUL_NODE** is set by this function, then it will
be used to specify on what node (or class of node) the job should be
run.
Here is an example of setting the failure message for jobs that check Here is an example of setting the failure message for jobs that check
whether a change merges cleanly:: whether a change merges cleanly::

2
tests/fixtures/custom_functions.py vendored Normal file
View File

@ -0,0 +1,2 @@
def select_debian_node(change, params):
params['ZUUL_NODE'] = 'debian'

11
tests/fixtures/layout.yaml vendored
View File

@ -1,3 +1,6 @@
includes:
- python-file: custom_functions.py
pipelines: pipelines:
- name: check - name: check
manager: IndependentPipelineManager manager: IndependentPipelineManager
@ -46,6 +49,8 @@ jobs:
- name: project-testfile - name: project-testfile
files: files:
- '.*-requires' - '.*-requires'
- name: node-project-test1
parameter-function: select_debian_node
project-templates: project-templates:
- name: test-one-and-two - name: test-one-and-two
@ -135,3 +140,9 @@ projects:
template: template:
- name: test-one-and-two - name: test-one-and-two
projectname: project projectname: project
- name: org/node-project
gate:
- node-project-merge:
- node-project-test1
- node-project-test2

6
tests/fixtures/zuul.conf vendored
View File

@ -1,7 +1,5 @@
[jenkins] [gearman]
server=https://jenkins.example.com server=127.0.0.1
user=jenkins
apikey=1234
[gerrit] [gerrit]
server=review.example.com server=review.example.com

1377
tests/test_scheduler.py
View File

File diff suppressed because it is too large Load Diff

1
tools/pip-requires
View File

@ -9,3 +9,4 @@ python-daemon
extras extras
statsd>=1.0.0,<3.0 statsd>=1.0.0,<3.0
voluptuous>=0.6,<0.7 voluptuous>=0.6,<0.7
http://tarballs.openstack.org/gear/gear-master.tar.gz#egg=gear

4
tox.ini
View File

@ -7,13 +7,14 @@ setenv = STATSD_HOST=localhost
STATSD_PORT=8125 STATSD_PORT=8125
deps = -r{toxinidir}/tools/pip-requires deps = -r{toxinidir}/tools/pip-requires
-r{toxinidir}/tools/test-requires -r{toxinidir}/tools/test-requires
commands = nosetests {posargs} commands = nosetests --logging-format="%(asctime)s %(name)-32s %(levelname)-8s %(message)s" {posargs}
[tox:jenkins] [tox:jenkins]
downloadcache = ~/cache/pip downloadcache = ~/cache/pip
[testenv:pep8] [testenv:pep8]
deps = pep8==1.3.3 deps = pep8==1.3.3
-r{toxinidir}/tools/pip-requires
commands = pep8 --ignore=E123,E125,E128 --repeat --show-source --exclude=.venv,.tox,dist,doc,build . commands = pep8 --ignore=E123,E125,E128 --repeat --show-source --exclude=.venv,.tox,dist,doc,build .
[testenv:cover] [testenv:cover]
@ -21,6 +22,7 @@ setenv = NOSE_WITH_COVERAGE=1
[testenv:pyflakes] [testenv:pyflakes]
deps = pyflakes deps = pyflakes
-r{toxinidir}/tools/pip-requires
commands = pyflakes zuul setup.py commands = pyflakes zuul setup.py
[testenv:venv] [testenv:venv]

18
zuul/cmd/server.py
View File

@ -1,3 +1,4 @@
#!/usr/bin/env python
# Copyright 2012 Hewlett-Packard Development Company, L.P. # Copyright 2012 Hewlett-Packard Development Company, L.P.
# Copyright 2013 OpenStack Foundation # Copyright 2013 OpenStack Foundation
# #
@ -88,7 +89,7 @@ class Server(object):
def test_config(self): def test_config(self):
# See comment at top of file about zuul imports # See comment at top of file about zuul imports
import zuul.scheduler import zuul.scheduler
import zuul.launcher.jenkins import zuul.launcher.gearman
import zuul.trigger.gerrit import zuul.trigger.gerrit
logging.basicConfig(level=logging.DEBUG) logging.basicConfig(level=logging.DEBUG)
@ -98,20 +99,24 @@ class Server(object):
def main(self): def main(self):
# See comment at top of file about zuul imports # See comment at top of file about zuul imports
import zuul.scheduler import zuul.scheduler
import zuul.launcher.jenkins import zuul.launcher.gearman
import zuul.trigger.gerrit import zuul.trigger.gerrit
import zuul.webapp
self.sched = zuul.scheduler.Scheduler() self.sched = zuul.scheduler.Scheduler()
jenkins = zuul.launcher.jenkins.Jenkins(self.config, self.sched) gearman = zuul.launcher.gearman.Gearman(self.config, self.sched)
gerrit = zuul.trigger.gerrit.Gerrit(self.config, self.sched) gerrit = zuul.trigger.gerrit.Gerrit(self.config, self.sched)
webapp = zuul.webapp.WebApp(self.sched)
self.sched.setLauncher(jenkins) self.sched.setLauncher(gearman)
self.sched.setTrigger(gerrit) self.sched.setTrigger(gerrit)
self.sched.start() self.sched.start()
self.sched.reconfigure(self.config) self.sched.reconfigure(self.config)
self.sched.resume() self.sched.resume()
webapp.start()
signal.signal(signal.SIGHUP, self.reconfigure_handler) signal.signal(signal.SIGHUP, self.reconfigure_handler)
signal.signal(signal.SIGUSR1, self.exit_handler) signal.signal(signal.SIGUSR1, self.exit_handler)
while True: while True:
@ -168,3 +173,8 @@ def main():
with daemon.DaemonContext(pidfile=pid): with daemon.DaemonContext(pidfile=pid):
server.setup_logging() server.setup_logging()
server.main() server.main()
if __name__ == "__main__":
sys.path.insert(0, '.')
main()

387
zuul/launcher/gearman.py Normal file
View File

@ -0,0 +1,387 @@
# Copyright 2012 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import gear
import json
import logging
import time
import threading
from uuid import uuid4
from zuul.model import Build
class GearmanCleanup(threading.Thread):
""" A thread that checks to see if outstanding builds have
completed without reporting back. """
log = logging.getLogger("zuul.JenkinsCleanup")
def __init__(self, gearman):
threading.Thread.__init__(self)
self.gearman = gearman
self.wake_event = threading.Event()
self._stopped = False
def stop(self):
self._stopped = True
self.wake_event.set()
def run(self):
while True:
self.wake_event.wait(300)
if self._stopped:
return
try:
self.gearman.lookForLostBuilds()
except:
self.log.exception("Exception checking builds:")
def getJobData(job):
if not len(job.data):
return {}
d = job.data[-1]
if not d:
return {}
return json.loads(d)
class ZuulGearmanClient(gear.Client):
def __init__(self, zuul_gearman):
super(ZuulGearmanClient, self).__init__()
self.__zuul_gearman = zuul_gearman
def handleWorkComplete(self, packet):
job = super(ZuulGearmanClient, self).handleWorkComplete(packet)
self.__zuul_gearman.onBuildCompleted(job)
return job
def handleWorkFail(self, packet):
job = super(ZuulGearmanClient, self).handleWorkFail(packet)
self.__zuul_gearman.onBuildCompleted(job)
return job
def handleWorkStatus(self, packet):
job = super(ZuulGearmanClient, self).handleWorkStatus(packet)
self.__zuul_gearman.onWorkStatus(job)
return job
def handleWorkData(self, packet):
job = super(ZuulGearmanClient, self).handleWorkData(packet)
self.__zuul_gearman.onWorkStatus(job)
return job
def handleDisconnect(self, job):
job = super(ZuulGearmanClient, self).handleDisconnect(job)
self.__zuul_gearman.onDisconnect(job)
def handleStatusRes(self, packet):
try:
job = super(ZuulGearmanClient, self).handleStatusRes(packet)
except gear.UnknownJobError:
handle = packet.getArgument(0)
for build in self.__zuul_gearman.builds:
if build.__gearman_job.handle == handle:
self.__zuul_gearman.onUnknownJob(job)
class Gearman(object):
log = logging.getLogger("zuul.Gearman")
negative_function_cache_ttl = 5
def __init__(self, config, sched):
self.sched = sched
self.builds = {}
self.meta_jobs = {} # A list of meta-jobs like stop or describe
server = config.get('gearman', 'server')
if config.has_option('gearman', 'port'):
port = config.get('gearman', 'port')
else:
port = 4730
self.gearman = ZuulGearmanClient(self)
self.gearman.addServer(server, port)
self.cleanup_thread = GearmanCleanup(self)
self.cleanup_thread.start()
self.function_cache = set()
self.function_cache_time = 0
def stop(self):
self.log.debug("Stopping")
self.cleanup_thread.stop()
self.cleanup_thread.join()
self.gearman.shutdown()
self.log.debug("Stopped")
def isJobRegistered(self, name):
if self.function_cache_time:
for connection in self.gearman.active_connections:
if connection.connect_time > self.function_cache_time:
self.function_cache = set()
self.function_cache_time = 0
break
if name in self.function_cache:
self.log.debug("Function %s is registered" % name)
return True
if ((time.time() - self.function_cache_time) <
self.negative_function_cache_ttl):
self.log.debug("Function %s is not registered "
"(negative ttl in effect)" % name)
return False
self.function_cache_time = time.time()
for connection in self.gearman.active_connections:
try:
req = gear.StatusAdminRequest()
connection.sendAdminRequest(req)
req.waitForResponse()
except Exception:
self.log.exception("Exception while checking functions")
continue
for line in req.response.split('\n'):
parts = [x.strip() for x in line.split()]
if not parts or parts[0] == '.':
continue
self.function_cache.add(parts[0])
if name in self.function_cache:
self.log.debug("Function %s is registered" % name)
return True
self.log.debug("Function %s is not registered" % name)
return False
def launch(self, job, change, pipeline, dependent_changes=[]):
self.log.info("Launch job %s for change %s with dependent changes %s" %
(job, change, dependent_changes))
dependent_changes = dependent_changes[:]
dependent_changes.reverse()
uuid = str(uuid4().hex)
params = dict(ZUUL_UUID=uuid,
ZUUL_PROJECT=change.project.name)
params['ZUUL_PIPELINE'] = pipeline.name
if hasattr(change, 'refspec'):
changes_str = '^'.join(
['%s:%s:%s' % (c.project.name, c.branch, c.refspec)
for c in dependent_changes + [change]])
params['ZUUL_BRANCH'] = change.branch
params['ZUUL_CHANGES'] = changes_str
params['ZUUL_REF'] = ('refs/zuul/%s/%s' %
(change.branch,
change.current_build_set.ref))
params['ZUUL_COMMIT'] = change.current_build_set.commit
zuul_changes = ' '.join(['%s,%s' % (c.number, c.patchset)
for c in dependent_changes + [change]])
params['ZUUL_CHANGE_IDS'] = zuul_changes
params['ZUUL_CHANGE'] = str(change.number)
params['ZUUL_PATCHSET'] = str(change.patchset)
if hasattr(change, 'ref'):
params['ZUUL_REFNAME'] = change.ref
params['ZUUL_OLDREV'] = change.oldrev
params['ZUUL_NEWREV'] = change.newrev
params['ZUUL_SHORT_OLDREV'] = change.oldrev[:7]
params['ZUUL_SHORT_NEWREV'] = change.newrev[:7]
params['ZUUL_REF'] = change.ref
params['ZUUL_COMMIT'] = change.newrev
# This is what we should be heading toward for parameters:
# required:
# ZUUL_UUID
# ZUUL_REF (/refs/zuul/..., /refs/tags/foo, master)
# ZUUL_COMMIT
# optional:
# ZUUL_PROJECT
# ZUUL_PIPELINE
# optional (changes only):
# ZUUL_BRANCH
# ZUUL_CHANGE
# ZUUL_CHANGE_IDS
# ZUUL_PATCHSET
# optional (ref updated only):
# ZUUL_OLDREV
# ZUUL_NEWREV
# ZUUL_SHORT_NEWREV
# ZUUL_SHORT_OLDREV
if callable(job.parameter_function):
job.parameter_function(change, params)
self.log.debug("Custom parameter function used for job %s, "
"change: %s, params: %s" % (job, change, params))
if 'ZUUL_NODE' in params:
name = "build:%s:%s" % (job.name, params['ZUUL_NODE'])
else:
name = "build:%s" % job.name
build = Build(job, uuid)
gearman_job = gear.Job(name, json.dumps(params),
unique=uuid)
build.__gearman_job = gearman_job
self.builds[uuid] = build
if not self.isJobRegistered(gearman_job.name):
self.log.error("Job %s is not registered with Gearman" %
gearman_job)
self.onBuildCompleted(gearman_job, 'LOST')
return build
try:
self.gearman.submitJob(gearman_job)
except Exception:
self.log.exception("Unable to submit job to Gearman")
self.onBuildCompleted(gearman_job, 'LOST')
return build
gearman_job.waitForHandle(30)
if not gearman_job.handle:
self.log.error("No job handle was received for %s after 30 seconds"
" marking as lost." %
gearman_job)
self.onBuildCompleted(gearman_job, 'LOST')
return build
def cancel(self, build):
self.log.info("Cancel build %s for job %s" % (build, build.job))
if build.number:
self.log.debug("Build %s has already started" % build)
self.cancelRunningBuild(build)
self.log.debug("Canceled running build %s" % build)
return
else:
self.log.debug("Build %s has not started yet" % build)
self.log.debug("Looking for build %s in queue" % build)
if self.cancelJobInQueue(build):
self.log.debug("Removed build %s from queue" % build)
return
self.log.debug("Still unable to find build %s to cancel" % build)
if build.number:
self.log.debug("Build %s has just started" % build)
self.cancelRunningBuild(build)
self.log.debug("Canceled just running build %s" % build)
else:
self.log.error("Build %s has not started but "
"was not found in queue" % build)
def onBuildCompleted(self, job, result=None):
if job.unique in self.meta_jobs:
del self.meta_jobs[job.unique]
return
build = self.builds.get(job.unique)
if build:
if result is None:
data = getJobData(job)
result = data.get('result')
self.log.info("Build %s complete, result %s" %
(job, result))
build.result = result
self.sched.onBuildCompleted(build)
# The test suite expects the build to be removed from the
# internal dict after it's added to the report queue.
del self.builds[job.unique]
else:
if not job.name.startswith("stop:"):
self.log.error("Unable to find build %s" % job.unique)
def onWorkStatus(self, job):
data = getJobData(job)
self.log.info("Build %s update" % job)
build = self.builds.get(job.unique)
if build:
self.log.debug("Found build %s" % build)
if not build.number:
self.log.info("Build %s started" % job)
build.url = data.get('full_url')
build.number = data.get('number')
build.__gearman_master = data.get('master')
self.sched.onBuildStarted(build)
build.fraction_complete = job.fraction_complete
else:
self.log.error("Unable to find build %s" % job.unique)
def onDisconnect(self, job):
self.log.info("Gearman job %s lost due to disconnect" % job)
self.onBuildCompleted(job, 'LOST')
def onUnknownJob(self, job):
self.log.info("Gearman job %s lost due to unknown handle" % job)
self.onBuildCompleted(job, 'LOST')
def cancelJobInQueue(self, build):
job = build.__gearman_job
req = gear.CancelJobAdminRequest(job.handle)
job.connection.sendAdminRequest(req)
req.waitForResponse()
self.log.debug("Response to cancel build %s request: %s" %
(build, req.response.strip()))
if req.response.startswith("OK"):
try:
del self.builds[job.unique]
except:
pass
return True
return False
def cancelRunningBuild(self, build):
stop_uuid = str(uuid4().hex)
stop_job = gear.Job("stop:%s" % build.__gearman_master,
build.uuid,
unique=stop_uuid)
self.meta_jobs[stop_uuid] = stop_job
self.log.debug("Submitting stop job: %s", stop_job)
self.gearman.submitJob(stop_job)
return True
def setBuildDescription(self, build, desc):
try:
name = "set_description:%s" % build.__gearman_master
except AttributeError:
# We haven't yet received the first data packet that tells
# us where the job is running.
return False
if not self.isJobRegistered(name):
return False
desc_uuid = str(uuid4().hex)
data = dict(unique_id=build.uuid,
html_description=desc)
desc_job = gear.Job(name, json.dumps(data), unique=desc_uuid)
self.meta_jobs[desc_uuid] = desc_job
self.log.debug("Submitting describe job: %s", desc_job)
self.gearman.submitJob(desc_job)
return True
def lookForLostBuilds(self):
self.log.debug("Looking for lost builds")
for build in self.builds.values():
if build.result:
# The build has finished, it will be removed
continue
job = build.__gearman_job
if not job.handle:
# The build hasn't been enqueued yet
continue
p = gear.Packet(gear.constants.REQ, gear.constants.GET_STATUS,
job.handle)
job.connection.sendPacket(p)

499
zuul/launcher/jenkins.py
View File

@ -1,499 +0,0 @@
# Copyright 2012 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
# So we can name this module "jenkins" and still load the "jenkins"
# system module
from __future__ import absolute_import
import json
import logging
import pprint
import threading
import time
import urllib # for extending jenkins lib
import urllib2 # for extending jenkins lib
from uuid import uuid4
import jenkins
from paste import httpserver
from webob import Request
from zuul.model import Build
# The amount of time we tolerate a change in build status without
# receiving a notification
JENKINS_GRACE_TIME = 60
class JenkinsCallback(threading.Thread):
log = logging.getLogger("zuul.JenkinsCallback")
def __init__(self, jenkins):
threading.Thread.__init__(self)
self.jenkins = jenkins
def run(self):
httpserver.serve(self.app, host='0.0.0.0', port='8001')
def app(self, environ, start_response):
request = Request(environ)
if request.path == '/jenkins_endpoint':
self.jenkins_endpoint(request)
start_response('200 OK', [('content-type', 'text/html')])
return ['Zuul good.']
elif request.path == '/status':
try:
ret = self.jenkins.sched.formatStatusHTML()
except:
self.log.exception("Exception formatting status:")
raise
start_response('200 OK', [('content-type', 'text/html')])
return [ret]
elif request.path == '/status.json':
try:
ret = self.jenkins.sched.formatStatusJSON()
except:
self.log.exception("Exception formatting status:")
raise
start_response('200 OK', [('content-type', 'application/json'),
('Access-Control-Allow-Origin', '*')])
return [ret]
else:
start_response('200 OK', [('content-type', 'text/html')])
return ['Zuul good.']
def jenkins_endpoint(self, request):
try:
data = json.loads(request.body)
except:
self.log.exception("Exception handling Jenkins notification:")
raise # let wsgi handler process the issue
if data:
self.log.debug("Received data from Jenkins: \n%s" %
(pprint.pformat(data)))
build = data.get('build')
if build:
phase = build.get('phase')
status = build.get('status')
url = build.get('full_url')
number = build.get('number')
params = build.get('parameters')
if params:
# UUID is deprecated in favor of ZUUL_UUID
uuid = params.get('ZUUL_UUID') or params.get('UUID')
if (status and url and uuid and phase and
phase == 'COMPLETED'):
self.jenkins.onBuildCompleted(uuid,
status,
url,
number)
if (phase and phase == 'STARTED'):
self.jenkins.onBuildStarted(uuid, url, number)
class JenkinsCleanup(threading.Thread):
""" A thread that checks to see if outstanding builds have
completed without reporting back. """
log = logging.getLogger("zuul.JenkinsCleanup")
def __init__(self, jenkins):
threading.Thread.__init__(self)
self.jenkins = jenkins
self.wake_event = threading.Event()
self._stopped = False
def stop(self):
self._stopped = True
self.wake_event.set()
def run(self):
while True:
self.wake_event.wait(180)
if self._stopped:
return
try:
self.jenkins.lookForLostBuilds()
except:
self.log.exception("Exception checking builds:")
STOP_BUILD = 'job/%(name)s/%(number)s/stop'
CANCEL_QUEUE = 'queue/item/%(number)s/cancelQueue'
BUILD_INFO = 'job/%(name)s/%(number)s/api/json?depth=0'
BUILD_DESCRIPTION = 'job/%(name)s/%(number)s/submitDescription'
DEBUG = False
class ExtendedJenkins(jenkins.Jenkins):
def jenkins_open(self, req):
'''
Utility routine for opening an HTTP request to a Jenkins server.
'''
try:
if self.auth:
req.add_header('Authorization', self.auth)
return urllib2.urlopen(req).read()
except urllib2.HTTPError, e:
if DEBUG:
print e.msg
print e.fp.read()
raise
def stop_build(self, name, number):
'''
Stop a running Jenkins build.
@param name: Name of Jenkins job
@type name: str
@param number: Jenkins build number for the job
@type number: int
'''
request = urllib2.Request(self.server + STOP_BUILD % locals())
self.jenkins_open(request)
def cancel_queue(self, number):
'''
Cancel a queued build.
@param number: Jenkins queue number for the build
@type number: int
'''
# Jenkins returns a 302 from this URL, unless Referer is not set,
# then you get a 404.
request = urllib2.Request(self.server + CANCEL_QUEUE % locals(),
urllib.urlencode({}),
headers={'Referer': self.server})
self.jenkins_open(request)
def get_build_info(self, name, number):
'''
Get information for a build.
@param name: Name of Jenkins job
@type name: str
@param number: Jenkins build number for the job
@type number: int
@return: dictionary
'''
request = urllib2.Request(self.server + BUILD_INFO % locals())
return json.loads(self.jenkins_open(request))
def set_build_description(self, name, number, description):
'''
Get information for a build.
@param name: Name of Jenkins job
@type name: str
@param number: Jenkins build number for the job
@type number: int
@param description: Bulid description to set
@type description: str
'''
params = urllib.urlencode({'description': description})
request = urllib2.Request(self.server + BUILD_DESCRIPTION % locals(),
params)
self.jenkins_open(request)
class Jenkins(object):
log = logging.getLogger("zuul.Jenkins")
launch_retry_timeout = 5
def __init__(self, config, sched):
self.sched = sched
self.builds = {}
server = config.get('jenkins', 'server')
user = config.get('jenkins', 'user')
apikey = config.get('jenkins', 'apikey')
self.jenkins = ExtendedJenkins(server, user, apikey)
self.callback_thread = JenkinsCallback(self)
self.callback_thread.start()
self.cleanup_thread = JenkinsCleanup(self)
self.cleanup_thread.start()
def stop(self):
self.cleanup_thread.stop()
self.cleanup_thread.join()
#TODO: remove dependent_changes
def launch(self, job, change, pipeline, dependent_changes=[]):
self.log.info("Launch job %s for change %s with dependent changes %s" %
(job, change, dependent_changes))
dependent_changes = dependent_changes[:]
dependent_changes.reverse()
uuid = str(uuid4().hex)
params = dict(UUID=uuid, # deprecated
ZUUL_UUID=uuid,
GERRIT_PROJECT=change.project.name, # deprecated
ZUUL_PROJECT=change.project.name)
params['ZUUL_PIPELINE'] = pipeline.name
if hasattr(change, 'refspec'):
changes_str = '^'.join(
['%s:%s:%s' % (c.project.name, c.branch, c.refspec)
for c in dependent_changes + [change]])
params['GERRIT_BRANCH'] = change.branch # deprecated
params['ZUUL_BRANCH'] = change.branch
params['GERRIT_CHANGES'] = changes_str # deprecated
params['ZUUL_CHANGES'] = changes_str
params['ZUUL_REF'] = ('refs/zuul/%s/%s' %
(change.branch,
change.current_build_set.ref))
params['ZUUL_COMMIT'] = change.current_build_set.commit
zuul_changes = ' '.join(['%s,%s' % (c.number, c.patchset)
for c in dependent_changes + [change]])
params['ZUUL_CHANGE_IDS'] = zuul_changes
params['ZUUL_CHANGE'] = str(change.number)
params['ZUUL_PATCHSET'] = str(change.patchset)
if hasattr(change, 'ref'):
params['GERRIT_REFNAME'] = change.ref # deprecated
params['ZUUL_REFNAME'] = change.ref
params['GERRIT_OLDREV'] = change.oldrev # deprecated
params['ZUUL_OLDREV'] = change.oldrev
params['GERRIT_NEWREV'] = change.newrev # deprecated
params['ZUUL_NEWREV'] = change.newrev
params['ZUUL_SHORT_OLDREV'] = change.oldrev[:7]
params['ZUUL_SHORT_NEWREV'] = change.newrev[:7]
params['ZUUL_REF'] = change.ref
params['ZUUL_COMMIT'] = change.newrev
# This is what we should be heading toward for parameters:
# required:
# ZUUL_UUID
# ZUUL_REF (/refs/zuul/..., /refs/tags/foo, master)
# ZUUL_COMMIT
# optional:
# ZUUL_PROJECT
# ZUUL_PIPELINE
# optional (changes only):
# ZUUL_BRANCH
# ZUUL_CHANGE
# ZUUL_CHANGE_IDS
# ZUUL_PATCHSET
# optional (ref updated only):
# ZUUL_OLDREV
# ZUUL_NEWREV
# ZUUL_SHORT_NEWREV
# ZUUL_SHORT_OLDREV
if callable(job.parameter_function):
job.parameter_function(change, params)
self.log.debug("Custom parameter function used for job %s, "
"change: %s, params: %s" % (job, change, params))
build = Build(job, uuid)
# We can get the started notification on another thread before
# this is done so we add the build even before we trigger the
# job on Jenkins.
self.builds[uuid] = build
# Sometimes Jenkins may erroneously respond with a 404. Handle
# that by retrying for 30 seconds.
launched = False
errored = False
for count in range(6):
try:
self.jenkins.build_job(job.name, parameters=params)
launched = True
break
except:
errored = True
self.log.exception("Exception launching build %s for "
"job %s for change %s (will retry):" %
(build, job, change))
time.sleep(self.launch_retry_timeout)
if errored:
if launched:
self.log.error("Finally able to launch %s" % build)
else:
self.log.error("Unable to launch %s, even after retrying, "
"declaring lost" % build)
# To keep the queue moving, declare this as a lost build
# so that the change will get dropped.
self.onBuildCompleted(build.uuid, 'LOST', None, None)
return build
def findBuildInQueue(self, build):
for item in self.jenkins.get_queue_info():
if 'actions' not in item:
continue
for action in item['actions']:
if 'parameters' not in action:
continue
parameters = action['parameters']
for param in parameters:
# UUID is deprecated in favor of ZUUL_UUID
if ((param['name'] in ['ZUUL_UUID', 'UUID'])
and build.uuid == param['value']):
return item
return False
def cancel(self, build):
self.log.info("Cancel build %s for job %s" % (build, build.job))
if build.number:
self.log.debug("Build %s has already started" % build)
self.jenkins.stop_build(build.job.name, build.number)
self.log.debug("Canceled running build %s" % build)
return
else:
self.log.debug("Build %s has not started yet" % build)
self.log.debug("Looking for build %s in queue" % build)
item = self.findBuildInQueue(build)
if item:
self.log.debug("Found queue item %s for build %s" %
(item['id'], build))
try:
self.jenkins.cancel_queue(item['id'])
self.log.debug("Canceled queue item %s for build %s" %
(item['id'], build))
return
except:
self.log.exception("Exception canceling queue item %s "
"for build %s" % (item['id'], build))
self.log.debug("Still unable to find build %s to cancel" % build)
if build.number:
self.log.debug("Build %s has just started" % build)
self.jenkins.stop_build(build.job.name, build.number)
self.log.debug("Canceled just running build %s" % build)
else:
self.log.error("Build %s has not started but "
"was not found in queue" % build)
def setBuildDescription(self, build, description):
if not build.number:
return
try:
self.jenkins.set_build_description(build.job.name,
build.number,
description)
except:
self.log.exception("Exception setting build description for %s" %
build)
def onBuildCompleted(self, uuid, status, url, number):
self.log.info("Build %s #%s complete, status %s" %
(uuid, number, status))
build = self.builds.get(uuid)
if build:
self.log.debug("Found build %s" % build)
del self.builds[uuid]
if url:
build.url = url
build.result = status
build.number = number
self.sched.onBuildCompleted(build)
else:
self.log.error("Unable to find build %s" % uuid)
def onBuildStarted(self, uuid, url, number):
self.log.info("Build %s #%s started, url: %s" % (uuid, number, url))
build = self.builds.get(uuid)
if build:
self.log.debug("Found build %s" % build)
build.url = url
build.number = number
self.sched.onBuildStarted(build)
else:
self.log.error("Unable to find build %s" % uuid)
def lookForLostBuilds(self):
self.log.debug("Looking for lost builds")
lostbuilds = []
for build in self.builds.values():
if build.result:
# The build has finished, it will be removed
continue
if build.number:
# The build has started; see if it has finished
try:
info = self.jenkins.get_build_info(build.job.name,
build.number)
if hasattr(build, '_jenkins_missing_build_info'):
del build._jenkins_missing_build_info
except:
self.log.exception("Exception getting info for %s" % build)
# We can't look it up in jenkins. That could be transient.
# If it keeps up, assume it's permanent.
if hasattr(build, '_jenkins_missing_build_info'):
missing_time = build._jenkins_missing_build_info
if time.time() - missing_time > JENKINS_GRACE_TIME:
self.log.debug("Lost build %s because "
"it has started but "
"the build URL is not working" %
build)
lostbuilds.append(build)
else:
build._jenkins_missing_build_info = time.time()
continue
if not info:
self.log.debug("Lost build %s because "
"it started but "
"info can not be retreived" % build)
lostbuilds.append(build)
continue
if info['building']:
# It has not finished.
continue
if info['duration'] == 0:
# Possible jenkins bug -- not building, but no duration
self.log.debug("Possible jenkins bug with build %s: "
"not building, but no duration is set "
"Build info %s:" % (build,
pprint.pformat(info)))
continue
finish_time = (info['timestamp'] + info['duration']) / 1000
if time.time() - finish_time > JENKINS_GRACE_TIME:
self.log.debug("Lost build %s because "
"it finished more than 5 minutes ago. "
"Build info %s:" % (build,
pprint.pformat(info)))
lostbuilds.append(build)
continue
# Give it more time
else:
# The build has not started
if time.time() - build.launch_time < JENKINS_GRACE_TIME:
# It just started, give it a bit
continue
info = self.findBuildInQueue(build)
if info:
# It's in the queue. All good.
continue
if build.number:
# We just got notified it started
continue
# It may have just started. If we keep ending up here,
# assume the worst.
if hasattr(build, '_jenkins_missing_from_queue'):
missing_time = build._jenkins_missing_from_queue
if time.time() - missing_time > JENKINS_GRACE_TIME:
self.log.debug("Lost build %s because "
"it has not started and "
"is not in the queue" % build)
lostbuilds.append(build)
continue
else:
build._jenkins_missing_from_queue = time.time()
for build in lostbuilds:
self.log.error("Declaring %s lost" % build)
self.onBuildCompleted(build.uuid, 'LOST', None, None)

1
zuul/model.py
View File

@ -457,6 +457,7 @@ class Build(object):
self.launch_time = time.time() self.launch_time = time.time()
self.start_time = None self.start_time = None
self.end_time = None self.end_time = None
self.fraction_complete = None
def __repr__(self): def __repr__(self):
return '<Build %s of %s>' % (self.uuid, self.job.name) return '<Build %s of %s>' % (self.uuid, self.job.name)

50
zuul/webapp.py Normal file
View File

@ -0,0 +1,50 @@
# Copyright 2012 Hewlett-Packard Development Company, L.P.
# Copyright 2013 OpenStack Foundation
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import logging
import threading
from paste import httpserver
from webob import Request
class WebApp(threading.Thread):
log = logging.getLogger("zuul.WebApp")
def __init__(self, scheduler):
threading.Thread.__init__(self)
self.scheduler = scheduler
def run(self):
self.server = httpserver.serve(self.app, host='0.0.0.0', port='8001',
start_loop=False)
self.server.serve_forever()
def stop(self):
self.server.server_close()
def app(self, environ, start_response):
request = Request(environ)
if request.path == '/status.json':
try:
ret = self.scheduler.formatStatusJSON()
except:
self.log.exception("Exception formatting status:")
raise
start_response('200 OK', [('content-type', 'application/json'),
('Access-Control-Allow-Origin', '*')])
return [ret]
else:
start_response('404 Not Found', [('content-type', 'text/plain')])
return ['Not found.']