zuul
/
zuul
Code Issues Proposed changes
Browse Source

Merge "Make zuul more worker agnostic"

changes/36/70636/7
Jenkins 8 years ago committed by Gerrit Code Review
parent
9a39d8be3b e898716cb8
commit
666dfa6f45
6 changed files with 68 additions and 68 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 117
      doc/source/launchers.rst
  2. 6
      doc/source/merger.rst
  3. 8
      doc/source/statsd.rst
  4. 2
      etc/status/public_html/index.html
  5. 1
      requirements.txt
  6. 2
      zuul/launcher/gearman.py

117
doc/source/launchers.rst
Unescape View File

@ -22,61 +22,14 @@ launcher -- if it can trigger jobs, cancel them, and receive success
or failure reports, it should be able to be used with Zuul. Patches
to this effect are welcome.
Gearman
-------
Gearman_ is a general-purpose protocol for distributing jobs to any
number of workers. Zuul works with Gearman by sending specific
information with job requests to Gearman, and expects certain
information to be returned on completion. This protocol is described
in `Zuul-Gearman Protocol`_.
The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul
by providing an interface between Jenkins and Gearman. In this
configuration, Zuul asks Gearman to run jobs, and Gearman can then
distribute those jobs to any number of Jenkins systems (including
multiple Jenkins masters).
In order for Zuul to run any jobs, you will need a running Gearman
server. Zuul includes a Gearman server, and it is recommended that it
be used as it supports the following features needed by Zuul:
* Canceling jobs in the queue (admin protocol command "cancel job").
* Strict FIFO queue operation (gearmand's round-robin mode may be
sufficient, but is untested).
To enable the built-in server, see the ``gearman_server`` section of
``zuul.conf``. Be sure that the host allows connections from Zuul and
any workers (e.g., Jenkins masters) on TCP port 4730, and nowhere else
(as the Gearman protocol does not include any provision for
authentication).
Gearman Jenkins Plugin
----------------------
The `Gearman Plugin`_ can be installed in Jenkins in order to
facilitate Jenkins running jobs for Zuul. Install the plugin and
configure it with the hostname or IP address of your Gearman server
and the port on which it is listening (4730 by default). It will
automatically register all known Jenkins jobs as functions that Zuul
can invoke via Gearman.
Any number of masters can be configured in this way, and Gearman will
distribute jobs to all of them as appropriate.
No special Jenkins job configuration is needed to support triggering
by Zuul.
Zuul Parameters
---------------
Zuul will pass some parameters with every job it launches. The
Gearman Plugin will ensure these are supplied as Jenkins build
parameters, so they will be available for use in the job configuration
as well as to the running job as environment variables. Builds can
be triggered either by an action on a change or by a reference update.
Both events share a common set of parameters and more specific
parameters as follows:
Zuul will pass some parameters with every job it launches. These are
for workers to be able to get the repositories into the state they are
intended to be tested in. Builds can be triggered either by an action
on a change or by a reference update. Both events share a common set
of parameters and more specific parameters as follows:
Common parameters
~~~~~~~~~~~~~~~~~
@ -158,8 +111,57 @@ And finally a reference being altered::
Your jobs can check whether the parameters are ``000000`` to act
differently on each kind of event.
Gearman
-------
Gearman_ is a general-purpose protocol for distributing jobs to any
number of workers. Zuul works with Gearman by sending specific
information with job requests to Gearman, and expects certain
information to be returned on completion. This protocol is described
in `Zuul-Gearman Protocol`_.
In order for Zuul to run any jobs, you will need a running Gearman
server. Zuul includes a Gearman server, and it is recommended that it
be used as it supports the following features needed by Zuul:
* Canceling jobs in the queue (admin protocol command "cancel job").
* Strict FIFO queue operation (gearmand's round-robin mode may be
sufficient, but is untested).
To enable the built-in server, see the ``gearman_server`` section of
``zuul.conf``. Be sure that the host allows connections from Zuul and
any workers (e.g., Jenkins masters) on TCP port 4730, and nowhere else
(as the Gearman protocol does not include any provision for
authentication).
Gearman Jenkins Plugin
~~~~~~~~~~~~~~~~~~~~~~
The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul
by providing an interface between Jenkins and Gearman. In this
configuration, Zuul asks Gearman to run jobs, and Gearman can then
distribute those jobs to any number of Jenkins systems (including
multiple Jenkins masters).
The `Gearman Plugin`_ can be installed in Jenkins in order to
facilitate Jenkins running jobs for Zuul. Install the plugin and
configure it with the hostname or IP address of your Gearman server
and the port on which it is listening (4730 by default). It will
automatically register all known Jenkins jobs as functions that Zuul
can invoke via Gearman.
Any number of masters can be configured in this way, and Gearman will
distribute jobs to all of them as appropriate.
No special Jenkins job configuration is needed to support triggering
by Zuul.
The Gearman Plugin will ensure the `Zuul Parameters`_ are supplied as
Jenkins build parameters, so they will be available for use in the job
configuration as well as to the running job as environment variables.
Jenkins git plugin configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In order to test the correct build, configure the Jenkins Git SCM
plugin as follows::
@ -184,9 +186,8 @@ script to prepare the workspace for its integration testing:
https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh
Zuul-Gearman Protocol
---------------------
~~~~~~~~~~~~~~~~~~~~~
This section is only relevant if you intend to implement a new kind of
worker that runs jobs for Zuul via Gearman. If you just want to use
@ -195,7 +196,7 @@ Jenkins, see `Gearman Jenkins Plugin`_ instead.
The Zuul protocol as used with Gearman is as follows:
Starting Builds
~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^
To start a build, Zuul invokes a Gearman function with the following
format:
@ -277,7 +278,7 @@ appropriate. A WORK_EXCEPTION packet will be interpreted as a
WORK_FAIL, but the exception will be logged in Zuul's error log.
Stopping Builds
~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^
If Zuul needs to abort a build already in progress, it will invoke the
following function through Gearman:
@ -300,7 +301,7 @@ The original job is expected to complete with a WORK_DATA and
WORK_FAIL packet as described in `Starting Builds`_.
Build Descriptions
~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^
In order to update the job running system with a description of the
current state of all related builds, the job runner may optionally

6
doc/source/merger.rst
Unescape View File

@ -33,8 +33,8 @@ and checking it out. The parameters that provide this information are
described in :ref:`launchers`.
These references need to be made available via a Git repository that
is available to Jenkins. This is accomplished by serving Zuul's Git
repositories directly.
is available to workers (such as Jenkins). This is accomplished by
serving Zuul's Git repositories directly.
Serving Zuul Git Repos
~~~~~~~~~~~~~~~~~~~~~~
@ -42,7 +42,7 @@ Serving Zuul Git Repos
Zuul maintains its own copies of any needed Git repositories in the
directory specified by ``git_dir`` in the ``merger`` section of
zuul.conf (by default, /var/lib/zuul/git). To directly serve Zuul's
Git repositories in order to provide Zuul refs for Jenkins, you can
Git repositories in order to provide Zuul refs for workers, you can
configure Apache to do so using the following directives::
SetEnv GIT_PROJECT_ROOT /var/lib/zuul/git

8
doc/source/statsd.rst
Unescape View File

@ -66,10 +66,10 @@ The metrics are emitted by the Zuul scheduler (`zuul/scheduler.py`):
#. **<jobname>** The triggered job name.
#. **<build result>** Result as defined in your triggering system. For
Jenkins that would be SUCCESS, FAILURE, UNSTABLE, LOST. The
metrics holds both an increasing counter and a timing reporting
the duration of the build. Whenever the result is a SUCCESS or
FAILURE, Zuul will additionally report the duration of the
build as a timing event.
metrics holds both an increasing counter and a timing
reporting the duration of the build. Whenever the result is a
SUCCESS or FAILURE, Zuul will additionally report the duration
of the build as a timing event.
#. **resident_time** timing representing how long the Change has been
known by Zuul (which includes build time and Zuul overhead).

2
etc/status/public_html/index.html
Unescape View File

@ -130,7 +130,7 @@ under the License.
<body>
<div class="container">
<h1>Zuul Status</h1>
<p>Real-time status monitor of Zuul, the pipeline manager between Gerrit and Jenkins.</p>
<p>Real-time status monitor of Zuul, the pipeline manager between Gerrit and Workers.</p>
<div class="zuul-container" id="zuul-container"></div>
</div>

1
requirements.txt
Unescape View File

@ -1,7 +1,6 @@
pbr>=0.5.21,<1.0
argparse
python-jenkins
PyYAML>=3.1.0
Paste
WebOb>=1.2.3,<1.3

2
zuul/launcher/gearman.py
Unescape View File

@ -27,7 +27,7 @@ from zuul.model import Build
class GearmanCleanup(threading.Thread):
""" A thread that checks to see if outstanding builds have
completed without reporting back. """
log = logging.getLogger("zuul.JenkinsCleanup")
log = logging.getLogger("zuul.GearmanCleanup")
def __init__(self, gearman):
threading.Thread.__init__(self)

Write Preview
Loading…
Cancel
Save