Make zuul more worker agnostic
* Remove mention of Jenkins from the codebase * Remove unused dependency on python-jenkins * Update docs to be suggestive of Jenkins but not require Change-Id: I15f5462466899fb520aa66669f2e0b1d4fa1b98a
This commit is contained in:
Joshua Hesketh
parent
0781020be2
commit
e898716cb8
@ -22,61 +22,14 @@ launcher -- if it can trigger jobs, cancel them, and receive success
|
||||
or failure reports, it should be able to be used with Zuul. Patches
|
||||
to this effect are welcome.
|
||||
|
||||
Gearman
|
||||
-------
|
||||
|
||||
Gearman_ is a general-purpose protocol for distributing jobs to any
|
||||
number of workers. Zuul works with Gearman by sending specific
|
||||
information with job requests to Gearman, and expects certain
|
||||
information to be returned on completion. This protocol is described
|
||||
in `Zuul-Gearman Protocol`_.
|
||||
|
||||
The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul
|
||||
by providing an interface between Jenkins and Gearman. In this
|
||||
configuration, Zuul asks Gearman to run jobs, and Gearman can then
|
||||
distribute those jobs to any number of Jenkins systems (including
|
||||
multiple Jenkins masters).
|
||||
|
||||
In order for Zuul to run any jobs, you will need a running Gearman
|
||||
server. Zuul includes a Gearman server, and it is recommended that it
|
||||
be used as it supports the following features needed by Zuul:
|
||||
|
||||
* Canceling jobs in the queue (admin protocol command "cancel job").
|
||||
* Strict FIFO queue operation (gearmand's round-robin mode may be
|
||||
sufficient, but is untested).
|
||||
|
||||
To enable the built-in server, see the ``gearman_server`` section of
|
||||
``zuul.conf``. Be sure that the host allows connections from Zuul and
|
||||
any workers (e.g., Jenkins masters) on TCP port 4730, and nowhere else
|
||||
(as the Gearman protocol does not include any provision for
|
||||
authentication).
|
||||
|
||||
Gearman Jenkins Plugin
|
||||
----------------------
|
||||
|
||||
The `Gearman Plugin`_ can be installed in Jenkins in order to
|
||||
facilitate Jenkins running jobs for Zuul. Install the plugin and
|
||||
configure it with the hostname or IP address of your Gearman server
|
||||
and the port on which it is listening (4730 by default). It will
|
||||
automatically register all known Jenkins jobs as functions that Zuul
|
||||
can invoke via Gearman.
|
||||
|
||||
Any number of masters can be configured in this way, and Gearman will
|
||||
distribute jobs to all of them as appropriate.
|
||||
|
||||
No special Jenkins job configuration is needed to support triggering
|
||||
by Zuul.
|
||||
|
||||
Zuul Parameters
|
||||
---------------
|
||||
|
||||
Zuul will pass some parameters with every job it launches. The
|
||||
Gearman Plugin will ensure these are supplied as Jenkins build
|
||||
parameters, so they will be available for use in the job configuration
|
||||
as well as to the running job as environment variables. Builds can
|
||||
be triggered either by an action on a change or by a reference update.
|
||||
Both events share a common set of parameters and more specific
|
||||
parameters as follows:
|
||||
Zuul will pass some parameters with every job it launches. These are
|
||||
for workers to be able to get the repositories into the state they are
|
||||
intended to be tested in. Builds can be triggered either by an action
|
||||
on a change or by a reference update. Both events share a common set
|
||||
of parameters and more specific parameters as follows:
|
||||
|
||||
Common parameters
|
||||
~~~~~~~~~~~~~~~~~
|
||||
@ -158,8 +111,57 @@ And finally a reference being altered::
|
||||
Your jobs can check whether the parameters are ``000000`` to act
|
||||
differently on each kind of event.
|
||||
|
||||
Gearman
|
||||
-------
|
||||
|
||||
Gearman_ is a general-purpose protocol for distributing jobs to any
|
||||
number of workers. Zuul works with Gearman by sending specific
|
||||
information with job requests to Gearman, and expects certain
|
||||
information to be returned on completion. This protocol is described
|
||||
in `Zuul-Gearman Protocol`_.
|
||||
|
||||
In order for Zuul to run any jobs, you will need a running Gearman
|
||||
server. Zuul includes a Gearman server, and it is recommended that it
|
||||
be used as it supports the following features needed by Zuul:
|
||||
|
||||
* Canceling jobs in the queue (admin protocol command "cancel job").
|
||||
* Strict FIFO queue operation (gearmand's round-robin mode may be
|
||||
sufficient, but is untested).
|
||||
|
||||
To enable the built-in server, see the ``gearman_server`` section of
|
||||
``zuul.conf``. Be sure that the host allows connections from Zuul and
|
||||
any workers (e.g., Jenkins masters) on TCP port 4730, and nowhere else
|
||||
(as the Gearman protocol does not include any provision for
|
||||
authentication).
|
||||
|
||||
Gearman Jenkins Plugin
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul
|
||||
by providing an interface between Jenkins and Gearman. In this
|
||||
configuration, Zuul asks Gearman to run jobs, and Gearman can then
|
||||
distribute those jobs to any number of Jenkins systems (including
|
||||
multiple Jenkins masters).
|
||||
|
||||
The `Gearman Plugin`_ can be installed in Jenkins in order to
|
||||
facilitate Jenkins running jobs for Zuul. Install the plugin and
|
||||
configure it with the hostname or IP address of your Gearman server
|
||||
and the port on which it is listening (4730 by default). It will
|
||||
automatically register all known Jenkins jobs as functions that Zuul
|
||||
can invoke via Gearman.
|
||||
|
||||
Any number of masters can be configured in this way, and Gearman will
|
||||
distribute jobs to all of them as appropriate.
|
||||
|
||||
No special Jenkins job configuration is needed to support triggering
|
||||
by Zuul.
|
||||
|
||||
The Gearman Plugin will ensure the `Zuul Parameters`_ are supplied as
|
||||
Jenkins build parameters, so they will be available for use in the job
|
||||
configuration as well as to the running job as environment variables.
|
||||
|
||||
Jenkins git plugin configuration
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In order to test the correct build, configure the Jenkins Git SCM
|
||||
plugin as follows::
|
||||
@ -184,9 +186,8 @@ script to prepare the workspace for its integration testing:
|
||||
|
||||
https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh
|
||||
|
||||
|
||||
Zuul-Gearman Protocol
|
||||
---------------------
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This section is only relevant if you intend to implement a new kind of
|
||||
worker that runs jobs for Zuul via Gearman. If you just want to use
|
||||
@ -195,7 +196,7 @@ Jenkins, see `Gearman Jenkins Plugin`_ instead.
|
||||
The Zuul protocol as used with Gearman is as follows:
|
||||
|
||||
Starting Builds
|
||||
~~~~~~~~~~~~~~~
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
To start a build, Zuul invokes a Gearman function with the following
|
||||
format:
|
||||
@ -277,7 +278,7 @@ appropriate. A WORK_EXCEPTION packet will be interpreted as a
|
||||
WORK_FAIL, but the exception will be logged in Zuul's error log.
|
||||
|
||||
Stopping Builds
|
||||
~~~~~~~~~~~~~~~
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
If Zuul needs to abort a build already in progress, it will invoke the
|
||||
following function through Gearman:
|
||||
@ -300,7 +301,7 @@ The original job is expected to complete with a WORK_DATA and
|
||||
WORK_FAIL packet as described in `Starting Builds`_.
|
||||
|
||||
Build Descriptions
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In order to update the job running system with a description of the
|
||||
current state of all related builds, the job runner may optionally
|
||||
|
||||
@ -33,8 +33,8 @@ and checking it out. The parameters that provide this information are
|
||||
described in :ref:`launchers`.
|
||||
|
||||
These references need to be made available via a Git repository that
|
||||
is available to Jenkins. This is accomplished by serving Zuul's Git
|
||||
repositories directly.
|
||||
is available to workers (such as Jenkins). This is accomplished by
|
||||
serving Zuul's Git repositories directly.
|
||||
|
||||
Serving Zuul Git Repos
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
@ -42,7 +42,7 @@ Serving Zuul Git Repos
|
||||
Zuul maintains its own copies of any needed Git repositories in the
|
||||
directory specified by ``git_dir`` in the ``merger`` section of
|
||||
zuul.conf (by default, /var/lib/zuul/git). To directly serve Zuul's
|
||||
Git repositories in order to provide Zuul refs for Jenkins, you can
|
||||
Git repositories in order to provide Zuul refs for workers, you can
|
||||
configure Apache to do so using the following directives::
|
||||
|
||||
SetEnv GIT_PROJECT_ROOT /var/lib/zuul/git
|
||||
|
||||
@ -66,10 +66,10 @@ The metrics are emitted by the Zuul scheduler (`zuul/scheduler.py`):
|
||||
#. **<jobname>** The triggered job name.
|
||||
#. **<build result>** Result as defined in your triggering system. For
|
||||
Jenkins that would be SUCCESS, FAILURE, UNSTABLE, LOST. The
|
||||
metrics holds both an increasing counter and a timing reporting
|
||||
the duration of the build. Whenever the result is a SUCCESS or
|
||||
FAILURE, Zuul will additionally report the duration of the
|
||||
build as a timing event.
|
||||
metrics holds both an increasing counter and a timing
|
||||
reporting the duration of the build. Whenever the result is a
|
||||
SUCCESS or FAILURE, Zuul will additionally report the duration
|
||||
of the build as a timing event.
|
||||
|
||||
#. **resident_time** timing representing how long the Change has been
|
||||
known by Zuul (which includes build time and Zuul overhead).
|
||||
|
||||
@ -130,7 +130,7 @@ under the License.
|
||||
<body>
|
||||
<div class="container">
|
||||
<h1>Zuul Status</h1>
|
||||
<p>Real-time status monitor of Zuul, the pipeline manager between Gerrit and Jenkins.</p>
|
||||
<p>Real-time status monitor of Zuul, the pipeline manager between Gerrit and Workers.</p>
|
||||
|
||||
<div class="zuul-container" id="zuul-container"></div>
|
||||
</div>
|
||||
|
||||
@ -1,7 +1,6 @@
|
||||
pbr>=0.5.21,<1.0
|
||||
|
||||
argparse
|
||||
python-jenkins
|
||||
PyYAML>=3.1.0
|
||||
Paste
|
||||
WebOb>=1.2.3,<1.3
|
||||
|
||||
@ -27,7 +27,7 @@ from zuul.model import Build
|
||||
class GearmanCleanup(threading.Thread):
|
||||
""" A thread that checks to see if outstanding builds have
|
||||
completed without reporting back. """
|
||||
log = logging.getLogger("zuul.JenkinsCleanup")
|
||||
log = logging.getLogger("zuul.GearmanCleanup")
|
||||
|
||||
def __init__(self, gearman):
|
||||
threading.Thread.__init__(self)
|
||||
|
||||
Loading…
Reference in New Issue
Block a user