Merge "Make zuul more worker agnostic"
This commit is contained in:
commit
666dfa6f45
|
@ -22,61 +22,14 @@ launcher -- if it can trigger jobs, cancel them, and receive success
|
||||||
or failure reports, it should be able to be used with Zuul. Patches
|
or failure reports, it should be able to be used with Zuul. Patches
|
||||||
to this effect are welcome.
|
to this effect are welcome.
|
||||||
|
|
||||||
Gearman
|
|
||||||
-------
|
|
||||||
|
|
||||||
Gearman_ is a general-purpose protocol for distributing jobs to any
|
|
||||||
number of workers. Zuul works with Gearman by sending specific
|
|
||||||
information with job requests to Gearman, and expects certain
|
|
||||||
information to be returned on completion. This protocol is described
|
|
||||||
in `Zuul-Gearman Protocol`_.
|
|
||||||
|
|
||||||
The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul
|
|
||||||
by providing an interface between Jenkins and Gearman. In this
|
|
||||||
configuration, Zuul asks Gearman to run jobs, and Gearman can then
|
|
||||||
distribute those jobs to any number of Jenkins systems (including
|
|
||||||
multiple Jenkins masters).
|
|
||||||
|
|
||||||
In order for Zuul to run any jobs, you will need a running Gearman
|
|
||||||
server. Zuul includes a Gearman server, and it is recommended that it
|
|
||||||
be used as it supports the following features needed by Zuul:
|
|
||||||
|
|
||||||
* Canceling jobs in the queue (admin protocol command "cancel job").
|
|
||||||
* Strict FIFO queue operation (gearmand's round-robin mode may be
|
|
||||||
sufficient, but is untested).
|
|
||||||
|
|
||||||
To enable the built-in server, see the ``gearman_server`` section of
|
|
||||||
``zuul.conf``. Be sure that the host allows connections from Zuul and
|
|
||||||
any workers (e.g., Jenkins masters) on TCP port 4730, and nowhere else
|
|
||||||
(as the Gearman protocol does not include any provision for
|
|
||||||
authentication).
|
|
||||||
|
|
||||||
Gearman Jenkins Plugin
|
|
||||||
----------------------
|
|
||||||
|
|
||||||
The `Gearman Plugin`_ can be installed in Jenkins in order to
|
|
||||||
facilitate Jenkins running jobs for Zuul. Install the plugin and
|
|
||||||
configure it with the hostname or IP address of your Gearman server
|
|
||||||
and the port on which it is listening (4730 by default). It will
|
|
||||||
automatically register all known Jenkins jobs as functions that Zuul
|
|
||||||
can invoke via Gearman.
|
|
||||||
|
|
||||||
Any number of masters can be configured in this way, and Gearman will
|
|
||||||
distribute jobs to all of them as appropriate.
|
|
||||||
|
|
||||||
No special Jenkins job configuration is needed to support triggering
|
|
||||||
by Zuul.
|
|
||||||
|
|
||||||
Zuul Parameters
|
Zuul Parameters
|
||||||
---------------
|
---------------
|
||||||
|
|
||||||
Zuul will pass some parameters with every job it launches. The
|
Zuul will pass some parameters with every job it launches. These are
|
||||||
Gearman Plugin will ensure these are supplied as Jenkins build
|
for workers to be able to get the repositories into the state they are
|
||||||
parameters, so they will be available for use in the job configuration
|
intended to be tested in. Builds can be triggered either by an action
|
||||||
as well as to the running job as environment variables. Builds can
|
on a change or by a reference update. Both events share a common set
|
||||||
be triggered either by an action on a change or by a reference update.
|
of parameters and more specific parameters as follows:
|
||||||
Both events share a common set of parameters and more specific
|
|
||||||
parameters as follows:
|
|
||||||
|
|
||||||
Common parameters
|
Common parameters
|
||||||
~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~
|
||||||
|
@ -158,8 +111,57 @@ And finally a reference being altered::
|
||||||
Your jobs can check whether the parameters are ``000000`` to act
|
Your jobs can check whether the parameters are ``000000`` to act
|
||||||
differently on each kind of event.
|
differently on each kind of event.
|
||||||
|
|
||||||
|
Gearman
|
||||||
|
-------
|
||||||
|
|
||||||
|
Gearman_ is a general-purpose protocol for distributing jobs to any
|
||||||
|
number of workers. Zuul works with Gearman by sending specific
|
||||||
|
information with job requests to Gearman, and expects certain
|
||||||
|
information to be returned on completion. This protocol is described
|
||||||
|
in `Zuul-Gearman Protocol`_.
|
||||||
|
|
||||||
|
In order for Zuul to run any jobs, you will need a running Gearman
|
||||||
|
server. Zuul includes a Gearman server, and it is recommended that it
|
||||||
|
be used as it supports the following features needed by Zuul:
|
||||||
|
|
||||||
|
* Canceling jobs in the queue (admin protocol command "cancel job").
|
||||||
|
* Strict FIFO queue operation (gearmand's round-robin mode may be
|
||||||
|
sufficient, but is untested).
|
||||||
|
|
||||||
|
To enable the built-in server, see the ``gearman_server`` section of
|
||||||
|
``zuul.conf``. Be sure that the host allows connections from Zuul and
|
||||||
|
any workers (e.g., Jenkins masters) on TCP port 4730, and nowhere else
|
||||||
|
(as the Gearman protocol does not include any provision for
|
||||||
|
authentication).
|
||||||
|
|
||||||
|
Gearman Jenkins Plugin
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul
|
||||||
|
by providing an interface between Jenkins and Gearman. In this
|
||||||
|
configuration, Zuul asks Gearman to run jobs, and Gearman can then
|
||||||
|
distribute those jobs to any number of Jenkins systems (including
|
||||||
|
multiple Jenkins masters).
|
||||||
|
|
||||||
|
The `Gearman Plugin`_ can be installed in Jenkins in order to
|
||||||
|
facilitate Jenkins running jobs for Zuul. Install the plugin and
|
||||||
|
configure it with the hostname or IP address of your Gearman server
|
||||||
|
and the port on which it is listening (4730 by default). It will
|
||||||
|
automatically register all known Jenkins jobs as functions that Zuul
|
||||||
|
can invoke via Gearman.
|
||||||
|
|
||||||
|
Any number of masters can be configured in this way, and Gearman will
|
||||||
|
distribute jobs to all of them as appropriate.
|
||||||
|
|
||||||
|
No special Jenkins job configuration is needed to support triggering
|
||||||
|
by Zuul.
|
||||||
|
|
||||||
|
The Gearman Plugin will ensure the `Zuul Parameters`_ are supplied as
|
||||||
|
Jenkins build parameters, so they will be available for use in the job
|
||||||
|
configuration as well as to the running job as environment variables.
|
||||||
|
|
||||||
Jenkins git plugin configuration
|
Jenkins git plugin configuration
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
In order to test the correct build, configure the Jenkins Git SCM
|
In order to test the correct build, configure the Jenkins Git SCM
|
||||||
plugin as follows::
|
plugin as follows::
|
||||||
|
@ -184,9 +186,8 @@ script to prepare the workspace for its integration testing:
|
||||||
|
|
||||||
https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh
|
https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh
|
||||||
|
|
||||||
|
|
||||||
Zuul-Gearman Protocol
|
Zuul-Gearman Protocol
|
||||||
---------------------
|
~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
This section is only relevant if you intend to implement a new kind of
|
This section is only relevant if you intend to implement a new kind of
|
||||||
worker that runs jobs for Zuul via Gearman. If you just want to use
|
worker that runs jobs for Zuul via Gearman. If you just want to use
|
||||||
|
@ -195,7 +196,7 @@ Jenkins, see `Gearman Jenkins Plugin`_ instead.
|
||||||
The Zuul protocol as used with Gearman is as follows:
|
The Zuul protocol as used with Gearman is as follows:
|
||||||
|
|
||||||
Starting Builds
|
Starting Builds
|
||||||
~~~~~~~~~~~~~~~
|
^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
To start a build, Zuul invokes a Gearman function with the following
|
To start a build, Zuul invokes a Gearman function with the following
|
||||||
format:
|
format:
|
||||||
|
@ -277,7 +278,7 @@ appropriate. A WORK_EXCEPTION packet will be interpreted as a
|
||||||
WORK_FAIL, but the exception will be logged in Zuul's error log.
|
WORK_FAIL, but the exception will be logged in Zuul's error log.
|
||||||
|
|
||||||
Stopping Builds
|
Stopping Builds
|
||||||
~~~~~~~~~~~~~~~
|
^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
If Zuul needs to abort a build already in progress, it will invoke the
|
If Zuul needs to abort a build already in progress, it will invoke the
|
||||||
following function through Gearman:
|
following function through Gearman:
|
||||||
|
@ -300,7 +301,7 @@ The original job is expected to complete with a WORK_DATA and
|
||||||
WORK_FAIL packet as described in `Starting Builds`_.
|
WORK_FAIL packet as described in `Starting Builds`_.
|
||||||
|
|
||||||
Build Descriptions
|
Build Descriptions
|
||||||
~~~~~~~~~~~~~~~~~~
|
^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
In order to update the job running system with a description of the
|
In order to update the job running system with a description of the
|
||||||
current state of all related builds, the job runner may optionally
|
current state of all related builds, the job runner may optionally
|
||||||
|
|
|
@ -33,8 +33,8 @@ and checking it out. The parameters that provide this information are
|
||||||
described in :ref:`launchers`.
|
described in :ref:`launchers`.
|
||||||
|
|
||||||
These references need to be made available via a Git repository that
|
These references need to be made available via a Git repository that
|
||||||
is available to Jenkins. This is accomplished by serving Zuul's Git
|
is available to workers (such as Jenkins). This is accomplished by
|
||||||
repositories directly.
|
serving Zuul's Git repositories directly.
|
||||||
|
|
||||||
Serving Zuul Git Repos
|
Serving Zuul Git Repos
|
||||||
~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
@ -42,7 +42,7 @@ Serving Zuul Git Repos
|
||||||
Zuul maintains its own copies of any needed Git repositories in the
|
Zuul maintains its own copies of any needed Git repositories in the
|
||||||
directory specified by ``git_dir`` in the ``merger`` section of
|
directory specified by ``git_dir`` in the ``merger`` section of
|
||||||
zuul.conf (by default, /var/lib/zuul/git). To directly serve Zuul's
|
zuul.conf (by default, /var/lib/zuul/git). To directly serve Zuul's
|
||||||
Git repositories in order to provide Zuul refs for Jenkins, you can
|
Git repositories in order to provide Zuul refs for workers, you can
|
||||||
configure Apache to do so using the following directives::
|
configure Apache to do so using the following directives::
|
||||||
|
|
||||||
SetEnv GIT_PROJECT_ROOT /var/lib/zuul/git
|
SetEnv GIT_PROJECT_ROOT /var/lib/zuul/git
|
||||||
|
|
|
@ -66,10 +66,10 @@ The metrics are emitted by the Zuul scheduler (`zuul/scheduler.py`):
|
||||||
#. **<jobname>** The triggered job name.
|
#. **<jobname>** The triggered job name.
|
||||||
#. **<build result>** Result as defined in your triggering system. For
|
#. **<build result>** Result as defined in your triggering system. For
|
||||||
Jenkins that would be SUCCESS, FAILURE, UNSTABLE, LOST. The
|
Jenkins that would be SUCCESS, FAILURE, UNSTABLE, LOST. The
|
||||||
metrics holds both an increasing counter and a timing reporting
|
metrics holds both an increasing counter and a timing
|
||||||
the duration of the build. Whenever the result is a SUCCESS or
|
reporting the duration of the build. Whenever the result is a
|
||||||
FAILURE, Zuul will additionally report the duration of the
|
SUCCESS or FAILURE, Zuul will additionally report the duration
|
||||||
build as a timing event.
|
of the build as a timing event.
|
||||||
|
|
||||||
#. **resident_time** timing representing how long the Change has been
|
#. **resident_time** timing representing how long the Change has been
|
||||||
known by Zuul (which includes build time and Zuul overhead).
|
known by Zuul (which includes build time and Zuul overhead).
|
||||||
|
|
|
@ -130,7 +130,7 @@ under the License.
|
||||||
<body>
|
<body>
|
||||||
<div class="container">
|
<div class="container">
|
||||||
<h1>Zuul Status</h1>
|
<h1>Zuul Status</h1>
|
||||||
<p>Real-time status monitor of Zuul, the pipeline manager between Gerrit and Jenkins.</p>
|
<p>Real-time status monitor of Zuul, the pipeline manager between Gerrit and Workers.</p>
|
||||||
|
|
||||||
<div class="zuul-container" id="zuul-container"></div>
|
<div class="zuul-container" id="zuul-container"></div>
|
||||||
</div>
|
</div>
|
||||||
|
|
|
@ -1,7 +1,6 @@
|
||||||
pbr>=0.5.21,<1.0
|
pbr>=0.5.21,<1.0
|
||||||
|
|
||||||
argparse
|
argparse
|
||||||
python-jenkins
|
|
||||||
PyYAML>=3.1.0
|
PyYAML>=3.1.0
|
||||||
Paste
|
Paste
|
||||||
WebOb>=1.2.3,<1.3
|
WebOb>=1.2.3,<1.3
|
||||||
|
|
|
@ -27,7 +27,7 @@ from zuul.model import Build
|
||||||
class GearmanCleanup(threading.Thread):
|
class GearmanCleanup(threading.Thread):
|
||||||
""" A thread that checks to see if outstanding builds have
|
""" A thread that checks to see if outstanding builds have
|
||||||
completed without reporting back. """
|
completed without reporting back. """
|
||||||
log = logging.getLogger("zuul.JenkinsCleanup")
|
log = logging.getLogger("zuul.GearmanCleanup")
|
||||||
|
|
||||||
def __init__(self, gearman):
|
def __init__(self, gearman):
|
||||||
threading.Thread.__init__(self)
|
threading.Thread.__init__(self)
|
||||||
|
|
Loading…
Reference in New Issue