From e898716cb8c7c6156ae377b2912ca1696990f5e9 Mon Sep 17 00:00:00 2001 From: Joshua Hesketh Date: Thu, 13 Mar 2014 13:05:33 +1100 Subject: [PATCH] Make zuul more worker agnostic * Remove mention of Jenkins from the codebase * Remove unused dependency on python-jenkins * Update docs to be suggestive of Jenkins but not require Change-Id: I15f5462466899fb520aa66669f2e0b1d4fa1b98a --- doc/source/launchers.rst | 117 +++++++++++++++--------------- doc/source/merger.rst | 6 +- doc/source/statsd.rst | 8 +- etc/status/public_html/index.html | 2 +- requirements.txt | 1 - zuul/launcher/gearman.py | 2 +- 6 files changed, 68 insertions(+), 68 deletions(-) diff --git a/doc/source/launchers.rst b/doc/source/launchers.rst index d6e8b64c6e..1c00b50e71 100644 --- a/doc/source/launchers.rst +++ b/doc/source/launchers.rst @@ -22,61 +22,14 @@ launcher -- if it can trigger jobs, cancel them, and receive success or failure reports, it should be able to be used with Zuul. Patches to this effect are welcome. -Gearman -------- - -Gearman_ is a general-purpose protocol for distributing jobs to any -number of workers. Zuul works with Gearman by sending specific -information with job requests to Gearman, and expects certain -information to be returned on completion. This protocol is described -in `Zuul-Gearman Protocol`_. - -The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul -by providing an interface between Jenkins and Gearman. In this -configuration, Zuul asks Gearman to run jobs, and Gearman can then -distribute those jobs to any number of Jenkins systems (including -multiple Jenkins masters). - -In order for Zuul to run any jobs, you will need a running Gearman -server. Zuul includes a Gearman server, and it is recommended that it -be used as it supports the following features needed by Zuul: - -* Canceling jobs in the queue (admin protocol command "cancel job"). -* Strict FIFO queue operation (gearmand's round-robin mode may be - sufficient, but is untested). - -To enable the built-in server, see the ``gearman_server`` section of -``zuul.conf``. Be sure that the host allows connections from Zuul and -any workers (e.g., Jenkins masters) on TCP port 4730, and nowhere else -(as the Gearman protocol does not include any provision for -authentication). - -Gearman Jenkins Plugin ----------------------- - -The `Gearman Plugin`_ can be installed in Jenkins in order to -facilitate Jenkins running jobs for Zuul. Install the plugin and -configure it with the hostname or IP address of your Gearman server -and the port on which it is listening (4730 by default). It will -automatically register all known Jenkins jobs as functions that Zuul -can invoke via Gearman. - -Any number of masters can be configured in this way, and Gearman will -distribute jobs to all of them as appropriate. - -No special Jenkins job configuration is needed to support triggering -by Zuul. - Zuul Parameters --------------- -Zuul will pass some parameters with every job it launches. The -Gearman Plugin will ensure these are supplied as Jenkins build -parameters, so they will be available for use in the job configuration -as well as to the running job as environment variables. Builds can -be triggered either by an action on a change or by a reference update. -Both events share a common set of parameters and more specific -parameters as follows: +Zuul will pass some parameters with every job it launches. These are +for workers to be able to get the repositories into the state they are +intended to be tested in. Builds can be triggered either by an action +on a change or by a reference update. Both events share a common set +of parameters and more specific parameters as follows: Common parameters ~~~~~~~~~~~~~~~~~ @@ -158,8 +111,57 @@ And finally a reference being altered:: Your jobs can check whether the parameters are ``000000`` to act differently on each kind of event. +Gearman +------- + +Gearman_ is a general-purpose protocol for distributing jobs to any +number of workers. Zuul works with Gearman by sending specific +information with job requests to Gearman, and expects certain +information to be returned on completion. This protocol is described +in `Zuul-Gearman Protocol`_. + +In order for Zuul to run any jobs, you will need a running Gearman +server. Zuul includes a Gearman server, and it is recommended that it +be used as it supports the following features needed by Zuul: + +* Canceling jobs in the queue (admin protocol command "cancel job"). +* Strict FIFO queue operation (gearmand's round-robin mode may be + sufficient, but is untested). + +To enable the built-in server, see the ``gearman_server`` section of +``zuul.conf``. Be sure that the host allows connections from Zuul and +any workers (e.g., Jenkins masters) on TCP port 4730, and nowhere else +(as the Gearman protocol does not include any provision for +authentication). + +Gearman Jenkins Plugin +~~~~~~~~~~~~~~~~~~~~~~ + +The `Gearman Jenkins Plugin`_ makes it easy to use Jenkins with Zuul +by providing an interface between Jenkins and Gearman. In this +configuration, Zuul asks Gearman to run jobs, and Gearman can then +distribute those jobs to any number of Jenkins systems (including +multiple Jenkins masters). + +The `Gearman Plugin`_ can be installed in Jenkins in order to +facilitate Jenkins running jobs for Zuul. Install the plugin and +configure it with the hostname or IP address of your Gearman server +and the port on which it is listening (4730 by default). It will +automatically register all known Jenkins jobs as functions that Zuul +can invoke via Gearman. + +Any number of masters can be configured in this way, and Gearman will +distribute jobs to all of them as appropriate. + +No special Jenkins job configuration is needed to support triggering +by Zuul. + +The Gearman Plugin will ensure the `Zuul Parameters`_ are supplied as +Jenkins build parameters, so they will be available for use in the job +configuration as well as to the running job as environment variables. + Jenkins git plugin configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In order to test the correct build, configure the Jenkins Git SCM plugin as follows:: @@ -184,9 +186,8 @@ script to prepare the workspace for its integration testing: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh - Zuul-Gearman Protocol ---------------------- +~~~~~~~~~~~~~~~~~~~~~ This section is only relevant if you intend to implement a new kind of worker that runs jobs for Zuul via Gearman. If you just want to use @@ -195,7 +196,7 @@ Jenkins, see `Gearman Jenkins Plugin`_ instead. The Zuul protocol as used with Gearman is as follows: Starting Builds -~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^ To start a build, Zuul invokes a Gearman function with the following format: @@ -277,7 +278,7 @@ appropriate. A WORK_EXCEPTION packet will be interpreted as a WORK_FAIL, but the exception will be logged in Zuul's error log. Stopping Builds -~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^ If Zuul needs to abort a build already in progress, it will invoke the following function through Gearman: @@ -300,7 +301,7 @@ The original job is expected to complete with a WORK_DATA and WORK_FAIL packet as described in `Starting Builds`_. Build Descriptions -~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^ In order to update the job running system with a description of the current state of all related builds, the job runner may optionally diff --git a/doc/source/merger.rst b/doc/source/merger.rst index 4c445c685c..e7a9666bf9 100644 --- a/doc/source/merger.rst +++ b/doc/source/merger.rst @@ -33,8 +33,8 @@ and checking it out. The parameters that provide this information are described in :ref:`launchers`. These references need to be made available via a Git repository that -is available to Jenkins. This is accomplished by serving Zuul's Git -repositories directly. +is available to workers (such as Jenkins). This is accomplished by +serving Zuul's Git repositories directly. Serving Zuul Git Repos ~~~~~~~~~~~~~~~~~~~~~~ @@ -42,7 +42,7 @@ Serving Zuul Git Repos Zuul maintains its own copies of any needed Git repositories in the directory specified by ``git_dir`` in the ``merger`` section of zuul.conf (by default, /var/lib/zuul/git). To directly serve Zuul's -Git repositories in order to provide Zuul refs for Jenkins, you can +Git repositories in order to provide Zuul refs for workers, you can configure Apache to do so using the following directives:: SetEnv GIT_PROJECT_ROOT /var/lib/zuul/git diff --git a/doc/source/statsd.rst b/doc/source/statsd.rst index d657fb200f..f789d612a1 100644 --- a/doc/source/statsd.rst +++ b/doc/source/statsd.rst @@ -66,10 +66,10 @@ The metrics are emitted by the Zuul scheduler (`zuul/scheduler.py`): #. **** The triggered job name. #. **** Result as defined in your triggering system. For Jenkins that would be SUCCESS, FAILURE, UNSTABLE, LOST. The - metrics holds both an increasing counter and a timing reporting - the duration of the build. Whenever the result is a SUCCESS or - FAILURE, Zuul will additionally report the duration of the - build as a timing event. + metrics holds both an increasing counter and a timing + reporting the duration of the build. Whenever the result is a + SUCCESS or FAILURE, Zuul will additionally report the duration + of the build as a timing event. #. **resident_time** timing representing how long the Change has been known by Zuul (which includes build time and Zuul overhead). diff --git a/etc/status/public_html/index.html b/etc/status/public_html/index.html index f5efce0d24..d09f44fe31 100644 --- a/etc/status/public_html/index.html +++ b/etc/status/public_html/index.html @@ -130,7 +130,7 @@ under the License.

Zuul Status

-

Real-time status monitor of Zuul, the pipeline manager between Gerrit and Jenkins.

+

Real-time status monitor of Zuul, the pipeline manager between Gerrit and Workers.

diff --git a/requirements.txt b/requirements.txt index 92bb296ad9..4e9f29c210 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,7 +1,6 @@ pbr>=0.5.21,<1.0 argparse -python-jenkins PyYAML>=3.1.0 Paste WebOb>=1.2.3,<1.3 diff --git a/zuul/launcher/gearman.py b/zuul/launcher/gearman.py index 3638add3e8..081db2b9bb 100644 --- a/zuul/launcher/gearman.py +++ b/zuul/launcher/gearman.py @@ -27,7 +27,7 @@ from zuul.model import Build class GearmanCleanup(threading.Thread): """ A thread that checks to see if outstanding builds have completed without reporting back. """ - log = logging.getLogger("zuul.JenkinsCleanup") + log = logging.getLogger("zuul.GearmanCleanup") def __init__(self, gearman): threading.Thread.__init__(self)