236 lines
6.4 KiB
ReStructuredText
236 lines
6.4 KiB
ReStructuredText
:title: Jenkins Job Builder
|
|
|
|
.. _jjb:
|
|
|
|
Jenkins Job Builder
|
|
###################
|
|
|
|
Jenkins Job Builder is a system for configuring Jenkins jobs using
|
|
simple YAML files stored in Git.
|
|
|
|
At a Glance
|
|
===========
|
|
|
|
:Hosts:
|
|
* http://zm*.openstack.org
|
|
:Configuration:
|
|
* :config:`jenkins/jobs/`
|
|
:Projects:
|
|
* https://git.openstack.org/cgit/openstack-infra/jenkins-job-builder
|
|
:Bugs:
|
|
* https://storyboard.openstack.org/#!/project/723
|
|
:Resources:
|
|
* `Reference Manual <http://docs.openstack.org/infra/jenkins-job-builder>`_
|
|
|
|
Overview
|
|
========
|
|
|
|
In order to make the process of managing thousands of Jenkins jobs
|
|
easier, Jenkins Job Builder was designed to take YAML based
|
|
configurations and convert those into jobs that are injected into
|
|
Jenkins.
|
|
|
|
The documentation below describes how the OpenStack Infrastructure
|
|
team uses the Jenkins Job Builder in our environment.
|
|
|
|
Configuring Projects
|
|
====================
|
|
|
|
The YAML scripts to make this work are stored in the
|
|
:config:`jenkins/jobs/` directory of the project-config repository.
|
|
In this directory you can have four different types of yaml config
|
|
files:
|
|
|
|
* Jenkins Jobs Defaults in ``defaults.yaml``.
|
|
* Jenkins Jobs Macros to give larger config sections meaningful names in
|
|
``macros.yaml``.
|
|
* Project specific configurations in ``project_name.yaml``.
|
|
* Job template configurations. Need a ``projects.yaml`` file to
|
|
specify how the templates should be filled out and templates go in
|
|
``template_name.yaml``.
|
|
|
|
YAML Format
|
|
===========
|
|
|
|
Defaults
|
|
--------
|
|
|
|
Example defaults config:
|
|
|
|
.. code-block:: yaml
|
|
|
|
- defaults:
|
|
name: global
|
|
project-type: freestyle
|
|
concurrent: true
|
|
|
|
wrappers:
|
|
- timeout:
|
|
timeout: 30
|
|
fail: true
|
|
- timestamps
|
|
|
|
logrotate:
|
|
daysToKeep: 1
|
|
numToKeep: -1
|
|
artifactDaysToKeep: -1
|
|
artifactNumToKeep: -1
|
|
|
|
This config starts with the ``- defaults::`` line. This specifies that this
|
|
section contains default values rather than job specifications. In this
|
|
section we specify a useful set of defaults including a default description
|
|
indicating Puppet manages these jobs, jobs are allowed to run concurrently,
|
|
and a thirty minute job timeout.
|
|
|
|
Macros
|
|
------
|
|
|
|
Macros exist to give meaningful names to blocks of configuration that can be
|
|
used in job configs in place of the blocks they name. For example:
|
|
|
|
.. code-block:: yaml
|
|
|
|
- builder:
|
|
name: git-prep
|
|
builders:
|
|
- shell: "/slave_scripts/git-prep.sh"
|
|
|
|
- builder:
|
|
name: docs
|
|
builders:
|
|
- shell: "/slave_scripts/run-docs.sh"
|
|
|
|
- publisher:
|
|
name: console-log
|
|
publishers:
|
|
- scp:
|
|
site: 'scp-server'
|
|
files:
|
|
- target: 'logs/$JOB_NAME/$BUILD_NUMBER'
|
|
copy-console: true
|
|
copy-after-failure: true
|
|
|
|
In this block of code we define two builder macros and one publisher macro.
|
|
Each macro has a name and using that name in a job config is equivalent to
|
|
having the yaml below the name in place of the name in the job config. The next
|
|
section shows how you can use these macros.
|
|
|
|
Job Config
|
|
----------
|
|
|
|
Example job config:
|
|
|
|
.. code-block:: yaml
|
|
|
|
- job:
|
|
name: example-docs
|
|
node: node-label
|
|
|
|
triggers:
|
|
- zuul
|
|
|
|
builders:
|
|
- git-prep
|
|
- docs
|
|
|
|
publishers:
|
|
- scp:
|
|
site: 'scp-server'
|
|
files:
|
|
- target: 'dir/ectory'
|
|
source: 'build/html/foo'
|
|
keep-hierarchy: true
|
|
- console-log
|
|
|
|
Each job specification begins with ``-job:``. Under this section you can
|
|
specify the job details like name, node, etc. Any detail defined in the
|
|
defaults section that is not defined under this job will be included as well.
|
|
In addition to attribute details you can also specify how jenkins should
|
|
perform this job. What trigger methods should be used, the build steps,
|
|
jenkins publishing steps and so on. The macros defined earlier make this easy
|
|
and simple.
|
|
|
|
Job Templates
|
|
-------------
|
|
|
|
Job templates allow you to specify a job config once with arguments that are
|
|
replaced with the values specified in ``projects.yaml``. This allows you to
|
|
reuse job configs across many projects. First you need a templated job config:
|
|
|
|
.. code-block:: yaml
|
|
|
|
- job-template:
|
|
name: '{name}-docs'
|
|
|
|
triggers:
|
|
- zuul
|
|
|
|
builders:
|
|
- git-prep
|
|
- docs
|
|
|
|
publishers:
|
|
- scp:
|
|
site: 'scp-server'
|
|
files:
|
|
- target: 'dir/ectory'
|
|
source: 'build/html/foo'
|
|
keep-hierarchy: true
|
|
- console-log
|
|
|
|
node: '{node}'
|
|
|
|
|
|
- job-group:
|
|
name: python-jobs
|
|
jobs:
|
|
- '{name}-docs'
|
|
|
|
This takes the previous ``example-docs`` job and templatizes it. This will
|
|
allow us to easily create ``example1-docs`` and ``example2-docs`` jobs.
|
|
Each job template begins with ``- job-template:`` and the job specification is
|
|
identical to the previous one, but we have introduced variable arguments. In
|
|
this case ``{name}`` is a variable value that will be replaced. The values for
|
|
name will be defined in the ``projects.yaml`` file.
|
|
|
|
The ``- job-group:`` section is not strictly necessary but allows you to group
|
|
many job templates with the same variable arguments under one name.
|
|
|
|
The ``projects.yaml`` pulls all of the magic together. It specifies the
|
|
arguments to and instantiates the job templates as real jobs. For example:
|
|
|
|
.. code-block:: yaml
|
|
|
|
- project:
|
|
name: example1
|
|
node: bare-trusty
|
|
|
|
jobs:
|
|
- python-jobs
|
|
|
|
- project:
|
|
name: example2
|
|
node: bare-centos6
|
|
|
|
jobs:
|
|
- {name}-docs
|
|
|
|
Each project using templated jobs should have its own ``- project:`` section.
|
|
Under this sections there should be a ``jobs:`` section with a list of job
|
|
templates or job groups to be used by this project. Other values under the
|
|
``- project:`` section define the arguments to the templates lised under
|
|
``jobs:``. In this case we are giving the docs template ``name`` and ``node``
|
|
values.
|
|
|
|
Notice that example1 makes use of the job group and example2 makes use of the
|
|
job template.
|
|
|
|
Zuul
|
|
====
|
|
|
|
In our environment, we no longer use Jenkins to execute jobs. Zuul
|
|
itself, via Ansible, runs the actual workload. Zuul reads JJB config
|
|
files in order to define its jobs, so, aside from the detail of not
|
|
actually using Jenkins or creating any jobs in it, the use of JJB to
|
|
configure jobs in Zuul is the same.
|