system-config/doc/jenkins_jobs.rst

Jenkins Job Builder

Overview

In order to make the process of managing hundreds of Jenkins Jobs easier a Python based utility was designed to take YAML based configurations and convert those into jobs that are injected into Jenkins.

Adding a project

The YAML scripts to make this work are stored in the openstack-ci-puppet repository in the modules/jenkins_jobs/files/projects/site/project.yaml directory. Where site is either openstack or stackforge and project is the name of the project the YAML file is for.

Once the YAML file is added the puppet module needs to be told that the project is there. For example:

class { "jenkins_jobs":
  site => "stackforge",
  projects => ['reddwarf', 'ceilometer']
}

In this example the YAML files for reddwarf and ceilometer in the stackforge projects directory will be executed.

YAML Format

The bare minimum YAML needs to look like this:

---
modules:
  - properties
  - scm
  - assignednode
  - trigger_none
  - builders
  - publisher_none

main:
  name: 'job-name'
  review_site: 'review.stackforge.org'
  github_org: 'stackforge'
  project: 'project'
  authenticatedBuild: 'false'
  disabled: 'false'

or for a templated project:

project:
  template: 'python_jobs'

values:
  name: 'cinder'
  disabled: 'false'
  github_org: 'openstack'
  review_site: 'review.openstack.org'
  publisher_site: 'nova.openstack.org'

The first example starts with ---, this signifies the start of a job, there can be multiple jobs per project file. The file does not need to start with the --- but jobs do need to be separated by it. Each YAML file can contain any combination of templated or normal jobs.

In the first example the modules entry is an array of modules that should be loaded for this job. Modules are located in the modules/jenkins_jobs/files/modules/ directory and are python scripts to generate the required XML. Each module has a comment near the top showing the required YAML to support that module. The follow modules are required to generate a correct XML that Jenkins will support:

• properties (supplies the <properties> XML data)
• scm (supplies the <scm> XML data, required even is scm is not used
• trigger* (a trigger module is required)
• builders
• publisher* (a publisher module is required)

Each module also requires a main section which has the main data for the modules, inside this there is:

• name - the name of the job
• review_site - review.openstack.org or review.stackforge.org
• github_org - the parent of the github branch for the project (typically openstack or stackforge
• project - the name of the project
• authenticatedBuild - whether or not you need to be authenticated to hit the build button
• disabled - whether or not this job should be disabled

In the templated example there is the project tag to specify that this is a templated project. The template value specified a template file found in the modules/jenkins_jobs/files/templates directory. The template will look like a regular set of jobs but contain values in caps surrounded by '@' symbols. The template process takes the parameters specified in the values section and replaces the values surrounded by the '@' symbol.

As an example in the template:

main:
  name: 'gate-@NAME@-pep8'

Using the above example of a templated job the @NAME@ would be replaced with cinder.

Testing a Job

Once a new YAML file has been created its output can be tested by using the jenkins_jobs.py script directly. For example:

$ python jenkins_jobs.py test projects/openstack/cinder.yml

This will spit out the XML that would normally be sent directly to Jenkins.

Job Caching

The Jenkins Jobs builder maintains a special YAML file in ~/.jenkins_jobs_cache.yml. This contains an MD5 of every generated XML that it builds. If it finds the XML is different then it will proceed to send this to Jenkins, otherwise it is skipped. If a job is accidentally deleted then this file should be modified or removed.

Sending a Job to Jenkins

The Jenkins Jobs builder talks to Jenkins using the Jenkins API. This means that it can create and modify jobs directly without the need to restart or reload the Jenkins server. It also means that Jenkins will verify the XML and cause the Jenkins Jobs builder to fail if there is a problem.

For this to work a configuration file is needed. This needs to be stored in /root/secret-files/jenkins_jobs.ini and puppet will automatically put it in the right place. The format for this file is as follows:

[jenkins]
user=username
password=password
url=jenkins_url

The password can be obtained by logging into the Jenkins user, clicking on your username in the top-right, clicking on Configure and then Show API Token. This API Token is your password for the API.

Adding a Module

Modules need to contain a class with the same name as the filename. The basic layout is:

import xml.etree.ElementTree as XML

class my_module(object):
    def __init__(self, data):
        self.data = data

    def gen_xml(self, xml_parent):

The __init__ function will be provided with data which is a Python dictionary representing the YAML data for the job.

The gen_xml function will be provided with xml_parent which is an XML ElementTree object to be modified.