project templating system
On setup where Zuul ends up triggering hundreds of projects, you end up having projects using roughly the same pipeline/jobs. Whenever one want to add a job in all the similiar project, he has to edit each project one by one. To save some precious time, this patch introduces the concept of project templates. It lets you define a set of pipeline and attached jobs though the job names can be passed parameters defined on a per project basis. Thus, updating similiar projects is all about editing a single template. A basic example is provided in the documentation. The voluptuous schema has been updated. It does check whether all parameters are properly passed to a template but does NOT check whether the resulting job name exist. The parameter expansion in templates is borrowed from Jenkins Job Builder (deep_format function). It has been tweaked to also expand dictionary keys. Layout test plan: $ nosetests -m layout --nocapture Test layout file validation ... <...> bad_template1.yaml required key not provided @ data['projects'][0]['template']['project'] bad_template2.yaml extra keys not allowed @ data['projects'][0]['template']['extraparam'] good_template1.yaml ok <...> $ A basic test hasbeen added to verify whether a project-template properly triggers its tests: $ nosetests --nocapture \ tests/test_scheduler.py:testScheduler.test_job_from_templates_launched Test whether a job generated via a template can be launched ... ok ---------------------------------------------------------------------- Ran 1 test in 0.863s OK $ Change-Id: Ib82e4719331c204de87fbb4b20c198842b7e32f4 Reviewed-on: https://review.openstack.org/21881 Reviewed-by: Jeremy Stanley <fungi@yuggoth.org> Reviewed-by: James E. Blair <corvus@inaugust.com> Approved: James E. Blair <corvus@inaugust.com> Tested-by: Jenkins
This commit is contained in:
parent
3c5e5b507f
commit
80edd5a8fa
|
@ -504,6 +504,40 @@ can help avoid running unnecessary jobs.
|
|||
|
||||
.. seealso:: The OpenStack Zuul configuration for a comprehensive example: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml
|
||||
|
||||
Project Templates
|
||||
"""""""""""""""""
|
||||
|
||||
Whenever you have lot of similiar projects (such as plugins for a project) you
|
||||
will most probably want to use the same pipeline configurations. The
|
||||
project templates let you define pipelines and job name templates to trigger.
|
||||
One can then just apply the template on its project which make it easier to
|
||||
update several similiar projects. As an example::
|
||||
|
||||
project-templates:
|
||||
# Name of the template
|
||||
- name: plugin-triggering
|
||||
# Definition of pipelines just like for a `project`
|
||||
check:
|
||||
- '{jobprefix}-merge':
|
||||
- '{jobprefix}-pep8'
|
||||
- '{jobprefix}-pyflakes'
|
||||
gate:
|
||||
- '{jobprefix}-merge':
|
||||
- '{jobprefix}-unittest'
|
||||
- '{jobprefix}-pep8'
|
||||
- '{jobprefix}-pyflakes'
|
||||
|
||||
In your projects definition, you will then apply the template using the template
|
||||
key::
|
||||
|
||||
projects:
|
||||
- name: plugin/foobar
|
||||
template:
|
||||
- name: plugin-triggering
|
||||
jobprefix: plugin-foobar
|
||||
|
||||
You can pass several parameters to a template. A ``parameter`` value will be
|
||||
used for expansion of ``{parameter}`` in the template strings.
|
||||
|
||||
logging.conf
|
||||
~~~~~~~~~~~~
|
||||
|
|
|
@ -47,6 +47,12 @@ jobs:
|
|||
files:
|
||||
- '.*-requires'
|
||||
|
||||
project-templates:
|
||||
- name: test-one-and-two
|
||||
check:
|
||||
- '{projectname}-test1'
|
||||
- '{projectname}-test2'
|
||||
|
||||
projects:
|
||||
- name: org/project
|
||||
merge-mode: cherry-pick
|
||||
|
@ -124,3 +130,8 @@ projects:
|
|||
- nonvoting-project-test2
|
||||
post:
|
||||
- nonvoting-project-post
|
||||
|
||||
- name: org/templated-project
|
||||
template:
|
||||
- name: test-one-and-two
|
||||
projectname: project
|
||||
|
|
|
@ -0,0 +1,19 @@
|
|||
# Template is going to be called but missing a parameter
|
||||
|
||||
pipelines:
|
||||
- name: 'check'
|
||||
manager: IndependentPipelineManager
|
||||
trigger:
|
||||
- event: patchset-created
|
||||
|
||||
project-templates:
|
||||
- name: template-generic
|
||||
check:
|
||||
# Template uses the 'project' parameter' which must
|
||||
- '{project}-merge'
|
||||
|
||||
projects:
|
||||
- name: organization/project
|
||||
template:
|
||||
- name: template-generic
|
||||
# Here we 'forgot' to pass 'project'
|
|
@ -0,0 +1,22 @@
|
|||
# Template is going to be called with an extra parameter
|
||||
|
||||
pipelines:
|
||||
- name: 'check'
|
||||
manager: IndependentPipelineManager
|
||||
trigger:
|
||||
- event: patchset-created
|
||||
|
||||
project-templates:
|
||||
- name: template-generic
|
||||
check:
|
||||
# Template only uses the 'project' parameter'
|
||||
- '{project}-merge'
|
||||
|
||||
projects:
|
||||
- name: organization/project
|
||||
template:
|
||||
- name: template-generic
|
||||
project: 'MyProjectName'
|
||||
# Feed an extra parameters which is not going to be used
|
||||
# by the template. That is an error.
|
||||
extraparam: 'IShouldNotBeSet'
|
|
@ -0,0 +1,13 @@
|
|||
# Template refers to an unexisting pipeline
|
||||
|
||||
pipelines:
|
||||
# We have no pipelines at all
|
||||
|
||||
project-templates:
|
||||
- name: template-generic
|
||||
unexisting-pipeline: # pipeline does not exist
|
||||
|
||||
projects:
|
||||
- name: organization/project
|
||||
template:
|
||||
- name: template-generic
|
|
@ -0,0 +1,16 @@
|
|||
pipelines:
|
||||
- name: 'check'
|
||||
manager: IndependentPipelineManager
|
||||
trigger:
|
||||
- event: patchset-created
|
||||
|
||||
project-templates:
|
||||
- name: template-generic
|
||||
check:
|
||||
- '{project}-merge'
|
||||
|
||||
projects:
|
||||
- name: organization/project
|
||||
template:
|
||||
- name: template-generic
|
||||
project: 'myproject'
|
|
@ -689,6 +689,7 @@ class testScheduler(unittest.TestCase):
|
|||
init_repo("org/project3")
|
||||
init_repo("org/one-job-project")
|
||||
init_repo("org/nonvoting-project")
|
||||
init_repo("org/templated-project")
|
||||
self.config = CONFIG
|
||||
|
||||
self.statsd = FakeStatsd()
|
||||
|
@ -1376,6 +1377,20 @@ class testScheduler(unittest.TestCase):
|
|||
assert B.reported == 2
|
||||
self.assertEmptyQueues()
|
||||
|
||||
def test_job_from_templates_launched(self):
|
||||
"Test whether a job generated via a template can be launched"
|
||||
A = self.fake_gerrit.addFakeChange(
|
||||
'org/templated-project', 'master', 'A')
|
||||
self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
|
||||
self.waitUntilSettled()
|
||||
jobs = self.fake_jenkins.job_history
|
||||
job_names = [x.name for x in jobs]
|
||||
|
||||
assert 'project-test1' in job_names
|
||||
assert 'project-test2' in job_names
|
||||
assert jobs[0].result == 'SUCCESS'
|
||||
assert jobs[1].result == 'SUCCESS'
|
||||
|
||||
def test_dependent_changes_dequeue(self):
|
||||
"Test that dependent patches are not needlessly tested"
|
||||
A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
|
||||
|
|
|
@ -1,4 +1,6 @@
|
|||
# Copyright 2013 OpenStack Foundation
|
||||
# Copyright 2013 Antoine "hashar" Musso
|
||||
# Copyright 2013 Wikimedia Foundation Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
|
@ -13,6 +15,7 @@
|
|||
# under the License.
|
||||
|
||||
import voluptuous as v
|
||||
import string
|
||||
|
||||
|
||||
# Several forms accept either a single item or a list, this makes
|
||||
|
@ -55,6 +58,9 @@ class LayoutSchema(object):
|
|||
}
|
||||
pipelines = [pipeline]
|
||||
|
||||
project_template = {v.Required('name'): str}
|
||||
project_templates = [project_template]
|
||||
|
||||
job = {v.Required('name'): str,
|
||||
'failure-message': str,
|
||||
'success-message': str,
|
||||
|
@ -80,21 +86,90 @@ class LayoutSchema(object):
|
|||
else:
|
||||
self.job_name.validate(path, self.job_name.schema, value)
|
||||
|
||||
def validateTemplateCalls(self, calls):
|
||||
""" Verify a project pass the parameters required
|
||||
by a project-template
|
||||
"""
|
||||
for call in calls:
|
||||
schema = self.templates_schemas[call.get('name')]
|
||||
schema(call)
|
||||
|
||||
def collectFormatParam(self, tree):
|
||||
"""In a nested tree of string, dict and list, find out any named
|
||||
parameters that might be used by str.format(). This is used to find
|
||||
out whether projects are passing all the required parameters when
|
||||
using a project template.
|
||||
|
||||
Returns a set() of all the named parameters found.
|
||||
"""
|
||||
parameters = set()
|
||||
if isinstance(tree, str):
|
||||
# parse() returns a tuple of
|
||||
# (literal_text, field_name, format_spec, conversion)
|
||||
# We are just looking for field_name
|
||||
parameters = set([t[1] for t in string.Formatter().parse(tree)
|
||||
if t[1] is not None])
|
||||
elif isinstance(tree, list):
|
||||
for item in tree:
|
||||
parameters.update(self.collectFormatParam(item))
|
||||
elif isinstance(tree, dict):
|
||||
for item in tree:
|
||||
parameters.update(self.collectFormatParam(tree[item]))
|
||||
|
||||
return parameters
|
||||
|
||||
def getSchema(self, data):
|
||||
pipelines = data.get('pipelines')
|
||||
if not pipelines:
|
||||
pipelines = []
|
||||
pipelines = [p['name'] for p in pipelines if 'name' in p]
|
||||
|
||||
# Whenever a project uses a template, it better have to exist
|
||||
project_templates = data.get('project-templates', [])
|
||||
template_names = [t['name'] for t in project_templates
|
||||
if 'name' in t]
|
||||
|
||||
# A project using a template must pass all parameters to it.
|
||||
# We first collect each templates parameters and craft a new
|
||||
# schema for each of the template. That will later be used
|
||||
# by validateTemplateCalls().
|
||||
self.templates_schemas = {}
|
||||
for t_name in template_names:
|
||||
# Find out the parameters used inside each templates:
|
||||
template = [t for t in project_templates
|
||||
if t['name'] == t_name]
|
||||
template_parameters = self.collectFormatParam(template)
|
||||
|
||||
# Craft the templates schemas
|
||||
schema = {v.Required('name'): v.Any(*template_names)}
|
||||
for required_param in template_parameters:
|
||||
# add this template parameters as requirements:
|
||||
schema.update({v.Required(required_param): str})
|
||||
|
||||
# Register the schema for validateTemplateCalls()
|
||||
self.templates_schemas[t_name] = v.Schema(schema)
|
||||
|
||||
project = {'name': str,
|
||||
'merge-mode': v.Any('cherry-pick'),
|
||||
'template': self.validateTemplateCalls,
|
||||
}
|
||||
|
||||
# And project should refers to existing pipelines
|
||||
for p in pipelines:
|
||||
project[p] = self.validateJob
|
||||
projects = [project]
|
||||
|
||||
# Sub schema to validate a project template has existing
|
||||
# pipelines and jobs.
|
||||
project_template = {'name': str}
|
||||
for p in pipelines:
|
||||
project_template[p] = self.validateJob
|
||||
project_templates = [project_template]
|
||||
# Gather our sub schemas
|
||||
schema = v.Schema({'includes': self.includes,
|
||||
v.Required('pipelines'): self.pipelines,
|
||||
'jobs': self.jobs,
|
||||
'project-templates': project_templates,
|
||||
v.Required('projects'): projects,
|
||||
})
|
||||
return schema
|
||||
|
@ -116,3 +191,6 @@ class LayoutValidator(object):
|
|||
if 'jobs' in data:
|
||||
self.checkDuplicateNames(data['jobs'], ['jobs'])
|
||||
self.checkDuplicateNames(data['projects'], ['projects'])
|
||||
if 'project-templates' in data:
|
||||
self.checkDuplicateNames(
|
||||
data['project-templates'], ['project-templates'])
|
||||
|
|
|
@ -1,5 +1,7 @@
|
|||
# Copyright 2012 Hewlett-Packard Development Company, L.P.
|
||||
# Copyright 2013 OpenStack Foundation
|
||||
# Copyright 2013 Antoine "hashar" Musso
|
||||
# Copyright 2013 Wikimedia Foundation Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
|
@ -32,6 +34,28 @@ import merger
|
|||
statsd = extras.try_import('statsd.statsd')
|
||||
|
||||
|
||||
def deep_format(obj, paramdict):
|
||||
"""Apply the paramdict via str.format() to all string objects found within
|
||||
the supplied obj. Lists and dicts are traversed recursively.
|
||||
|
||||
Borrowed from Jenkins Job Builder project"""
|
||||
if isinstance(obj, str):
|
||||
ret = obj.format(**paramdict)
|
||||
elif isinstance(obj, list):
|
||||
ret = []
|
||||
for item in obj:
|
||||
ret.append(deep_format(item, paramdict))
|
||||
elif isinstance(obj, dict):
|
||||
ret = {}
|
||||
for item in obj:
|
||||
exp_item = item.format(**paramdict)
|
||||
|
||||
ret[exp_item] = deep_format(obj[item], paramdict)
|
||||
else:
|
||||
ret = obj
|
||||
return ret
|
||||
|
||||
|
||||
class Scheduler(threading.Thread):
|
||||
log = logging.getLogger("zuul.Scheduler")
|
||||
|
||||
|
@ -55,6 +79,7 @@ class Scheduler(threading.Thread):
|
|||
self.pipelines = {}
|
||||
self.jobs = {}
|
||||
self.projects = {}
|
||||
self.project_templates = {}
|
||||
self.metajobs = {}
|
||||
|
||||
def stop(self):
|
||||
|
@ -126,6 +151,16 @@ class Scheduler(threading.Thread):
|
|||
toList(trigger.get('email_filter')))
|
||||
manager.event_filters.append(f)
|
||||
|
||||
for project_template in data.get('project-templates', []):
|
||||
# Make sure the template only contains valid pipelines
|
||||
tpl = dict(
|
||||
(pipe_name, project_template.get(pipe_name))
|
||||
for pipe_name in self.pipelines.keys()
|
||||
if pipe_name in project_template
|
||||
)
|
||||
self.project_templates[project_template.get('name')] \
|
||||
= tpl
|
||||
|
||||
for config_job in data.get('jobs', []):
|
||||
job = self.getJob(config_job['name'])
|
||||
# Be careful to only set attributes explicitly present on
|
||||
|
@ -177,6 +212,17 @@ class Scheduler(threading.Thread):
|
|||
|
||||
for config_project in data.get('projects', []):
|
||||
project = Project(config_project['name'])
|
||||
|
||||
for requested_template in config_project.get('template', []):
|
||||
# Fetch the template from 'project-templates'
|
||||
tpl = self.project_templates.get(
|
||||
requested_template.get('name'))
|
||||
# Expand it with the project context
|
||||
expanded = deep_format(tpl, requested_template)
|
||||
# Finally merge the expansion with whatever has been already
|
||||
# defined for this project
|
||||
config_project.update(expanded)
|
||||
|
||||
self.projects[config_project['name']] = project
|
||||
mode = config_project.get('merge-mode')
|
||||
if mode and mode == 'cherry-pick':
|
||||
|
|
Loading…
Reference in New Issue