Require a base job

This makes base jobs required and allows for a per-tenant default.

Story: 2001110
Task: 4793
Change-Id: I26ffddad8358c156cfac749ce98af70f3447f671
This commit is contained in:
James E. Blair 2017-08-07 09:52:45 -07:00
parent 0fb72dce11
commit 2bab6e7361
102 changed files with 520 additions and 9 deletions

View File

@ -165,3 +165,12 @@ configuration. An example tenant definition is:
can be limited to the protected branches which are gated. This
is a tenant wide setting and can be overridden per project.
This currently only affects GitHub projects.
.. attr:: default-parent
:default: base
If a job is defined without an explicit :attr:`job.parent`
attribute, this job will be configured as the job's parent.
This allows an administrator to configure a default base job to
implement local policies such as node setup and artifact
publishing.

View File

@ -6,6 +6,13 @@ Glossary
.. glossary::
:sorted:
base job
A job with no parent. A base job may only be defined in a
:term:`config-project`. Multiple base jobs may be defined, but
each tenant has a single default job which will be used as the
parent of any job which does not specify one explicitly.
check
By convention, the name of a pipeline which performs pre-merge

View File

@ -17,9 +17,10 @@ they specify one of two security contexts for that project. A
*config-project* is one which is primarily tasked with holding
configuration information and job content for Zuul. Jobs which are
defined in a config-project are run with elevated privileges, and all
Zuul configuration items are available for use. It is expected that
changes to config-projects will undergo careful scrutiny before being
merged.
Zuul configuration items are available for use. Base jobs (that is,
jobs without a parent) may only be defined in config-projects. It is
expected that changes to config-projects will undergo careful scrutiny
before being merged.
An *untrusted-project* is a project whose primary focus is not to
operate Zuul, but rather it is one of the projects being tested or
@ -439,6 +440,12 @@ specialization before arriving at a particular job. A job may inherit
from any other job in any project (however, if the other job is marked
as ``final``, some attributes may not be overidden).
A job with no parent is called a *base job* and may only be defined in
a :term:`config-project`. Every other job must have a parent, and so
ultimately, all jobs must have an inheritance path which terminates at
a base job. Each tenant has a default parent job which will be used
if no explicit parent is specified.
Jobs also support a concept called variance. The first time a job
definition appears is called the reference definition of the job.
Subsequent job definitions with the same name are called variants.
@ -503,11 +510,17 @@ Here is an example of two job definitions:
this name to use as the main playbook for the job. This name is
also referenced later in a project pipeline configuration.
.. TODO: figure out how to link the parent default to tenant.default.parent
.. attr:: parent
:default: Tenant default-parent
Specifies a job to inherit from. The parent job can be defined
in this or any other project. Any attributes not specified on
a job will be collected from its parent.
in this or any other project. Any attributes not specified on a
job will be collected from its parent. If no value is supplied
here, the job specified by :attr:`tenant.default-parent` will be
used. If **parent** is set to ``null`` (which is only valid in
a :term:`config-project`), this is a :term:`base job`.
.. attr:: description

View File

@ -69,6 +69,10 @@
data:
value: vartest_secret
- job:
name: base
parent: null
- job:
name: base-urls
success-url: https://success.example.com/zuul-logs/{build.uuid}/

View File

@ -0,0 +1,24 @@
- pipeline:
name: check
manager: independent
trigger:
gerrit:
- event: patchset-created
success:
gerrit:
Verified: 1
failure:
gerrit:
Verified: -1
- job:
name: my-base
parent: null
tags:
- mybase
- job:
name: other-base
parent: null
tags:
- otherbase

View File

@ -0,0 +1,13 @@
- job:
name: my-job
- job:
name: other-job
parent: other-base
- project:
name: org/project
check:
jobs:
- my-job
- other-job

View File

@ -0,0 +1 @@
test

View File

@ -0,0 +1,2 @@
- hosts: all
tasks: []

View File

@ -0,0 +1,2 @@
- hosts: all
tasks: []

View File

@ -0,0 +1,9 @@
- tenant:
name: tenant-one
default-parent: my-base
source:
gerrit:
config-projects:
- common-config
untrusted-projects:
- org/project

View File

@ -1,2 +1,6 @@
- job:
name: base
parent: null
- job:
name: trusted-zuul.yaml-job

View File

@ -12,6 +12,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: data-return

View File

@ -19,6 +19,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: A

View File

@ -12,6 +12,10 @@
gerrit:
verified: -1
- job:
name: base
parent: null
- job:
name: dd-big-empty-file

View File

@ -26,6 +26,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project-test1

View File

@ -11,6 +11,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project-test1

View File

@ -10,3 +10,7 @@
failure:
gerrit:
Verified: -1
- job:
name: base
parent: null

View File

@ -69,6 +69,10 @@
type: approved
trigger: {}
- job:
name: base
parent: null
- job:
name: common-config-test

View File

@ -31,6 +31,10 @@
- compute1
- compute2
- job:
name: base
parent: null
- job:
name: single-inventory
nodes:

View File

@ -32,6 +32,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: project-test1

View File

@ -26,8 +26,13 @@
gerrit:
Verified: 0
- job:
name: base
parent: null
- job:
name: project-gerrit
- job:
name: project1-github

View File

@ -10,3 +10,7 @@
failure:
gerrit:
Verified: -1
- job:
name: base
parent: null

View File

@ -11,6 +11,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: python27
nodes:

View File

@ -35,6 +35,7 @@
- job:
name: base
parent: null
timeout: 30
nodes:
- name: controller

View File

@ -12,6 +12,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: python27
pre-run: playbooks/pre

View File

@ -76,6 +76,10 @@
gerrit:
- event: ref-updated
- job:
name: base
parent: null
- job:
name: job1

View File

@ -30,6 +30,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project1-job

View File

@ -32,6 +32,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project1-job

View File

@ -32,6 +32,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project1-job

View File

@ -30,6 +30,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project1-job

View File

@ -46,6 +46,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project1-job

View File

@ -49,6 +49,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project-job

View File

@ -30,6 +30,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project1-job

View File

@ -32,6 +32,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project1-job

View File

@ -36,6 +36,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project1-job

View File

@ -32,6 +32,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: common-config-test

View File

@ -19,6 +19,10 @@
name: test-semaphore-two
max: 2
- job:
name: base
parent: null
- job:
name: project-test1

View File

@ -11,6 +11,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project-test1

View File

@ -13,6 +13,7 @@
- job:
name: base
parent: null
- job:
name: test2

View File

@ -40,6 +40,10 @@
- event: ref-updated
ref: ^(?!refs/).*$
- job:
name: base
parent: null
- job:
name: project-merge
hold-following-changes: true

View File

@ -1,2 +1,6 @@
- job:
name: base
parent: null
- job:
name: project-test1

View File

@ -14,6 +14,10 @@
resultsdb:
resultsdb_failures:
- job:
name: base
parent: null
- job:
name: project-merge

View File

@ -11,6 +11,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: python27
vars:

View File

@ -16,6 +16,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: docs-draft-test
success-url: http://docs-draft.example.org/{change.number:.2}/{change.number}/{change.patchset}/{pipeline.name}/{job.name}/{build.uuid:.7}/publish-docs/

View File

@ -1,3 +1,7 @@
- job:
name: base
parent: null
- job:
name: project-test1

View File

@ -11,6 +11,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: common-config-job

View File

@ -17,3 +17,7 @@
start:
github:
comment: true
- job:
name: base
parent: null

View File

@ -43,6 +43,10 @@
another_gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project-test1

View File

@ -11,6 +11,10 @@
alt_voting_gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project-test1

View File

@ -45,6 +45,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: project-check

View File

@ -43,6 +43,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project-check

View File

@ -11,6 +11,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project-test2
nodes:

View File

@ -17,8 +17,13 @@
failure:
github: {}
- job:
name: base
parent: null
- job:
name: project-test1
- job:
name: project-test2

View File

@ -27,16 +27,25 @@
failure:
github: {}
- job:
name: base
parent: null
- job:
name: project1-test
- job:
name: project2-test
- job:
name: project3-test
- job:
name: project4-test
- job:
name: project5-test
- job:
name: project6-test

View File

@ -15,10 +15,16 @@
github:
unlabel: 'merge'
- job:
name: base
parent: null
- job:
name: project-test1
- job:
name: project-test2
- job:
name: project-merge
failure-message: Unable to merge change

View File

@ -8,6 +8,10 @@
- opened
- changed
- job:
name: base
parent: null
- job:
name: one-job-project-merge

View File

@ -15,6 +15,10 @@
to: you@example.com
disable-after-consecutive-failures: 3
- job:
name: base
parent: null
- job:
name: project-test1
nodes:

View File

@ -7,6 +7,10 @@
ref: ^(?!refs/).*$
ignore-deletes: false
- job:
name: base
parent: null
- job:
name: project-post
nodes:

View File

@ -6,6 +6,10 @@
- event: pull_request
action: opened
- job:
name: base
parent: null
- job:
name: project-test1
files:

View File

@ -25,6 +25,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: project-test1
# success-url: http://logs.exxxample.com/{change.number}/{change.patchset}/{pipeline.name}/{job.name}

View File

@ -5,6 +5,10 @@
timer:
- time: '* * * * * */1'
- job:
name: base
parent: null
- job:
name: project-bitrot
nodes:

View File

@ -12,6 +12,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project1-merge

View File

@ -11,6 +11,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project-test-irrelevant-starts-empty

View File

@ -11,6 +11,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project-test-irrelevant-files

View File

@ -19,6 +19,10 @@
unlabel:
- 'test'
- job:
name: base
parent: null
- job:
name: project-labels

View File

@ -19,6 +19,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: project-merge
hold-following-changes: true

View File

@ -11,6 +11,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project-merge
hold-following-changes: true

View File

@ -11,6 +11,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project-merge
hold-following-changes: true

View File

@ -32,6 +32,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: project-merge
hold-following-changes: true

View File

@ -46,6 +46,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: project-merge
hold-following-changes: true

View File

@ -13,6 +13,10 @@
merge: true
comment: false
- job:
name: base
parent: null
- project:
name: org/project
merge:

View File

@ -11,6 +11,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: project-testfile
files:

View File

@ -32,6 +32,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: gate-noop

View File

@ -20,6 +20,10 @@
gerrit:
- event: ref-updated
- job:
name: base
parent: null
- job:
name: project-test1

View File

@ -19,6 +19,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: nonvoting-project-merge
hold-following-changes: true

View File

@ -9,6 +9,10 @@
failure:
gerrit: {}
- job:
name: base
parent: null
- job:
name: project-merge
hold-following-changes: true

View File

@ -40,6 +40,10 @@
- event: ref-updated
ref: ^(?!refs/).*$
- job:
name: base
parent: null
- job:
name: one-job-project-merge
hold-following-changes: true

View File

@ -14,8 +14,13 @@
- event: push
ref: ^refs/tags/.*$
- job:
name: base
parent: null
- job:
name: project-post
- job:
name: project-tag

View File

@ -24,6 +24,10 @@
window-decrease-type: exponential
window-decrease-factor: 2
- job:
name: base
parent: null
- job:
name: project-merge

View File

@ -32,6 +32,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: integration
required-projects:

View File

@ -7,6 +7,10 @@
gerrit:
- event: ref-updated
- job:
name: base
parent: null
- job:
name: integration
branches: master

View File

@ -7,6 +7,10 @@
gerrit:
- event: ref-updated
- job:
name: base
parent: null
- job:
name: integration
override-branch: stable/havana

View File

@ -6,6 +6,10 @@
- event: ref-updated
ref: ^(?!refs/).*$
- job:
name: base
parent: null
- job:
name: integration
required-projects:

View File

@ -32,6 +32,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: integration
required-projects:

View File

@ -5,6 +5,10 @@
timer:
- time: '* * * * * */1'
- job:
name: base
parent: null
- job:
name: integration
branches: master

View File

@ -5,6 +5,10 @@
timer:
- time: '* * * * * */1'
- job:
name: base
parent: null
- job:
name: integration
required-projects:

View File

@ -32,6 +32,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: integration
required-projects:

View File

@ -32,6 +32,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: project-merge
hold-following-changes: true

View File

@ -78,6 +78,10 @@
comment: true
status: 'failure'
- job:
name: base
parent: null
- job:
name: project-test1

View File

@ -22,8 +22,13 @@
github_ent:
status: 'success'
- job:
name: base
parent: null
- job:
name: project1-test1
- job:
name: project2-test2

View File

@ -183,24 +183,37 @@
github:
comment: true
- job:
name: base
parent: null
- job:
name: project1-pipeline
- job:
name: project2-trigger
- job:
name: project3-reviewusername
- job:
name: project4-reviewreq
- job:
name: project5-reviewuserstate
- job:
name: project6-newerthan
- job:
name: project7-olderthan
- job:
name: project8-requireopen
- job:
name: project9-requirecurrent
- job:
name: project10-label

View File

@ -11,6 +11,10 @@
label:
- 'tests passed'
- job:
name: base
parent: null
- job:
name: project-reviews

View File

@ -38,6 +38,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: project-merge
hold-following-changes: true

View File

@ -11,6 +11,10 @@
gerrit:
Verified: -1
- job:
name: base
parent: null
- job:
name: merge
tags:

View File

@ -32,6 +32,10 @@
Verified: 0
precedence: high
- job:
name: base
parent: null
- job:
name: project-merge
hold-following-changes: true

View File

@ -10,6 +10,10 @@
from: zuul_from@example.com
subject: Periodic check for {change.project} succeeded
- job:
name: base
parent: null
- job:
name: project-bitrot-stable-old
success-url: http://logs.example.com/{job.name}/{build.number}

View File

@ -18,6 +18,10 @@
timer:
- time: '* * * * * */1'
- job:
name: base
parent: null
- job:
name: project-test1

View File

@ -305,5 +305,6 @@ class TestConfigConflict(ZuulTestCase):
tenant = self.sched.abide.tenants.get('tenant-one')
jobs = sorted(tenant.layout.jobs.keys())
self.assertEquals(
['noop', 'trusted-zuul.yaml-job', 'untrusted-zuul.yaml-job'],
['base', 'noop', 'trusted-zuul.yaml-job',
'untrusted-zuul.yaml-job'],
jobs)

View File

@ -65,6 +65,7 @@ class TestJob(BaseTestCase):
'_source_context': self.context,
'_start_mark': self.start_mark,
'name': 'job',
'parent': None,
'irrelevant-files': [
'^docs/.*$'
]})
@ -184,6 +185,7 @@ class TestJob(BaseTestCase):
'_source_context': self.context,
'_start_mark': self.start_mark,
'name': 'base',
'parent': None,
'timeout': 30,
'pre-run': 'base-pre',
'post-run': 'base-post',
@ -389,6 +391,7 @@ class TestJob(BaseTestCase):
'_source_context': self.context,
'_start_mark': self.start_mark,
'name': 'base',
'parent': None,
'timeout': 30,
})
layout.addJob(base)
@ -487,6 +490,7 @@ class TestJob(BaseTestCase):
'_source_context': self.context,
'_start_mark': self.start_mark,
'name': 'base',
'parent': None,
'timeout': 30,
})
layout.addJob(base)
@ -565,6 +569,7 @@ class TestJob(BaseTestCase):
'_source_context': self.context,
'_start_mark': self.start_mark,
'name': 'base',
'parent': None,
'timeout': 30,
})
layout.addJob(base)
@ -614,6 +619,7 @@ class TestJob(BaseTestCase):
base = configloader.JobParser.fromYaml(tenant, layout, {
'_source_context': base_context,
'_start_mark': self.start_mark,
'parent': None,
'name': 'base',
})
layout.addJob(base)
@ -639,6 +645,7 @@ class TestJob(BaseTestCase):
'_source_context': self.context,
'_start_mark': self.start_mark,
'name': 'job',
'parent': None,
'allowed-projects': ['project'],
})
self.layout.addJob(job)
@ -679,6 +686,7 @@ class TestJob(BaseTestCase):
'_source_context': self.context,
'_start_mark': self.start_mark,
'name': 'job',
'parent': None,
})
auth = model.AuthContext()
auth.secrets.append('foo')

View File

@ -1120,3 +1120,43 @@ class TestMaxNodesPerJob(AnsibleZuulTestCase):
self.waitUntilSettled()
self.assertNotIn("exceeds tenant max-nodes", B.messages[0],
"B should not fail because of nodes limit")
class TestBaseJobs(ZuulTestCase):
tenant_config_file = 'config/base-jobs/main.yaml'
def test_multiple_base_jobs(self):
A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
self.waitUntilSettled()
self.assertHistory([
dict(name='my-job', result='SUCCESS', changes='1,1'),
dict(name='other-job', result='SUCCESS', changes='1,1'),
], ordered=False)
self.assertEqual(self.getJobFromHistory('my-job').
parameters['zuul']['jobtags'],
['mybase'])
self.assertEqual(self.getJobFromHistory('other-job').
parameters['zuul']['jobtags'],
['otherbase'])
def test_untrusted_base_job(self):
"""Test that a base job may not be defined in an untrusted repo"""
in_repo_conf = textwrap.dedent(
"""
- job:
name: fail-base
parent: null
""")
file_dict = {'.zuul.yaml': in_repo_conf}
A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A',
files=file_dict)
self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
self.waitUntilSettled()
self.assertEqual(A.reported, 1,
"A should report failure")
self.assertEqual(A.patchsets[0]['approvals'][0]['value'], "-1")
self.assertIn('Base jobs must be defined in config projects',
A.messages[0])
self.assertHistory([])

Some files were not shown because too many files have changed in this diff Show More