Check out implicit branch in timer jobs

So that we may re-use the same jobs for pre and post merge tests,
enqueue an item for every branch of every timer-triggered project
and checkout that branch before running the job.  This means that
rather than having a job for gate plus a job for each stable branch,
we hav just have a single job which runs with different content.

The old method is still supported using override branches.

This updates the model to include Change, Branch, Tag, and Ref
objects which can be used as the value of Item.change.  Branch,
Tag, and Ref are all very similar, but the distinction may help
us ensure that we're encoding the right information about the items
we are enqueing.  This is important for branch matching in pipelines
and is also used to provide job variables.

Change-Id: I5c41d2dcbbbd1c17d68074cd7480e6ab83f884ea
This commit is contained in:
James E. Blair 2017-07-19 11:56:55 -07:00
parent a8b35774b3
commit 2103778ae4
17 changed files with 373 additions and 102 deletions

View File

@ -107,8 +107,163 @@ Zuul Variables
~~~~~~~~~~~~~~
Zuul supplies not only the variables specified by the job definition
to Ansible, but also some variables from the executor itself. They
are:
to Ansible, but also some variables from the Zuul itself.
When a pipeline is triggered an action, it enqueues items which may
vary based on the pipeline's configuration. For example, when a new
change is created, that change may be enqueued into the pipeline,
while a tag may be enqueued into the pipeline when it is pushed.
Information about these items is available to jobs. All of the items
enqueued in a pipeline are git references, and therefore share some
attributes in common. But other attributes may vary based on the type
of item.
All items provide the following information as Ansible variables:
**zuul.uuid**
The UUID of the build. A build is a single execution of a job.
When an item is enqueued into a pipeline, this usually results in
one build of each job configured for that item's project. However,
items may be re-enqueued in which case another build may run. In
dependent pipelines, the same job may run multiple times for the
same item as circumstances change ahead in the queue. Each time a
job is run, for whatever reason, it is acompanied with a new
unique id.
.. TODO: rename build
**zuul.buildset**
The build set UUID. When Zuul runs jobs for an item, the collection
of those jobs is known as a buildset. If the configuration of items
ahead in a dependent pipeline changes, Zuul creates a new buildset
and restarts all of the jobs.
**zuul.ref**
The git ref of the item. This will be the full path (e.g.,
'refs/heads/master' or 'refs/changes/...').
**zuul.pipeline**
The name of the pipeline in which the job is being run.
**zuul.job**
The name of the job being run.
**zuul.project**
The item's project. This is a data structure with the following
fields:
**zuul.project.name**
The name of the project, excluding hostname. E.g., `org/project`.
**zuul.project.canonical_hostname**
The canonical hostname where the project lives. E.g.,
`git.example.com`.
**zuul.project.canonical_name**
The full canonical name of the project including hostname. E.g.,
`git.example.com/org/project`.
**zuul.tenant**
The name of the current Zuul tenant.
**zuul.tags**
A list of tags associated with the job. Not to be confused with git
tags, these are simply free-form text fields that can be used by the
job for reporting or classification purposes.
.. TODO: rename jobtags
**zuul.items**
A data structure representing the items being tested with this
change.
.. TODO: implement and document items
Change Items
++++++++++++
A change to the repository. Most often, this will be a git reference
which has not yet been merged into the repository (e.g., a gerrit
change or a GitHub pull request). The following additional variables
are available:
**zuul.branch**
The target branch of the change (without the `refs/heads/` prefix).
**zuul.change**
The identifier for the change.
**zuul.patchset**
The patchset identifier for the change. If a change is revised,
this will have a different value.
Branch Items
++++++++++++
This represents a branch tip. This item may have been enqueued
because the branch was updated (via a change having merged, or a
direct push). Or it may have been enqueued by a timer for the purpose
of verifying the current condition of the branch. The following
additional variables are available:
**zuul.branch**
The name of the item's branch (without the `refs/heads/` prefix).
**zuul.oldrev**
If the item was enqueued as the result of a change merging or being
pushed to the branch, the git sha of the old revision will be
included here. Otherwise, this value will not be present.
**zuul.newrev**
If the item was enqueued as the result of a change merging or being
pushed to the branch, the git sha of the new revision will be
included here. Otherwise, this value will not be present.
Tag Items
+++++++++
This represents a git tag. The item may have been enqueued because a
tag was created or deleted. The following additional variables are
available:
**zuul.tag**
The name of the item's tag (without the `refs/tags/` prefix).
**zuul.oldrev**
If the item was enqueued as the result of a tag being created or
deleted the git sha of the old revision will be included here.
Otherwise, this value will not be present.
**zuul.newrev**
If the item was enqueued as the result of a tag being created or
deleted the git sha of the new revision will be included here.
Otherwise, this value will not be present.
Ref Items
+++++++++
This represents a git reference that is neither a change, branch, or
tag. Note that all items include a `ref` attribute which may be used
to identify the ref. The following additional variables are
available:
**zuul.oldrev**
If the item was enqueued as the result of a ref being created,
deleted, or changed the git sha of the old revision will be included
here. Otherwise, this value will not be present.
**zuul.newrev**
If the item was enqueued as the result of a ref being created,
deleted, or changed the git sha of the new revision will be included
here. Otherwise, this value will not be present.
Working Directory
+++++++++++++++++
Additionally, some information about the working directory and the
executor running the job is available:
**zuul.executor.hostname**
The hostname of the executor.

View File

@ -1068,8 +1068,10 @@ class BuildHistory(object):
self.__dict__.update(kw)
def __repr__(self):
return ("<Completed build, result: %s name: %s uuid: %s changes: %s>" %
(self.result, self.name, self.uuid, self.changes))
return ("<Completed build, result: %s name: %s uuid: %s "
"changes: %s ref: %s>" %
(self.result, self.name, self.uuid,
self.changes, self.ref))
class FakeStatsd(threading.Thread):
@ -1344,6 +1346,7 @@ class RecordingAnsibleJob(zuul.executor.server.AnsibleJob):
self.executor_server.build_history.append(
BuildHistory(name=build.name, result=result, changes=build.changes,
node=build.node, uuid=build.unique,
ref=build.parameters['zuul']['ref'],
parameters=build.parameters, jobdir=build.jobdir,
pipeline=build.parameters['ZUUL_PIPELINE'])
)

View File

@ -6,20 +6,14 @@
- time: '* * * * * */1'
- job:
name: project-bitrot-stable-old
name: project-bitrot
nodes:
- name: static
label: ubuntu-xenial
- job:
name: project-bitrot-stable-older
nodes:
- name: static
label: ubuntu-trusty
- project:
name: org/project
periodic:
jobs:
- project-bitrot-stable-old
- project-bitrot-stable-older
- project-bitrot

View File

@ -24,17 +24,11 @@
name: project-test1
- job:
name: project-bitrot-stable-old
name: project-bitrot
nodes:
- name: static
label: ubuntu-xenial
- job:
name: project-bitrot-stable-older
nodes:
- name: static
label: ubuntu-trusty
- project:
name: org/project
check:
@ -42,5 +36,4 @@
- project-test1
periodic:
jobs:
- project-bitrot-stable-old
- project-bitrot-stable-older
- project-bitrot

View File

@ -0,0 +1,19 @@
- pipeline:
name: periodic
manager: independent
trigger:
timer:
- time: '* * * * * */1'
- job:
name: integration
branches: master
override-branch: stable/havana
required-projects:
- org/project1
- project:
name: org/project1
periodic:
jobs:
- integration

View File

@ -7,7 +7,6 @@
- job:
name: integration
override-branch: stable/havana
required-projects:
- org/project1

View File

@ -25,17 +25,11 @@
name: project-test2
- job:
name: project-bitrot-stable-old
name: project-bitrot
nodes:
- name: static
label: ubuntu-xenial
- job:
name: project-bitrot-stable-older
nodes:
- name: static
label: ubuntu-trusty
- project:
name: org/project
check:
@ -44,5 +38,4 @@
- project-test2
periodic:
jobs:
- project-bitrot-stable-old
- project-bitrot-stable-older
- project-bitrot

View File

@ -248,6 +248,46 @@ class TestExecutorRepos(ZuulTestCase):
self.assertBuildStates(states, projects)
def test_periodic_override(self):
# This test can not use simple_layout because it must start
# with a configuration which does not include a
# timer-triggered job so that we have an opportunity to set
# the hold flag before the first job.
# This tests that we can override the branch in a timer
# trigger (mostly to ensure backwards compatability for jobs).
self.executor_server.hold_jobs_in_build = True
# Start timer trigger - also org/project
self.commitConfigUpdate('common-config',
'layouts/repo-checkout-timer-override.yaml')
self.sched.reconfigure(self.config)
p1 = 'review.example.com/org/project1'
projects = [p1]
self.create_branch('org/project1', 'stable/havana')
# The pipeline triggers every second, so we should have seen
# several by now.
time.sleep(5)
self.waitUntilSettled()
# Stop queuing timer triggered jobs so that the assertions
# below don't race against more jobs being queued.
self.commitConfigUpdate('common-config',
'layouts/repo-checkout-no-timer.yaml')
self.sched.reconfigure(self.config)
self.assertEquals(1, len(self.builds), "One build is running")
upstream = self.getUpstreamRepos(projects)
states = [
{p1: dict(commit=str(upstream[p1].commit('stable/havana')),
branch='stable/havana'),
},
]
self.assertBuildStates(states, projects)
def test_periodic(self):
# This test can not use simple_layout because it must start
# with a configuration which does not include a
@ -274,14 +314,19 @@ class TestExecutorRepos(ZuulTestCase):
'layouts/repo-checkout-no-timer.yaml')
self.sched.reconfigure(self.config)
self.assertEquals(1, len(self.builds), "One build is running")
self.assertEquals(2, len(self.builds), "Two builds are running")
upstream = self.getUpstreamRepos(projects)
states = [
{p1: dict(commit=str(upstream[p1].commit('stable/havana')),
branch='stable/havana'),
},
{p1: dict(commit=str(upstream[p1].commit('master')),
branch='master'),
},
]
if self.builds[0].parameters['zuul']['ref'] == 'refs/heads/master':
states = list(reversed(states))
self.assertBuildStates(states, projects)

View File

@ -12,11 +12,14 @@
# License for the specific language governing permissions and limitations
# under the License.
import os
import re
from testtools.matchers import MatchesRegex, StartsWith
import urllib
import time
import git
from tests.base import ZuulTestCase, simple_layout, random_sha1
@ -94,7 +97,16 @@ class TestGithubDriver(ZuulTestCase):
def test_tag_event(self):
self.executor_server.hold_jobs_in_build = True
sha = random_sha1()
self.create_branch('org/project', 'tagbranch')
files = {'README.txt': 'test'}
self.addCommitToRepo('org/project', 'test tag',
files, branch='tagbranch', tag='newtag')
path = os.path.join(self.upstream_root, 'org/project')
repo = git.Repo(path)
tag = repo.tags['newtag']
sha = tag.commit.hexsha
del repo
self.fake_github.emitEvent(
self.fake_github.getPushEvent('org/project', 'refs/tags/newtag',
new_rev=sha))

View File

@ -1806,17 +1806,17 @@ class TestScheduler(ZuulTestCase):
self.commitConfigUpdate('common-config', 'layouts/no-timer.yaml')
self.sched.reconfigure(self.config)
self.assertEqual(len(self.builds), 2, "Two timer jobs")
self.assertEqual(len(self.builds), 1, "One timer job")
A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
self.waitUntilSettled()
self.assertEqual(len(self.builds), 3, "One change plus two timer jobs")
self.assertEqual(len(self.builds), 2, "One change plus one timer job")
self.fake_gerrit.addEvent(A.getChangeAbandonedEvent())
self.waitUntilSettled()
self.assertEqual(len(self.builds), 2, "Two timer jobs remain")
self.assertEqual(len(self.builds), 1, "One timer job remains")
self.executor_server.release()
self.waitUntilSettled()
@ -2777,6 +2777,7 @@ class TestScheduler(ZuulTestCase):
# with a configuration which does not include a
# timer-triggered job so that we have an opportunity to set
# the hold flag before the first job.
self.create_branch('org/project', 'stable')
self.executor_server.hold_jobs_in_build = True
self.commitConfigUpdate('common-config', 'layouts/timer.yaml')
self.sched.reconfigure(self.config)
@ -2803,10 +2804,12 @@ class TestScheduler(ZuulTestCase):
self.executor_server.release()
self.waitUntilSettled()
self.assertEqual(self.getJobFromHistory(
'project-bitrot-stable-old').result, 'SUCCESS')
self.assertEqual(self.getJobFromHistory(
'project-bitrot-stable-older').result, 'SUCCESS')
self.assertHistory([
dict(name='project-bitrot', result='SUCCESS',
ref='refs/heads/master'),
dict(name='project-bitrot', result='SUCCESS',
ref='refs/heads/stable'),
], ordered=False)
data = json.loads(data)
status_jobs = set()
@ -2816,8 +2819,7 @@ class TestScheduler(ZuulTestCase):
for change in head:
for job in change['jobs']:
status_jobs.add(job['name'])
self.assertIn('project-bitrot-stable-old', status_jobs)
self.assertIn('project-bitrot-stable-older', status_jobs)
self.assertIn('project-bitrot', status_jobs)
def test_idle(self):
"Test that frequent periodic jobs work"
@ -2846,12 +2848,12 @@ class TestScheduler(ZuulTestCase):
'layouts/no-timer.yaml')
self.sched.reconfigure(self.config)
self.waitUntilSettled()
self.assertEqual(len(self.builds), 2,
self.assertEqual(len(self.builds), 1,
'Timer builds iteration #%d' % x)
self.executor_server.release('.*')
self.waitUntilSettled()
self.assertEqual(len(self.builds), 0)
self.assertEqual(len(self.history), x * 2)
self.assertEqual(len(self.history), x)
@simple_layout('layouts/smtp.yaml')
def test_check_smtp_pool(self):

View File

@ -667,7 +667,8 @@ class TestPrePlaybooks(AnsibleZuulTestCase):
self.assertFalse(os.path.exists(pre_flag_path))
post_flag_path = os.path.join(self.test_root, build.uuid +
'.post.flag')
self.assertTrue(os.path.exists(post_flag_path))
self.assertTrue(os.path.exists(post_flag_path),
"The file %s should exist" % post_flag_path)
class TestBrokenConfig(ZuulTestCase):

View File

@ -26,7 +26,7 @@ import queue
import voluptuous as v
from zuul.connection import BaseConnection
from zuul.model import Ref
from zuul.model import Ref, Tag, Branch
from zuul import exceptions
from zuul.driver.gerrit.gerritmodel import GerritChange, GerritTriggerEvent
@ -293,7 +293,34 @@ class GerritConnection(BaseConnection):
if event.change_number:
change = self._getChange(event.change_number, event.patch_number,
refresh=refresh)
elif event.ref and event.ref.startswith('refs/tags/'):
project = self.source.getProject(event.project_name)
change = Tag(project)
change.tag = event.ref[len('refs/tags/'):]
change.ref = event.ref
change.oldrev = event.oldrev
change.newrev = event.newrev
change.url = self._getGitwebUrl(project, sha=event.newrev)
elif event.ref and not event.ref.startswith('refs/'):
# Gerrit ref-updated events don't have branch prefixes.
project = self.source.getProject(event.project_name)
change = Branch(project)
change.branch = event.ref
change.ref = 'refs/heads/' + event.ref
change.oldrev = event.oldrev
change.newrev = event.newrev
change.url = self._getGitwebUrl(project, sha=event.newrev)
elif event.ref and event.ref.startswith('refs/heads/'):
# From the timer trigger
project = self.source.getProject(event.project_name)
change = Branch(project)
change.ref = event.ref
change.branch = event.branch
change.oldrev = event.oldrev
change.newrev = event.newrev
change.url = self._getGitwebUrl(project, sha=event.newrev)
elif event.ref:
# catch-all ref (ie, not a branch or head)
project = self.source.getProject(event.project_name)
change = Ref(project)
change.ref = event.ref
@ -301,14 +328,8 @@ class GerritConnection(BaseConnection):
change.newrev = event.newrev
change.url = self._getGitwebUrl(project, sha=event.newrev)
else:
project = self.source.getProject(event.project_name)
change = Ref(project)
branch = event.branch or 'master'
change.ref = 'refs/heads/%s' % branch
refs = self.getInfoRefs(project)
change.oldrev = refs[change.ref]
change.newrev = refs[change.ref]
change.url = self._getGitwebUrl(project, sha=change.newrev)
self.log.warning("Unable to get change for %s" % (event,))
change = None
return change
def _getChange(self, number, patchset, refresh=False, history=None):

View File

@ -32,7 +32,7 @@ import github3
from github3.exceptions import MethodNotAllowed
from zuul.connection import BaseConnection
from zuul.model import Ref
from zuul.model import Ref, Branch, Tag
from zuul.exceptions import MergeFailure
from zuul.driver.github.githubmodel import PullRequest, GithubTriggerEvent
@ -506,16 +506,21 @@ class GithubConnection(BaseConnection):
change.source_event = event
change.is_current_patchset = (change.pr.get('head').get('sha') ==
event.patch_number)
elif event.ref:
change = Ref(project)
else:
if event.ref and event.ref.startswith('refs/tags/'):
change = Tag(project)
change.tag = event.ref[len('refs/tags/'):]
elif event.ref and event.ref.startswith('refs/heads/'):
change = Branch(project)
change.branch = event.ref[len('refs/heads/'):]
else:
change = Ref(project)
change.ref = event.ref
change.oldrev = event.oldrev
change.newrev = event.newrev
change.url = self.getGitwebUrl(project, sha=event.newrev)
change.source_event = event
change.files = self.getPushedFileNames(event)
else:
change = Ref(project)
return change
def _getChange(self, project, number, patchset=None, refresh=False,

View File

@ -80,15 +80,18 @@ class TimerDriver(Driver, TriggerInterface):
def _onTrigger(self, tenant, pipeline_name, timespec):
for project_name in tenant.layout.project_configs.keys():
project_hostname, project_name = project_name.split('/', 1)
event = TimerTriggerEvent()
event.type = 'timer'
event.timespec = timespec
event.forced_pipeline = pipeline_name
event.project_hostname = project_hostname
event.project_name = project_name
self.log.debug("Adding event %s" % event)
self.sched.addEvent(event)
(trusted, project) = tenant.getProject(project_name)
for branch in project.source.getProjectBranches(project):
event = TimerTriggerEvent()
event.type = 'timer'
event.timespec = timespec
event.forced_pipeline = pipeline_name
event.project_hostname = project.canonical_hostname
event.project_name = project.name
event.ref = 'refs/heads/%s' % branch
event.branch = branch
self.log.debug("Adding event %s" % event)
self.sched.addEvent(event)
def stop(self):
if self.apsched:

View File

@ -156,19 +156,26 @@ class ExecutorClient(object):
canonical_name=item.change.project.canonical_name)
zuul_params = dict(uuid=uuid,
ref=item.current_build_set.ref,
buildset=item.current_build_set.uuid,
ref=item.change.ref,
pipeline=pipeline.name,
job=job.name,
project=project,
tenant=tenant.name,
tags=' '.join(sorted(job.tags)))
if hasattr(item.change, 'branch'):
zuul_params['branch'] = item.change.branch
if hasattr(item.change, 'tag'):
zuul_params['tag'] = item.change.tag
if hasattr(item.change, 'number'):
zuul_params['change'] = item.change.number
if hasattr(item.change, 'patchset'):
zuul_params['patchset'] = item.change.patchset
if hasattr(item.change, 'oldrev'):
zuul_params['oldrev'] = item.change.oldrev
if hasattr(item.change, 'newrev'):
zuul_params['newrev'] = item.change.newrev
# Legacy environment variables
params = dict(ZUUL_UUID=uuid,
ZUUL_PROJECT=item.change.project.name)

View File

@ -838,8 +838,18 @@ class AnsibleJob(object):
for project in args['projects']:
repo = repos[project['canonical_name']]
# If this project is the Zuul project and this is a ref
# rather than a change, checkout the ref.
if (project['canonical_name'] ==
args['zuul']['project']['canonical_name'] and
(not args['zuul'].get('branch')) and
args['zuul'].get('ref')):
ref = args['zuul']['ref']
else:
ref = None
self.checkoutBranch(repo,
project['name'],
ref,
args['branch'],
args['override_branch'],
project['override_branch'],
@ -909,7 +919,7 @@ class AnsibleJob(object):
repo.setRef('refs/heads/' + branch, commit)
return True
def checkoutBranch(self, repo, project_name, zuul_branch,
def checkoutBranch(self, repo, project_name, ref, zuul_branch,
job_branch, project_override_branch,
project_default_branch):
branches = repo.getBranches()
@ -921,6 +931,16 @@ class AnsibleJob(object):
self.log.info("Checking out %s job branch %s",
project_name, job_branch)
repo.checkoutLocalBranch(job_branch)
elif ref and ref.startswith('refs/heads/'):
b = ref[len('refs/heads/'):]
self.log.info("Checking out %s branch ref %s",
project_name, b)
repo.checkoutLocalBranch(b)
elif ref and ref.startswith('refs/tags/'):
t = ref[len('refs/tags/'):]
self.log.info("Checking out %s tag ref %s",
project_name, t)
repo.checkout(t)
elif zuul_branch and zuul_branch in branches:
self.log.info("Checking out %s zuul branch %s",
project_name, zuul_branch)

View File

@ -1817,19 +1817,17 @@ class QueueItem(object):
oldrev = None
newrev = None
refspec = None
branch = None
if hasattr(self.change, 'number'):
number = self.change.number
patchset = self.change.patchset
refspec = self.change.refspec
branch = self.change.branch
elif hasattr(self.change, 'newrev'):
if hasattr(self.change, 'newrev'):
oldrev = self.change.oldrev
newrev = self.change.newrev
branch = self.change.ref
else:
oldrev = None
newrev = None
branch = None
if hasattr(self.change, 'branch'):
branch = self.change.branch
source = self.change.project.source
connection_name = source.connection.connection_name
project = self.change.project
@ -1855,32 +1853,26 @@ class Ref(object):
self.ref = None
self.oldrev = None
self.newrev = None
self.files = []
def getBasePath(self):
base_path = ''
if hasattr(self, 'ref'):
base_path = "%s/%s" % (self.newrev[:2], self.newrev)
return base_path
def _id(self):
return self.newrev
def __repr__(self):
rep = None
if self.newrev == '0000000000000000000000000000000000000000':
rep = '<Ref 0x%x deletes %s from %s' % (
id(self), self.ref, self.oldrev)
rep = '<%s 0x%x deletes %s from %s' % (
type(self).__name__,
id(self), self.ref, self.oldrev)
elif self.oldrev == '0000000000000000000000000000000000000000':
rep = '<Ref 0x%x creates %s on %s>' % (
id(self), self.ref, self.newrev)
rep = '<%s 0x%x creates %s on %s>' % (
type(self).__name__,
id(self), self.ref, self.newrev)
else:
# Catch all
rep = '<Ref 0x%x %s updated %s..%s>' % (
id(self), self.ref, self.oldrev, self.newrev)
rep = '<%s 0x%x %s updated %s..%s>' % (
type(self).__name__,
id(self), self.ref, self.oldrev, self.newrev)
return rep
def equals(self, other):
@ -1913,11 +1905,24 @@ class Ref(object):
newrev=self.newrev)
class Change(Ref):
class Branch(Ref):
"""An existing branch state for a Project."""
def __init__(self, project):
super(Branch, self).__init__(project)
self.branch = None
class Tag(Ref):
"""An existing tag state for a Project."""
def __init__(self, project):
super(Tag, self).__init__(project)
self.tag = None
class Change(Branch):
"""A proposed new state for a Project."""
def __init__(self, project):
super(Change, self).__init__(project)
self.branch = None
self.number = None
self.url = None
self.patchset = None
@ -1941,12 +1946,6 @@ class Change(Ref):
def __repr__(self):
return '<Change 0x%x %s>' % (id(self), self._id())
def getBasePath(self):
if hasattr(self, 'refspec'):
return "%s/%s/%s" % (
str(self.number)[-2:], self.number, self.patchset)
return super(Change, self).getBasePath()
def equals(self, other):
if self.number == other.number and self.patchset == other.patchset:
return True