zuul/zuul
Code Issues Proposed changes

Add include-vars job attribute

Browse Source 
This adds a job attribute which may be used to instruct Zuul to load
variables from a file in a target git repo before running a job's
playbook.

The intended use case is jobs which are heavily reliant on variables
for their configuration and where these veriables change quite
frequently.  When users change job variables, a tenant reconfiguration
is required in order to apply the new values.  In practice, the
new values do not typically change Zuul's behavior, only the input
data supplied to the job.

To make this case more efficient, this allows users to "offload" Zuul
variables into files that are in the repo itself.  Changes to these
dedicated variable files will not trigger Zuul reconfiguration, but
they will achieve the same effect when running jobs.

An obvious question is "why not add an include_vars to playbooks?".
This is a reasonable solution, but it does require some extra work
to anticipate the situations where doing so would be useful.
Additionally, because Zuul runs multiple playbooks, in order to
achieve full parity with Zuul job variables, the import would need
to occur very early in the job (perhaps a base-job pre-run playbook)
and use cacheable facts.  This suggests that the base job itself
might need to provide a Zuul job-variable-based API to load variable
files.  At this point, it is no longer a simple solution, nor is it
as user-friendly as support include-vars in Zuul.

As mentioned above, the primary use case is offloading what would
otherwise be Zuul job variables to a file.  This change attempts
to encode a behavior that is most similar to what happens with
job.vars.  That is to say, if someone moves an existing "job.vars"
variable to a "job.include-vars" file, they should ideally get the
same behavior.

To that end, include-vars assumes (unless otherwise specified) that
the file it should read is in the same repo where the job containing
the include-vars entry is defined.

Users can achieve more advanced effects by specifying that the
file should be read from a different project.

Finally, users may also use this feature in centrally-defined jobs
that may apply to any repo by specifying that the file should be
read from the project under test.  As an example, this would allow
patterns where data such as python or nodejs versions may be defined
in a file in the repo instead of as a project variable.

Change-Id: I7afb0f515e87b884f1e548b3bcf798884e3714e1
This commit is contained in:
James E. Blair 2024-09-18 16:00:18 -07:00
parent d195b3cb65
commit 8e01541279
19 changed files with 513 additions and 52 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

73
doc/source/config/job.rst
View File

@ -967,6 +967,79 @@ Here is an example of two job definitions:
api:
baz: "this variable is visible on api1 and api2"
.. attr:: include-vars
A list of files from which to read variables.
The value may be supplied as a list or a single item, and each
value may be a string or a dictionary described below. If
supplied as a string, it is treated as the
:attr:`job.include-vars.name`.
Files are read in order, with later variable values overriding
earlier ones. Variables specified by :attr:`job.vars` and
related attributes will override variables read from files.
The file should be in YAML or JSON format.
Supports override control. The default is ``!inherit``: values
are appended without duplication.
.. attr:: name
:required:
The name (relative to the root of the repository) of the file
to read.
.. attr:: project
The name of the project containing the file to read. If this
is left unspecified, the project containing the current job
definition is used. This option is mutually exclusive with
:attr:`job.include-vars.zuul-project`.
.. attr:: required
:default: true
A boolean indicating whether this file is required to be
present. If this is set to ``true`` and the file is not
present, it is considered an error and the job result will
reflect this.
.. attr:: zuul-project
:default: false
A boolean indicating that instead of using a specified
project, the project associated with the change under test
(which can be found in the :var:`zuul.project` variable)
should be used. This permits the definition of jobs that may
be centrally defined and used globally to read variables
from files in the projects upon which they run.
This option is mutually exclusive with
:attr:`job.include-vars.project`.
An example using job-vars:
.. code-block:: yaml
- job:
name: central-job
description: |
This job reads versions.yaml from whatever project
it is used to test.
include-vars:
- name: versions.yaml
zuul-project: true
- job:
name: integration-job
description: |
This job reads data/product-versions.yaml from the repo
that contains this very job definition, no matter which
project runs the job.
include-vars: data/product-versions.yaml
.. attr:: dependencies
A list of other jobs upon which this job depends. Zuul will not

11
doc/source/job-content.rst
View File

@ -94,6 +94,7 @@ order of precedence is:
#. :ref:`Secrets <user_jobs_secrets>`
#. :ref:`Job variables <user_jobs_job_variables>`
#. :ref:`Project variables <user_jobs_project_variables>`
#. :ref:`File variables <user_jobs_file_variables>`
#. :ref:`Parent job results <user_jobs_parent_results>`
Meaning that a site-wide variable with the same name as any other will
@ -201,6 +202,16 @@ a project.
vars:
var_for_all_jobs: override
.. _user_jobs_file_variables:
File Variables
~~~~~~~~~~~~~~
Any variables specified in files loaded from project repositories
(using the :attr:`job.include-vars` attribute) are available to jobs
as Ansible host variables in the same way as :ref:`job variables
<user_jobs_job_variables>`.
.. _user_jobs_parent_results:
Parent Job Results

10
releasenotes/notes/include-vars-b6ed95f1b5b88a3a.yaml Normal file
View File

@ -0,0 +1,10 @@
---
features:
- |
Jobs may now read variables from an in-repo file using the
:attr:`job.include-vars` job attribute. This allows jobs with
many (or frequently changing) variables to load their variables
from a file in order to reduce the complexity of Zuul
configuration. Because these files are read when the job is
executed, changes to their values will not cause Zuul tenant
reconfigurations.

1
tests/fixtures/config/include-vars/git/common-config/playbooks/run.yaml vendored Normal file
View File

@ -0,0 +1 @@
---

43
tests/fixtures/config/include-vars/git/common-config/zuul.yaml vendored Normal file
View File

@ -0,0 +1,43 @@
- pipeline:
name: check
manager: independent
trigger:
gerrit:
- event: patchset-created
- event: comment-added
comment: '^(Patch Set [0-9]+:\n\n)?(?i:recheck)$'
success:
gerrit:
Verified: 1
failure:
gerrit:
Verified: -1
- job:
name: base
parent: null
nodeset:
nodes:
- name: controller
label: label1
run: playbooks/run.yaml
- job:
name: zuul-project
include-vars:
- name: zuul-vars.yaml
zuul-project: true
- job:
name: zuul-project-required
include-vars:
- name: missing-vars.yaml
zuul-project: true
required: true
- job:
name: zuul-project-missing-ok
include-vars:
- name: missing-vars.yaml
zuul-project: true
required: false

1
tests/fixtures/config/include-vars/git/org_project1/README vendored Normal file
View File

@ -0,0 +1 @@
test

4
tests/fixtures/config/include-vars/git/org_project1/project1-vars.yaml vendored Normal file
View File

@ -0,0 +1,4 @@
project1: true
job_var_precedence: include-vars
project_var_precedence: include-vars
parent_var_precedence: include-vars

26
tests/fixtures/config/include-vars/git/org_project1/zuul.yaml vendored Normal file
View File

@ -0,0 +1,26 @@
- job:
name: parent-job
description: Returns data to test precedence
- job:
name: same-project
include-vars: project1-vars.yaml
vars:
job_var_precedence: job-vars
- job:
name: other-project
include-vars:
- project: org/project2
name: project2-vars.yaml
- project:
vars:
project_var_precedence: project-vars
check:
jobs:
- parent-job
- same-project:
dependencies:
- parent-job
- other-project

1
tests/fixtures/config/include-vars/git/org_project2/README vendored Normal file
View File

@ -0,0 +1 @@
test

1
tests/fixtures/config/include-vars/git/org_project2/project2-vars.yaml vendored Normal file
View File

@ -0,0 +1 @@
project2: true

1
tests/fixtures/config/include-vars/git/org_project2/zuul-vars.yaml vendored Normal file
View File

@ -0,0 +1 @@
zuulproject: true

6
tests/fixtures/config/include-vars/git/org_project2/zuul.yaml vendored Normal file
View File

@ -0,0 +1,6 @@
- project:
check:
jobs:
- zuul-project
- zuul-project-required
- zuul-project-missing-ok

9
tests/fixtures/config/include-vars/main.yaml vendored Normal file
View File

@ -0,0 +1,9 @@
- tenant:
name: tenant-one
source:
gerrit:
config-projects:
- common-config
untrusted-projects:
- org/project1
- org/project2

109
tests/unit/test_v3.py
View File

@ -10790,3 +10790,112 @@ class TestSuperproject(ZuulTestCase):
self.assertIn('the only project definition permitted is',
A.messages[0])
self.assertHistory([])
class TestIncludeVars(ZuulTestCase):
tenant_config_file = 'config/include-vars/main.yaml'
def getBuildInventory(self, name):
build = self.getBuildByName(name)
inv_path = os.path.join(build.jobdir.root, 'ansible', 'inventory.yaml')
with open(inv_path, 'r') as f:
inventory = yaml.safe_load(f)
return inventory
def test_include_vars(self):
# Test including from the same project and another project
self.executor_server.hold_jobs_in_build = True
A = self.fake_gerrit.addFakeChange('org/project1', 'master', 'A')
self.executor_server.returnData(
'parent-job', A,
{
'parent_var_precedence': 'parent-vars',
'parent_vars': True,
}
)
self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
self.waitUntilSettled("waiting for parent job to be running")
self.executor_server.release('parent-job')
self.waitUntilSettled("waiting for remaining jobs to start")
inventory = self.getBuildInventory('same-project')
ivars = inventory['all']['vars']
# We read a variable from the expected file
self.assertTrue(ivars['project1'])
# Job and project vars have higher precedence than file
self.assertEqual('job-vars', ivars['job_var_precedence'])
self.assertEqual('project-vars', ivars['project_var_precedence'])
# Files have higher precedence than returned parent data
self.assertEqual('include-vars', ivars['parent_var_precedence'])
# Make sure we did get returned parent data
self.assertTrue(ivars['parent_vars'])
inventory = self.getBuildInventory('other-project')
ivars = inventory['all']['vars']
self.assertTrue(ivars['project2'])
self.executor_server.hold_jobs_in_build = False
self.executor_server.release()
self.waitUntilSettled()
self.assertHistory([
dict(name='parent-job', result='SUCCESS', changes='1,1'),
dict(name='same-project', result='SUCCESS', changes='1,1'),
dict(name='other-project', result='SUCCESS', changes='1,1'),
], ordered=False)
def test_include_vars_zuul_project(self):
# Test zuul-project vars (required and not)
self.executor_server.hold_jobs_in_build = True
A = self.fake_gerrit.addFakeChange('org/project2', 'master', 'A')
self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
self.waitUntilSettled()
# Required and present
inventory = self.getBuildInventory('zuul-project')
ivars = inventory['all']['vars']
self.assertTrue(ivars['zuulproject'])
# Optional and not present
inventory = self.getBuildInventory('zuul-project-missing-ok')
ivars = inventory['all']['vars']
self.assertFalse('zuulproject' in ivars)
self.executor_server.hold_jobs_in_build = False
self.executor_server.release()
self.waitUntilSettled()
self.assertHistory([
dict(name='zuul-project', result='SUCCESS', changes='1,1'),
dict(name='zuul-project-missing-ok',
result='SUCCESS', changes='1,1'),
], ordered=False)
# Required and not present
self.assertTrue(re.search(
'- zuul-project-required .* ERROR Required vars file '
'missing-vars.yaml not found',
A.messages[-1]))
def test_include_vars_config_error(self):
# Test the mutual exclusion of project and zuul-project
in_repo_conf = textwrap.dedent(
"""
- job:
name: badjob
include-vars:
- name: foo.yaml
project: org/project
zuul-project: true
""")
file_dict = {'zuul.yaml': in_repo_conf}
A = self.fake_gerrit.addFakeChange('org/project1', 'master', 'A',
files=file_dict)
self.fake_gerrit.addEvent(A.getPatchsetCreatedEvent(1))
self.waitUntilSettled()
self.assertEqual(A.reported, 1,
"A should report failure")
self.assertEqual(A.patchsets[0]['approvals'][0]['value'], "-1")
self.assertIn('extra keys not allowed', A.messages[0])

11
tests/unit/test_web.py
View File

@ -372,6 +372,7 @@ class TestWeb(BaseTestWeb):
'group_variables': {},
'host_variables': {},
'image_build_name': None,
'include_vars': [],
'intermediate': False,
'irrelevant_files': [],
'match_on_config_updates': True,
@ -511,6 +512,7 @@ class TestWeb(BaseTestWeb):
'extra_variables': {},
'group_variables': {},
'host_variables': {},
'include_vars': [],
'variant_description': '',
'voting': True,
'workspace_scheme': 'golang'
@ -566,6 +568,7 @@ class TestWeb(BaseTestWeb):
'extra_variables': {},
'group_variables': {},
'host_variables': {},
'include_vars': [],
'variant_description': 'stable',
'voting': True,
'workspace_scheme': 'golang'
@ -614,6 +617,7 @@ class TestWeb(BaseTestWeb):
'extra_variables': {},
'group_variables': {},
'host_variables': {},
'include_vars': [],
'variant_description': '',
'voting': True,
'workspace_scheme': 'golang'
@ -743,6 +747,7 @@ class TestWeb(BaseTestWeb):
'extra_variables': {},
'group_variables': {},
'host_variables': {},
'include_vars': [],
'variant_description': '',
'voting': True,
'workspace_scheme': 'golang'}],
@ -786,6 +791,7 @@ class TestWeb(BaseTestWeb):
'extra_variables': {},
'group_variables': {},
'host_variables': {},
'include_vars': [],
'variant_description': '',
'voting': True,
'workspace_scheme': 'golang'}],
@ -829,6 +835,7 @@ class TestWeb(BaseTestWeb):
'extra_variables': {},
'group_variables': {},
'host_variables': {},
'include_vars': [],
'variant_description': '',
'voting': True,
'workspace_scheme': 'golang'}],
@ -872,6 +879,7 @@ class TestWeb(BaseTestWeb):
'extra_variables': {},
'group_variables': {},
'host_variables': {},
'include_vars': [],
'variant_description': '',
'voting': True,
'workspace_scheme': 'golang'}]]
@ -942,6 +950,7 @@ class TestWeb(BaseTestWeb):
'extra_variables': {},
'group_variables': {},
'host_variables': {},
'include_vars': [],
'variant_description': '',
'voting': True,
'workspace_scheme': 'golang'}
@ -1298,6 +1307,7 @@ class TestWeb(BaseTestWeb):
'change_url': None,
'child_jobs': [],
'event_id': None,
'include_vars': [],
},
'workspace_scheme': 'golang',
}
@ -1347,6 +1357,7 @@ class TestWeb(BaseTestWeb):
'change_url': None,
'child_jobs': [],
'event_id': None,
'include_vars': [],
'items': [],
'job': 'noop',
'jobtags': [],

49
zuul/configloader.py
View File

@ -730,15 +730,33 @@ class JobParser(object):
'semaphores': to_list(str),
}
playbook_def = to_list(vs.Any(str, complex_playbook_def))
post_run_playbook_def = {
vs.Required('name'): str,
'semaphores': to_list(str),
'cleanup': bool,
}
playbook_def = to_list(vs.Any(str, complex_playbook_def))
post_run_playbook_def = to_list(vs.Any(str, post_run_playbook_def))
complex_include_vars_project_def = {
vs.Required('name'): str,
'project': str,
'required': bool,
}
complex_include_vars_zuul_project_def = {
vs.Required('name'): str,
'zuul-project': bool,
'required': bool,
}
include_vars_def = to_list(vs.Any(str,
complex_include_vars_project_def,
complex_include_vars_zuul_project_def,
))
# Attributes of a job that can also be used in Project and ProjectTemplate
job_attributes = {'parent': vs.Any(str, None),
'final': bool,
@ -783,6 +801,7 @@ class JobParser(object):
'extra-vars': override_value(ansible_vars_dict),
'host-vars': override_value({str: ansible_vars_dict}),
'group-vars': override_value({str: ansible_vars_dict}),
'include-vars': override_value(include_vars_def),
'dependencies': override_list(
vs.Any(job_dependency, str)),
'allowed-projects': to_list(str),
@ -1123,6 +1142,34 @@ class JobParser(object):
check_varnames(conf_vars)
setattr(job, job_attr, conf_vars)
with self.pcontext.confAttr(conf, 'include-vars') as conf_include_vars:
if isinstance(conf_include_vars, yaml.OverrideValue):
job.override_control['include_vars'] =\
conf_include_vars.override
conf_include_vars = conf_include_vars.value
if conf_include_vars:
include_vars = []
for iv in as_list(conf_include_vars):
if isinstance(iv, str):
iv = {'name': iv}
project_cn = None
if not iv.get('zuul-project', False):
pname = iv.get(
'project',
conf['_source_context'].project_canonical_name)
(trusted, project) = self.pcontext.tenant.getProject(
pname)
if project is None:
raise ProjectNotFoundError(pname)
project_cn = project.canonical_name
job_include_vars = model.JobIncludeVars(
iv['name'],
project_cn,
iv.get('required', True),
)
include_vars.append(job_include_vars)
job.include_vars = tuple(include_vars)
allowed_projects = conf.get('allowed-projects', None)
# See note above at "post-review".
if allowed_projects and not job.allowed_projects:

1
zuul/executor/common.py
View File

@ -66,6 +66,7 @@ def construct_build_params(uuid, connections, job, item, pipeline,
tenant=tenant.name,
event_id=item.event.zuul_event_id if item.event else None,
jobtags=sorted(job.tags),
include_vars=job.include_vars,
))
if hasattr(change, 'message'):
zuul_params['message'] = strings.b64encode(change.message)

131
zuul/executor/server.py
View File

@ -1,5 +1,5 @@
# Copyright 2014 OpenStack Foundation
# Copyright 2021-2023 Acme Gating, LLC
# Copyright 2021-2024 Acme Gating, LLC
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
@ -1092,9 +1092,8 @@ class AnsibleJob(object):
max_attempts = self.arguments["max_attempts"]
self.retry_limit = self.arguments["zuul"]["attempts"] >= max_attempts
parent_data = self.arguments["parent_data"]
self.normal_vars = Job._deepUpdate(parent_data.copy(),
self.job.variables)
# We don't set normal_vars until we load the include-vars
# files after preparing repos.
self.secret_vars = self.arguments["secret_parent_data"]
def run(self):
@ -1390,42 +1389,33 @@ class AnsibleJob(object):
tasks = []
projects = set()
with open(self.jobdir.job_output_file, 'a') as job_output:
job_output.write("{now} | Updating repositories\n".format(
now=datetime.datetime.now()
))
# Make sure all projects used by the job are updated...
# Make sure all projects used by the job are updated ...
for project in args['projects']:
self.log.debug("Updating project %s" % (project,))
key = (project['connection'], project['name'])
projects.add(key)
# ... as well as all playbook and role projects ...
for playbook in self.job.all_playbooks:
key = (playbook['connection'], playbook['project'])
projects.add(key)
for role in playbook['roles']:
key = (role['connection'], role['project'])
projects.add(key)
# ... and include-vars projects.
for iv in self.job.include_vars:
key = (iv['connection'], iv['project'])
projects.add(key)
for (connection, project) in projects:
self.log.debug("Updating project %s %s", connection, project)
tasks.append(self.executor_server.update(
project['connection'], project['name'],
repo_state=self.repo_state,
connection, project, repo_state=self.repo_state,
zuul_event_id=self.zuul_event_id,
build=self.build_request.uuid,
span_context=tracing.getSpanContext(
trace.get_current_span()),
))
projects.add((project['connection'], project['name']))
# ...as well as all playbook and role projects.
repos = []
for playbook in self.job.all_playbooks:
repos.append(playbook)
repos += playbook['roles']
for repo in repos:
key = (repo['connection'], repo['project'])
if key not in projects:
self.log.debug("Updating playbook or role %s" % (
repo['project'],))
tasks.append(self.executor_server.update(
*key, repo_state=self.repo_state,
zuul_event_id=self.zuul_event_id,
build=self.build_request.uuid,
span_context=tracing.getSpanContext(
trace.get_current_span()),
))
projects.add(key)
for task in tasks:
task.wait()
@ -1580,6 +1570,8 @@ class AnsibleJob(object):
self._send_aborted()
return
self.loadIncludeVars()
# We set the nodes to "in use" as late as possible. So in case
# the build failed during the checkout phase, the node is
# still untouched and nodepool can re-allocate it to a
@ -2489,6 +2481,26 @@ class AnsibleJob(object):
# It is neither a bare role, nor a collection of roles
raise RoleNotFoundError("Unable to find role in %s" % (path,))
def selectBranchForProject(self, project, project_default_branch):
# Find if the project is one of the job-specified projects.
# If it is, we can honor the project checkout-override options.
args = self.arguments
args_project = {}
for p in args['projects']:
if (p['canonical_name'] == project.canonical_name):
args_project = p
break
return self.resolveBranch(
project.canonical_name,
None,
args['branch'],
self.job.override_branch,
self.job.override_checkout,
args_project.get('override_branch'),
args_project.get('override_checkout'),
project_default_branch)
def prepareZuulRole(self, jobdir_playbook, role, args, role_info):
self.log.debug("Prepare zuul role for %s" % (role,))
# Check out the role repo if needed
@ -2509,23 +2521,8 @@ class AnsibleJob(object):
role_info.checkout_description = 'playbook branch'
role_info.checkout = branch
else:
# Find if the project is one of the job-specified projects.
# If it is, we can honor the project checkout-override options.
args_project = {}
for p in args['projects']:
if (p['canonical_name'] == project.canonical_name):
args_project = p
break
branch, selected_desc = self.resolveBranch(
project.canonical_name,
None,
args['branch'],
self.job.override_branch,
self.job.override_checkout,
args_project.get('override_branch'),
args_project.get('override_checkout'),
role['project_default_branch'])
branch, selected_desc = self.selectBranchForProject(
project, role['project_default_branch'])
self.log.debug("Role using %s %s", selected_desc, branch)
role_info.checkout_description = selected_desc
role_info.checkout = branch
@ -2678,6 +2675,40 @@ class AnsibleJob(object):
known_hosts.write('%s\n' % key)
return zuul_resources
def loadIncludeVars(self):
parent_data = self.arguments["parent_data"]
normal_vars = parent_data.copy()
for iv in self.job.include_vars:
source = self.executor_server.connections.getSource(
iv['connection'])
project = source.getProject(iv['project'])
branch, selected_desc = self.selectBranchForProject(
project, iv['project_default_branch'])
self.log.debug("Include-vars project %s using %s %s",
project.canonical_name, selected_desc, branch)
if not iv['trusted']:
path = self.checkoutUntrustedProject(project, branch,
self.arguments)
else:
path = self.checkoutTrustedProject(project, branch,
self.arguments)
path = os.path.join(path, iv['name'])
try:
with open(path) as f:
self.log.debug("Loading vars from %s", path)
new_vars = yaml.safe_load(f)
normal_vars = Job._deepUpdate(normal_vars, new_vars)
except FileNotFoundError:
self.log.info("Vars file %s not found", path)
if iv['required']:
raise ExecutorError(
f"Required vars file {iv['name']} not found")
self.normal_vars = Job._deepUpdate(normal_vars,
self.job.variables)
def prepareVars(self, args, zuul_resources):
normal_vars = self.normal_vars.copy()
check_varnames(normal_vars)

77
zuul/model.py
View File

@ -3009,6 +3009,7 @@ class FrozenJob(zkobject.ZKObject):
'deduplicate',
'failure_output',
'image_build_name',
'include_vars',
)
job_data_attributes = ('artifact_data',
@ -3181,6 +3182,8 @@ class FrozenJob(zkobject.ZKObject):
for (project_name, job_project)
in data['required_projects'].items()}
# MODEL_API <= 31
data.setdefault('include_vars', [])
data['provides'] = frozenset(data['provides'])
data['requires'] = frozenset(data['requires'])
data['tags'] = frozenset(data['tags'])
@ -3433,6 +3436,7 @@ class Job(ConfigObject):
d['deduplicate'] = self.deduplicate
d['failure_output'] = self.failure_output
d['image_build_name'] = self.image_build_name
d['include_vars'] = list(map(lambda x: x.toDict(), self.include_vars))
if self.isBase():
d['parent'] = None
elif self.parent:
@ -3491,6 +3495,7 @@ class Job(ConfigObject):
extra_variables={},
host_variables={},
group_variables={},
include_vars=(),
nodeset=Job.empty_nodeset,
workspace=None,
pre_run=(),
@ -3527,6 +3532,7 @@ class Job(ConfigObject):
override_control['extra_variables'] = False
override_control['host_variables'] = False
override_control['group_variables'] = False
override_control['include_vars'] = False
override_control['required_projects'] = False
override_control['failure_output'] = False
@ -3567,6 +3573,11 @@ class Job(ConfigObject):
project_canonical_names.update(self._projectsFromPlaybooks(
itertools.chain(self.pre_run, [self.run[0]], self.post_run,
self.cleanup_run), with_implicit=True))
project_canonical_names.update({
iv.project_canonical_name
for iv in self.include_vars
if iv.project_canonical_name
})
# Return a sorted list so the order is deterministic for
# comparison.
return sorted(project_canonical_names)
@ -3600,6 +3611,25 @@ class Job(ConfigObject):
role['project'] = role_project.name
return d
def _freezeIncludeVars(self, tenant, layout, change, include_vars):
project_cname = (include_vars.project_canonical_name or
change.project.canonical_name)
(trusted, project) = tenant.getProject(project_cname)
project_metadata = layout.getProjectMetadata(project_cname)
if project_metadata:
default_branch = project_metadata.default_branch
else:
default_branch = 'master'
connection = project.source.connection
return dict(
name=include_vars.name,
connection=connection.connection_name,
project=project.name,
project_default_branch=default_branch,
trusted=trusted,
required=include_vars.required,
)
def _deduplicateSecrets(self, context, frozen_playbooks):
# At the end of this method, the values in the playbooks'
# secrets dictionary will be mutated to either be an integer
@ -3696,6 +3726,9 @@ class Job(ConfigObject):
# redacted.
kw['secrets'] = self._deduplicateSecrets(context, frozen_playbooks)
kw['affected_projects'] = self._getAffectedProjects(tenant)
# Fill in the zuul project for any include-vars that don't specify it
kw['include_vars'] = [self._freezeIncludeVars(
tenant, layout, change, iv) for iv in kw['include_vars']]
kw['config_hash'] = self.getConfigHash(tenant)
# Ensure that the these attributes are exactly equal to what
# would be deserialized on another scheduler.
@ -4035,7 +4068,8 @@ class Job(ConfigObject):
'roles', 'variables', 'extra_variables',
'host_variables', 'group_variables',
'required_projects', 'allowed_projects',
'semaphores', 'failure_output']):
'semaphores', 'failure_output',
'include_vars']):
setattr(self, k, other._get(k))
# Don't set final above so that we don't trip an error halfway
@ -4132,6 +4166,15 @@ class Job(ConfigObject):
other.cleanup_run, layout, semaphore_handler)
self.cleanup_run = other_cleanup_run + self.cleanup_run
self.updateVariables(other)
if other._get('include_vars') is not None:
if other.override_control['include_vars']:
include_vars = other.include_vars
else:
include_vars = list(self.include_vars)
for iv in other.include_vars:
if iv not in include_vars:
include_vars.append(iv)
self.include_vars = tuple(include_vars)
if other._get('required_projects') is not None:
self.updateProjects(other)
if (other._get('allowed_projects') is not None and
@ -4237,6 +4280,38 @@ class Job(ConfigObject):
return False
class JobIncludeVars(ConfigObject):
""" A reference to a variables file from a job. """
def __init__(self, name, project_canonical_name, required):
super().__init__()
self.name = name
# The repo to look for the file in, or None for the zuul project
self.project_canonical_name = project_canonical_name
self.required = required
def toDict(self):
d = dict()
d['name'] = self.name
d['project_canonical_name'] = self.project_canonical_name
d['required'] = self.required
return d
@classmethod
def fromDict(cls, data):
return cls(data['name'],
data['canonical_project_name'],
data['required'])
def __hash__(self):
return hash(json.dumps(self.toDict(), sort_keys=True))
def __eq__(self, other):
if not isinstance(other, JobIncludeVars):
return False
return self.toDict() == other.toDict()
class JobProject(ConfigObject):
""" A reference to a project from a job. """