zuul/zuul
Code Issues Proposed changes

Freeze job dependencies with job graph

Browse Source 
This is part of the circular dependency refactor.

Update the job graph to record job dependencies when it is frozen,
and store these dependencies by uuid.  This means our dependency
graph points to actual frozen jobs rather than mere job names.

This is a pre-requisite to being able to disambiguate dependencies
later when a queue item supports multiple jobs with the same name.

The behavior where we would try to unwind an addition to the job
graph if it failed is removed.  This was originally written with the
idea that we would try to run as many jobs as possible if there was
a config error.  That was pre-zuul-v3 behavior.  Long since, in all
cases when we actually encounter an error adding to the job graph,
we bail and report that to the user.  No longer handling that
case simplifies the code somewhat and makes it more future-proof
(while complicating one of the tests that relied on that behavior
as a shortcut).

This attempts to handle upgrades by emulating the old behavior
if a job graph was created on an older model version.  Since it
relies on frozen job uuids, it also attempts to handle the case
where a frozenjob does not have a uuid (which is a very recent
model change and likely to end up in the same upgrade for some
users) by emulating the old behavior.

Change-Id: I0070a07fcb5af950651404fa8ae66ea18c6ca006
This commit is contained in:
James E. Blair
2023-11-28 14:07:29 -08:00
parent 27a4beb698
commit 071c48c5ae
7 changed files with 365 additions and 93 deletions
Download Patch File Download Diff File Expand all files Collapse all files

29
zuul/manager/__init__.py
View File

@@ -1017,7 +1017,7 @@ class PipelineManager(metaclass=ABCMeta):
def _getPausedParent(self, build_set, job):
job_graph = build_set.job_graph
if job_graph:
for parent in job_graph.getParentJobsRecursively(job.name):
for parent in job_graph.getParentJobsRecursively(job):
build = build_set.getBuild(parent.name)
if build.paused:
return build
@@ -1929,6 +1929,27 @@ class PipelineManager(metaclass=ABCMeta):
self._resumeBuilds(build.build_set)
return True
def _legacyGetJob(self, item, job):
# TODO (model_api<21): The "this_job" indirection can be
# removed when the circular dependency refactor is complete.
# Until then, we can't assume that the build (and therefore
# job) is within the current buildset (if it has been
# deduplicated).
try:
this_uuid = item.current_build_set.job_graph.getUuidForJob(
job.name)
except ValueError:
# This doesn't currently raise a ValueError, it just
# returns None, but that could easily change during
# refactoring so let's go ahead and handle both.
this_uuid = None
if this_uuid is None:
# This is the model_api < 18 case, we're going to end up
# looking up dependencies by name anyway so the specific
# frozen job object doesn't matter.
return job
return item.current_build_set.job_graph.getJobFromUuid(this_uuid)
def _resumeBuilds(self, build_set):
"""
Resumes all paused builds of a buildset that may be resumed.
@@ -1940,10 +1961,11 @@ class PipelineManager(metaclass=ABCMeta):
child_builds = []
for item in self._getItemsWithBuild(build):
job_graph = item.current_build_set.job_graph
_this_job = self._legacyGetJob(item, build.job)
child_builds += [
item.current_build_set.builds.get(x.name)
for x in job_graph.getDependentJobsRecursively(
build.job.name)]
_this_job)]
all_completed = True
for child_build in child_builds:
if not child_build or not child_build.result:
@@ -1962,7 +1984,8 @@ class PipelineManager(metaclass=ABCMeta):
def _resetDependentBuilds(self, build_set, build):
job_graph = build_set.job_graph
for job in job_graph.getDependentJobsRecursively(build.job.name):
_this_job = self._legacyGetJob(build_set.item, build.job)
for job in job_graph.getDependentJobsRecursively(_this_job):
self.sched.cancelJob(build_set, job)
build = build_set.getBuild(job.name)
if build: