Add job option to hold following changes.

By setting 'hold-following-changes' to True on .*-merge jobs,
jobs for following changes in the dependend change queue won't be
launched until all of the -merge jobs ahead of them complete
successfully.  This prevents wasting resources on testing changes
that have no chance of merging as tested.  It gives up a small
amount of parallelization in order to achieve this (merge jobs will
now run in series, while other jobs may continue to run in parallel).

This should also help ensure that jobs nearer the front of the queue
run sooner than jobs at the back.  Currently, the nondeterminism of
job run-time can cause Jenkins to use limited resources on the back
of the queue instead of the front.

Also fixes these bugs:

* The success/failure messages specified were not being used.
* If a parameter function was applied to a meta-job, it would not
  be applied to jobs that matched the meta-job.

Change-Id: I03cc3e25c554d8aeb1ddb478afaed168a961962f

doc/source/zuul.rst

@@ -268,6 +268,18 @@ each job as it builds a list from the project specification.
 **success-message (optional)**
   The message that should be reported to Gerrit if the job fails.
 
+**hold-following-changes (optional)**
+  This is a boolean that indicates that changes that follow this
+  change in a dependent change queue should wait until this job
+  succeeds before launching.  If this is applied to a very short job
+  that can predict whether longer jobs will fail early, this can be
+  used to reduce the number of jobs that Zuul will launch and
+  ultimately have to cancel.  In that case, a small amount of
+  paralellization of jobs is traded for more efficient use of testing
+  resources.  On the other hand, to apply this to a long running job
+  would largely defeat the parallelization of dependent change testing
+  that is the main feature of Zuul.  The default is False.
+
 **branch (optional)**
   This job should only be run on matching branches.  This field is
   treated as a regular expression and multiple branches may be

zuul/model.py

@@ -56,10 +56,12 @@ class ChangeQueue(object):
 
 class Job(object):
     def __init__(self, name):
+        # If you add attributes here, be sure to add them to the copy method.
         self.name = name
         self.failure_message = None
         self.success_message = None
         self.parameter_function = None
+        self.hold_following_changes = False
         self.event_filters = []
 
     def __str__(self):
@@ -71,6 +73,8 @@ class Job(object):
     def copy(self, other):
         self.failure_message = other.failure_message
         self.success_message = other.failure_message
+        self.parameter_function = other.parameter_function
+        self.hold_following_changes = other.hold_following_changes
         self.event_filters = other.event_filters[:]
 
     def eventMatches(self, event):
@@ -374,6 +378,10 @@ class Change(object):
         for job in self._filterJobs(self.project.getJobs(self.queue_name)):
             build = self.current_build_set.getBuild(job.name)
             result = build.result
+            if result == 'SUCCESS' and job.success_message:
+                result = job.success_message
+            elif result == 'FAILURE' and job.failure_message:
+                result = job.failure_message
             url = build.url
             if not url:
                 url = job.name
@@ -404,8 +412,27 @@ class Change(object):
                 fakebuild.result = 'SKIPPED'
                 self.addBuild(fakebuild)
 
+    def isHoldingFollowingChanges(self):
+        tree = self.project.getJobTreeForQueue(self.queue_name)
+        for job in self._filterJobs(tree.getJobs()):
+            if not job.hold_following_changes:
+                continue
+            build = self.current_build_set.getBuild(job.name)
+            if not build:
+                return True
+            if build.result != 'SUCCESS':
+                return True
+        if not self.change_ahead:
+            return False
+        return self.change_ahead.isHoldingFollowingChanges()
+
     def _findJobsToRun(self, job_trees):
         torun = []
+        if self.change_ahead:
+            # Only run our jobs if any 'hold' jobs on the change ahead
+            # have completed successfully.
+            if self.change_ahead.isHoldingFollowingChanges():
+                return []
         for tree in job_trees:
             job = tree.job
             if not job.eventMatches(self.event):

zuul/scheduler.py

@@ -102,6 +102,9 @@ class Scheduler(threading.Thread):
             m = config_job.get('success-message', None)
             if m:
                 job.success_message = m
+            m = config_job.get('hold-following-changes', False)
+            if m:
+                job.hold_following_changes = True
             fname = config_job.get('parameter-function', None)
             if fname:
                 func = self._config_env.get(fname, None)
@@ -373,7 +376,11 @@ class BaseQueueManager(object):
                     efilters += str(e)
                 if efilters:
                     efilters = ' ' + efilters
-                self.log.info("%s%s%s" % (istr, repr(tree.job), efilters))
+                hold = ''
+                if tree.job.hold_following_changes:
+                    hold = ' [hold]'
+                self.log.info("%s%s%s%s" % (istr, repr(tree.job),
+                                            efilters, hold))
             for x in tree.job_trees:
                 log_jobs(x, indent + 2)