Add ZUUL_COMMIT.

To facilitate using zuul with just the jenkins git plugin, add
the ZUUL_COMMIT parameter.  Also, always include ZUUL_REF, whether
the job is pre- or post-commit.  If it's pre, it will be a
refs/zuul ref, if it's post, it will be refs/tags or a branch
name.

Change-Id: I88c38a28dcd552b2540095d36caacd10acf167b8
Reviewed-on: https://review.openstack.org/13934
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins

doc/source/launchers.rst

@@ -63,44 +63,71 @@ Zuul will pass some parameters to Jenkins for every job it launches.
 Check **This build is parameterized**, and add the following fields
 with the type **String Parameter**:
 
-**UUID**
+**ZUUL_UUID**
   Zuul provided key to link builds with Gerrit events
-**GERRIT_PROJECT**
+**ZUUL_REF**
-  Zuul provided project name
+  Zuul provided ref that includes commit(s) to build
-**GERRIT_BRANCH**
+**ZUUL_COMMIT**
-  Zuul provided branch name
+  The commit SHA1 at the head of ZUUL_REF
-**GERRIT_CHANGES**
-  Zuul provided list of dependent changes to merge
 
-You may find it useful to use the ``GERRIT_*`` variables in your job.
+Those are the only required parameters.  The UUID is needed for Zuul
-In particular, ``GERRIT_CHANGES`` indicates the change or changes that
+to keep track of the build, and the REF and COMMIT parameters are for
-should be tested.  If Zuul has decided that more than one change
+use in preparing the git repo for the build.  The following parameters
-should be merged and tested together, they will all be listed in
+will be sent for all builds, but are not required so you do not need
-``GERRIT_CHANGES``.  The format for the description of one change is::
+to configure Jenkins to accept them if you do not plan on using them:
 
-  project:branch:refspec
+**ZUUL_PROJECT**
+  The project that triggered this build
+**ZUUL_PIPELINE**
+  The Zuul pipeline that is building this job
 
-And multiple changes are separated by a carat ("^").  E.g.::
+The following parameters are optional and will only be provided for
+builds associated with changes (i.e., in response to patchset-created
+or comment-added events):
 
-  testproject:master:refs/changes/20/420/1^testproject:master:refs/changes/21/421/1"
+**ZUUL_BRANCH**
+  The target branch for the change that triggered this build
+**ZUUL_CHANGE**
+  The Gerrit change ID for the change that triggered this build
+**ZUUL_CHANGE_IDS**
+  All of the Gerrit change IDs that are included in this build (useful
+  when the DependentPipelineManager combines changes for testing)
+**ZUUL_PATCHSET**
+  The Gerrit patchset number for the change that triggered this build
 
-The OpenStack project uses the following script to update the
+The following parameters are optional and will only be provided for
-repository in a workspace and merge appropriate changes:
+post-merge (ref-updated) builds:
 
-  https://github.com/openstack/openstack-ci-puppet/blob/master/modules/jenkins/files/slave_scripts/gerrit-git-prep.sh
+**ZUUL_OLDREV**
+  The SHA1 of the old revision at this ref (recall the ref name is
+  in ZUUL_REF)
+**ZUUL_NEWREV**
+  The SHA1 of the new revision at this ref (recall the ref name is
+  in ZUUL_REF)
+**ZUUL_SHORT_OLDREV**
+  The shortened (7 character) SHA1 of the old revision
+**ZUUL_SHORT_NEWREV**
+  The shortened (7 character) SHA1 of the new revision
 
-Gerrit events that do not include a change (e.g., ref-updated events
+In order to test the correct build, configure the Jenkins Git SCM
-which are emitted after a git ref is updated (i.e., a commit is merged
+plugin as follows::
-to master)) require a slightly different set of parameters:
 
-**UUID**
+  Source Code Management:
-  Zuul provided key to link builds with Gerrit events
+    Git
-**GERRIT_PROJECT**
+      Repositories:
-  Zuul provided project name
+        Repository URL:  <your Gerrit or Zuul repository URL>
-**GERRIT_REFNAME**
+          Advanced:
-  Zuul provided ref name
+            Refspec: ${ZUUL_REF}
-**GERRIT_OLDREV**
+      Branches to build:
-  Zuul provided old reference for ref-updated
+        Branch Specifier: ${ZUUL_COMMIT}
-**GERRIT_NEWREV**
+	  Advanced:
-    Zuul provided new reference for ref-updated
+	    Clean after checkout: True
 
+That should be sufficient for a job that only builds a single project.
+If you have multiple interrelated projects (i.e., they share a Zuul
+Change Queue) that are built together, you may be able to configure
+the Git plugin to prepare them, or you may chose to use a shell script
+instead.  The OpenStack project uses the following script to prepare
+the workspace for its integration testing:
+
+  https://github.com/openstack-ci/devstack-gate/blob/master/devstack-vm-gate-wrap.sh

tests/test_scheduler.py

@@ -128,11 +128,15 @@ def job_has_changes(*args):
     path = os.path.join(GIT_ROOT, project)
     repo = git.Repo(path)
     ref = job.parameters['ZUUL_REF']
+    sha = job.parameters['ZUUL_COMMIT']
     repo_messages = [c.message.strip() for c in repo.iter_commits(ref)]
+    repo_shas = [c.hexsha for c in repo.iter_commits(ref)]
     commit_messages = ['%s-1' % commit.subject for commit in commits]
     for msg in commit_messages:
         if msg not in repo_messages:
             return False
+    if repo_shas[0] != sha:
+        return False
     return True
 
 

zuul/launcher/jenkins.py

@@ -219,21 +219,23 @@ class Jenkins(object):
         dependent_changes = dependent_changes[:]
         dependent_changes.reverse()
         uuid = str(uuid4().hex)
-        params = dict(UUID=uuid,
+        params = dict(UUID=uuid,  # deprecated
-                      GERRIT_PROJECT=change.project.name,
+                      ZUUL_UUID=uuid,
+                      GERRIT_PROJECT=change.project.name,  # deprecated
                       ZUUL_PROJECT=change.project.name)
         params['ZUUL_PIPELINE'] = pipeline.name
         if hasattr(change, 'refspec'):
             changes_str = '^'.join(
                 ['%s:%s:%s' % (c.project.name, c.branch, c.refspec)
                  for c in dependent_changes + [change]])
-            params['GERRIT_BRANCH'] = change.branch
+            params['GERRIT_BRANCH'] = change.branch  # deprecated
             params['ZUUL_BRANCH'] = change.branch
-            params['GERRIT_CHANGES'] = changes_str
+            params['GERRIT_CHANGES'] = changes_str   # deprecated
             params['ZUUL_CHANGES'] = changes_str
             params['ZUUL_REF'] = ('refs/zuul/%s/%s' %
                                   (change.branch,
                                    change.current_build_set.ref))
+            params['ZUUL_COMMIT'] = change.current_build_set.commit
 
             zuul_changes = ' '.join(['%s,%s' % (c.number, c.patchset)
                                      for c in dependent_changes + [change]])
@@ -241,14 +243,41 @@ class Jenkins(object):
             params['ZUUL_CHANGE'] = str(change.number)
             params['ZUUL_PATCHSET'] = str(change.patchset)
         if hasattr(change, 'ref'):
-            params['GERRIT_REFNAME'] = change.ref
+            params['GERRIT_REFNAME'] = change.ref   # deprecated
             params['ZUUL_REFNAME'] = change.ref
-            params['GERRIT_OLDREV'] = change.oldrev
+            params['GERRIT_OLDREV'] = change.oldrev   # deprecated
             params['ZUUL_OLDREV'] = change.oldrev
-            params['GERRIT_NEWREV'] = change.newrev
+            params['GERRIT_NEWREV'] = change.newrev   # deprecated
             params['ZUUL_NEWREV'] = change.newrev
+            params['ZUUL_SHORT_OLDREV'] = change.oldrev[:7]
             params['ZUUL_SHORT_NEWREV'] = change.newrev[:7]
 
+            params['ZUUL_REF'] = change.ref
+            params['ZUUL_COMMIT'] = change.newrev
+
+        # This is what we should be heading toward for parameters:
+
+        # required:
+        # ZUUL_UUID
+        # ZUUL_REF (/refs/zuul/..., /refs/tags/foo, master)
+        # ZUUL_COMMIT
+
+        # optional:
+        # ZUUL_PROJECT
+        # ZUUL_PIPELINE
+
+        # optional (changes only):
+        # ZUUL_BRANCH
+        # ZUUL_CHANGE
+        # ZUUL_CHANGE_IDS
+        # ZUUL_PATCHSET
+
+        # optional (ref updated only):
+        # ZUUL_OLDREV
+        # ZUUL_NEWREV
+        # ZUUL_SHORT_NEWREV
+        # ZUUL_SHORT_OLDREV
+
         if callable(job.parameter_function):
             job.parameter_function(change, params)
             self.log.debug("Custom parameter function used for job %s, "

zuul/merger.py

@@ -81,6 +81,7 @@ class Repo(object):
 
     def setZuulRef(self, ref, commit):
         self.repo.refs[ref].commit = commit
+        return self.repo.refs[ref].commit
 
     def push(self, local, remote):
         self.log.debug("Pushing %s:%s to %s " % (local, remote,
@@ -112,6 +113,7 @@ class Merger(object):
 
     def mergeChanges(self, changes, target_ref=None, mode=None):
         projects = {}
+        commit = None
         # Reset all repos involved in the change set
         for change in changes:
             branches = projects.get(change.project, [])
@@ -148,7 +150,11 @@ class Merger(object):
                     repo.merge(change.refspec)
                 elif mode == model.CHERRY_PICK:
                     repo.cherryPick(change.refspec)
-                repo.setZuulRef(change.branch + '/' + target_ref, 'HEAD')
+                # Keep track of the last commit, it's the commit that
+                # will be passed to jenkins because it's the commit
+                # for the triggering change
+                commit = repo.setZuulRef(change.branch + '/' + target_ref,
+                                         'HEAD').hexsha
             except:
                 self.log.info("Unable to merge %s" % change)
                 return False
@@ -169,4 +175,4 @@ class Merger(object):
                         self.log.error("Ref %s did not show up in repo" % ref)
                         return False
 
-        return True
+        return commit

zuul/model.py

@@ -402,6 +402,7 @@ class BuildSet(object):
         self.next_build_set = None
         self.previous_build_set = None
         self.ref = None
+        self.commit = None
         self.unable_to_merge = False
 
     def setConfiguration(self):
@@ -416,9 +417,6 @@ class BuildSet(object):
         if not self.ref:
             self.ref = 'Z' + uuid4().hex
 
-    def getRef(self):
-        return self.ref
-
     def addBuild(self, build):
         self.builds[build.job.name] = build
         build.build_set = self

zuul/scheduler.py

@@ -531,18 +531,18 @@ class BasePipelineManager(object):
 
     def _launchJobs(self, change, jobs):
         self.log.debug("Launching jobs for change %s" % change)
-        ref = change.current_build_set.getRef()
+        ref = change.current_build_set.ref
         if hasattr(change, 'refspec') and not ref:
             change.current_build_set.setConfiguration()
-            ref = change.current_build_set.getRef()
+            ref = change.current_build_set.ref
             mode = model.MERGE_IF_NECESSARY
-            merged = self.sched.merger.mergeChanges([change], ref, mode=mode)
+            commit = self.sched.merger.mergeChanges([change], ref, mode=mode)
-            if not merged:
+            if not commit:
                 self.log.info("Unable to merge change %s" % change)
                 self.pipeline.setUnableToMerge(change)
                 self.possiblyReportChange(change)
                 return
-
+            change.current_build_set.commit = commit
         for job in self.pipeline.findJobsToRun(change):
             self.log.debug("Found job %s for change %s" % (job, change))
             try:
@@ -956,19 +956,20 @@ class DependentPipelineManager(BasePipelineManager):
 
     def _launchJobs(self, change, jobs):
         self.log.debug("Launching jobs for change %s" % change)
-        ref = change.current_build_set.getRef()
+        ref = change.current_build_set.ref
         if hasattr(change, 'refspec') and not ref:
             change.current_build_set.setConfiguration()
-            ref = change.current_build_set.getRef()
+            ref = change.current_build_set.ref
             dependent_changes = self._getDependentChanges(change)
             dependent_changes.reverse()
             all_changes = dependent_changes + [change]
-            merged = self.sched.merger.mergeChanges(all_changes, ref)
+            commit = self.sched.merger.mergeChanges(all_changes, ref)
-            if not merged:
+            if not commit:
                 self.log.info("Unable to merge changes %s" % all_changes)
                 self.pipeline.setUnableToMerge(change)
                 self.possiblyReportChange(change)
                 return
+            change.current_build_set.commit = commit
         #TODO: remove this line after GERRIT_CHANGES is gone
         dependent_changes = self._getDependentChanges(change)
         for job in jobs: