If a new branch is created off of an existing commit then the rev
(commit) for the new branch is likely already in the repos on the
executors. Unfortunately we were checking if a repo needs to be updated
based only on the presence of the commit and not whether or not the ref
(branch) exists.
This means in some cases zuul would fall back to the default branch for
the repo instead of testing changes proposed to new branches because it
thought that the local executor git repo did not need to be updated. Not
updating the local git repo results in the branch not being created
which fails branch validity checks and we fall back to the default.
You can see that happening in this log snippet:
2019-08-26 16:35:40,747 DEBUG zuul.ExecutorServer: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] Got executor:execute job: b405f247cf3a441b924bce4c35b2f635
2019-08-26 16:35:41,255 INFO zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Beginning job openstack-tox-py36 for ref refs/changes/58/677258/5 (change https://review.opendev.org/677258)
2019-08-26 16:35:41,255 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Job root: /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635
2019-08-26 16:35:41,256 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Updating project {'connection': 'gerrit', 'canonical_name': 'opendev.org/x/ranger-agent', 'override_branch': None, 'default_branch': 'master', 'name': 'x/ranger-agent', 'override_checkout': None}
2019-08-26 16:35:41,258 INFO zuul.ExecutorServer: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Updating repo gerrit/x/ranger-agent
2019-08-26 16:35:41,406 INFO zuul.Merger: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Skipping updating local repository gerrit/x/ranger-agent
2019-08-26 16:35:41,419 DEBUG zuul.ExecutorServer: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Finished updating repo gerrit/x/ranger-agent
2019-08-26 16:35:41,998 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Git updates complete
2019-08-26 16:35:42,960 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Cloning gerrit/x/ranger-agent
2019-08-26 16:35:43,032 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Cloning from /var/lib/zuul/executor-git/opendev.org/x/ranger-agent to /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,202 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Merging for change 677258,5
2019-08-26 16:35:43,202 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Processing ref refs/changes/58/677258/5 for project gerrit/x/ranger-agent / python3 uuid 2ebedc56e15d47118f60e2e248b19d62
2019-08-26 16:35:43,203 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] No base commit found for ('gerrit', 'x/ranger-agent', 'python3')
2019-08-26 16:35:43,203 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Resetting repository /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,207 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Updating repository /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,570 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Reset to kilo
2019-08-26 16:35:43,573 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Restore repo state for project gerrit/x/ranger-agent
2019-08-26 16:35:43,594 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/python3 at e796eb53956f1e83ddc53a8dce9cf3957b3b59ef in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,617 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/master at e796eb53956f1e83ddc53a8dce9cf3957b3b59ef in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,619 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/ocata at e796eb53956f1e83ddc53a8dce9cf3957b3b59ef in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,620 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/newton at 9e01ee002f36ecae2a21ded4e9e14de00c78de6c in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,621 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/kilo at 6ef65d8d15b730129ff86c83d6103d97f65193e9 in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:43,624 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Delete reference refs/remotes/origin/HEAD
2019-08-26 16:35:43,697 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Updating remote reference origin/python3 to e796eb53956f1e83ddc53a8dce9cf3957b3b59ef
2019-08-26 16:35:43,702 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Checking out e796eb53956f1e83ddc53a8dce9cf3957b3b59ef
2019-08-26 16:35:43,974 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Merging refs/changes/58/677258/5 with args ['-s', 'resolve', 'FETCH_HEAD']
2019-08-26 16:35:44,099 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/master at e796eb53956f1e83ddc53a8dce9cf3957b3b59ef in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:44,258 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/ocata at e796eb53956f1e83ddc53a8dce9cf3957b3b59ef in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:44,290 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/newton at 9e01ee002f36ecae2a21ded4e9e14de00c78de6c in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:44,323 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/python3 at e5ef4f738e6d916103c178aedb65e6aa7d2e9dfd in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:44,348 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Create reference refs/heads/kilo at 6ef65d8d15b730129ff86c83d6103d97f65193e9 in /var/lib/zuul/builds/b405f247cf3a441b924bce4c35b2f635/work/src/opendev.org/x/ranger-agent
2019-08-26 16:35:44,511 INFO zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Checking out opendev.org/x/ranger-agent project default branch master
2019-08-26 16:35:44,515 DEBUG zuul.AnsibleJob: [e: 9e01fe2c653a4c7ea9d121e41c2ed5f7] [build: b405f247cf3a441b924bce4c35b2f635] Checking out master
Change-Id: I1858a2ce54312b16524db915d093ccef2b0f892e
061c10505b