zuul/zuul-jobs
Code Issues Proposed changes

Retry failed git pushses on workspace setup

Browse Source 
We've seen network/ssh failures to push one of the many git repos into
the workspace. Followup pushes to other repos succeed. Considering that
subsequent pushes do succeed allow for 3 retries to get these workspaces
set up.

Logs below show this happening. The first push of the repo under test
fails (openstack/tripleo-heat-templates) then the push for openstack
requirements succeeds.

  LOOP [mirror-workspace-git-repos : Synchronize src repos to workspace directory]
  Timeout exception waiting for the logger. Please check connectivity to [158.69.64.232:19885]
  ubuntu-xenial -> localhost | ssh: connect to host 158.69.64.232 port 22: No route to host
  ubuntu-xenial -> localhost | fatal: Could not read from remote repository.
  ubuntu-xenial -> localhost |
  ubuntu-xenial -> localhost | Please make sure you have the correct access rights
  ubuntu-xenial -> localhost | and the repository exists.
  ubuntu-xenial -> localhost | ssh: connect to host 158.69.64.232 port 22: No route to host
  ubuntu-xenial -> localhost | fatal: Could not read from remote repository.
  ubuntu-xenial -> localhost |
  ubuntu-xenial -> localhost | Please make sure you have the correct access rights
  ubuntu-xenial -> localhost | and the repository exists.
  ubuntu-xenial -> localhost | To git+ssh://zuul@158.69.64.232//home/zuul/src/git.openstack.org/openstack/requirements
  ubuntu-xenial -> localhost |    66b2988..740ae1a  master -> master
  ubuntu-xenial -> localhost |  * [new branch]      stable/newton -> stable/newton
  ubuntu-xenial -> localhost |  * [new branch]      stable/ocata -> stable/ocata
  ubuntu-xenial -> localhost |  * [new branch]      stable/pike -> stable/pike
  ubuntu-xenial -> localhost |  * [new branch]      stable/queens -> stable/queens
  ubuntu-xenial -> localhost |  * [new branch]      stable/rocky -> stable/rocky
  ubuntu-xenial -> localhost |  * [new branch]      origin/HEAD -> origin/HEAD
  ubuntu-xenial -> localhost |  * [new branch]      origin/master -> origin/master
  ubuntu-xenial -> localhost |  * [new branch]      origin/stable/newton -> origin/stable/newton
  ubuntu-xenial -> localhost |  * [new branch]      origin/stable/ocata -> origin/stable/ocata
  ubuntu-xenial -> localhost |  * [new branch]      origin/stable/pike -> origin/stable/pike
  ubuntu-xenial -> localhost |  * [new branch]      origin/stable/queens -> origin/stable/queens
  ubuntu-xenial -> localhost |  * [new branch]      origin/stable/rocky -> origin/stable/rocky
  ubuntu-xenial -> localhost | ok: Item: Runtime: 0:00:01.828724
  ubuntu-xenial -> localhost | changed: All items complete

  ubuntu-xenial -> localhost | ssh: connect to host 158.69.64.232 port 22: No route to host
  ubuntu-xenial -> localhost | fatal: Could not read from remote repository.
  ubuntu-xenial -> localhost |
  ubuntu-xenial -> localhost | Please make sure you have the correct access rights
  ubuntu-xenial -> localhost | and the repository exists.

  http://logs.openstack.org/56/605356/17/check/openstack-tox-linters/4323abf/ara-report/result/bb14d3fe-338b-4d7a-852f-e66c60d13493/

Related-Bug: 1384373
Change-Id: I6c34d1a6e34762f78b9dd2a2d2a07ceb131a64cc
This commit is contained in:
Clark Boylan 2018-10-05 08:52:51 -07:00
parent 43868ae828
commit b5ceee2004
1 changed files with 7 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
roles/mirror-workspace-git-repos/tasks/main.yaml
View File

@ -12,6 +12,13 @@
chdir: "{{ zuul.executor.work_root }}/{{ item.value.src_dir }}"
with_dict: "{{ zuul.projects }}"
delegate_to: localhost
# We occasionally see git pushes in the middle of this loop fail then
# subsequent pushes for other repos succeed. The entire loop ends up
# failing because one of the pushes failed. Mitigate this by retrying
# on failure.
register: git_push
until: git_push is success
retries: 3
# ANSIBLE0006: Skip linting since it triggers on the "git" command,
# but push is not supported by ansible git module.
tags: