zuul
/
zuul
Code Issues Proposed changes

Revert "Enable maintainConnectionCache" 

This reverts commit 517fd7741d.

2017-09-08 00:12:07,550 ERROR zuul.Scheduler: Exception in management event:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/zuul/scheduler.py", line 783, in process_management_queue
    self._doTenantReconfigureEvent(event)
  File "/usr/local/lib/python3.5/dist-packages/zuul/scheduler.py", line 485, in _doTenantReconfigureEvent
    self._reconfigureTenant(tenant)
  File "/usr/local/lib/python3.5/dist-packages/zuul/scheduler.py", line 588, in _reconfigureTenant
    self.maintainConnectionCache()
  File "/usr/local/lib/python3.5/dist-packages/zuul/scheduler.py", line 731, in maintainConnectionCache
    connection.maintainCache(relevant)
  File "/usr/local/lib/python3.5/dist-packages/zuul/driver/github/githubconnection.py", line 574, in maintainCache
    for key, change in self._change_cache.items():

This causes tenant reconfiguration to abort halfway through which can
cause changes to get stuck in a gate pipeline.

Specifically to reproduce this error, we need two changes in gate, the first
is a config (zuul.yaml) change, and some pull requests in the github cache.

When the first change lands, zuul will perform a reconfiguration and fail.
Then when the second change completes jobs, it will report and claim to be
removed from the queue, but it will still actually be there.

Change-Id: I32e4a4f864ad16be391f57793f1a3abecdd6eee8
changes/21/502121/1
James E. Blair 2017-09-08 09:34:37 -07:00
parent 18d3169a36
commit 25796c229c
1 changed files with 4 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
zuul/scheduler.py
View File

@ -585,7 +585,8 @@ class Scheduler(threading.Thread):
self._reenqueueTenant(old_tenant, tenant)
self.maintainConnectionCache()
# TODOv3(jeblair): update for tenants
# self.maintainConnectionCache()
self.connections.reconfigureDrivers(tenant)
# TODOv3(jeblair): remove postconfig calls?
@ -718,6 +719,7 @@ class Scheduler(threading.Thread):
self.run_handler_lock.release()
def maintainConnectionCache(self):
# TODOv3(jeblair): update for tenants
relevant = set()
for tenant in self.abide.tenants.values():
for pipeline in tenant.layout.pipelines.values():
@ -727,7 +729,7 @@ class Scheduler(threading.Thread):
for item in pipeline.getAllItems():
relevant.add(item.change)
relevant.update(item.change.getRelatedChanges())
for connection in self.connections.connections.values():
for connection in self.connections.values():
connection.maintainCache(relevant)
self.log.debug(
"End maintain connection cache for: %s" % connection)