zuul
/
zuul
Code Issues Proposed changes

Delay getting many files to mergers 

In the past GitHub only returned at max 300 changed files of a
PR. Therefore zuul has a mechanism to delay getting the file list
using the mergers. Latest GitHub Enterprise seems to have removed this
limitation completely. This leads to large delays in the event
processing if there are incoming events for PRs with many files
(e.g. requesting files for a PR with 30000 files means requesting 300
pages of changed files and leads to a roughly 5 minute delay).

In order to prevent zuul limit this to at max 10 pages of changed
files and if there are more delay to the mergers.

Change-Id: I910ec545c4375103a5ffe5da106c7f5648aa5c72
This commit is contained in:
Tobias Henkel 2020-10-08 16:43:55 +02:00
parent 92a552ec67
commit 904d20c576
No known key found for this signature in database
GPG Key ID: 03750DEC158E5FA2
1 changed files with 11 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
zuul/driver/github/githubconnection.py
View File

@ -1649,7 +1649,17 @@ class GithubConnection(BaseConnection):
number, owner, proj))
pr = probj.as_dict()
try:
pr['files'] = [f.filename for f in probj.files()]
if pr.get('changed_files', 0) > 999:
# Don't request more than ten pages. If we exceed this we
# need to determine the files via the mergers asynchronously
# in order to not block the event processing by iterating on
# too many pages.
self.log.warning('Pull request #%s of %s/%s has too many '
'files. Files will be requested '
'asynchronously', number, owner, proj)
pr['files'] = None
else:
pr['files'] = [f.filename for f in probj.files()]
except github3.exceptions.ServerError as exc:
# NOTE: For PRs with a lot of lines changed, Github will return
# an error (HTTP 500) because it can't generate the diff.