zuul/zuul
Code Issues Proposed changes

Add support for the Gerrit checks plugin

Browse Source 
This adds initial support for the Gerrit checks plugin.
Development of that plugin is still in progress, and hopefully it
(and our support for it) will change over time.  Because we expect
to change how we interact with it in the near future, this is
documented as experimental support for now.  A release note is
intentionally omitted -- that's more appropriate when we remove
the 'experimental' label.

Change-Id: Ida0cdef682ca2ce117617eacfb67f371426a3131
This commit is contained in:
James E. Blair
2019-09-06 14:50:05 -07:00
parent 48aa3ebd98
commit e78e948284
11 changed files with 853 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

150
doc/source/admin/drivers/gerrit.rst
View File

@@ -334,10 +334,158 @@ order to be enqueued into the pipeline.
approval:
- Code-Review: [-1, -2]
Reference pipelines configuration
Reference Pipelines Configuration
---------------------------------
Here is an example of standard pipelines you may want to define:
.. literalinclude:: ../examples/zuul-config/zuul.d/gerrit-reference-pipelines.yaml
:language: yaml
Checks Plugin Support (Experimental)
------------------------------------
The Gerrit driver has experimental support for Gerrit's `checks`
plugin. Neither the `checks` plugin itself nor Zuul's support for it
are stable yet, and this is not recommended for production use. If
you wish to help develop this support, you should expect to be able to
upgrade both Zuul and Gerrit frequently as the two systems evolve. No
backward-compatible support will be provided and configurations may
need to be updated frequently.
Caveats include (but are not limited to):
* This documentation is brief.
* Access control for the `checks` API in Gerrit depends on a single
global administrative permission, ``administrateCheckers``. This is
required in order to use the `checks` API and can not be restricted
by project. This means that any system using the `checks` API can
interfere with any other.
* Checkers are restricted to a single project. This means that a
system with many projects will require many checkers to be defined
in Gerrit -- one for each project+pipeline.
* No support is provided for attaching checks to tags or commits,
meaning that tag, release, and post pipelines are unable to be used
with the `checks` API and must rely on `stream-events`.
* Sub-checks are not implemented yet, so in order to see the results
of individual jobs on a change, users must either follow the
buildset link, or the pipeline must be configured to leave a
traditional comment.
* Familiarity with the `checks` API is recommended.
* Checkers may not be permanently deleted from Gerrit (only
"soft-deleted" so they no longer apply), so any experiments you
perform on a production system will leave data there forever.
In order to use the `checks` API, you must have HTTP access configured
in `zuul.conf`.
There are two ways to configure a pipeline for the `checks` API:
directly referencing the checker UUID, or referencing it's scheme. It
is hoped that once multi-repository checks are supported, that an
administrator will be able to configure a single checker in Gerrit for
each Zuul pipeline, and those checkers can apply to all repositories.
If and when that happens, we will be able to reference the checker
UUID directly in Zuul's pipeline configuration. If you only have a
single project, you may find this approach acceptable now.
To use this approach, create a checker named ``zuul:check`` and
configure a pipeline like this:
.. code-block:: yaml
- pipeline:
name: check
manager: independent
trigger:
gerrit:
- event: pending-check
uuid: 'zuul:check'
enqueue:
gerrit:
checks_api:
uuid: 'zuul:check'
state: SCHEDULED
message: 'Change has been enqueued in check'
start:
gerrit:
checks_api:
uuid: 'zuul:check'
state: RUNNING
message: 'Jobs have started running'
no-jobs:
gerrit:
checks_api:
uuid: 'zuul:check'
state: NOT_RELEVANT
message: 'Change has no jobs configured'
success:
gerrit:
checks_api:
uuid: 'zuul:check'
state: SUCCESSFUL
message: 'Change passed all voting jobs'
failure:
gerrit:
checks_api:
uuid: 'zuul:check'
state: FAILED
message: 'Change failed'
For a system with multiple repositories and one or more checkers for
each repository, the `scheme` approach is recommended. To use this,
create a checker for each pipeline in each repository. Give them
names such as ``zuul_check:project1``, ``zuul_gate:project1``,
``zuul_check:project2``, etc. The part before the ``:`` is the
`scheme`. Then create a pipeline like this:
.. code-block:: yaml
- pipeline:
name: check
manager: independent
trigger:
gerrit:
- event: pending-check
scheme: 'zuul_check'
enqueue:
gerrit:
checks_api:
scheme: 'zuul_check'
state: SCHEDULED
message: 'Change has been enqueued in check'
start:
gerrit:
checks_api:
scheme: 'zuul_check'
state: RUNNING
message: 'Jobs have started running'
no-jobs:
gerrit:
checks_api:
scheme: 'zuul_check'
state: NOT_RELEVANT
message: 'Change has no jobs configured'
success:
gerrit:
checks_api:
scheme: 'zuul_check'
state: SUCCESSFUL
message: 'Change passed all voting jobs'
failure:
gerrit:
checks_api:
scheme: 'zuul_check'
state: FAILED
message: 'Change failed'
This will match and report to the appropriate checker for a given
repository based on the scheme you provided.
.. The original design doc may be of use during development:
https://gerrit-review.googlesource.com/c/gerrit/+/214733

140
tests/base.py
View File

@@ -15,6 +15,7 @@
import configparser
from contextlib import contextmanager
import copy
import datetime
import errno
import gc
@@ -191,6 +192,8 @@ class FakeGerritChange(object):
self.fail_merge = False
self.messages = []
self.comments = []
self.checks = {}
self.checks_history = []
self.data = {
'branch': branch,
'comments': self.comments,
@@ -286,6 +289,30 @@ class FakeGerritChange(object):
self.patchsets.append(d)
self.data['submitRecords'] = self.getSubmitRecords()
def setCheck(self, checker, reset=False, **kw):
if reset:
self.checks[checker] = {'state': 'NOT_STARTED',
'created': str(datetime.datetime.now())}
chk = self.checks[checker]
chk['updated'] = str(datetime.datetime.now())
for (key, default) in [
('state', None),
('repository', self.project),
('change_number', self.number),
('patch_set_id', self.latest_patchset),
('checker_uuid', checker),
('message', None),
('url', None),
('started', None),
('finished', None),
]:
val = kw.get(key, chk.get(key, default))
if val is not None:
chk[key] = val
elif key in chk:
del chk[key]
self.checks_history.append(copy.deepcopy(self.checks))
def addComment(self, filename, line, message, name, email, username,
comment_range=None):
comment = {
@@ -521,6 +548,12 @@ class GerritWebServer(object):
log = logging.getLogger("zuul.test.FakeGerritConnection")
review_re = re.compile('/a/changes/(.*?)/revisions/(.*?)/review')
submit_re = re.compile('/a/changes/(.*?)/submit')
pending_checks_re = re.compile(
r'/a/plugins/checks/checks\.pending/\?'
r'query=checker:(.*?)\+\(state:(.*?)\)')
update_checks_re = re.compile(
r'/a/changes/(.*)/revisions/(.*?)/checks/(.*)')
list_checkers_re = re.compile('/a/plugins/checks/checkers/')
def do_POST(self):
path = self.path
@@ -536,6 +569,23 @@ class GerritWebServer(object):
m = self.submit_re.match(path)
if m:
return self.submit(m.group(1), data)
m = self.update_checks_re.match(path)
if m:
return self.update_checks(
m.group(1), m.group(2), m.group(3), data)
self.send_response(500)
self.end_headers()
def do_GET(self):
path = self.path
self.log.debug("Got GET %s", path)
m = self.pending_checks_re.match(path)
if m:
return self.get_pending_checks(m.group(1), m.group(2))
m = self.list_checkers_re.match(path)
if m:
return self.list_checkers()
self.send_response(500)
self.end_headers()
@@ -555,7 +605,7 @@ class GerritWebServer(object):
return self._404()
message = data['message']
action = data['labels']
action = data.get('labels', {})
comments = data.get('comments', {})
fake_gerrit._test_handle_review(
int(change.data['number']), message, action, comments)
@@ -574,6 +624,54 @@ class GerritWebServer(object):
self.send_response(200)
self.end_headers()
def update_checks(self, change_id, revision, checker, data):
self.log.debug("Update checks %s %s %s",
change_id, revision, checker)
change = self._get_change(change_id)
if not change:
return self._404()
change.setCheck(checker, **data)
self.send_response(200)
# TODO: return the real data structure, but zuul
# ignores this now.
self.end_headers()
def get_pending_checks(self, checker, state):
self.log.debug("Get pending checks %s %s", checker, state)
ret = []
for c in fake_gerrit.changes.values():
if checker not in c.checks:
continue
patchset_pending_checks = {}
if c.checks[checker]['state'] == state:
patchset_pending_checks[checker] = {
'state': c.checks[checker]['state'],
}
if patchset_pending_checks:
ret.append({
'patch_set': {
'repository': c.project,
'change_number': c.number,
'patch_set_id': c.latest_patchset,
},
'pending_checks': patchset_pending_checks,
})
self.send_data(ret)
def list_checkers(self):
self.log.debug("Get checkers")
self.send_data(fake_gerrit.fake_checkers)
def send_data(self, data):
data = json.dumps(data).encode('utf-8')
data = b")]}'\n" + data
self.send_response(200)
self.send_header('Content-Type', 'application/json')
self.send_header('Content-Length', len(data))
self.end_headers()
self.wfile.write(data)
def log_message(self, fmt, *args):
self.log.debug(fmt, *args)
@@ -589,6 +687,23 @@ class GerritWebServer(object):
self.thread.join()
class FakeGerritPoller(gerritconnection.GerritPoller):
"""A Fake Gerrit poller for use in tests.
This subclasses
:py:class:`~zuul.connection.gerrit.GerritPoller`.
"""
poll_interval = 1
def _run(self, *args, **kw):
r = super(FakeGerritPoller, self)._run(*args, **kw)
# Set the event so tests can confirm that the poller has run
# after they changed something.
self.connection._poller_event.set()
return r
class FakeGerritConnection(gerritconnection.GerritConnection):
"""A Fake Gerrit connection for use in tests.
@@ -598,9 +713,10 @@ class FakeGerritConnection(gerritconnection.GerritConnection):
"""
log = logging.getLogger("zuul.test.FakeGerritConnection")
_poller_class = FakeGerritPoller
def __init__(self, driver, connection_name, connection_config,
changes_db=None, upstream_root=None):
changes_db=None, upstream_root=None, poller_event=None):
if connection_config.get('password'):
self.web_server = GerritWebServer(self)
@@ -619,6 +735,11 @@ class FakeGerritConnection(gerritconnection.GerritConnection):
self.changes = changes_db
self.queries = []
self.upstream_root = upstream_root
self.fake_checkers = []
self._poller_event = poller_event
def addFakeChecker(self, **kw):
self.fake_checkers.append(kw)
def addFakeChange(self, project, branch, subject, status='NEW',
files=None, parent=None):
@@ -690,12 +811,12 @@ class FakeGerritConnection(gerritconnection.GerritConnection):
}
return event
def review(self, change, message, action={}, file_comments={},
def review(self, item, message, action={}, file_comments={},
zuul_event_id=None):
if self.web_server:
return super(FakeGerritConnection, self).review(
change, message, action, file_comments)
self._test_handle_review(int(change.number), message, action)
item, message, action, file_comments)
self._test_handle_review(int(item.change.number), message, action)
def _test_handle_review(self, change_number, message, action,
file_comments=None):
@@ -2929,6 +3050,7 @@ class ZuulTestCase(BaseTestCase):
self.sched.trigger_event_queue,
self.sched.management_event_queue
]
self.poller_events = {}
self.configure_connections()
self.sched.registerConnections(self.connections)
@@ -2992,9 +3114,11 @@ class ZuulTestCase(BaseTestCase):
def getGerritConnection(driver, name, config):
db = self.gerrit_changes_dbs.setdefault(config['server'], {})
event = self.poller_events.setdefault(name, threading.Event())
con = FakeGerritConnection(driver, name, config,
changes_db=db,
upstream_root=self.upstream_root)
upstream_root=self.upstream_root,
poller_event=event)
if con.web_server:
self.addCleanup(con.web_server.stop)
@@ -3628,6 +3752,10 @@ class ZuulTestCase(BaseTestCase):
self.executor_server.lock.release()
self.sched.wake_event.wait(0.1)
def waitForPoll(self, poller, timeout=30):
self.poller_events[poller].clear()
self.poller_events[poller].wait(30)
def logState(self):
""" Log the current state of the system """
self.log.info("Begin state dump --------------------")

94
tests/fixtures/layouts/gerrit-checks-nojobs.yaml vendored Normal file
View File

@@ -0,0 +1,94 @@
- pipeline:
name: check
manager: independent
trigger:
gerrit:
- event: pending-check
uuid: 'zuul:check'
enqueue:
gerrit:
checks_api:
uuid: 'zuul:check'
state: SCHEDULED
message: 'Change has been enqueued in check'
start:
gerrit:
checks_api:
uuid: 'zuul:check'
state: RUNNING
message: 'Jobs have started running'
no-jobs:
gerrit:
checks_api:
uuid: 'zuul:check'
state: NOT_RELEVANT
message: 'Change has no jobs configured'
success:
gerrit:
checks_api:
uuid: 'zuul:check'
state: SUCCESSFUL
message: 'Change passed all voting jobs'
failure:
gerrit:
checks_api:
uuid: 'zuul:check'
state: FAILED
message: 'Change failed'
- pipeline:
name: gate
manager: dependent
trigger:
gerrit:
- event: pending-check
uuid: 'zuul:gate'
enqueue:
gerrit:
checks_api:
uuid: 'zuul:gate'
state: SCHEDULED
message: 'Change has been enqueued in gate'
start:
gerrit:
Verified: 0
checks_api:
uuid: 'zuul:gate'
state: RUNNING
message: 'Jobs have started running'
no-jobs:
gerrit:
checks_api:
uuid: 'zuul:gate'
state: NOT_RELEVANT
message: 'Change has no jobs configured'
success:
gerrit:
Verified: 2
submit: true
checks_api:
uuid: 'zuul:gate'
state: SUCCESSFUL
message: 'Change passed all voting jobs'
failure:
gerrit:
Verified: -2
checks_api:
uuid: 'zuul:gate'
state: FAILED
message: 'Change failed'
- job:
name: test-job
parent: null
run: test-job.yaml
branches: otherbranch
- project:
name: org/project
check:
jobs:
- test-job
gate:
jobs:
- test-job

48
tests/fixtures/layouts/gerrit-checks-scheme.yaml vendored Normal file
View File

@@ -0,0 +1,48 @@
- pipeline:
name: check
manager: independent
trigger:
gerrit:
- event: pending-check
scheme: 'zuul_check'
enqueue:
gerrit:
checks_api:
scheme: 'zuul_check'
state: SCHEDULED
message: 'Change has been enqueued in check'
start:
gerrit:
checks_api:
scheme: 'zuul_check'
state: RUNNING
message: 'Jobs have started running'
no-jobs:
gerrit:
checks_api:
scheme: 'zuul_check'
state: NOT_RELEVANT
message: 'Change has no jobs configured'
success:
gerrit:
checks_api:
scheme: 'zuul_check'
state: SUCCESSFUL
message: 'Change passed all voting jobs'
failure:
gerrit:
checks_api:
scheme: 'zuul_check'
state: FAILED
message: 'Change failed'
- job:
name: test-job
parent: null
run: test-job.yaml
- project:
name: org/project
check:
jobs:
- test-job

93
tests/fixtures/layouts/gerrit-checks.yaml vendored Normal file
View File

@@ -0,0 +1,93 @@
- pipeline:
name: check
manager: independent
trigger:
gerrit:
- event: pending-check
uuid: 'zuul:check'
enqueue:
gerrit:
checks_api:
uuid: 'zuul:check'
state: SCHEDULED
message: 'Change has been enqueued in check'
start:
gerrit:
checks_api:
uuid: 'zuul:check'
state: RUNNING
message: 'Jobs have started running'
no-jobs:
gerrit:
checks_api:
uuid: 'zuul:check'
state: NOT_RELEVANT
message: 'Change has no jobs configured'
success:
gerrit:
checks_api:
uuid: 'zuul:check'
state: SUCCESSFUL
message: 'Change passed all voting jobs'
failure:
gerrit:
checks_api:
uuid: 'zuul:check'
state: FAILED
message: 'Change failed'
- pipeline:
name: gate
manager: dependent
trigger:
gerrit:
- event: pending-check
uuid: 'zuul:gate'
enqueue:
gerrit:
checks_api:
uuid: 'zuul:gate'
state: SCHEDULED
message: 'Change has been enqueued in gate'
start:
gerrit:
Verified: 0
checks_api:
uuid: 'zuul:gate'
state: RUNNING
message: 'Jobs have started running'
no-jobs:
gerrit:
checks_api:
uuid: 'zuul:gate'
state: NOT_RELEVANT
message: 'Change has no jobs configured'
success:
gerrit:
Verified: 2
submit: true
checks_api:
uuid: 'zuul:gate'
state: SUCCESSFUL
message: 'Change passed all voting jobs'
failure:
gerrit:
Verified: -2
checks_api:
uuid: 'zuul:gate'
state: FAILED
message: 'Change failed'
- job:
name: test-job
parent: null
run: test-job.yaml
- project:
name: org/project
check:
jobs:
- test-job
gate:
jobs:
- test-job

99
tests/unit/test_gerrit.py
View File

@@ -17,7 +17,9 @@ import textwrap
from unittest import mock
import tests.base
from tests.base import BaseTestCase, ZuulTestCase, AnsibleZuulTestCase
from tests.base import (
BaseTestCase, ZuulTestCase, AnsibleZuulTestCase,
simple_layout)
from zuul.driver.gerrit import GerritDriver
from zuul.driver.gerrit.gerritconnection import GerritConnection
@@ -236,3 +238,98 @@ class TestFileComments(AnsibleZuulTestCase):
"A should have a validation error reported")
self.assertIn('invalid file missingfile.txt', A.messages[0],
"A should have file error reported")
class TestChecksApi(ZuulTestCase):
config_file = 'zuul-gerrit-web.conf'
@simple_layout('layouts/gerrit-checks.yaml')
def test_check_pipeline(self):
A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
A.setCheck('zuul:check', reset=True)
self.waitForPoll('gerrit')
self.waitUntilSettled()
self.assertEqual(A.checks_history[0]['zuul:check']['state'],
'NOT_STARTED')
self.assertEqual(A.checks_history[1]['zuul:check']['state'],
'SCHEDULED')
self.assertEqual(A.checks_history[2]['zuul:check']['state'],
'RUNNING')
self.assertEqual(A.checks_history[3]['zuul:check']['state'],
'SUCCESSFUL')
self.assertEqual(len(A.checks_history), 4)
self.assertTrue(isinstance(
A.checks_history[3]['zuul:check']['started'], str))
self.assertTrue(isinstance(
A.checks_history[3]['zuul:check']['finished'], str))
self.assertTrue(
A.checks_history[3]['zuul:check']['finished'] >
A.checks_history[3]['zuul:check']['started'])
self.assertEqual(A.checks_history[3]['zuul:check']['message'],
'Change passed all voting jobs')
self.assertHistory([
dict(name='test-job', result='SUCCESS', changes='1,1')])
@simple_layout('layouts/gerrit-checks.yaml')
def test_gate_pipeline(self):
A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
A.addApproval('Code-Review', 2)
A.addApproval('Approved', 1)
A.setCheck('zuul:gate', reset=True)
self.waitForPoll('gerrit')
self.waitUntilSettled()
self.assertEqual(A.checks_history[0]['zuul:gate']['state'],
'NOT_STARTED')
self.assertEqual(A.checks_history[1]['zuul:gate']['state'],
'SCHEDULED')
self.assertEqual(A.checks_history[2]['zuul:gate']['state'],
'RUNNING')
self.assertEqual(A.checks_history[3]['zuul:gate']['state'],
'SUCCESSFUL')
self.assertEqual(len(A.checks_history), 4)
self.assertHistory([
dict(name='test-job', result='SUCCESS', changes='1,1')])
self.assertEqual(A.data['status'], 'MERGED')
@simple_layout('layouts/gerrit-checks-scheme.yaml')
def test_check_pipeline_scheme(self):
self.fake_gerrit.addFakeChecker(uuid='zuul_check:abcd',
repository='org/project',
status='ENABLED')
self.sched.reconfigure(self.config)
self.waitUntilSettled()
A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
A.setCheck('zuul_check:abcd', reset=True)
self.waitForPoll('gerrit')
self.waitUntilSettled()
self.assertEqual(A.checks_history[0]['zuul_check:abcd']['state'],
'NOT_STARTED')
self.assertEqual(A.checks_history[1]['zuul_check:abcd']['state'],
'SCHEDULED')
self.assertEqual(A.checks_history[2]['zuul_check:abcd']['state'],
'RUNNING')
self.assertEqual(A.checks_history[3]['zuul_check:abcd']['state'],
'SUCCESSFUL')
self.assertEqual(len(A.checks_history), 4)
self.assertHistory([
dict(name='test-job', result='SUCCESS', changes='1,1')])
@simple_layout('layouts/gerrit-checks-nojobs.yaml')
def test_no_jobs(self):
A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
A.setCheck('zuul:check', reset=True)
self.waitForPoll('gerrit')
self.waitUntilSettled()
self.assertEqual(A.checks_history[0]['zuul:check']['state'],
'NOT_STARTED')
self.assertEqual(A.checks_history[1]['zuul:check']['state'],
'SCHEDULED')
self.assertEqual(A.checks_history[2]['zuul:check']['state'],
'NOT_RELEVANT')
self.assertEqual(len(A.checks_history), 3)
self.assertEqual(A.data['status'], 'NEW')

19
zuul/driver/gerrit/__init__.py
View File

@@ -18,12 +18,31 @@ from zuul.driver.gerrit import gerritconnection
from zuul.driver.gerrit import gerrittrigger
from zuul.driver.gerrit import gerritsource
from zuul.driver.gerrit import gerritreporter
from zuul.driver.util import to_list
class GerritDriver(Driver, ConnectionInterface, TriggerInterface,
SourceInterface, ReporterInterface):
name = 'gerrit'
def reconfigure(self, tenant):
connection_checker_map = {}
for pipeline in tenant.layout.pipelines.values():
for trigger in pipeline.triggers:
if isinstance(trigger, gerrittrigger.GerritTrigger):
con = trigger.connection
checkers = connection_checker_map.setdefault(con, [])
for trigger_item in to_list(trigger.config):
if trigger_item['event'] == 'pending-check':
d = {}
if 'uuid' in trigger_item:
d['uuid'] = trigger_item['uuid']
elif 'scheme' in trigger_item:
d['scheme'] = trigger_item['scheme']
checkers.append(d)
for (con, checkers) in connection_checker_map.items():
con.setWatchedCheckers(checkers)
def getConnection(self, name, config):
return gerritconnection.GerritConnection(self, name, config)

213
zuul/driver/gerrit/gerritconnection.py
View File

@@ -13,6 +13,7 @@
# License for the specific language governing permissions and limitations
# under the License.
import datetime
import json
import re
import re2
@@ -80,6 +81,7 @@ class GerritEventConnector(threading.Thread):
log = get_annotated_logger(self.log, event)
event.type = data.get('type')
event.uuid = data.get('uuid')
# This catches when a change is merged, as it could potentially
# have merged layout info which will need to be read in.
# Ideally this would be done with a refupdate event so as to catch
@@ -131,6 +133,7 @@ class GerritEventConnector(threading.Thread):
'ref-replication-scheduled': None,
'topic-changed': 'changer',
'project-created': None, # Gerrit 2.14
'pending-check': None, # Gerrit 3.0+
}
event.account = None
if event.type in accountfield_from_type:
@@ -294,6 +297,58 @@ class GerritWatcher(threading.Thread):
self._stopped = True
class GerritPoller(threading.Thread):
# Poll gerrit without stream-events
log = logging.getLogger("gerrit.GerritPoller")
poll_interval = 30
def __init__(self, connection):
threading.Thread.__init__(self)
self.connection = connection
self._stopped = False
self._stop_event = threading.Event()
def _makeEvent(self, change, uuid, check):
return {'type': 'pending-check',
'uuid': uuid,
'change': {
'project': change['patch_set']['repository'],
'number': change['patch_set']['change_number'],
},
'patchSet': {
'number': change['patch_set']['patch_set_id'],
}}
def _run(self):
try:
for checker in self.connection.watched_checkers:
changes = self.connection.get(
'plugins/checks/checks.pending/?'
'query=checker:%s+(state:NOT_STARTED)' % checker)
for change in changes:
for uuid, check in change['pending_checks'].items():
event = self._makeEvent(change, uuid, check)
self.connection.addEvent(event)
except Exception:
self.log.exception("Exception on Gerrit poll with %s:",
self.connection.connection_name)
def run(self):
last_start = time.time()
while not self._stopped:
next_start = last_start + self.poll_interval
self._stop_event.wait(max(next_start - time.time(), 0))
if self._stopped:
break
last_start = time.time()
self._run()
def stop(self):
self.log.debug("Stopping watcher")
self._stopped = True
self._stop_event.set()
class GerritConnection(BaseConnection):
driver_name = 'gerrit'
log = logging.getLogger("zuul.GerritConnection")
@@ -305,6 +360,7 @@ class GerritConnection(BaseConnection):
r"@{|\.\.|\.$|^@$|/$|^/|//+") # everything else we can check with re2
replication_timeout = 300
replication_retry_interval = 5
_poller_class = GerritPoller
def __init__(self, driver, connection_name, connection_config):
super(GerritConnection, self).__init__(driver, connection_name,
@@ -325,8 +381,11 @@ class GerritConnection(BaseConnection):
self.keyfile = self.connection_config.get('sshkey', None)
self.keepalive = int(self.connection_config.get('keepalive', 60))
self.watcher_thread = None
self.poller_thread = None
self.event_queue = queue.Queue()
self.client = None
self.watched_checkers = []
self.project_checker_map = {}
self.baseurl = self.connection_config.get(
'baseurl', 'https://%s' % self.server).rstrip('/')
@@ -362,6 +421,42 @@ class GerritConnection(BaseConnection):
self.auth = authclass(
self.user, self.password)
def setWatchedCheckers(self, checkers_to_watch):
self.log.debug("Setting watched checkers to %s", checkers_to_watch)
self.watched_checkers = set()
self.project_checker_map = {}
schemes_to_watch = set()
uuids_to_watch = set()
for x in checkers_to_watch:
if 'scheme' in x:
schemes_to_watch.add(x['scheme'])
if 'uuid' in x:
uuids_to_watch.add(x['uuid'])
if schemes_to_watch:
# get a list of all configured checkers
try:
configured_checkers = self.get('plugins/checks/checkers/')
except Exception:
self.log.exception("Unable to get checkers")
configured_checkers = []
# filter it through scheme matches in checkers_to_watch
for checker in configured_checkers:
if checker['status'] != 'ENABLED':
continue
checker_scheme, checker_id = checker['uuid'].split(':')
repo = checker['repository']
repo = self.canonical_hostname + '/' + repo
# map scheme matches to project names
if checker_scheme in schemes_to_watch:
repo_checkers = self.project_checker_map.setdefault(
repo, set())
repo_checkers.add(checker['uuid'])
self.watched_checkers.add(checker['uuid'])
# add uuids from checkers_to_watch
for x in uuids_to_watch:
self.watched_checkers.add(x)
def toDict(self):
d = super().toDict()
d.update({
@@ -375,6 +470,28 @@ class GerritConnection(BaseConnection):
def url(self, path):
return self.baseurl + '/a/' + path
def get(self, path):
url = self.url(path)
self.log.debug('GET: %s' % (url,))
r = self.session.get(
url,
verify=self.verify_ssl,
auth=self.auth, timeout=TIMEOUT,
headers={'User-Agent': self.user_agent})
self.log.debug('Received: %s %s' % (r.status_code, r.text,))
if r.status_code != 200:
raise Exception("Received response %s" % (r.status_code,))
ret = None
if r.text and len(r.text) > 4:
try:
ret = json.loads(r.text[4:])
except Exception:
self.log.exception(
"Unable to parse result %s from post to %s" %
(r.text, url))
raise
return ret
def post(self, path, data):
url = self.url(path)
self.log.debug('POST: %s' % (url,))
@@ -836,17 +953,18 @@ class GerritConnection(BaseConnection):
def eventDone(self):
self.event_queue.task_done()
def review(self, change, message, action={},
def review(self, item, message, action={},
file_comments={}, zuul_event_id=None):
if self.session:
meth = self.review_http
else:
meth = self.review_ssh
return meth(change, message, action=action,
return meth(item, message, action=action,
file_comments=file_comments, zuul_event_id=zuul_event_id)
def review_ssh(self, change, message, action={},
def review_ssh(self, item, message, action={},
file_comments={}, zuul_event_id=None):
change = item.change
project = change.project.name
cmd = 'gerrit review --project %s' % project
if message:
@@ -861,14 +979,56 @@ class GerritConnection(BaseConnection):
out, err = self._ssh(cmd, zuul_event_id=zuul_event_id)
return err
def review_http(self, change, message, action={},
def report_checks(self, log, item, changeid, checkinfo):
change = item.change
checkinfo = checkinfo.copy()
uuid = checkinfo.pop('uuid', None)
scheme = checkinfo.pop('scheme', None)
if uuid is None:
uuids = self.project_checker_map.get(
change.project.canonical_name, set())
for u in uuids:
if u.split(':')[0] == scheme:
uuid = u
break
if uuid is None:
log.error("Unable to find matching checker for %s %s",
item, checkinfo)
return
def fmt(t):
return str(datetime.datetime.fromtimestamp(t))
if item.enqueue_time:
checkinfo['started'] = fmt(item.enqueue_time)
if item.report_time:
checkinfo['finished'] = fmt(item.report_time)
url = item.formatStatusUrl()
if url:
checkinfo['url'] = url
if checkinfo:
for x in range(1, 4):
try:
self.post('changes/%s/revisions/%s/checks/%s' %
(changeid, change.commit, uuid),
checkinfo)
break
except Exception:
log.exception("Error submitting check data to gerrit, "
"attempt %s", x)
time.sleep(x * 10)
def review_http(self, item, message, action={},
file_comments={}, zuul_event_id=None):
change = item.change
log = get_annotated_logger(self.log, zuul_event_id)
data = dict(message=message,
strict_labels=False)
submit = False
labels = {}
for key, val in action.items():
if key == 'checks_api':
continue
if val is True:
if key == 'submit':
submit = True
@@ -883,21 +1043,27 @@ class GerritConnection(BaseConnection):
urllib.parse.quote(str(change.project), safe=''),
urllib.parse.quote(str(change.branch), safe=''),
change.id)
for x in range(1, 4):
try:
self.post('changes/%s/revisions/%s/review' %
(changeid, change.commit),
data)
break
except Exception:
log.exception("Error submitting data to gerrit, attempt %s", x)
time.sleep(x * 10)
if 'checks_api' in action:
self.report_checks(log, item, changeid, action['checks_api'])
if (message or data.get('labels') or data.get('comments')):
for x in range(1, 4):
try:
self.post('changes/%s/revisions/%s/review' %
(changeid, change.commit),
data)
break
except Exception:
log.exception(
"Error submitting data to gerrit, attempt %s", x)
time.sleep(x * 10)
if change.is_current_patchset and submit:
try:
self.post('changes/%s/submit' % (changeid,), {})
except Exception:
log.exception("Error submitting data to gerrit, attempt %s", x)
time.sleep(x * 10)
for x in range(1, 4):
try:
self.post('changes/%s/submit' % (changeid,), {})
except Exception:
log.exception(
"Error submitting data to gerrit, attempt %s", x)
time.sleep(x * 10)
def query(self, query, event=None):
args = '--all-approvals --comments --commit-message'
@@ -1078,11 +1244,13 @@ class GerritConnection(BaseConnection):
def onLoad(self):
self.log.debug("Starting Gerrit Connection/Watchers")
self._start_watcher_thread()
self._start_poller_thread()
self._start_event_connector()
def onStop(self):
self.log.debug("Stopping Gerrit Connection/Watchers")
self._stop_watcher_thread()
self._stop_poller_thread()
self._stop_event_connector()
def _stop_watcher_thread(self):
@@ -1100,6 +1268,15 @@ class GerritConnection(BaseConnection):
keepalive=self.keepalive)
self.watcher_thread.start()
def _stop_poller_thread(self):
if self.poller_thread:
self.poller_thread.stop()
self.poller_thread.join()
def _start_poller_thread(self):
self.poller_thread = self._poller_class(self)
self.poller_thread.start()
def _stop_event_connector(self):
if self.gerrit_event_connector:
self.gerrit_event_connector.stop()

18
zuul/driver/gerrit/gerritmodel.py
View File

@@ -35,6 +35,8 @@ class GerritTriggerEvent(TriggerEvent):
def __init__(self):
super(GerritTriggerEvent, self).__init__()
self.approvals = []
self.uuid = None
self.scheme = None
def __repr__(self):
ret = '<GerritTriggerEvent %s %s' % (self.type,
@@ -154,8 +156,8 @@ class GerritApprovalFilter(object):
class GerritEventFilter(EventFilter, GerritApprovalFilter):
def __init__(self, trigger, types=[], branches=[], refs=[],
event_approvals={}, comments=[], emails=[], usernames=[],
required_approvals=[], reject_approvals=[],
ignore_deletes=True):
required_approvals=[], reject_approvals=[], uuid=None,
scheme=None, ignore_deletes=True):
EventFilter.__init__(self, trigger)
@@ -176,6 +178,8 @@ class GerritEventFilter(EventFilter, GerritApprovalFilter):
self.emails = [re.compile(x) for x in emails]
self.usernames = [re.compile(x) for x in usernames]
self.event_approvals = event_approvals
self.uuid = uuid
self.scheme = scheme
self.ignore_deletes = ignore_deletes
def __repr__(self):
@@ -183,6 +187,10 @@ class GerritEventFilter(EventFilter, GerritApprovalFilter):
if self._types:
ret += ' types: %s' % ', '.join(self._types)
if self.uuid:
ret += ' uuid: %s' % (self.uuid,)
if self.scheme:
ret += ' scheme: %s' % (self.scheme,)
if self._branches:
ret += ' branches: %s' % ', '.join(self._branches)
if self._refs:
@@ -217,6 +225,12 @@ class GerritEventFilter(EventFilter, GerritApprovalFilter):
if self.types and not matches_type:
return False
if event.type == 'pending-check':
if self.uuid and event.uuid != self.uuid:
return False
if self.scheme and event.uuid.split(':')[0] != self.scheme:
return False
# branches are ORed
matches_branch = False
for branch in self.branches:

2
zuul/driver/gerrit/gerritreporter.py
View File

@@ -61,7 +61,7 @@ class GerritReporter(BaseReporter):
item.change._ref_sha = item.change.project.source.getRefSha(
item.change.project, 'refs/heads/' + item.change.branch)
return self.connection.review(item.change, message, self.config,
return self.connection.review(item, message, self.config,
comments, zuul_event_id=item.event)
def getSubmitAllowNeeds(self):

7
zuul/driver/gerrit/gerrittrigger.py
View File

@@ -56,6 +56,8 @@ class GerritTrigger(BaseTrigger):
reject_approvals=to_list(
trigger.get('reject-approval')
),
uuid=trigger.get('uuid'),
scheme=trigger.get('scheme'),
ignore_deletes=ignore_deletes
)
efilters.append(f)
@@ -80,7 +82,10 @@ def getSchema():
'change-restored',
'change-merged',
'comment-added',
'ref-updated')),
'ref-updated',
'pending-check')),
'uuid': str,
'scheme': str,
'comment_filter': scalar_or_list(str),
'comment': scalar_or_list(str),
'email_filter': scalar_or_list(str),