Support multiple triggers
Add the ability for Zuul to accept inputs from multiple trigger sources simultaneously. Pipelines are associated with exactly one trigger, which must now be named in the configuration file. Co-Authored-By: Monty Taylor <mordred@inaugust.com> Change-Id: Ief2b31a7b8d85d30817f2747c1e2635f71ea24b9
This commit is contained in:
parent
3cd25034b6
commit
6c358e72ea
8
NEWS.rst
8
NEWS.rst
|
@ -11,13 +11,19 @@ Since 1.3.0:
|
||||||
QueueItem has the full context for why the change is being run
|
QueueItem has the full context for why the change is being run
|
||||||
(including the pipeline, items ahead and behind, etc.). The Change
|
(including the pipeline, items ahead and behind, etc.). The Change
|
||||||
is still available via the "change" attribute on the QueueItem. The
|
is still available via the "change" attribute on the QueueItem. The
|
||||||
second argument is now the Job that is about to be run, ande the
|
second argument is now the Job that is about to be run, and the
|
||||||
parameter dictionary is shifted to the third position.
|
parameter dictionary is shifted to the third position.
|
||||||
|
|
||||||
* The ZUUL_SHORT_* parameters have been removed (the same
|
* The ZUUL_SHORT_* parameters have been removed (the same
|
||||||
functionality may be achieved with a custom parameter function that
|
functionality may be achieved with a custom parameter function that
|
||||||
matches all jobs).
|
matches all jobs).
|
||||||
|
|
||||||
|
* Multiple triggers are now supported, in principle (though only
|
||||||
|
Gerrit is defined currently). Your layout.yaml file will need to
|
||||||
|
change to add the key "gerrit:" inside of the "triggers:" list to
|
||||||
|
specify a Gerrit trigger (and facilitate adding other kinds of
|
||||||
|
triggers later). See the sample layout.yaml.
|
||||||
|
|
||||||
* The default behavior is now to immediately dequeue changes that have
|
* The default behavior is now to immediately dequeue changes that have
|
||||||
merge conflicts, even those not at the head of the queue. To enable
|
merge conflicts, even those not at the head of the queue. To enable
|
||||||
the old behavior (which would wait until the conflicting change was
|
the old behavior (which would wait until the conflicting change was
|
||||||
|
|
13
TESTING.rst
13
TESTING.rst
|
@ -67,6 +67,19 @@ List Failing Tests
|
||||||
.tox/py27/bin/activate
|
.tox/py27/bin/activate
|
||||||
testr failing --list
|
testr failing --list
|
||||||
|
|
||||||
|
Hanging Tests
|
||||||
|
-------------
|
||||||
|
|
||||||
|
The following will run each test in turn and print the name of the
|
||||||
|
test as it is run::
|
||||||
|
|
||||||
|
. .tox/py27/bin/activate
|
||||||
|
testr run --subunit | subunit2pyunit
|
||||||
|
|
||||||
|
You can compare the output of that to::
|
||||||
|
|
||||||
|
python -m testtools.run discover --list
|
||||||
|
|
||||||
Need More Info?
|
Need More Info?
|
||||||
---------------
|
---------------
|
||||||
|
|
||||||
|
|
|
@ -178,7 +178,8 @@ explanation of each of the parameters::
|
||||||
- name: check
|
- name: check
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: patchset-created
|
gerrit:
|
||||||
|
- event: patchset-created
|
||||||
success:
|
success:
|
||||||
verified: 1
|
verified: 1
|
||||||
failure:
|
failure:
|
||||||
|
@ -253,51 +254,55 @@ explanation of each of the parameters::
|
||||||
DependentPipelineManager, see: :doc:`gating`.
|
DependentPipelineManager, see: :doc:`gating`.
|
||||||
|
|
||||||
**trigger**
|
**trigger**
|
||||||
This describes what Gerrit events should be placed in the pipeline.
|
Exactly one trigger source must be supplied for each pipeline.
|
||||||
Triggers are not exclusive -- matching events may be placed in
|
Triggers are not exclusive -- matching events may be placed in
|
||||||
multiple pipelines, and they will behave independently in each of the
|
multiple pipelines, and they will behave independently in each of
|
||||||
pipelines they match. Multiple triggers may be listed. Further
|
the pipelines they match. You may select from the following:
|
||||||
parameters describe the kind of events that match:
|
|
||||||
|
|
||||||
*event*
|
**gerrit**
|
||||||
The event name from gerrit. Examples: ``patchset-created``,
|
This describes what Gerrit events should be placed in the
|
||||||
``comment-added``, ``ref-updated``. This field is treated as a
|
pipeline. Multiple gerrit triggers may be listed. Further
|
||||||
regular expression.
|
parameters describe the kind of events that match:
|
||||||
|
|
||||||
*branch*
|
*event*
|
||||||
The branch associated with the event. Example: ``master``. This
|
The event name from gerrit. Examples: ``patchset-created``,
|
||||||
field is treated as a regular expression, and multiple branches may
|
``comment-added``, ``ref-updated``. This field is treated as a
|
||||||
be listed.
|
regular expression.
|
||||||
|
|
||||||
*ref*
|
*branch*
|
||||||
On ref-updated events, the branch parameter is not used, instead the
|
The branch associated with the event. Example: ``master``. This
|
||||||
ref is provided. Currently Gerrit has the somewhat idiosyncratic
|
field is treated as a regular expression, and multiple branches may
|
||||||
behavior of specifying bare refs for branch names (e.g., ``master``),
|
be listed.
|
||||||
but full ref names for other kinds of refs (e.g., ``refs/tags/foo``).
|
|
||||||
Zuul matches what you put here exactly against what Gerrit
|
|
||||||
provides. This field is treated as a regular expression, and
|
|
||||||
multiple refs may be listed.
|
|
||||||
|
|
||||||
*approval*
|
*ref*
|
||||||
This is only used for ``comment-added`` events. It only matches if
|
On ref-updated events, the branch parameter is not used, instead the
|
||||||
the event has a matching approval associated with it. Example:
|
ref is provided. Currently Gerrit has the somewhat idiosyncratic
|
||||||
``code-review: 2`` matches a ``+2`` vote on the code review category.
|
behavior of specifying bare refs for branch names (e.g., ``master``),
|
||||||
Multiple approvals may be listed.
|
but full ref names for other kinds of refs (e.g., ``refs/tags/foo``).
|
||||||
|
Zuul matches what you put here exactly against what Gerrit
|
||||||
|
provides. This field is treated as a regular expression, and
|
||||||
|
multiple refs may be listed.
|
||||||
|
|
||||||
*email_filter*
|
*approval*
|
||||||
This is used for any event. It takes a regex applied on the performer
|
This is only used for ``comment-added`` events. It only matches if
|
||||||
email, i.e Gerrit account email address. If you want to specify
|
the event has a matching approval associated with it. Example:
|
||||||
several email filters, you must use a YAML list. Make sure to use non
|
``code-review: 2`` matches a ``+2`` vote on the code review category.
|
||||||
greedy matchers and to escapes dots!
|
Multiple approvals may be listed.
|
||||||
Example: ``email_filter: ^.*?@example\.org$``.
|
|
||||||
|
|
||||||
*comment_filter*
|
*email_filter*
|
||||||
This is only used for ``comment-added`` events. It accepts a list of
|
This is used for any event. It takes a regex applied on the performer
|
||||||
regexes that are searched for in the comment string. If any of these
|
email, i.e Gerrit account email address. If you want to specify
|
||||||
regexes matches a portion of the comment string the trigger is
|
several email filters, you must use a YAML list. Make sure to use non
|
||||||
matched. ``comment_filter: retrigger`` will match when comments
|
greedy matchers and to escapes dots!
|
||||||
containing 'retrigger' somewhere in the comment text are added to a
|
Example: ``email_filter: ^.*?@example\.org$``.
|
||||||
change.
|
|
||||||
|
*comment_filter*
|
||||||
|
This is only used for ``comment-added`` events. It accepts a list of
|
||||||
|
regexes that are searched for in the comment string. If any of these
|
||||||
|
regexes matches a portion of the comment string the trigger is
|
||||||
|
matched. ``comment_filter: retrigger`` will match when comments
|
||||||
|
containing 'retrigger' somewhere in the comment text are added to a
|
||||||
|
change.
|
||||||
|
|
||||||
**dequeue-on-new-patchset**
|
**dequeue-on-new-patchset**
|
||||||
Normally, if a new patchset is uploaded to a change that is in a
|
Normally, if a new patchset is uploaded to a change that is in a
|
||||||
|
|
|
@ -2,7 +2,8 @@ pipelines:
|
||||||
- name: check
|
- name: check
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: patchset-created
|
gerrit:
|
||||||
|
- event: patchset-created
|
||||||
success:
|
success:
|
||||||
verified: 1
|
verified: 1
|
||||||
failure:
|
failure:
|
||||||
|
@ -11,8 +12,9 @@ pipelines:
|
||||||
- name: tests
|
- name: tests
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: patchset-created
|
gerrit:
|
||||||
email_filter: ^.*@example.org$
|
- event: patchset-created
|
||||||
|
email_filter: ^.*@example.org$
|
||||||
success:
|
success:
|
||||||
verified: 1
|
verified: 1
|
||||||
failure:
|
failure:
|
||||||
|
@ -21,15 +23,17 @@ pipelines:
|
||||||
- name: post
|
- name: post
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: ref-updated
|
gerrit:
|
||||||
ref: ^(?!refs/).*$
|
- event: ref-updated
|
||||||
|
ref: ^(?!refs/).*$
|
||||||
|
|
||||||
- name: gate
|
- name: gate
|
||||||
manager: DependentPipelineManager
|
manager: DependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: comment-added
|
gerrit:
|
||||||
approval:
|
- event: comment-added
|
||||||
- approved: 1
|
approval:
|
||||||
|
- approved: 1
|
||||||
start:
|
start:
|
||||||
verified: 0
|
verified: 0
|
||||||
success:
|
success:
|
||||||
|
|
|
@ -2,7 +2,8 @@ pipelines:
|
||||||
- name: check
|
- name: check
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: patchset-created
|
gerrit:
|
||||||
|
- event: patchset-created
|
||||||
success:
|
success:
|
||||||
verified: 1
|
verified: 1
|
||||||
failure:
|
failure:
|
||||||
|
@ -11,16 +12,18 @@ pipelines:
|
||||||
- name: post
|
- name: post
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: ref-updated
|
gerrit:
|
||||||
ref: ^(?!refs/).*$
|
- event: ref-updated
|
||||||
|
ref: ^(?!refs/).*$
|
||||||
|
|
||||||
- name: gate
|
- name: gate
|
||||||
manager: DependentPipelineManager
|
manager: DependentPipelineManager
|
||||||
failure-message: Build failed. For information on how to proceed, see http://wiki.example.org/Test_Failures
|
failure-message: Build failed. For information on how to proceed, see http://wiki.example.org/Test_Failures
|
||||||
trigger:
|
trigger:
|
||||||
- event: comment-added
|
gerrit:
|
||||||
approval:
|
- event: comment-added
|
||||||
- approved: 1
|
approval:
|
||||||
|
- approved: 1
|
||||||
success:
|
success:
|
||||||
verified: 2
|
verified: 2
|
||||||
submit: true
|
submit: true
|
||||||
|
|
|
@ -5,7 +5,8 @@ pipelines:
|
||||||
- name: check
|
- name: check
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: patchset-created
|
gerrit:
|
||||||
|
- event: patchset-created
|
||||||
success:
|
success:
|
||||||
verified: 1
|
verified: 1
|
||||||
failure:
|
failure:
|
||||||
|
@ -14,16 +15,18 @@ pipelines:
|
||||||
- name: post
|
- name: post
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: ref-updated
|
gerrit:
|
||||||
ref: ^(?!refs/).*$
|
- event: ref-updated
|
||||||
|
ref: ^(?!refs/).*$
|
||||||
|
|
||||||
- name: gate
|
- name: gate
|
||||||
manager: DependentPipelineManager
|
manager: DependentPipelineManager
|
||||||
failure-message: Build failed. For information on how to proceed, see http://wiki.example.org/Test_Failures
|
failure-message: Build failed. For information on how to proceed, see http://wiki.example.org/Test_Failures
|
||||||
trigger:
|
trigger:
|
||||||
- event: comment-added
|
gerrit:
|
||||||
approval:
|
- event: comment-added
|
||||||
- approved: 1
|
approval:
|
||||||
|
- approved: 1
|
||||||
success:
|
success:
|
||||||
verified: 2
|
verified: 2
|
||||||
submit: true
|
submit: true
|
||||||
|
@ -37,14 +40,16 @@ pipelines:
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
dequeue-on-new-patchset: false
|
dequeue-on-new-patchset: false
|
||||||
trigger:
|
trigger:
|
||||||
- event: comment-added
|
gerrit:
|
||||||
approval:
|
- event: comment-added
|
||||||
- approved: 1
|
approval:
|
||||||
|
- approved: 1
|
||||||
|
|
||||||
- name: dup1
|
- name: dup1
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: change-restored
|
gerrit:
|
||||||
|
- event: change-restored
|
||||||
success:
|
success:
|
||||||
verified: 1
|
verified: 1
|
||||||
failure:
|
failure:
|
||||||
|
@ -53,7 +58,8 @@ pipelines:
|
||||||
- name: dup2
|
- name: dup2
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: change-restored
|
gerrit:
|
||||||
|
- event: change-restored
|
||||||
success:
|
success:
|
||||||
verified: 1
|
verified: 1
|
||||||
failure:
|
failure:
|
||||||
|
@ -64,9 +70,10 @@ pipelines:
|
||||||
manager: DependentPipelineManager
|
manager: DependentPipelineManager
|
||||||
failure-message: Build failed. For information on how to proceed, see http://wiki.example.org/Test_Failures
|
failure-message: Build failed. For information on how to proceed, see http://wiki.example.org/Test_Failures
|
||||||
trigger:
|
trigger:
|
||||||
- event: comment-added
|
gerrit:
|
||||||
approval:
|
- event: comment-added
|
||||||
- approved: 1
|
approval:
|
||||||
|
- approved: 1
|
||||||
success:
|
success:
|
||||||
verified: 2
|
verified: 2
|
||||||
submit: true
|
submit: true
|
||||||
|
|
|
@ -2,7 +2,8 @@ pipelines:
|
||||||
- name: check
|
- name: check
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: non-event
|
gerrit:
|
||||||
|
- event: non-event
|
||||||
|
|
||||||
projects:
|
projects:
|
||||||
- name: foo
|
- name: foo
|
||||||
|
|
|
@ -2,8 +2,9 @@ pipelines:
|
||||||
- name: check
|
- name: check
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- approval:
|
gerrit:
|
||||||
- approved: 1
|
- approval:
|
||||||
|
- approved: 1
|
||||||
|
|
||||||
projects:
|
projects:
|
||||||
- name: foo
|
- name: foo
|
||||||
|
|
|
@ -2,8 +2,9 @@ pipelines:
|
||||||
- name: check
|
- name: check
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: comment-added
|
gerrit:
|
||||||
approved: 1
|
- event: comment-added
|
||||||
|
approved: 1
|
||||||
|
|
||||||
projects:
|
projects:
|
||||||
- name: foo
|
- name: foo
|
||||||
|
|
|
@ -4,7 +4,8 @@ pipelines:
|
||||||
- name: 'check'
|
- name: 'check'
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: patchset-created
|
gerrit:
|
||||||
|
- event: patchset-created
|
||||||
|
|
||||||
project-templates:
|
project-templates:
|
||||||
- name: template-generic
|
- name: template-generic
|
||||||
|
|
|
@ -4,7 +4,8 @@ pipelines:
|
||||||
- name: 'check'
|
- name: 'check'
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: patchset-created
|
gerrit:
|
||||||
|
- event: patchset-created
|
||||||
|
|
||||||
project-templates:
|
project-templates:
|
||||||
- name: template-generic
|
- name: template-generic
|
||||||
|
|
|
@ -5,7 +5,8 @@ pipelines:
|
||||||
- name: check
|
- name: check
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: patchset-created
|
gerrit:
|
||||||
|
- event: patchset-created
|
||||||
success:
|
success:
|
||||||
verified: 1
|
verified: 1
|
||||||
failure:
|
failure:
|
||||||
|
@ -14,17 +15,19 @@ pipelines:
|
||||||
- name: post
|
- name: post
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: ref-updated
|
gerrit:
|
||||||
ref: ^(?!refs/).*$
|
- event: ref-updated
|
||||||
|
ref: ^(?!refs/).*$
|
||||||
|
|
||||||
- name: gate
|
- name: gate
|
||||||
manager: DependentPipelineManager
|
manager: DependentPipelineManager
|
||||||
success-message: Your change is awesome.
|
success-message: Your change is awesome.
|
||||||
failure-message: Build failed. For information on how to proceed, see http://wiki.example.org/Test_Failures
|
failure-message: Build failed. For information on how to proceed, see http://wiki.example.org/Test_Failures
|
||||||
trigger:
|
trigger:
|
||||||
- event: comment-added
|
gerrit:
|
||||||
approval:
|
- event: comment-added
|
||||||
- approved: 1
|
approval:
|
||||||
|
- approved: 1
|
||||||
success:
|
success:
|
||||||
verified: 2
|
verified: 2
|
||||||
code-review: 1
|
code-review: 1
|
||||||
|
|
|
@ -2,7 +2,8 @@ pipelines:
|
||||||
- name: 'check'
|
- name: 'check'
|
||||||
manager: IndependentPipelineManager
|
manager: IndependentPipelineManager
|
||||||
trigger:
|
trigger:
|
||||||
- event: patchset-created
|
gerrit:
|
||||||
|
- event: patchset-created
|
||||||
|
|
||||||
project-templates:
|
project-templates:
|
||||||
- name: template-generic
|
- name: template-generic
|
||||||
|
|
|
@ -28,7 +28,7 @@ FIXTURE_DIR = os.path.join(os.path.dirname(__file__),
|
||||||
LAYOUT_RE = re.compile(r'^(good|bad)_.*\.yaml$')
|
LAYOUT_RE = re.compile(r'^(good|bad)_.*\.yaml$')
|
||||||
|
|
||||||
|
|
||||||
class testScheduler(testtools.TestCase):
|
class TestLayoutValidator(testtools.TestCase):
|
||||||
def test_layouts(self):
|
def test_layouts(self):
|
||||||
"""Test layout file validation"""
|
"""Test layout file validation"""
|
||||||
print
|
print
|
||||||
|
|
|
@ -761,7 +761,7 @@ class TestScheduler(testtools.TestCase):
|
||||||
self.webapp = zuul.webapp.WebApp(self.sched, port=0)
|
self.webapp = zuul.webapp.WebApp(self.sched, port=0)
|
||||||
|
|
||||||
self.sched.setLauncher(self.launcher)
|
self.sched.setLauncher(self.launcher)
|
||||||
self.sched.setTrigger(self.gerrit)
|
self.sched.registerTrigger(self.gerrit)
|
||||||
|
|
||||||
self.sched.start()
|
self.sched.start()
|
||||||
self.sched.reconfigure(self.config)
|
self.sched.reconfigure(self.config)
|
||||||
|
@ -777,7 +777,7 @@ class TestScheduler(testtools.TestCase):
|
||||||
|
|
||||||
def assertFinalState(self):
|
def assertFinalState(self):
|
||||||
# Make sure that the change cache is cleared
|
# Make sure that the change cache is cleared
|
||||||
self.assertEqual(len(self.sched.trigger._change_cache.keys()), 0)
|
self.assertEqual(len(self.gerrit._change_cache.keys()), 0)
|
||||||
self.assertEmptyQueues()
|
self.assertEmptyQueues()
|
||||||
|
|
||||||
def shutdown(self):
|
def shutdown(self):
|
||||||
|
@ -1440,9 +1440,9 @@ class TestScheduler(testtools.TestCase):
|
||||||
self.fake_gerrit.addEvent(B.addApproval('APRV', 1))
|
self.fake_gerrit.addEvent(B.addApproval('APRV', 1))
|
||||||
self.waitUntilSettled()
|
self.waitUntilSettled()
|
||||||
|
|
||||||
self.log.debug("len %s " % self.sched.trigger._change_cache.keys())
|
self.log.debug("len %s " % self.gerrit._change_cache.keys())
|
||||||
# there should still be changes in the cache
|
# there should still be changes in the cache
|
||||||
self.assertNotEqual(len(self.sched.trigger._change_cache.keys()), 0)
|
self.assertNotEqual(len(self.gerrit._change_cache.keys()), 0)
|
||||||
|
|
||||||
self.worker.hold_jobs_in_build = False
|
self.worker.hold_jobs_in_build = False
|
||||||
self.worker.release()
|
self.worker.release()
|
||||||
|
@ -1457,21 +1457,19 @@ class TestScheduler(testtools.TestCase):
|
||||||
"Test whether a change is ready to merge"
|
"Test whether a change is ready to merge"
|
||||||
# TODO: move to test_gerrit (this is a unit test!)
|
# TODO: move to test_gerrit (this is a unit test!)
|
||||||
A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
|
A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
|
||||||
a = self.sched.trigger.getChange(1, 2)
|
trigger = self.sched.layout.pipelines['gate'].trigger
|
||||||
|
a = self.sched.triggers['gerrit'].getChange(1, 2)
|
||||||
mgr = self.sched.layout.pipelines['gate'].manager
|
mgr = self.sched.layout.pipelines['gate'].manager
|
||||||
self.assertFalse(
|
self.assertFalse(trigger.canMerge(a, mgr.getSubmitAllowNeeds()))
|
||||||
self.sched.trigger.canMerge(a, mgr.getSubmitAllowNeeds()))
|
|
||||||
|
|
||||||
A.addApproval('CRVW', 2)
|
A.addApproval('CRVW', 2)
|
||||||
a = self.sched.trigger.getChange(1, 2, refresh=True)
|
a = trigger.getChange(1, 2, refresh=True)
|
||||||
self.assertFalse(
|
self.assertFalse(trigger.canMerge(a, mgr.getSubmitAllowNeeds()))
|
||||||
self.sched.trigger.canMerge(a, mgr.getSubmitAllowNeeds()))
|
|
||||||
|
|
||||||
A.addApproval('APRV', 1)
|
A.addApproval('APRV', 1)
|
||||||
a = self.sched.trigger.getChange(1, 2, refresh=True)
|
a = trigger.getChange(1, 2, refresh=True)
|
||||||
self.assertTrue(
|
self.assertTrue(trigger.canMerge(a, mgr.getSubmitAllowNeeds()))
|
||||||
self.sched.trigger.canMerge(a, mgr.getSubmitAllowNeeds()))
|
trigger.maintainCache([])
|
||||||
self.sched.trigger.maintainCache([])
|
|
||||||
|
|
||||||
def test_build_configuration(self):
|
def test_build_configuration(self):
|
||||||
"Test that zuul merges the right commits for testing"
|
"Test that zuul merges the right commits for testing"
|
||||||
|
@ -2347,6 +2345,7 @@ class TestScheduler(testtools.TestCase):
|
||||||
def test_test_config(self):
|
def test_test_config(self):
|
||||||
"Test that we can test the config"
|
"Test that we can test the config"
|
||||||
sched = zuul.scheduler.Scheduler()
|
sched = zuul.scheduler.Scheduler()
|
||||||
|
sched.registerTrigger(None, 'gerrit')
|
||||||
sched.testConfig(CONFIG.get('zuul', 'layout_config'))
|
sched.testConfig(CONFIG.get('zuul', 'layout_config'))
|
||||||
|
|
||||||
def test_build_description(self):
|
def test_build_description(self):
|
||||||
|
|
|
@ -105,6 +105,7 @@ class Server(object):
|
||||||
|
|
||||||
logging.basicConfig(level=logging.DEBUG)
|
logging.basicConfig(level=logging.DEBUG)
|
||||||
self.sched = zuul.scheduler.Scheduler()
|
self.sched = zuul.scheduler.Scheduler()
|
||||||
|
self.sched.registerTrigger(None, 'gerrit')
|
||||||
layout = self.sched.testConfig(self.config.get('zuul',
|
layout = self.sched.testConfig(self.config.get('zuul',
|
||||||
'layout_config'))
|
'layout_config'))
|
||||||
if not job_list_path:
|
if not job_list_path:
|
||||||
|
@ -165,7 +166,7 @@ class Server(object):
|
||||||
webapp = zuul.webapp.WebApp(self.sched)
|
webapp = zuul.webapp.WebApp(self.sched)
|
||||||
|
|
||||||
self.sched.setLauncher(gearman)
|
self.sched.setLauncher(gearman)
|
||||||
self.sched.setTrigger(gerrit)
|
self.sched.registerTrigger(gerrit)
|
||||||
|
|
||||||
self.sched.start()
|
self.sched.start()
|
||||||
self.sched.reconfigure(self.config)
|
self.sched.reconfigure(self.config)
|
||||||
|
|
|
@ -35,18 +35,21 @@ class LayoutSchema(object):
|
||||||
|
|
||||||
variable_dict = v.Schema({}, extra=True)
|
variable_dict = v.Schema({}, extra=True)
|
||||||
|
|
||||||
trigger = {v.Required('event'): toList(v.Any('patchset-created',
|
gerrit_trigger = {v.Required('event'):
|
||||||
'change-abandoned',
|
toList(v.Any('patchset-created',
|
||||||
'change-restored',
|
'change-abandoned',
|
||||||
'change-merged',
|
'change-restored',
|
||||||
'comment-added',
|
'change-merged',
|
||||||
'ref-updated')),
|
'comment-added',
|
||||||
'comment_filter': toList(str),
|
'ref-updated')),
|
||||||
'email_filter': toList(str),
|
'comment_filter': toList(str),
|
||||||
'branch': toList(str),
|
'email_filter': toList(str),
|
||||||
'ref': toList(str),
|
'branch': toList(str),
|
||||||
'approval': toList(variable_dict),
|
'ref': toList(str),
|
||||||
}
|
'approval': toList(variable_dict),
|
||||||
|
}
|
||||||
|
|
||||||
|
trigger = v.Required(v.Any({'gerrit': toList(gerrit_trigger)}))
|
||||||
|
|
||||||
pipeline = {v.Required('name'): str,
|
pipeline = {v.Required('name'): str,
|
||||||
v.Required('manager'): manager,
|
v.Required('manager'): manager,
|
||||||
|
@ -56,7 +59,7 @@ class LayoutSchema(object):
|
||||||
'failure-message': str,
|
'failure-message': str,
|
||||||
'dequeue-on-new-patchset': bool,
|
'dequeue-on-new-patchset': bool,
|
||||||
'dequeue-on-conflict': bool,
|
'dequeue-on-conflict': bool,
|
||||||
'trigger': toList(trigger),
|
'trigger': trigger,
|
||||||
'success': variable_dict,
|
'success': variable_dict,
|
||||||
'failure': variable_dict,
|
'failure': variable_dict,
|
||||||
'start': variable_dict,
|
'start': variable_dict,
|
||||||
|
|
|
@ -47,6 +47,7 @@ class Pipeline(object):
|
||||||
self.manager = None
|
self.manager = None
|
||||||
self.queues = []
|
self.queues = []
|
||||||
self.precedence = PRECEDENCE_NORMAL
|
self.precedence = PRECEDENCE_NORMAL
|
||||||
|
self.trigger = None
|
||||||
|
|
||||||
def __repr__(self):
|
def __repr__(self):
|
||||||
return '<Pipeline %s>' % self.name
|
return '<Pipeline %s>' % self.name
|
||||||
|
@ -724,6 +725,7 @@ class TriggerEvent(object):
|
||||||
# common
|
# common
|
||||||
self.type = None
|
self.type = None
|
||||||
self.project_name = None
|
self.project_name = None
|
||||||
|
self.trigger_name = None
|
||||||
# Representation of the user account that performed the event.
|
# Representation of the user account that performed the event.
|
||||||
self.account = None
|
self.account = None
|
||||||
# patchset-created, comment-added, etc.
|
# patchset-created, comment-added, etc.
|
||||||
|
|
|
@ -74,7 +74,7 @@ class Scheduler(threading.Thread):
|
||||||
self._exit = False
|
self._exit = False
|
||||||
self._stopped = False
|
self._stopped = False
|
||||||
self.launcher = None
|
self.launcher = None
|
||||||
self.trigger = None
|
self.triggers = dict()
|
||||||
self.config = None
|
self.config = None
|
||||||
self._maintain_trigger_cache = False
|
self._maintain_trigger_cache = False
|
||||||
|
|
||||||
|
@ -141,20 +141,24 @@ class Scheduler(threading.Thread):
|
||||||
manager.success_action = conf_pipeline.get('success')
|
manager.success_action = conf_pipeline.get('success')
|
||||||
manager.failure_action = conf_pipeline.get('failure')
|
manager.failure_action = conf_pipeline.get('failure')
|
||||||
manager.start_action = conf_pipeline.get('start')
|
manager.start_action = conf_pipeline.get('start')
|
||||||
for trigger in toList(conf_pipeline['trigger']):
|
# TODO: move this into triggers (may require pluggable
|
||||||
approvals = {}
|
# configuration)
|
||||||
for approval_dict in toList(trigger.get('approval')):
|
if 'gerrit' in conf_pipeline['trigger']:
|
||||||
for k, v in approval_dict.items():
|
pipeline.trigger = self.triggers['gerrit']
|
||||||
approvals[k] = v
|
for trigger in toList(conf_pipeline['trigger']['gerrit']):
|
||||||
f = EventFilter(types=toList(trigger['event']),
|
approvals = {}
|
||||||
branches=toList(trigger.get('branch')),
|
for approval_dict in toList(trigger.get('approval')):
|
||||||
refs=toList(trigger.get('ref')),
|
for k, v in approval_dict.items():
|
||||||
approvals=approvals,
|
approvals[k] = v
|
||||||
comment_filters=
|
f = EventFilter(types=toList(trigger['event']),
|
||||||
toList(trigger.get('comment_filter')),
|
branches=toList(trigger.get('branch')),
|
||||||
email_filters=
|
refs=toList(trigger.get('ref')),
|
||||||
toList(trigger.get('email_filter')))
|
approvals=approvals,
|
||||||
manager.event_filters.append(f)
|
comment_filters=
|
||||||
|
toList(trigger.get('comment_filter')),
|
||||||
|
email_filters=
|
||||||
|
toList(trigger.get('email_filter')))
|
||||||
|
manager.event_filters.append(f)
|
||||||
|
|
||||||
for project_template in data.get('project-templates', []):
|
for project_template in data.get('project-templates', []):
|
||||||
# Make sure the template only contains valid pipelines
|
# Make sure the template only contains valid pipelines
|
||||||
|
@ -272,17 +276,23 @@ class Scheduler(threading.Thread):
|
||||||
else:
|
else:
|
||||||
sshkey = None
|
sshkey = None
|
||||||
|
|
||||||
self.merger = merger.Merger(self.trigger, merge_root, push_refs,
|
# TODO: The merger should have an upstream repo independent of
|
||||||
|
# triggers, and then each trigger should provide a fetch
|
||||||
|
# location.
|
||||||
|
self.merger = merger.Merger(self.triggers['gerrit'],
|
||||||
|
merge_root, push_refs,
|
||||||
sshkey, merge_email, merge_name)
|
sshkey, merge_email, merge_name)
|
||||||
for project in self.layout.projects.values():
|
for project in self.layout.projects.values():
|
||||||
url = self.trigger.getGitUrl(project)
|
url = self.triggers['gerrit'].getGitUrl(project)
|
||||||
self.merger.addProject(project, url)
|
self.merger.addProject(project, url)
|
||||||
|
|
||||||
def setLauncher(self, launcher):
|
def setLauncher(self, launcher):
|
||||||
self.launcher = launcher
|
self.launcher = launcher
|
||||||
|
|
||||||
def setTrigger(self, trigger):
|
def registerTrigger(self, trigger, name=None):
|
||||||
self.trigger = trigger
|
if name is None:
|
||||||
|
name = trigger.name
|
||||||
|
self.triggers[name] = trigger
|
||||||
|
|
||||||
def getProject(self, name):
|
def getProject(self, name):
|
||||||
self.layout_lock.acquire()
|
self.layout_lock.acquire()
|
||||||
|
@ -518,7 +528,8 @@ class Scheduler(threading.Thread):
|
||||||
relevant.add(item.change)
|
relevant.add(item.change)
|
||||||
relevant.update(item.change.getRelatedChanges())
|
relevant.update(item.change.getRelatedChanges())
|
||||||
self.log.debug("Trigger cache size: %s" % len(relevant))
|
self.log.debug("Trigger cache size: %s" % len(relevant))
|
||||||
self.trigger.maintainCache(relevant)
|
for trigger in self.triggers.values():
|
||||||
|
trigger.maintainCache(relevant)
|
||||||
|
|
||||||
def process_event_queue(self):
|
def process_event_queue(self):
|
||||||
self.log.debug("Fetching trigger event")
|
self.log.debug("Fetching trigger event")
|
||||||
|
@ -540,7 +551,8 @@ class Scheduler(threading.Thread):
|
||||||
self.merger.updateRepo(project)
|
self.merger.updateRepo(project)
|
||||||
|
|
||||||
for pipeline in self.layout.pipelines.values():
|
for pipeline in self.layout.pipelines.values():
|
||||||
change = event.getChange(project, self.trigger)
|
change = event.getChange(project,
|
||||||
|
self.triggers.get(event.trigger_name))
|
||||||
if event.type == 'patchset-created':
|
if event.type == 'patchset-created':
|
||||||
pipeline.manager.removeOldVersionsOfChange(change)
|
pipeline.manager.removeOldVersionsOfChange(change)
|
||||||
if pipeline.manager.eventMatches(event):
|
if pipeline.manager.eventMatches(event):
|
||||||
|
@ -709,7 +721,7 @@ class BasePipelineManager(object):
|
||||||
msg = "Starting %s jobs." % self.pipeline.name
|
msg = "Starting %s jobs." % self.pipeline.name
|
||||||
if self.sched.config.has_option('zuul', 'status_url'):
|
if self.sched.config.has_option('zuul', 'status_url'):
|
||||||
msg += "\n" + self.sched.config.get('zuul', 'status_url')
|
msg += "\n" + self.sched.config.get('zuul', 'status_url')
|
||||||
ret = self.sched.trigger.report(change, msg, self.start_action)
|
ret = self.pipeline.trigger.report(change, msg, self.start_action)
|
||||||
if ret:
|
if ret:
|
||||||
self.log.error("Reporting change start %s received: %s" %
|
self.log.error("Reporting change start %s received: %s" %
|
||||||
(change, ret))
|
(change, ret))
|
||||||
|
@ -1025,8 +1037,8 @@ class BasePipelineManager(object):
|
||||||
succeeded = self.pipeline.didAllJobsSucceed(item)
|
succeeded = self.pipeline.didAllJobsSucceed(item)
|
||||||
merged = (not ret)
|
merged = (not ret)
|
||||||
if merged:
|
if merged:
|
||||||
merged = self.sched.trigger.isMerged(item.change,
|
merged = self.pipeline.trigger.isMerged(item.change,
|
||||||
item.change.branch)
|
item.change.branch)
|
||||||
self.log.info("Reported change %s status: all-succeeded: %s, "
|
self.log.info("Reported change %s status: all-succeeded: %s, "
|
||||||
"merged: %s" % (item.change, succeeded, merged))
|
"merged: %s" % (item.change, succeeded, merged))
|
||||||
if not (succeeded and merged):
|
if not (succeeded and merged):
|
||||||
|
@ -1054,7 +1066,7 @@ class BasePipelineManager(object):
|
||||||
try:
|
try:
|
||||||
self.log.info("Reporting change %s, action: %s" %
|
self.log.info("Reporting change %s, action: %s" %
|
||||||
(item.change, action))
|
(item.change, action))
|
||||||
ret = self.sched.trigger.report(item.change, report, action)
|
ret = self.pipeline.trigger.report(item.change, report, action)
|
||||||
if ret:
|
if ret:
|
||||||
self.log.error("Reporting change %s received: %s" %
|
self.log.error("Reporting change %s received: %s" %
|
||||||
(item.change, ret))
|
(item.change, ret))
|
||||||
|
@ -1307,8 +1319,8 @@ class DependentPipelineManager(BasePipelineManager):
|
||||||
self.log.info(" %s" % queue)
|
self.log.info(" %s" % queue)
|
||||||
|
|
||||||
def isChangeReadyToBeEnqueued(self, change):
|
def isChangeReadyToBeEnqueued(self, change):
|
||||||
if not self.sched.trigger.canMerge(change,
|
if not self.pipeline.trigger.canMerge(change,
|
||||||
self.getSubmitAllowNeeds()):
|
self.getSubmitAllowNeeds()):
|
||||||
self.log.debug("Change %s can not merge, ignoring" % change)
|
self.log.debug("Change %s can not merge, ignoring" % change)
|
||||||
return False
|
return False
|
||||||
return True
|
return True
|
||||||
|
@ -1320,8 +1332,8 @@ class DependentPipelineManager(BasePipelineManager):
|
||||||
self.log.debug(" Changeish does not support dependencies")
|
self.log.debug(" Changeish does not support dependencies")
|
||||||
return
|
return
|
||||||
for needs in change.needed_by_changes:
|
for needs in change.needed_by_changes:
|
||||||
if self.sched.trigger.canMerge(needs,
|
if self.pipeline.trigger.canMerge(needs,
|
||||||
self.getSubmitAllowNeeds()):
|
self.getSubmitAllowNeeds()):
|
||||||
self.log.debug(" Change %s needs %s and is ready to merge" %
|
self.log.debug(" Change %s needs %s and is ready to merge" %
|
||||||
(needs, change))
|
(needs, change))
|
||||||
to_enqueue.append(needs)
|
to_enqueue.append(needs)
|
||||||
|
@ -1358,8 +1370,8 @@ class DependentPipelineManager(BasePipelineManager):
|
||||||
if self.isChangeAlreadyInQueue(change.needs_change):
|
if self.isChangeAlreadyInQueue(change.needs_change):
|
||||||
self.log.debug(" Needed change is already ahead in the queue")
|
self.log.debug(" Needed change is already ahead in the queue")
|
||||||
return True
|
return True
|
||||||
if self.sched.trigger.canMerge(change.needs_change,
|
if self.pipeline.trigger.canMerge(change.needs_change,
|
||||||
self.getSubmitAllowNeeds()):
|
self.getSubmitAllowNeeds()):
|
||||||
self.log.debug(" Change %s is needed" %
|
self.log.debug(" Change %s is needed" %
|
||||||
change.needs_change)
|
change.needs_change)
|
||||||
return change.needs_change
|
return change.needs_change
|
||||||
|
|
|
@ -25,11 +25,12 @@ class GerritEventConnector(threading.Thread):
|
||||||
|
|
||||||
log = logging.getLogger("zuul.GerritEventConnector")
|
log = logging.getLogger("zuul.GerritEventConnector")
|
||||||
|
|
||||||
def __init__(self, gerrit, sched):
|
def __init__(self, gerrit, sched, trigger):
|
||||||
super(GerritEventConnector, self).__init__()
|
super(GerritEventConnector, self).__init__()
|
||||||
self.daemon = True
|
self.daemon = True
|
||||||
self.gerrit = gerrit
|
self.gerrit = gerrit
|
||||||
self.sched = sched
|
self.sched = sched
|
||||||
|
self.trigger = trigger
|
||||||
self._stopped = False
|
self._stopped = False
|
||||||
|
|
||||||
def stop(self):
|
def stop(self):
|
||||||
|
@ -42,6 +43,7 @@ class GerritEventConnector(threading.Thread):
|
||||||
return
|
return
|
||||||
event = TriggerEvent()
|
event = TriggerEvent()
|
||||||
event.type = data.get('type')
|
event.type = data.get('type')
|
||||||
|
event.trigger_name = self.trigger.name
|
||||||
change = data.get('change')
|
change = data.get('change')
|
||||||
if change:
|
if change:
|
||||||
event.project_name = change.get('project')
|
event.project_name = change.get('project')
|
||||||
|
@ -85,9 +87,9 @@ class GerritEventConnector(threading.Thread):
|
||||||
# Call getChange for the side effect of updating the
|
# Call getChange for the side effect of updating the
|
||||||
# cache. Note that this modifies Change objects outside
|
# cache. Note that this modifies Change objects outside
|
||||||
# the main thread.
|
# the main thread.
|
||||||
self.sched.trigger.getChange(event.change_number,
|
self.trigger.getChange(event.change_number,
|
||||||
event.patch_number,
|
event.patch_number,
|
||||||
refresh=True)
|
refresh=True)
|
||||||
|
|
||||||
self.sched.addEvent(event)
|
self.sched.addEvent(event)
|
||||||
self.gerrit.eventDone()
|
self.gerrit.eventDone()
|
||||||
|
@ -103,6 +105,7 @@ class GerritEventConnector(threading.Thread):
|
||||||
|
|
||||||
|
|
||||||
class Gerrit(object):
|
class Gerrit(object):
|
||||||
|
name = 'gerrit'
|
||||||
log = logging.getLogger("zuul.Gerrit")
|
log = logging.getLogger("zuul.Gerrit")
|
||||||
replication_timeout = 60
|
replication_timeout = 60
|
||||||
replication_retry_interval = 5
|
replication_retry_interval = 5
|
||||||
|
@ -128,7 +131,7 @@ class Gerrit(object):
|
||||||
self.gerrit = gerrit.Gerrit(self.server, user, port, sshkey)
|
self.gerrit = gerrit.Gerrit(self.server, user, port, sshkey)
|
||||||
self.gerrit.startWatching()
|
self.gerrit.startWatching()
|
||||||
self.gerrit_connector = GerritEventConnector(
|
self.gerrit_connector = GerritEventConnector(
|
||||||
self.gerrit, sched)
|
self.gerrit, sched, self)
|
||||||
self.gerrit_connector.start()
|
self.gerrit_connector.start()
|
||||||
|
|
||||||
def stop(self):
|
def stop(self):
|
||||||
|
|
Loading…
Reference in New Issue