Zuul's new rate limiting feature would ignore jobs started that are
later moved outside the window (due to the window shrinking). Zuul
properly dealt with this jobs and changes when they slid back into the
window but it did so inefficiently. Go back to processing the entire
queue but check if each individual item is actionable before preparing
refs on it or starting jobs.
Change-Id: Ib76a68f9023652205003e0d164a78b8f67956adf
When changes report failure reduce the total number of changes that will
be tested concurrently in the pipeline queue the reported change
belonged too. Increase the number of changes that will be tested when
changes report success. This implements simple rate limiting which
should reduce resource thrash when tests are unstable.
Change-Id: Id092446c83649b3916751c4e4665d2adc75d0458
We want to report on "changes" triggered by a timer. Since those
aren't really changes, they are represented by a class called
NullChange which carries as much of the interface for a Change
object as possible (not much) so that they can be enqueued in
pipelines.
If a NullChange is reportable, then pretty much any kind of
change should be considered reportable. So remove the flag that
indicates that NullChanges and Refs are not reportable.
Only attempt to format a change report if there is an action
defined for that pipeline (in case the default formatting process
attempts to access a change attribute that is inappropriate).
Remove checks that try to avoid formatting or sending a report
based on attributes of the change (which are no longer relevant).
Add a test for using a timer trigger with an smtp reporter.
Add validation of the attributes that an smtp reporter can use in
the layout file.
Allow the operator to configure a Subject for smtp reports.
Change-Id: Icd067a7600c2922a318b9ade11b3946df4e53065
Currently we can filter pipeline triggers by email address but not
username. This is an issue for users that have no email addresses
such as Jenkins.
This patch adds a new filter "username_filter" to the gerrit trigger
section.
Change-Id: I66680ab7e9e5ff49466269175c8fb54aef30e016
Provide the short name of a project (anything after the last '/') to
project templates as the variable 'name'. If 'openstack/nova' invoked
a template, the variable 'name' would automatically be set to 'nova'
within the template.
Ideally this means that most template invocations in OpenStack's layout
will not need any variables defined.
Change-Id: I596744917c30c92041b8ea5b1f518d50fb64e59b
Voluptuous 0.7.0 introduced a backward API incompatible change in
upstream commit 475adebc:
https://github.com/alecthomas/voluptuous/commit/475adebc
The schema are precompiled and the validate_X have been removed,
voluptuous is now smart enough to detect the type of value being
validated and would call an internal validation method matching the
type.
Commit is contained since 0.7.0:
$ git tag --contains 475adebc
0.7.0
0.7.1
0.7.2
0.8.1
0.8.2
$
I tested it using etc/layout.yaml-sample and added an inexistent
pipeline to the project. The test yield:
voluptuous.MultipleInvalid: extra keys not allowed @
data['projects'][0]['nonpipe']
Debian has recently accepted voluptuous 0.8.2 in testing, so if we want
to package Zuul, we better have to upgrade our voluptuous requirement as
well. Ref: http://packages.qa.debian.org/v/voluptuous.html
Change-Id: I117ea644863b2e4a4dc3429aa81e868573382877
Update the scheduler algorithm to NNFI -- Nearest Non-Failing Item.
A stateless description of the algorithm is that jobs for every
item should always be run based on the repository state(s) set by
the nearest non-failing item ahead of it in its change queue.
This means that should an item fail (for any reason -- failure to
merge, a merge conflict, or job failure) changes after it will
have their builds canceled and restarted with the assumption that
the failed change will not merge but the nearest non-failing
change ahead will merge.
This should mean that dependent queues will always be running
jobs and no longer need to wait for a failing change to merge or
not merge before restarting jobs.
This removes the dequeue-on-conflict behavior because there is
now no cost to keeping an item that can not merge in the queue.
The documentation and associated test for this are removed.
This also removes the concept of severed heads because a failing
change at the head will not prevent other changes from proceeding
with their tests. If the jobs for the change at the head run
longer than following changes, it could still impact them while
it completes, but the reduction in code complexity is worth this
minor de-optimization.
The debugging representation of QueueItem is changed to make it
more useful.
Change-Id: I0d2d416fb0dd88647490ec06ed69deae71d39374
Utilises the new reporter plugin architecture to add support for
emailing success/failure messages based on layout.yaml.
This will assist in testing new gates as currently after a job has
finished if no report is sent back to gerrit then only the workers
logs can be consulted to see if it was successful. This will allow
developers to see exactly what zuul will return if they turn on
gerrit reporting.
Change-Id: I47ac038bbdffb0a0c75f8e63ff6978fd4b4d0a52
Allows multiple reports per a patchset to be sent to pluggable
destinations. These are configurable per pipeline and, if not
specified, defaults to the legacy behaviour of reporting back only
to gerrit.
Having multiple reporting methods means only certain success/failure
/start parameters will apply to certain reporters. Reporters are
listed as keys under each of those actions.
This means that each key under success/failure/start is a reporter and the
dictionaries under those are sent to the reporter to deal with.
Change-Id: I80d7539772e1485d5880132f22e55751b25ec198
We can more closely approximate Gerrit's behavior by using the
'resolve' git merge strategy. Make that the default, and leave
the previous behavior ('git merge') as an option. Also, finish
and correct the partially implemented plumbing for other merge
strategies (including cherry-pick).
(Note the previous unfinished implementation attempted to mimic
Gerrit's option names; the new implementation does not, but rather
documents the alignment. It's not a perfect translation anyway,
and this gives us more room to support other strategies not
currently supported by Gerrit).
Change-Id: Ie1ce4fde5980adf99bba69a5aa1d4e81026db676
Add the ability for Zuul to accept inputs from multiple trigger
sources simultaneously.
Pipelines are associated with exactly one trigger, which must now
be named in the configuration file.
Co-Authored-By: Monty Taylor <mordred@inaugust.com>
Change-Id: Ief2b31a7b8d85d30817f2747c1e2635f71ea24b9
Allow, eg, jobs in a gate pipeline to take precedence over
a check pipeline.
Change-Id: Idf91527704cc75b00a336291f91b908286f8e630
Reviewed-on: https://review.openstack.org/36552
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
On setup where Zuul ends up triggering hundreds of projects, you end up
having projects using roughly the same pipeline/jobs. Whenever one want
to add a job in all the similiar project, he has to edit each project
one by one.
To save some precious time, this patch introduces the concept of project
templates. It lets you define a set of pipeline and attached jobs
though the job names can be passed parameters defined on a per project
basis. Thus, updating similiar projects is all about editing a single
template.
A basic example is provided in the documentation.
The voluptuous schema has been updated. It does check whether all
parameters are properly passed to a template but does NOT check whether
the resulting job name exist.
The parameter expansion in templates is borrowed from Jenkins Job
Builder (deep_format function). It has been tweaked to also expand
dictionary keys.
Layout test plan:
$ nosetests -m layout --nocapture
Test layout file validation ...
<...>
bad_template1.yaml
required key not provided @
data['projects'][0]['template']['project']
bad_template2.yaml
extra keys not allowed @
data['projects'][0]['template']['extraparam']
good_template1.yaml
ok
<...>
$
A basic test hasbeen added to verify whether a project-template properly
triggers its tests:
$ nosetests --nocapture \
tests/test_scheduler.py:testScheduler.test_job_from_templates_launched
Test whether a job generated via a template can be launched ... ok
----------------------------------------------------------------------
Ran 1 test in 0.863s
OK
$
Change-Id: Ib82e4719331c204de87fbb4b20c198842b7e32f4
Reviewed-on: https://review.openstack.org/21881
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Add an additional job parameter, 'file', that will cause that
job to only run if the change touches files that match the
specification.
Change-Id: I8c8fd3d029e02e338fd1dd266443b9ac56c0e5ac
Reviewed-on: https://review.openstack.org/23710
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Voluptuous has a version 0.6 now which deprecates the any, match, and
required predicates and replaces them with Any, Match, and Required.
Update the minimum required version of Voluptuous and stop using the
deprecated predicates.
Change-Id: I554a6fda2c622a0b63f9247b1eae5eb494af225b
Reviewed-on: https://review.openstack.org/22167
Reviewed-by: Khai Do <zaro0508@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
When a new patchset is created for a change that is in a pipeline,
cancel running builds and dequeue that change (and possibly
dependent changes that can no longer merge).
Make this optional (and document the option).
Fixes bug 1022643.
Change-Id: I8f591956cf86645443e4b6075b8cdfc95a939e4f
Reviewed-on: https://review.openstack.org/20948
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
By setting parameters on a pipeline, you can change the default
"Build successful/failed" introductory messages left on Gerrit.
Change-Id: Ie05c8c87a47994faced4ee67fda4ab4e64ea0d20
Reviewed-on: https://review.openstack.org/19677
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Based on voluptuous library.
Basic validation should catch typos, missing or extra attributes.
Can be expanded to do more serious validation (ie, specifying
a comment in a trigger should require the event be comment-added).
Adds a command line option to validate a named layout file and
exit.
(Also add dist/ to .gitignore.)
Change-Id: Ia864ebde1765141d4d1a52bc77033689b6210e81
Reviewed-on: https://review.openstack.org/19443
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Clark Boylan
James E. Blair
Joshua Hesketh
Antoine Musso