When a new patchset is created for a change that is in a pipeline,
cancel running builds and dequeue that change (and possibly
dependent changes that can no longer merge).
Make this optional (and document the option).
Fixes bug 1022643.
Change-Id: I8f591956cf86645443e4b6075b8cdfc95a939e4f
Reviewed-on: https://review.openstack.org/20948
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
* Number of changes processed.
* Time each change spent in pipeline.
* Current number of changes in pipeline.
All of those per-pipeline, and per-pipeline, per-project.
Change-Id: I609ff33ab7c26fc43d323f8a0bbb61f93382af90
Reviewed-on: https://review.openstack.org/19764
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
By setting parameters on a pipeline, you can change the default
"Build successful/failed" introductory messages left on Gerrit.
Change-Id: Ie05c8c87a47994faced4ee67fda4ab4e64ea0d20
Reviewed-on: https://review.openstack.org/19677
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Based on voluptuous library.
Basic validation should catch typos, missing or extra attributes.
Can be expanded to do more serious validation (ie, specifying
a comment in a trigger should require the event be comment-added).
Adds a command line option to validate a named layout file and
exit.
(Also add dist/ to .gitignore.)
Change-Id: Ia864ebde1765141d4d1a52bc77033689b6210e81
Reviewed-on: https://review.openstack.org/19443
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Instead, use queue task_done calls to indicate that the scheduler
has finished processing events. This lets the tests know when
the queues are both empty and all requests have been handled.
Add a lock around reporting complete events in fake jenkins jobs
so that waitUntilSettled can be assured that no new events will
arrive.
Directly report LOST builds when a job doesn't exist, rather
than spawning a new thread (which was only done to work around
the lock).
Change-Id: I32ad46648c82d7458fb5be779c62ac5b57857674
Reviewed-on: https://review.openstack.org/19330
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
These occur at critical points in the code and should be protected
with exception handlers.
Make sure that a build actually has a start time before calculating
its duration.
Change-Id: I1ff1260da0b7b5095c6b075bd0d78c7a1490686a
Reviewed-on: https://review.openstack.org/19023
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
When a build is reported as lost, it has no start_time, so the
code that calculated the run-time for statsd threw an exception.
Correct that by only calculating the run time for jobs that have
really completed.
Add debug lines to onBuild* that would have helped diagnose this.
Keep track of the throwaway threads that are used to report LOST
builds so that the test suite can more correctly know when the
system is settled.
Add a test case for a single non-existent post job.
Add STATSD_* env variables in tox to make sure that the statsd
code paths are executed by tests.
Change-Id: I339443cbad7b0858d70e9180d63dfb701b8eaee9
Reviewed-on: https://review.openstack.org/18792
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
Reports statistics on job runtime, number/rate of jobs run, and
number/rate of individual gerrit events to statsd.
Change-Id: I0db9352610b4c268c95da98fd282d7f463945f90
Reviewed-on: https://review.openstack.org/18628
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Can be used for nifty ajax-style status pages.
Add optional description field to pipeline.
Change-Id: If5db3f6945f65f038833cbf9c783de5ffef63b49
Reviewed-on: https://review.openstack.org/18579
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Add the option to specify the URL pattern left in Gerrit, per-job.
Example: a job that builds documentation and uploads it to a staging
site can leave the URL of the staging site iff the job succeeds.
Change-Id: I34841e61e9116fd8d4de7ac09d8f9cfe36fe78ec
Reviewed-on: https://review.openstack.org/18264
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
On some setup we might restrict a pipeline to trusted users. The new
email_filter will matches the email coming from the event account
attribute.
In Gerrit, each event hold the account in a different field
name so I have simply added a map to find out the correct field.
email_filter works just like comment_filter, fields are ORed and are
considered to be regex.
Change-Id: I775f67b48d5f162106c024f94fe498a049b3fe94
Reviewed-on: https://review.openstack.org/17609
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
The gerrit trigger took an sshkey parameter, but the merger,
which uses GitPython, did not honor it. Add a GIT_SSH wrapper
for the merger to use to support arbitrary ssh keys for remote
git operation.
Change-Id: I43f0830905ee7103a2acbfa4b4f90959254f60b2
Reviewed-on: https://review.openstack.org/17269
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Paul Belanger <paul.belanger@polybeacon.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
To facilitate using zuul with just the jenkins git plugin, add
the ZUUL_COMMIT parameter. Also, always include ZUUL_REF, whether
the job is pre- or post-commit. If it's pre, it will be a
refs/zuul ref, if it's post, it will be refs/tags or a branch
name.
Change-Id: I88c38a28dcd552b2540095d36caacd10acf167b8
Reviewed-on: https://review.openstack.org/13934
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
When commenting on Gerrit that jobs are beginning include a link to the
zuul status page if one is configured.
Change-Id: Iba5f315f4baa006939dbdcb07b0758e570242ca8
Reviewed-on: https://review.openstack.org/13858
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Fixed all pep8 errors.
E125 is ignored because of false alarms.
Change-Id: I4da60409e0095c0896230cd01bda548ed2e3f741
Reviewed-on: https://review.openstack.org/13740
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
When Zuul tried to launch a non-existent jenkins job, the path
to mark the job as LOST involves calling back into the scheduler
from the launcher while holding the queue lock. That deadlocks,
so instead, spawn a thread to report the job as lost.
Change-Id: I6a5661638e7edba1c56eb0b3d6384283b3ecc4ed
Reviewed-on: https://review.openstack.org/12889
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Move most of the change queue logic into the model. This should
be a little cleaner, and it should be easier to follow what
happens when a build is complete.
Run launch jobs over the whole queue.
Collapse addChange into the base manager.
Makes reading the logic around adding a change simpler. Several
hooks are provided for the dependent manager subclass to extend
the method around dependent changes.
Move onbuildcompleted and reporting methods to base class.
Simplifies the code path when a build is completed. Move to a
more generalized model where updates are applied to changes as
received, then search for changes ready to be reported, then
search for jobs that need to be launched.
Also,
* test check queue functionality
* assert all queues are empty at ends of tests
* move formatting methods to pipeline/queue classes
* add queue-only mode notification to status screen
Change-Id: I336ac289af6ebd23567ac54b359c9f38af7f2ac7
Reviewed-on: https://review.openstack.org/12277
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Fix a bug where change objects were being compared incorrectly
when determining whether or not a change was in the queue
after a failed change had been dequeued. Added a test-case
that simulates the real-world conditions that exposed the bug.
Change-Id: I94a7915353335d80ab42b6c10c19595cb27788ae
Reviewed-on: https://review.openstack.org/12078
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: James E. Blair <corvus@inaugust.com>
Change-Id: Iadd7e7f22fa31282d820bac12ba00c8f94cb8369
Reviewed-on: https://review.openstack.org/11941
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
If url_pattern is set in zuul.conf, use it instead of the build
url supplied by jenkins when reporting a change to gerrit.
Change-Id: Ic66b1efb5c7eae0dbbfd1927ee2827d13729b568
Reviewed-on: https://review.openstack.org/11908
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Change-Id: I5a9ccc857c28d458e3a8fbac03a9d29cc49b1da0
Reviewed-on: https://review.openstack.org/11904
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
When a change at the head of the queue fails, immediately remove
it from the queue and re-launch jobs behind it -- there's no
need to wait for it to finish.
Change-Id: I5d8650c1aa6d56090175a7b90a7baf327f6b8158
Reviewed-on: https://review.openstack.org/11825
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
If a series of dependent changes are in the queue and one of them
fails tests and is dequeued, dequeue any changes behind it that
depend on it since they can't merge.
Change-Id: Iffe7964c2e76bc1649f14b62bf4b3fc391a8c8cb
Reviewed-on: https://review.openstack.org/11820
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
When all builds complete for a change, launch the jobs for the change
behind the completed change to allow any builds that were being held
until completion of the current change to run.
Change-Id: I69401d0d4b4ecd96d36584c182e4b06eaa5ccfc6
Reviewed-on: https://review.openstack.org/11816
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
So that the launcher can display/record useful information such as
pipeline, and included changes.
Change-Id: I8665c1d9df43030976f05ae7d7fe7529ccf56abf
Reviewed-on: https://review.openstack.org/11659
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
And the default -- so that Zuul doesn't need any special gerrit
privileges to run out of the box.
Change-Id: I36411b299c8363d509498ece8ea21f88665ca39a
Reviewed-on: https://review.openstack.org/11589
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Only merge changes if there is a refspec involved (not a ref-only change
like ref-updated). Also, only merge a change and launch jobs if a pipeline
actually supplies jobs for that change.
Add a test that ref-updated (eg, post) queues work.
Change-Id: Ic9d3089cc745aac21a6cf97a060a5181935abee9
Reviewed-on: https://review.openstack.org/11445
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
If Zuul is unable to merge a change, don't run any jobs, and report
the merge failure to gerrit directly (but still observing the
dependent change queue, in case a change ahead caused the merge
failure).
Adds a test for this situation.
Change-Id: I1ee2a8846b159db385019352cc04af2140db81af
Reviewed-on: https://review.openstack.org/11421
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Add git repo management and merging. When collecting changes to
be tested together, merge or cherry-pick those changes into the
zuul-managed repos, and create a unique ref for that configuration.
Pass the ref to Jenkins instead of the string description of the
changes, so that Jenkins only needs to checkout that one ref.
This moves the complexity of merging and managing multiple commits
out of Jenkins and into Zuul.
The GERRIT_CHANGES variable is deprecated (along with the rest of
the GERRIT_* variables) and will be removed in a future patch
(which will contain a documentation update).
Change-Id: I126c9030223c07a30f7092e2273ebd7605d9f3df
Reviewed-on: https://review.openstack.org/11349
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Change-Id: I03e9ec477b1d7493984fc6082bf30425ea8f5ac2
Reviewed-on: https://review.openstack.org/10981
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: James E. Blair <corvus@inaugust.com>
Change-Id: If14af44d13a0506501a88af0c12fca76862e1b8c
Reviewed-on: https://review.openstack.org/10974
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: James E. Blair <corvus@inaugust.com>
Change-Id: I4e8d5ca2a91595ba1ae3aaba4e64dd1b75665027
Reviewed-on: https://review.openstack.org/10972
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: James E. Blair <corvus@inaugust.com>
This is a refactoring of the data model with the following goals:
* Call top-level queues pipelines -- because too many other things
are already called queues. Pipelines convey the idea that there
are a number of tasks to be performed (jobs), and that those
tasks can be applied to different changes in parallel.
* Eliminate references to queue_name from within a Change.
Instead, methods that need to understand the pipeline that were
accessed previously via the change are now located in the
Pipeline class, taking a change as an argument. Essentially,
many methods involving changes (and builds, jobs, etc) must now
be called in the context of a pipeline.
* Add a changeish object to encompass the things that change and
ref events have in common.
Change-Id: Iaf8ed0991f3c5b2bf7ded2c340a60725f7f98eaf
Reviewed-on: https://review.openstack.org/10757
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
This fixes a potential race condition where a change may be reported
as merged in gerrit, but due to replication, the commit may not
yet show up in the repo. If a job is started during this time frame,
then the git repo will not be in the expected state for testing.
This now waits for the commit associated with the latest patchset
for the change to become the commit at ref/heads/branch. This
check obviously only works to verify that the merged commit is the
_most recently_ merged commit (ie, it does not perform the more
general check that the commit was merged at some time in the past).
That's okay here because zuul's merge commands are issued
synchronously from a single thread, so if zuul instructs gerrit
to merge a change, it should become that head, and no other changes
should be able to merge during that time.
Change-Id: I2213916c4850e7be80795cbed2dac29eacdb82bf
Reviewed-on: https://review.openstack.org/10673
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
* Test that patches are queued in gerrit dependency order.
* Test that we correctly decide when a change is able to merge.
Change-Id: Ib541314e7c956202a3f1f33a6b2185dd22f83f73
Reviewed-on: https://review.openstack.org/10633
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Add support for testing, and some initial functional tests.
Some threads in Zuul now have proper shutdown methods, since
that's needed for testing. The threads that are faked in the
testing infrastructure still lack shutdown methods.
Fix some bugs uncovered by tests:
Reverse the order of commits that are given to Jenkins (the
commits that preceded the one being tested were listed in
revers order, this corrects that).
Return 0 in reportChange if the change has already been
reported. The old return value of True was misinterpreted
as an error.
Change-Id: I4979d0efd8581061a2b856b892d4ecdb75284a1b
Reviewed-on: https://review.openstack.org/10572
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
For dependent change queues, test the changes in dependency order,
as reported by gerrit.
Effectively, this means that a change is enqueued only if:
* it can be merged
* the change it depends on has been merged or:
* can be merged
* is the current patchset
Also, once a change is enqueued, changes that depend on it are
examined to see if they can be merged (assuming this one succeeds).
If so, they are enqueued.
Change-Id: I917b2a2d1fc94c3aa2de406ed4b9f795a8abb079
Previously the change_ahead for the change_behind was set to None,
but then change_behind was not set to None as well. To properly
delete a change set both pointers to None.
Change-Id: If6d86759d3c10e88fca6e5f977456efd70bd0c32
Jobs for changes weren't being launched for DependentQueueManagers
because its addChange method called the superclass's addChange,
which had the same duplicate suppresion logic. This breaks
out the bulk of that into a private method that can be called
by both classes.
Change-Id: Ib47be82f255bab38ce03c7af7fb5b700e1956aaa
Jobs behind a failed change will need to be rerun so cancel them
to free up resources at the front of the change queue. When a change
at the front of the queue fails it is dequeued and all the jobs behind
it are rerun. When a change in the middle of the queue fails all the
jobs behind it are cancelled. Then the jobs will be rerun when a
change at the front of the queue is dequeued for failure (which may
end up being the change that caused the initial cancellation, or it
may be a change further up in the queue).
Change-Id: Ic0ebe15ec1a8d3a21ff04a6769243729683807ed
By setting 'hold-following-changes' to True on .*-merge jobs,
jobs for following changes in the dependend change queue won't be
launched until all of the -merge jobs ahead of them complete
successfully. This prevents wasting resources on testing changes
that have no chance of merging as tested. It gives up a small
amount of parallelization in order to achieve this (merge jobs will
now run in series, while other jobs may continue to run in parallel).
This should also help ensure that jobs nearer the front of the queue
run sooner than jobs at the back. Currently, the nondeterminism of
job run-time can cause Jenkins to use limited resources on the back
of the queue instead of the front.
Also fixes these bugs:
* The success/failure messages specified were not being used.
* If a parameter function was applied to a meta-job, it would not
be applied to jobs that matched the meta-job.
Change-Id: I03cc3e25c554d8aeb1ddb478afaed168a961962f
If A is a dependent change of B, and A and B finish test around
the same time, the following may happen:
Complete event for A is received.
A is marked as complete.
Complete event for B is received.
B is marked as complete.
Begin processing complete event for A
A is reported.
Since A merged, and B is dependent on A, zuul checks to see if
B should be reported (it may have been waiting on A).
B is complete, so it is reported.
End of processing A's complete event
Begin processing B's complete event
B is reported (but fails to report correctly).
To avoid this, record that a change has been succesfully reported,
(regardless of the outcome of tests), and if requested to report
the change again, simply return True, indicating that it has been
succesfully reported.
Change-Id: I5edbf158e7ef749cf855cadbacdc44161c054c59
Add a parameter-function attribute to jobs that specifies a function
that should be called to manipulate parameters passed to jenkins
before a job is launched.
Add the ability to include a python file to define such a function.
Finish implementing the "branch" attribute of jobs that lets you
specify whether a job should run on a particular branch.
Change-Id: I3f4d21ad5ac58a24d44a9a8437daa5c668967db9
A SIGUSR1 will cause zuul to queue new events, wait for existing
jobs to finish, then save the queue and exit.
It will likely take quite a while to complete (perhaps an hour),
so it's not implemented as a SIGTERM handler.
Can be used in an init script to implement a graceful restart.
Change-Id: I09fce571e971f16b5d20c5d69d595a05c7f6a4ba
James E. Blair