By setting parameters on a pipeline, you can change the default
"Build successful/failed" introductory messages left on Gerrit.
Change-Id: Ie05c8c87a47994faced4ee67fda4ab4e64ea0d20
Reviewed-on: https://review.openstack.org/19677
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Based on voluptuous library.
Basic validation should catch typos, missing or extra attributes.
Can be expanded to do more serious validation (ie, specifying
a comment in a trigger should require the event be comment-added).
Adds a command line option to validate a named layout file and
exit.
(Also add dist/ to .gitignore.)
Change-Id: Ia864ebde1765141d4d1a52bc77033689b6210e81
Reviewed-on: https://review.openstack.org/19443
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Instead, use queue task_done calls to indicate that the scheduler
has finished processing events. This lets the tests know when
the queues are both empty and all requests have been handled.
Add a lock around reporting complete events in fake jenkins jobs
so that waitUntilSettled can be assured that no new events will
arrive.
Directly report LOST builds when a job doesn't exist, rather
than spawning a new thread (which was only done to work around
the lock).
Change-Id: I32ad46648c82d7458fb5be779c62ac5b57857674
Reviewed-on: https://review.openstack.org/19330
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Zuul already treats LOST jobs as failures in much of the code. Do the
same in didAnyJobFail so that changes behind a change with a LOST job
get their jobs cancelled.
Change-Id: I1eb819a6cfb3b3ee0a2c131417b00dab1d83f29d
Reviewed-on: https://review.openstack.org/19270
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
These occur at critical points in the code and should be protected
with exception handlers.
Make sure that a build actually has a start time before calculating
its duration.
Change-Id: I1ff1260da0b7b5095c6b075bd0d78c7a1490686a
Reviewed-on: https://review.openstack.org/19023
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
When a build is reported as lost, it has no start_time, so the
code that calculated the run-time for statsd threw an exception.
Correct that by only calculating the run time for jobs that have
really completed.
Add debug lines to onBuild* that would have helped diagnose this.
Keep track of the throwaway threads that are used to report LOST
builds so that the test suite can more correctly know when the
system is settled.
Add a test case for a single non-existent post job.
Add STATSD_* env variables in tox to make sure that the statsd
code paths are executed by tests.
Change-Id: I339443cbad7b0858d70e9180d63dfb701b8eaee9
Reviewed-on: https://review.openstack.org/18792
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
Also, minor optimization to email filter matching.
Change-Id: Iedea0fc14b10a8f68ec52e38ceab4b2fb407e364
Reviewed-on: https://review.openstack.org/18515
Reviewed-by: Antoine Musso <hashar@free.fr>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Reports statistics on job runtime, number/rate of jobs run, and
number/rate of individual gerrit events to statsd.
Change-Id: I0db9352610b4c268c95da98fd282d7f463945f90
Reviewed-on: https://review.openstack.org/18628
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
So that status pages can easily link to gitweb urls of ref-update
triggered changes.
Change-Id: Id9b708a473d92c30727f66578a0174f18c657723
Reviewed-on: https://review.openstack.org/18602
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
Can be used for nifty ajax-style status pages.
Add optional description field to pipeline.
Change-Id: If5db3f6945f65f038833cbf9c783de5ffef63b49
Reviewed-on: https://review.openstack.org/18579
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Add the option to specify the URL pattern left in Gerrit, per-job.
Example: a job that builds documentation and uploads it to a staging
site can leave the URL of the staging site iff the job succeeds.
Change-Id: I34841e61e9116fd8d4de7ac09d8f9cfe36fe78ec
Reviewed-on: https://review.openstack.org/18264
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Some repositories might not have a master branch or could use a
different default HEAD. A git reset was made to remotes/master which
would produce a stacktrace.
Fix https://bugs.launchpad.net/zuul/+bug/1083154
Change-Id: Ie9390c0c5dd707d77fbfa8cff2078a9b31fd31ae
Reviewed-on: https://review.openstack.org/17239
Reviewed-by: Antoine Musso <hashar@free.fr>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
On some setup we might restrict a pipeline to trusted users. The new
email_filter will matches the email coming from the event account
attribute.
In Gerrit, each event hold the account in a different field
name so I have simply added a map to find out the correct field.
email_filter works just like comment_filter, fields are ORed and are
considered to be regex.
Change-Id: I775f67b48d5f162106c024f94fe498a049b3fe94
Reviewed-on: https://review.openstack.org/17609
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
The gerrit trigger took an sshkey parameter, but the merger,
which uses GitPython, did not honor it. Add a GIT_SSH wrapper
for the merger to use to support arbitrary ssh keys for remote
git operation.
Change-Id: I43f0830905ee7103a2acbfa4b4f90959254f60b2
Reviewed-on: https://review.openstack.org/17269
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Paul Belanger <paul.belanger@polybeacon.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
We're starting to use this elsewhere in OpenStack... avoids having to
do pyflakes workaround tricks.
Change-Id: Ic767677e87f30794d49baf100e3ded7555272df6
Reviewed-on: https://review.openstack.org/16851
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
If we put the code for zuul-server into the zuul module, then autodoc
will pick it up for documentation.
Change-Id: Idf2db82f706243c6e69984170b4783f1eece2237
Reviewed-on: https://review.openstack.org/16850
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Our code was expecting the Gerrit web interface to be available at the
root of the Gerrit server which is not always the case since people
could publish the Gerrit web interface under a subdirectory. In such
configuration, the existing assumption prevents Zuul from getting
references information.
This patch adds a new optional configuration variable 'baseurl' to the
'gerrit' section. It lets one override the default:
[gerrit]
server=gerrit.wikimedia.org
baseurl=https://gerrit.wikimedia.org/r
Updated Gerrit._getInfoRefs() to use 'baseurl' instead of 'server' when
creating the url to fetch references from. This is still falling back to
the old default whenever gerrit.baseurl is not set.
Change-Id: Ia9910acf46fe82c425911784412356598b3c7eac
Reviewed-on: https://review.openstack.org/16884
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
After successfully adding openstack versioning to jenkins-job-builder
this add the same support for zuul.
Change-Id: Ia5baab2b0d9392c1b3c70bf890eaf7c6a2ea5c29
Signed-off-by: Paul Belanger <paul.belanger@polybeacon.com>
Reviewed-on: https://review.openstack.org/16219
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
If a large change is fetched, it may arrive in only a pack file,
which tickles the invalid cache bug in git-python.
Fixes bug 1078946.
Change-Id: Ib912967e37c6aed906d86dc35a4b4a65e4f9c3bd
Reviewed-on: https://review.openstack.org/16199
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Change-Id: If1c43a2965c30e56b09a6cf85bcf8959c67d62c6
Reviewed-on: https://review.openstack.org/15927
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
GitPython when fetching refs may improperly read in git progress info
and interpret it as fetch info. When this happens an AssertionError is
thrown. However, since fetching seems to cache the ref locally
subsequent fetches work without running into the progress info problem.
So fetch again if an AssertionError is caught.
Upstream bugfix
https://github.com/gitpython-developers/GitPython/pull/42
Change-Id: I1768b836aade2853d4dd3b2f4cd1fce7ec035524
Reviewed-on: https://review.openstack.org/15219
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
The addition of ZUUL_COMMIT (commit 81515ad, change I88c38a28)
deprecated the UUID parameter in favor of ZUUL_UUID but did not honor
the new parameter.
This patch add ZUUL_UUID lookup in jenkins_endpoint() and
findBuildInQueue() with a fallback on UUID. Probably only needed on
jenkins_endpoint() when a Jenkins job lack the UUID parameter, but I
think we might as well add it in findBuildInQueue() just to be safe.
Also update the documentation to uses ZUUL_ prefixed variables where
relevant and added a note about GERRIT_PROJECT and UUID being
deprecated.
Change-Id: Ie0426f388cbd3a373af2110b7460aca6ff4f1751
Reviewed-on: https://review.openstack.org/14327
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
We always assume being able to get the references from a specific URL
although that might not be available (for example if Apache is not
properly setup). Logging an error with the URL we attempted to reach and
raising the urllib2 exception will give an obvious hint to the
administrator.
Change-Id: I0465f8ec0749f6f16d64ddb22dd2ad93dbc42c82
Reviewed-on: https://review.openstack.org/13815
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
To facilitate using zuul with just the jenkins git plugin, add
the ZUUL_COMMIT parameter. Also, always include ZUUL_REF, whether
the job is pre- or post-commit. If it's pre, it will be a
refs/zuul ref, if it's post, it will be refs/tags or a branch
name.
Change-Id: I88c38a28dcd552b2540095d36caacd10acf167b8
Reviewed-on: https://review.openstack.org/13934
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Change-Id: If46832e6c1d5538fad88c9503c248372cb5b9416
Reviewed-on: https://review.openstack.org/14096
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
In the case of ref deletion, the null sha is not very
unique, and could be present in another project. Make
sure to compare the project, ref, and sha to determine
if two ref-updated events are equal.
Change-Id: I6bd4a02f21bd31b91ef369783641ded014f7717c
Reviewed-on: https://review.openstack.org/13839
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
When commenting on Gerrit that jobs are beginning include a link to the
zuul status page if one is configured.
Change-Id: Iba5f315f4baa006939dbdcb07b0758e570242ca8
Reviewed-on: https://review.openstack.org/13858
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
I have been playing with zuul on port 8001 using curl:
curl http://127.0.0.1:8001/jenkins_endpoint
Had to `strace -f` the process to find out what was going wrong. Logging
the exception will simply make it easier to track possible issues coming
from the Jenkins Notification plugin..
Change-Id: I461146d0a6e7d63a0d8bedc3d4a6ad19e22548d6
Reviewed-on: https://review.openstack.org/13817
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Fixed all pep8 errors.
E125 is ignored because of false alarms.
Change-Id: I4da60409e0095c0896230cd01bda548ed2e3f741
Reviewed-on: https://review.openstack.org/13740
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
To make port setting of gerrit server in zuul.conf really work.
Change-Id: I6f44f21256391fd04e56c18c0b293d9767c5a6a9
Reviewed-on: https://review.openstack.org/13574
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Updated pep8 version requirement to 1.3.3
Fixed E127 errors
All ignores are to be removed in the next sequence of patches
Change-Id: Ia9e922b8873686a0f905f2548cc43d534ee1c912
Reviewed-on: https://review.openstack.org/13642
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Zhongyue Luo <zhongyue.nah@intel.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
This reverts commit afd39bb3cd
This change should be rebased against the dev branch instead and
submitted there. Once the production CI freeze ends immediately
following the OpenStack Folsom release, dev changes will be merged back
into master.
Change-Id: Iabc1a0b287f0a433ca297a0d5cd1e21dd670e1d3
Reviewed-on: https://review.openstack.org/13638
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Updated pep8 version requirement to 1.3.3
Fixed E127 errors
All ignores are to be removed in the next sequence of patches
Change-Id: I266498a2c64a215648a4a598b8cba5669049e37c
Reviewed-on: https://review.openstack.org/13384
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
Gerrit actually only speaks smart http, but we were assuming
dumb HTTP (which is provided in OpenStack's configuration).
Change-Id: Id299f53a6be6fc1670edf1da4b3353115bf1e31e
Reviewed-on: https://review.openstack.org/13146
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
When Zuul tried to launch a non-existent jenkins job, the path
to mark the job as LOST involves calling back into the scheduler
from the launcher while holding the queue lock. That deadlocks,
so instead, spawn a thread to report the job as lost.
Change-Id: I6a5661638e7edba1c56eb0b3d6384283b3ecc4ed
Reviewed-on: https://review.openstack.org/12889
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
The GitPython Repo object caches whether a given sha is held in an packed
or loose database. If the remote repository is repacked, that information
could be invalid. Therefore, each time the local repo is updated from the
remote, clear the object database cache.
See this file for the cache in action:
https://github.com/gitpython-developers/gitdb/blob/master/gitdb/db/base.py#L227
Change-Id: I547ba01d8cdc20f431e93e02736b498f744992c0
Reviewed-on: https://review.openstack.org/12597
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Change-Id: I6507896e9980eee4ac13ff8bd94f0e32947cd5d8
Reviewed-on: https://review.openstack.org/12279
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Move most of the change queue logic into the model. This should
be a little cleaner, and it should be easier to follow what
happens when a build is complete.
Run launch jobs over the whole queue.
Collapse addChange into the base manager.
Makes reading the logic around adding a change simpler. Several
hooks are provided for the dependent manager subclass to extend
the method around dependent changes.
Move onbuildcompleted and reporting methods to base class.
Simplifies the code path when a build is completed. Move to a
more generalized model where updates are applied to changes as
received, then search for changes ready to be reported, then
search for jobs that need to be launched.
Also,
* test check queue functionality
* assert all queues are empty at ends of tests
* move formatting methods to pipeline/queue classes
* add queue-only mode notification to status screen
Change-Id: I336ac289af6ebd23567ac54b359c9f38af7f2ac7
Reviewed-on: https://review.openstack.org/12277
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Fix a bug where change objects were being compared incorrectly
when determining whether or not a change was in the queue
after a failed change had been dequeued. Added a test-case
that simulates the real-world conditions that exposed the bug.
Change-Id: I94a7915353335d80ab42b6c10c19595cb27788ae
Reviewed-on: https://review.openstack.org/12078
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: James E. Blair <corvus@inaugust.com>
Change-Id: Iadd7e7f22fa31282d820bac12ba00c8f94cb8369
Reviewed-on: https://review.openstack.org/11941
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
If url_pattern is set in zuul.conf, use it instead of the build
url supplied by jenkins when reporting a change to gerrit.
Change-Id: Ic66b1efb5c7eae0dbbfd1927ee2827d13729b568
Reviewed-on: https://review.openstack.org/11908
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Change-Id: I5a9ccc857c28d458e3a8fbac03a9d29cc49b1da0
Reviewed-on: https://review.openstack.org/11904
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
When a change at the head of the queue fails, immediately remove
it from the queue and re-launch jobs behind it -- there's no
need to wait for it to finish.
Change-Id: I5d8650c1aa6d56090175a7b90a7baf327f6b8158
Reviewed-on: https://review.openstack.org/11825
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
If a series of dependent changes are in the queue and one of them
fails tests and is dequeued, dequeue any changes behind it that
depend on it since they can't merge.
Change-Id: Iffe7964c2e76bc1649f14b62bf4b3fc391a8c8cb
Reviewed-on: https://review.openstack.org/11820
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
When all builds complete for a change, launch the jobs for the change
behind the completed change to allow any builds that were being held
until completion of the current change to run.
Change-Id: I69401d0d4b4ecd96d36584c182e4b06eaa5ccfc6
Reviewed-on: https://review.openstack.org/11816
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
There may be a jenkins bug where a running build is not listed
as such in the api -- track these instances, and don't mark them
as lost without another indicator (build duration != 0).
Change-Id: Ib962e0274faf265c3a0d876a080b105804efa495
Reviewed-on: https://review.openstack.org/11773
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
So that the launcher can display/record useful information such as
pipeline, and included changes.
Change-Id: I8665c1d9df43030976f05ae7d7fe7529ccf56abf
Reviewed-on: https://review.openstack.org/11659
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
And the default -- so that Zuul doesn't need any special gerrit
privileges to run out of the box.
Change-Id: I36411b299c8363d509498ece8ea21f88665ca39a
Reviewed-on: https://review.openstack.org/11589
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
James E. Blair