Zuul's new rate limiting feature would ignore jobs started that are
later moved outside the window (due to the window shrinking). Zuul
properly dealt with this jobs and changes when they slid back into the
window but it did so inefficiently. Go back to processing the entire
queue but check if each individual item is actionable before preparing
refs on it or starting jobs.
Change-Id: Ib76a68f9023652205003e0d164a78b8f67956adf
This was causing a problem with window sizes on reconfiguration because
the ChangeQueue objects were persisting across the reload via the local
reference inside of QueueItem. Instead of adding more complexity to
reset those on reEnqueue, drop that and instead find the change queue
via the change's project when needed.
Also fix the fact that the QueueItem pipeline reference was not being
updated (it was set to None before a re-enqueue but then not set to
the new pipeline value).
Change-Id: I7f7050bfec985972ad7a1bc89da02d7b0753b798
When changes report failure reduce the total number of changes that will
be tested concurrently in the pipeline queue the reported change
belonged too. Increase the number of changes that will be tested when
changes report success. This implements simple rate limiting which
should reduce resource thrash when tests are unstable.
Change-Id: Id092446c83649b3916751c4e4665d2adc75d0458
When doing a doPromote we should keep the enqueue times of items.
However, the teardown and rebuild of the queues means that items
are fully destroyed and created a new.
Change the addChange api call so that it takes an optional enqueue
time to set on the item, which it can do internally once the new
item has gotten created.
This should address the issue where enqueue_times get lost when
we do a promote of the queue.
Now with unit tests! (Also ensured that if I removed the scheduler
change these tests failed, so the test is testing a correct thing)
Change-Id: I4dce23d00128280c7add922ca9ef5018b57d1cf3
Importing gear now imports statsd which initializes a socket
before the fork for geard. Move the gear import to avoid that
until a better solution is worked out.
Change-Id: I7c1d9911a3b9de072b4784c4af9543baf1aefc27
Current implementation does not support running more than one test
sequentially in the same test runner. Move message storage to the
test object and create a factory method for FakeSMTP that links
the FakeSMTP object to the message storage.
Also, make the timeout non-gentle. Gentle didn't seem to work.
Change-Id: I1eae42ceac4deb6eb30affd1d84f3e79c97ca2f4
As it stands, if multiple pipelines
have timers with the same timespec, they will all respond to
the events generated for all of them.
Instead of matching on the timespec, associate a timer trigger
with a pipeline directly.
Change-Id: I6a799cc3b59bd7527ace9ee1048bf633dcaa4cd9
We want to report on "changes" triggered by a timer. Since those
aren't really changes, they are represented by a class called
NullChange which carries as much of the interface for a Change
object as possible (not much) so that they can be enqueued in
pipelines.
If a NullChange is reportable, then pretty much any kind of
change should be considered reportable. So remove the flag that
indicates that NullChanges and Refs are not reportable.
Only attempt to format a change report if there is an action
defined for that pipeline (in case the default formatting process
attempts to access a change attribute that is inappropriate).
Remove checks that try to avoid formatting or sending a report
based on attributes of the change (which are no longer relevant).
Add a test for using a timer trigger with an smtp reporter.
Add validation of the attributes that an smtp reporter can use in
the layout file.
Allow the operator to configure a Subject for smtp reports.
Change-Id: Icd067a7600c2922a318b9ade11b3946df4e53065
It'll help us to determine whether we're running a version of zuul that has
added support for some new feature.
The version_string() from pbr's version_info used to collect zuul version.
Change-Id: Id451f15538258ab2dec8e3e8f000cff4a8b7b20d
Currently we can filter pipeline triggers by email address but not
username. This is an issue for users that have no email addresses
such as Jenkins.
This patch adds a new filter "username_filter" to the gerrit trigger
section.
Change-Id: I66680ab7e9e5ff49466269175c8fb54aef30e016
Provide the short name of a project (anything after the last '/') to
project templates as the variable 'name'. If 'openstack/nova' invoked
a template, the variable 'name' would automatically be set to 'nova'
within the template.
Ideally this means that most template invocations in OpenStack's layout
will not need any variables defined.
Change-Id: I596744917c30c92041b8ea5b1f518d50fb64e59b
Accept multiple template invocations per project, and also allow
adding individual jobs to a project that uses templates.
Change-Id: I6c668dd434c12bec96b9a27afd9fd2eca7a11d0a
Takes one or more changes and promotes them to the head of the queue.
Also, change the command line syntax for the enqueue command to accept
change IDs in the form 'change,patchset' in order to match the syntax
of promote, as well as be potentially more compatible with future
triggers.
Change-Id: Ic7ded9587c68217c060328bf4c3518e32fe659e3
Create a new management event queue to handle external requests
for actions that are not related to triggers or results.
Use this for reconfiguration events. Subsequent patches will use
it for other kinds of events.
Change-Id: Ia018d9a7acc35aaf615ca85f75c9f4c630a5287f
Adds programoutput sphinx extension as a test dependency so doc
builds can include the program help text.
Change-Id: Iec2f09f710162614cbb393a5628204ddebe2e29f
When combining change queues, given 3 projects that were transitively
connected by shared jobs, depending on the order of processing, it
was possible for them not to be combined. To correct this, repeat
the combining operation until the resulting set can be combined no
further.
In order to make the test (and actual usage) behavior more deterministic,
the list of projects returned by the pipeline is now sorted by name.
A test is added for this.
Change-Id: If1386cad4118257efee9aa9918ad12a626927038
Add a command line client called 'zuul' that supports one command
to start with: 'enqueue'. It allows an operator (one with access
to the gearman server) to enqueue an arbitrary change in a specified
pipeline. It uses gearman to communicate with the Zuul server, which
now has an added RPC listener component to answer such requests via
gearman.
Add tests for the client RPC interface.
Raise an exception if a Gerrit query does not produce a change. Unlike
events from Gerrit, user (or admin) submitted events over the RPC bus
are more likely to reference invalid changes. To validate those, the
Gerrit trigger will raise an exception (and remove from its cache) changes
which prove to be invalid.
Change-Id: Ife07683a736c15f4db44a0f9881f3f71b78716b2
This patch adds the zuul_url option in zuul conf file
in order to pass ZUUL_URL to Jenkins, which will be used
by devstack-vm-gate-wrap.sh.
Documentation added in launchers.rst and zuul.rst
explaining how this new option works.
Change-Id: I840423cc06fdfdacd301d30be3e0b3e589e563e9
* zuul/merger.py: Zuul's merger.Repo is holding on to repo objects that
keep many file descriptors open. Turn all git.Repo objects into method
local references so that the git.Repo objects can be garbage collected
once these methods are popped off the stack. GCing these objects frees
the file descriptors associated with that repo. This adds overhead to
repo operations as each operation must do initial setup IO.
* tests/test_scheduler.py: Assert that no instances of git.Repo are
known to the python garbage collector. Do the assertion after running a
fully garbage collection run. This is a cheap check that repo objects
are being garbage collected properly at the end of each test.
Change-Id: I1d67981f32708a85af62ca622402de9fac0e1842
* zuul/merger.py: Previously zuul's merger only checked an initialized
flag to determine if a repo should be cloned. If the state of the disk
changes under zuul the repo may need to be recloned even if the
initialized flag is set. Check if the repo exists on disk to determine
if the repo should be cloned to deal with external state changes. Keep
initialized flag in order to preserve ability to change git repo
settings for user and email.
* tests/test_scheduler.py: Add a test that checks a repo is recloned
when removed from the filesystem under zuul.
* tests/fixtures/layout-repo-deleted.yaml: Layout fixture for new test
added above. This creates a new repo unused by any other tests so that
failures of this test do not interfere with other tests, they will run
independent of each other.
Change-Id: I14fb34a2916002cefef73e41ec9182a073d59ef3
ASCII art is fun but not savvy for project managers and directors. This
patch slightly enhance the 'gating' documentation with colored diagrams.
This is made possible via http://blockdiag.com/ by Takeshi KOMIYA who
even took the time to write a sphinx extension. The version dependency
is at the very least 0.5.5, but might be higher :/
Change-Id: Ibe3c2674a5dff2114c40a84ffdec8a8886b1b21b
When doing the layout validation, I ended up spammed with a few errors:
ERROR:zuul.Scheduler:Invalid reporter name gerrit
The issue is that my pipelines use 'gerrit' as reporter while it is not
registered when testing the config. I registered the 'smtp' reporter as
well and the error is gone.
Side effect: the layout validation output now dumps actions for
start/success/failure.
Change-Id: I271a2943fa3e846ae60d9b615cd3a1ac3815bb1b
Voluptuous 0.7.0 introduced a backward API incompatible change in
upstream commit 475adebc:
https://github.com/alecthomas/voluptuous/commit/475adebc
The schema are precompiled and the validate_X have been removed,
voluptuous is now smart enough to detect the type of value being
validated and would call an internal validation method matching the
type.
Commit is contained since 0.7.0:
$ git tag --contains 475adebc
0.7.0
0.7.1
0.7.2
0.8.1
0.8.2
$
I tested it using etc/layout.yaml-sample and added an inexistent
pipeline to the project. The test yield:
voluptuous.MultipleInvalid: extra keys not allowed @
data['projects'][0]['nonpipe']
Debian has recently accepted voluptuous 0.8.2 in testing, so if we want
to package Zuul, we better have to upgrade our voluptuous requirement as
well. Ref: http://packages.qa.debian.org/v/voluptuous.html
Change-Id: I117ea644863b2e4a4dc3429aa81e868573382877
Jenkins
Clark Boylan
James E. Blair
Sean Dague
Anita Kuno
Sergey Lukjanov
Joshua Hesketh
Paul Belanger
Arx Cruz
Antoine Musso
Michael Prokop