The "building_jobs" dictionary on pipeline managers is no longer
necessary because the object data model is sufficient to find all
of the running builds. Remove the unnecessary second list.
Some of the protections that checked that builds were in this
dictionary are also no longer needed as it is not possible to
receive a result for a build that the running Zuul did not
request since the change to use Gearman.
Change-Id: I10de7edcf8356885f5f2d2f45d7014582bcbc3cf
Connect it to Zuul via Gearman. Any number of mergers may be
deployed.
Directly find the pipeline for a build when processing a result,
so that the procedure is roughly the same for build and merge
results.
The timer trigger currently requires the gerrit trigger also be
configured. Make that explicit inside of the timer trigger so
that the scheduler API interaction with triggers is cleaner.
Change-Id: I69498813764753c97c426e42d17596c2ef1d87cf
Instead of processing all of the queues each time any event happens,
first process all of the events (which are really queued up just so
that they can modify the pipeline data safely within the main run loop)
and then run the queue processor (which will then launch or cancel jobs
based on the new state of the queues).
This adds a lock to the run processor which is only needed so that
the test suite can safely inspect the state of the scheduler to
determine whether it is idle.
The result queue is made somewhat more generic in preparation for
handling merge result events.
Change-Id: I6bbb52f77b070df04a110a9d61b426265b1e89cc
In order to split out the merger as a new component, remove any
direct Zuul model object manipulation, instead passing in a
lists of dictionaries for merge instructions. Change the merger
algorithm so that it is compatible with this new method and makes
no assumptions about whether it has merged any changes previously.
There is likely to be a minor performance impact as some information
which was previously kept in-memory will now be fetched from the git
index.
The merger is also no-longer pre-populated with clones of git repos
at startup. Some tests were adjusted to accomodate this.
Change-Id: I3df86ae36b4969d568d1fb03df1e6569553d1226
This reverts commit 87650fa736.
The problem of distributing the load of serving Zuul Git refs will
be addressed by separating the merger component so that it can be
scaled out. Remove this feature which is hopefully new enough that
no one is using it.
Change-Id: Id2be95c85f5c3464a66537ae3095024a964ee1c0
This is in preparation to move to separate merger process, which
will not have access to the trigger. Similar functionality can
be obtained using the new replication feature, and docs have been
updated to that effect.
Change-Id: I9397f7eb8466af464c8e7adb02d0a8d3eff04f9f
argparse does not support optional subparser, hence the `zuul` client
command would always require to be passed a sub command. That prevented
`zuul --version` from showing up with a 'too few arguments' error.
I found out add_argument() has a action='version' which would print the
content of the 'version' argument and exit.
Change-Id: I553d3b2ab7aa04865e44559135adc5a7640307ed
In order to render queue window information nicely in status pages pass
the window for each queue in the JSON status blob. Also, annotate each
change in a queue with an active flag indicating whether or not it is
currently within the window.
Change-Id: Iccba9fb307f98a4fc145402ecd0a38f3d75b3253
This feature allows Zuul to consider existing (or new) approval
votes associated with a change when determining whether an event
matches. For example, it can be used to require that a Verified
vote of a certain age be present before a change is enqueued in
a pipeline.
Change-Id: I81344713d71b345b08576334568b9c49c810c7e9
Add layout.yaml documentation for the rate limit window configuration
that can be applied to DependentPipelineManagers.
Change-Id: I6ea0346fd9780f57f6b4e0becaa6b0beacd6ba1a
Zuul's new rate limiting feature would ignore jobs started that are
later moved outside the window (due to the window shrinking). Zuul
properly dealt with this jobs and changes when they slid back into the
window but it did so inefficiently. Go back to processing the entire
queue but check if each individual item is actionable before preparing
refs on it or starting jobs.
Change-Id: Ib76a68f9023652205003e0d164a78b8f67956adf
This was causing a problem with window sizes on reconfiguration because
the ChangeQueue objects were persisting across the reload via the local
reference inside of QueueItem. Instead of adding more complexity to
reset those on reEnqueue, drop that and instead find the change queue
via the change's project when needed.
Also fix the fact that the QueueItem pipeline reference was not being
updated (it was set to None before a re-enqueue but then not set to
the new pipeline value).
Change-Id: I7f7050bfec985972ad7a1bc89da02d7b0753b798
When changes report failure reduce the total number of changes that will
be tested concurrently in the pipeline queue the reported change
belonged too. Increase the number of changes that will be tested when
changes report success. This implements simple rate limiting which
should reduce resource thrash when tests are unstable.
Change-Id: Id092446c83649b3916751c4e4665d2adc75d0458
The Sphinx documentation now make use of the `program-output` plugin
which would invoke commands to generate inline documentation. Ex:
program-output:: zuul --help
program-output:: zuul enqueue --help
We want the resulting output to correspond to Zuul source code, not the
command which is currently installed on the host running the doc. On my
setup sphinx would die out because it cant find the command 'zuul'.
The new tox environement 'doc' runs sphinx in a virtual env which will
have the proper zuul command.
The generated doc is not written under the /.tox directory but to
/doc/build/html for convenience.
Example usage:
tox -edoc && open doc/build/html/index.html
Change-Id: Ib0170f94bb2c09eb60e555a32e101e2e0959b18e
When doing a doPromote we should keep the enqueue times of items.
However, the teardown and rebuild of the queues means that items
are fully destroyed and created a new.
Change the addChange api call so that it takes an optional enqueue
time to set on the item, which it can do internally once the new
item has gotten created.
This should address the issue where enqueue_times get lost when
we do a promote of the queue.
Now with unit tests! (Also ensured that if I removed the scheduler
change these tests failed, so the test is testing a correct thing)
Change-Id: I4dce23d00128280c7add922ca9ef5018b57d1cf3
* ms timestamp collected each time Scheduler.reconfigure() completed
* last reconfigured timestamp reported through the status.json
* it could be useful to determine when zuul conf reloaded
Change-Id: I03c5a5734f2127ef40be9ec512c983b136508be7
Importing gear now imports statsd which initializes a socket
before the fork for geard. Move the gear import to avoid that
until a better solution is worked out.
Change-Id: I7c1d9911a3b9de072b4784c4af9543baf1aefc27
Current implementation does not support running more than one test
sequentially in the same test runner. Move message storage to the
test object and create a factory method for FakeSMTP that links
the FakeSMTP object to the message storage.
Also, make the timeout non-gentle. Gentle didn't seem to work.
Change-Id: I1eae42ceac4deb6eb30affd1d84f3e79c97ca2f4
As it stands, if multiple pipelines
have timers with the same timespec, they will all respond to
the events generated for all of them.
Instead of matching on the timespec, associate a timer trigger
with a pipeline directly.
Change-Id: I6a799cc3b59bd7527ace9ee1048bf633dcaa4cd9
We want to report on "changes" triggered by a timer. Since those
aren't really changes, they are represented by a class called
NullChange which carries as much of the interface for a Change
object as possible (not much) so that they can be enqueued in
pipelines.
If a NullChange is reportable, then pretty much any kind of
change should be considered reportable. So remove the flag that
indicates that NullChanges and Refs are not reportable.
Only attempt to format a change report if there is an action
defined for that pipeline (in case the default formatting process
attempts to access a change attribute that is inappropriate).
Remove checks that try to avoid formatting or sending a report
based on attributes of the change (which are no longer relevant).
Add a test for using a timer trigger with an smtp reporter.
Add validation of the attributes that an smtp reporter can use in
the layout file.
Allow the operator to configure a Subject for smtp reports.
Change-Id: Icd067a7600c2922a318b9ade11b3946df4e53065
It'll help us to determine whether we're running a version of zuul that has
added support for some new feature.
The version_string() from pbr's version_info used to collect zuul version.
Change-Id: Id451f15538258ab2dec8e3e8f000cff4a8b7b20d
Currently we can filter pipeline triggers by email address but not
username. This is an issue for users that have no email addresses
such as Jenkins.
This patch adds a new filter "username_filter" to the gerrit trigger
section.
Change-Id: I66680ab7e9e5ff49466269175c8fb54aef30e016
James E. Blair
Jenkins
Antoine Musso
Clark Boylan
Sean Dague
Sergey Lukjanov
Anita Kuno
Joshua Hesketh