zuul/zuul
Tobias Henkel 1fab39cc4b
Parallelize github event processing
The GitHub driver does an event pre processing before adding a trigger
event to the scheduler. This is currently done single threaded to
ensure that the trigger events are enqueued in order into the
scheduler. A problem is that this pre processing can take a few
seconds which limits the rate of events we can process. In order to
parallelize this while keeping the order of the trigger events we need
to do two things.

We keep consuming the event queue single threaded but instead of
processing and forwarding them directly to the scheduler we process
them in a thread pool and put the futures into a result queue. This
second queue can then again be processed single threaded and maintains
the correct ordering of the events.

Second updating the change cache currently assumes that it runs single
threaded. In order to avoid data races we need to lock this by the
change.

Change-Id: I08f31b99e2a58e51cef0de89edd98c957d7db87f
2019-06-11 10:38:51 +02:00
..
ansible Merge "Get playbook data from extra_vars" 2019-05-20 06:20:04 +00:00
cmd Fix for yappi output 2019-04-17 16:16:39 -04:00
connection Annotate some logs in the scheduler with event id 2019-05-17 06:06:11 +02:00
driver Parallelize github event processing 2019-06-11 10:38:51 +02:00
execution_context Add wrapper driver execution context 2017-08-18 16:35:12 -07:00
executor Merge "Annotate logs around build states" 2019-06-11 05:51:17 +00:00
lib jsonutil: do not encode SourceContext or ZuulMark 2019-06-05 10:47:05 +00:00
manager Merge "Annotate logs around finished builds" 2019-06-11 06:33:05 +00:00
merger Annotate builds with event id 2019-05-30 19:18:00 +02:00
reporter Filter file comments for existing files 2018-10-25 06:21:01 -07:00
source Cache branches in connections/sources 2018-08-09 16:02:02 -07:00
sphinx Add zuul-sphinx as a requirement 2017-08-07 14:56:17 -07:00
trigger Annotate logs in the zuul driver with event ids 2019-05-17 06:06:11 +02:00
web Revert "Create zuul/web/static on demand" 2019-06-04 09:57:04 -07:00
__init__.py Initial commit. 2012-05-29 14:49:32 -07:00
_setup_hook.py Revert "Create zuul/web/static on demand" 2019-06-04 09:57:04 -07:00
change_matcher.py Fix wrong matched project template 2018-08-02 12:19:15 +02:00
configloader.py Support fail-fast in project pipelines 2019-04-29 08:26:58 +02:00
exceptions.py mqtt: add basic reporter 2018-05-02 07:20:46 +00:00
model.py Merge "Annotate logs around build completion and cancellation" 2019-06-10 20:07:08 +00:00
nodepool.py Annotate logs around finished builds 2019-05-30 19:21:31 +02:00
rpcclient.py Use gearman client keepalive 2018-12-11 21:28:59 +01:00
rpclistener.py web: remove SafeLoader left-over from ZuulJSONEncoder 2019-06-05 10:03:54 +00:00
scheduler.py Annotate logs around finished builds 2019-05-30 19:21:31 +02:00
version.py Report git sha in status page version 2018-04-27 10:21:43 -07:00
zk.py Ensure correct lexical sorting of node requests 2019-03-14 15:39:23 +01:00