The Gatekeeper, or a project gating system
Go to file
Simon Westphahl 305d4dbab9
Handle dependency limit errors more gracefully
When the dependency graph exceeds the configured size we will raise an
exception. Currently we don't handle those exceptions and let them
bubble up to the pipeline processing loop in the scheduler.

When this happens during trigger event processing this is only aborting
the current pipeline handling run and the next scheduler will continue
processing the pipeline as usual.

However, in case where the item is already enqueued this exception can
block the pipeline processor and lead to a hanging pipeline:

ERROR zuul.Scheduler: Exception in pipeline processing:
Traceback (most recent call last):
  File "/opt/zuul/lib/python3.11/site-packages/zuul/scheduler.py", line 2370, in _process_pipeline
    while not self._stopped and pipeline.manager.processQueue():
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 1800, in processQueue
    item_changed, nnfi = self._processOneItem(
                         ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 1624, in _processOneItem
    self.getDependencyGraph(item.changes[0], dependency_graph, item.event,
  File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 822, in getDependencyGraph
    self.getDependencyGraph(needed_change, dependency_graph,
  File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 822, in getDependencyGraph
    self.getDependencyGraph(needed_change, dependency_graph,
  File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 822, in getDependencyGraph
    self.getDependencyGraph(needed_change, dependency_graph,
  [Previous line repeated 8 more times]
  File "/opt/zuul/lib/python3.11/site-packages/zuul/manager/__init__.py", line 813, in getDependencyGraph
    raise Exception("Dependency graph is too large")
Exception: Dependency graph is too large

To fix this, we'll handle the exception and remove the affected item.
We'll also handle the exception during enqueue and ignore the trigger
event in this case.

Change-Id: I210c5fa4c568f2bf03eedc18b3e9c9a022628dc3
2024-03-19 14:37:26 +01:00
doc Report topic to jobs as zuul.topic 2024-03-08 11:30:45 -07:00
etc Remove "sql connection" backwards compatability for database 2022-01-25 16:07:08 -08:00
playbooks Keep task stdout/stderr separate in result object 2023-08-17 16:22:41 -07:00
releasenotes/notes Report topic to jobs as zuul.topic 2024-03-08 11:30:45 -07:00
tests Handle dependency limit errors more gracefully 2024-03-19 14:37:26 +01:00
tools Add script to generate openapi spec 2024-03-09 11:25:40 -08:00
web Add script to generate openapi spec 2024-03-09 11:25:40 -08:00
zuul Handle dependency limit errors more gracefully 2024-03-19 14:37:26 +01:00
.coveragerc Revert "Revert "Switch to stestr"" 2018-05-17 08:33:40 -07:00
.dockerignore Add web/node_modules to dockerignore 2019-01-27 11:23:45 +01:00
.gitignore Add noxfile and switch to nox 2022-12-20 08:57:53 -08:00
.gitreview OpenDev Migration Patch 2019-04-19 19:25:28 +00:00
.mailmap Fix pep8 E127 violations 2012-09-26 14:23:10 +00:00
.stestr.conf Revert "Revert "Switch to stestr"" 2018-05-17 08:33:40 -07:00
.zuul.yaml Replace Ansible 6 with Ansible 9 2024-02-15 16:20:45 -08:00
COPYING Update README and add GPL license 2018-03-19 09:25:52 -07:00
Dockerfile Use bookworm container images 2023-08-28 10:13:53 -07:00
LICENSE Initial commit. 2012-05-29 14:49:32 -07:00
MANIFEST.in Optimize canMerge using graphql 2020-02-28 09:43:56 +01:00
README.rst Update README to point to Matrix, not IRC 2022-11-08 22:25:10 -08:00
TESTING.rst Update unit test container setup and instructions 2022-08-05 21:00:02 +00:00
bindep.txt Use Python 3.11 for stream-functional jobs 2023-07-29 08:54:33 -07:00
noxfile.py Reduce unit test concurrency 2024-02-15 16:18:51 -08:00
reno.yaml Add reno configuration settings 2020-07-22 08:45:46 -07:00
requirements.txt Pin kazoo to 2.9.0 2024-02-15 16:18:15 -08:00
setup.cfg Add a release note about Python 3.11 2023-07-27 17:09:54 -07:00
setup.py Partial sync with OpenStack requirements. 2013-09-25 15:30:37 -07:00
test-requirements.txt Add AWS Kinesis support 2023-07-25 11:04:19 -07:00
tox.ini Fix zk host env var for tests 2023-04-05 14:01:08 +02:00

README.rst

Zuul

Zuul is a project gating system.

The latest documentation for Zuul v3 is published at: https://zuul-ci.org/docs/zuul/

If you are looking for the Edge routing service named Zuul that is related to Netflix, it can be found here: https://github.com/Netflix/zuul

If you are looking for the Javascript testing tool named Zuul, it can be found here: https://github.com/defunctzombie/zuul

Getting Help

There are two Zuul-related mailing lists:

zuul-announce

A low-traffic announcement-only list to which every Zuul operator or power-user should subscribe.

zuul-discuss

General discussion about Zuul, including questions about how to use it, and future development.

You will also find Zuul developers on Matrix <https://matrix.to/#/#zuul:opendev.org>.

Contributing

To browse the latest code, see: https://opendev.org/zuul/zuul To clone the latest code, use git clone https://opendev.org/zuul/zuul

Bugs are handled at: https://storyboard.openstack.org/#!/project/zuul/zuul

Suspected security vulnerabilities are most appreciated if first reported privately following any of the supported mechanisms described at https://zuul-ci.org/docs/zuul/user/vulnerabilities.html

Code reviews are handled by gerrit at https://review.opendev.org

After creating a Gerrit account, use git review to submit patches. Example:

# Do your commits
$ git review
# Enter your username if prompted

Join us on Matrix to discuss development or usage.

License

Zuul is free software. Most of Zuul is licensed under the Apache License, version 2.0. Some parts of Zuul are licensed under the General Public License, version 3.0. Please see the license headers at the tops of individual source files.

Python Version Support

Zuul requires Python 3. It does not support Python 2.

Since Zuul uses Ansible to drive CI jobs, Zuul can run tests anywhere Ansible can, including Python 2 environments.