James E. Blair
f9047557df
When using submitWholeTopic, it is possible to add changes to a dependency cycle without actually updating all of the involved changes. Yet, in Zuul, each of the involved changes needs to have a link to all the others. To accomplish this, when we update information about a change in zuul, we querry gerrit for changes that are "submitted together" with that change. That includes all the changes in the topic. Then we update the information for every change returned from that query in order to set up the reverse linkage. When we update a change, we perform 5 gerrit queries: * The change info itself * Related changes (git parent/child) * Files list * Reverve commit dependencies (Depends-On that point to this change) * Submitted together We also might perform more queries depending on Depends-On fooders we find. When we update the submitted together changes, we perform all of these queries for each of the other changes (but we keep track of the ones we've done already, so we don't do them more than once). That means in a set of 3 changes, we would perform 15 queries, But that's only what we would do if the changes already existed in their final state. In reality, changes are added to a topic one at a time. That means that we will get a gerrit event for every change (either a patchset-created event, or a topic-updated event). Every time we get one of these events, we repeat the process. That means for a set of 3 changes, we would perform 5 queries for the first event, 10 for the second, and 15 for the third, for a total of 30 queries. The total number of queries performed for any number of changes added in this manner is: queries = sum([5*i for i in range(1, count+1)]) count=100 gives 25250 queries For a large number of changes, this could be debilitating to Zuul (and possibly Gerrit). To reduce the impact of this, since we know that when a change is added to a topic, it is not necessary to update anything about the other changes in the topic other than their dependencies, we can avoid running some queries on the other changes in the topic. Ideally, we would avoid running any queries and just presume that a change that appears in the submitted-together list is there because it is a member of the topic, however, the submitted-together list also includes git parents. And if submitWholeTopic is not enabled, and two changes share a topic and a git parent-child relationship, the results would appear the same as if submitWholeTopic was enabled. It is not possible to determine whether the dependency should be one-way (parent-child) or circular (submitWholeTopic) by examining the results from only one change. We need to at least run the submitted-together query for every change in the result set to see if our change appears in the submitted-together results in the others. This change reduces the queries run in the way described. That means the total number of queries is: queries = 4*count + sum([1*i for i in range(1, count+1)]) count=100 gives 10500 queries Which is a significant improvement, especially for larger numbers of changes. In implementing this change, two related bugs were fixed: * The fake gerrit driver now only returns unmerged changes in the submitted-together query (as Gerrit does). * Some query history checks in the Gerrit driver used different values (some used change objects, some used (id, patchset) tuples which caused some history checks to fail. It failed safe, meaning that we performed unecessary queries. With the correction in this change we should perform fewer gerrit queries for complex change dependency topologies regardless of wehther they involve submitWholeTopic. Change-Id: Ie105fb2ecb3f48a9f2e2f97a608537eb601bc688
Zuul
Zuul is a project gating system.
The latest documentation for Zuul v3 is published at: https://zuul-ci.org/docs/zuul/
If you are looking for the Edge routing service named Zuul that is related to Netflix, it can be found here: https://github.com/Netflix/zuul
If you are looking for the Javascript testing tool named Zuul, it can be found here: https://github.com/defunctzombie/zuul
Getting Help
There are two Zuul-related mailing lists:
- zuul-announce
-
A low-traffic announcement-only list to which every Zuul operator or power-user should subscribe.
- zuul-discuss
-
General discussion about Zuul, including questions about how to use it, and future development.
You will also find Zuul developers on Matrix <https://matrix.to/#/#zuul:opendev.org>.
Contributing
To browse the latest code, see: https://opendev.org/zuul/zuul To clone the latest code, use git clone https://opendev.org/zuul/zuul
Bugs are handled at: https://storyboard.openstack.org/#!/project/zuul/zuul
Suspected security vulnerabilities are most appreciated if first reported privately following any of the supported mechanisms described at https://zuul-ci.org/docs/zuul/user/vulnerabilities.html
Code reviews are handled by gerrit at https://review.opendev.org
After creating a Gerrit account, use git review to submit patches. Example:
# Do your commits
$ git review
# Enter your username if promptedJoin us on Matrix to discuss development or usage.
License
Zuul is free software. Most of Zuul is licensed under the Apache License, version 2.0. Some parts of Zuul are licensed under the General Public License, version 3.0. Please see the license headers at the tops of individual source files.
Python Version Support
Zuul requires Python 3. It does not support Python 2.
Since Zuul uses Ansible to drive CI jobs, Zuul can run tests anywhere Ansible can, including Python 2 environments.