When using submitWholeTopic, it is possible to add changes to a
dependency cycle without actually updating all of the involved
changes. Yet, in Zuul, each of the involved changes needs to
have a link to all the others. To accomplish this, when we
update information about a change in zuul, we querry gerrit for
changes that are "submitted together" with that change. That
includes all the changes in the topic. Then we update the
information for every change returned from that query in order to
set up the reverse linkage.
When we update a change, we perform 5 gerrit queries:
* The change info itself
* Related changes (git parent/child)
* Files list
* Reverve commit dependencies (Depends-On that point to this change)
* Submitted together
We also might perform more queries depending on Depends-On fooders we find.
When we update the submitted together changes, we perform all of
these queries for each of the other changes (but we keep track of
the ones we've done already, so we don't do them more than once).
That means in a set of 3 changes, we would perform 15 queries,
But that's only what we would do if the changes already existed in
their final state. In reality, changes are added to a topic one at a
time. That means that we will get a gerrit event for every change
(either a patchset-created event, or a topic-updated event). Every
time we get one of these events, we repeat the process.
That means for a set of 3 changes, we would perform 5 queries for the
first event, 10 for the second, and 15 for the third, for a total of
30 queries. The total number of queries performed for any number of
changes added in this manner is:
queries = sum([5*i for i in range(1, count+1)])
count=100 gives 25250 queries
For a large number of changes, this could be debilitating to Zuul (and
possibly Gerrit).
To reduce the impact of this, since we know that when a change is
added to a topic, it is not necessary to update anything about the
other changes in the topic other than their dependencies, we can avoid
running some queries on the other changes in the topic.
Ideally, we would avoid running any queries and just presume that a
change that appears in the submitted-together list is there because it
is a member of the topic, however, the submitted-together list also
includes git parents. And if submitWholeTopic is not enabled, and two
changes share a topic and a git parent-child relationship, the results
would appear the same as if submitWholeTopic was enabled. It is not
possible to determine whether the dependency should be one-way
(parent-child) or circular (submitWholeTopic) by examining the results
from only one change. We need to at least run the submitted-together
query for every change in the result set to see if our change appears
in the submitted-together results in the others.
This change reduces the queries run in the way described. That means
the total number of queries is:
queries = 4*count + sum([1*i for i in range(1, count+1)])
count=100 gives 10500 queries
Which is a significant improvement, especially for larger numbers of
changes.
In implementing this change, two related bugs were fixed:
* The fake gerrit driver now only returns unmerged changes in the
submitted-together query (as Gerrit does).
* Some query history checks in the Gerrit driver used different values
(some used change objects, some used (id, patchset) tuples which
caused some history checks to fail. It failed safe, meaning that we
performed unecessary queries. With the correction in this change we
should perform fewer gerrit queries for complex change dependency
topologies regardless of wehther they involve submitWholeTopic.
Change-Id: Ie105fb2ecb3f48a9f2e2f97a608537eb601bc688
f9047557df