MergeSuperSet: Expand topic first, then complete change set without topic

For computing the changes that should be submitted together with
submitWholeTopic enabled we iteratively discover changes to be submitted
by:

1. including predecessor changes
2. expanding the topics

This change swaps this order to:

1. expand the topics
2. include predecessor changes

In a follow-up change the second step (include predecessor changes)
should become an extension point so that in a multi-master setup a
plugin can decide to distribute the load for computing the predecessor
changes across multiple nodes.

In the multi-master setup at Google the load balancer distributes
requests to different nodes based on the project in the URL. This means
not all nodes have to load all repositories into cache which improves
performance a lot. However we still have performance problems with the
Submitted Together REST endpoint and for large topics which span several
hundreds of projects loading the list of changes that should be
submitted together can take several minutes. The problem is that the
request to the Submitted Together REST endpoint is not specific to one
project, but to include the predecessor change we have to open the
repository for all projects that are touched in the topics. This means
the one node node that handles the request has to open several hundreds
of repositories which is slow and also polutes the repository caches. To
fix this we want to implement the extension point for computing
predecessor changes in such a way that it distributes the load over
multiple nodes that each have the necessary repositories already cached.
These nodes can do the computation in parallel which should speed up the
operation further.

In this context it makes sense to swap the order of the computation
steps. The first step (expand the topics) doesn't require opening any
repository since the topic is resolved via the change index and hence
this step can be executed locally on the node that receives the request.
This means we first do locally everything we can before distributing the
work to compute predecessor changes across multiple nodes.

Change-Id: I8ea6fc18a7cf94e16282fa53507c725d8aaae8f9
Signed-off-by: Edwin Kempin <ekempin@google.com>
This commit is contained in:
Edwin Kempin
2017-10-10 13:45:45 +02:00
parent 14c3be0d4d
commit 375c96f2ff

View File

@@ -358,8 +358,8 @@ public class MergeSuperSet {
do {
oldSeen = seen;
changes = completeChangeSetWithoutTopic(db, changes, user);
changes = topicClosure(db, changes, user, topicsSeen, visibleTopicsSeen);
changes = completeChangeSetWithoutTopic(db, changes, user);
seen = topicsSeen.size() + visibleTopicsSeen.size();
} while (seen != oldSeen);