MergeOp: Multiply retry timeout by number of projects

RetryHelper uses a single Retryer for the entire operation, which in the
case of MergeOp may potentially include updating a large number of
repositories independently. This means that the first lock failure
encountered during MergeOp may have happened after eating up most or all
of the default 20s retry budget.

Work around this by increasing the retry timeout based on the total
number of projects in the ChangeSet.

Change-Id: Ie8f078d5691de56fd2bfc2946358f4a1f0930a06
This commit is contained in:
Dave Borowitz
2017-08-07 14:21:39 -04:00
parent c731351eaa
commit 3575a3771f
2 changed files with 10 additions and 1 deletions

View File

@@ -491,7 +491,12 @@ public class MergeOp implements AutoCloseable {
}
return null;
},
RetryHelper.options().listener(retryTracker).build());
RetryHelper.options()
.listener(retryTracker)
// Up to the entire submit operation is retried, including possibly many projects.
// Multiply the timeout by the number of projects we're actually attempting to submit.
.timeout(retryHelper.getDefaultTimeout().multipliedBy(cs.projects().size()))
.build());
if (projects > 1) {
topicMetrics.topicSubmissionsCompleted.increment();

View File

@@ -108,6 +108,10 @@ public class RetryHelper {
WaitStrategies.randomWait(50, MILLISECONDS));
}
public Duration getDefaultTimeout() {
return defaultTimeout;
}
public <T> T execute(Action<T> action) throws RestApiException, UpdateException {
return execute(action, defaults());
}