opendev/gerrit
Code Issues Proposed changes

Selectively ignore errors from NoteDbUpdateManager

Browse Source 
As we know by now, NoteDb writes can fail for all sorts of reasons.
In the case of writes via BatchUpdate, from the user perspective we
don't actually care if the NoteDb update fails. If NoteDb reads are
enabled, we want the change to get rebuilt on the next read, but that
will happen regardless of whether or not BatchUpdate throws an error,
because we've already saved the data in the canonical location in
ReviewDb. Users prefer not to see errors, so just ignore them.

Another somewhat related issue is what to do about errors during the
auto-rebuild codepath during read. These have been cropping up
unpleasantly frequently due to the behavior of the change screen,
which sends a bunch of parallel requests all reading from the same
change entity. This is racy and tends to result in a noticeable
subset of requests getting LOCK_FAILURE and throwing. In this case,
the caller does actually care that the change got successfully rebuilt
after we detected it as stale--we can't just ignore the error. But it
doesn't actually matter that *this caller* was the one to rebuild the
change. We can just reread the NoteDbChangeState from ReviewDb and
compare it to the new value of NoteDb after an exception to see if it
got rebuilt behind our backs. Only go arm's-length here and retry
once; rebuilding in a loop seems risky.

Both of these cases are actually quite straightforward to fix; the
annoying thing is injecting a test ChangeRebuilder implementation
that's capable of doing the update behind the caller's back.

Change-Id: I0b80c97552cbdbfc6de44a785b41ad7ad8ab05b0
This commit is contained in:
Dave Borowitz
2016-06-02 15:13:41 -04:00
parent c7bb599d75
commit aa03d40c1f
8 changed files with 223 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
gerrit-acceptance-tests/src/test/java/com/google/gerrit/acceptance/server/notedb/ChangeRebuilderIT.java
View File

@@ -24,6 +24,7 @@ import static org.eclipse.jgit.lib.Constants.OBJ_BLOB;
import com.google.common.collect.ImmutableList;
import com.google.gerrit.acceptance.AbstractDaemonTest;
import com.google.gerrit.acceptance.AcceptanceTestRequestScope;
import com.google.gerrit.acceptance.GerritConfig;
import com.google.gerrit.acceptance.PushOneCommit;
import com.google.gerrit.acceptance.TestAccount;
import com.google.gerrit.common.TimeUtil;
@@ -50,6 +51,7 @@ import com.google.gerrit.server.git.RepoRefCache;
import com.google.gerrit.server.notedb.ChangeBundle;
import com.google.gerrit.server.notedb.ChangeNotes;
import com.google.gerrit.server.notedb.NoteDbChangeState;
import com.google.gerrit.server.notedb.TestChangeRebuilderWrapper;
import com.google.gerrit.server.schema.DisabledChangesReviewDbWrapper;
import com.google.gerrit.testutil.NoteDbChecker;
import com.google.gerrit.testutil.NoteDbMode;
@@ -92,6 +94,9 @@ public class ChangeRebuilderIT extends AbstractDaemonTest {
@Inject
private Provider<PostReview> postReview;
@Inject
private TestChangeRebuilderWrapper rebuilderWrapper;
@Before
public void setUp() {
assume().that(NoteDbMode.readWrite()).isFalse();
@@ -352,6 +357,36 @@ public class ChangeRebuilderIT extends AbstractDaemonTest {
assertThat(actual.differencesFrom(expected)).isEmpty();
}
@Test
@GerritConfig(name = "noteDb.testRebuilderWrapper", value = "true")
public void rebuildIgnoresErrorIfChangeIsUpToDateAfter() throws Exception {
setNotesMigration(true, true);
PushOneCommit.Result r = createChange();
Change.Id id = r.getPatchSetId().getParentKey();
assertChangeUpToDate(true, id);
// Make a ReviewDb change behind NoteDb's back and ensure it's detected.
setNotesMigration(false, false);
gApi.changes().id(id.get()).topic(name("a-topic"));
setInvalidNoteDbState(id);
assertChangeUpToDate(false, id);
// Force the next rebuild attempt to fail but also rebuild the change in the
// background.
rebuilderWrapper.stealNextUpdate();
setNotesMigration(true, true);
assertThat(gApi.changes().id(id.get()).info().topic)
.isEqualTo(name("a-topic"));
assertChangeUpToDate(true, id);
// Check that the bundles are equal.
ChangeBundle actual = ChangeBundle.fromNotes(
plcUtil, notesFactory.create(dbProvider.get(), project, id));
ChangeBundle expected = ChangeBundle.fromReviewDb(unwrapDb(), id);
assertThat(actual.differencesFrom(expected)).isEmpty();
}
@Test
public void rebuildAutomaticallyWhenDraftsOutOfDate() throws Exception {
setNotesMigration(true, true);