opendev/gerrit
Code Issues Proposed changes

Store in index the state of all refs that contribute to a change

Browse Source 
The primary storage (ReviewDb or NoteDb) and the secondary index
(Lucene, etc.) are not updated atomically, so they can get out of sync
in various ways. For example, the server may crash before writing to
the index, or index updates for concurrent primary storage writes may
happen out of order. We want to be able to detect when this has
happened.

One component of the state of a change is the row version field in the
ReviewDb entity; this is already stored in the index in the serialized
Change entity. The remainder of a change's state comprises various
refs:

 * NoteDb refs, if NoteDb is the primary storage:
    * Change meta ref.
    * Draft comment refs.
    * Robot comment ref.
 * Edit refs in the project repo.
 * Star refs in All-Users.

Store each of these in the index in a field REF_STATE, with the simple
format "<project>:<ref>:<sha>".

In addition to storing the refs that *are* present, we also need to
keep track of which refs are *not* present. For example, if only one
user is listed in the EDITBY field, then there must exist an edit ref
for that user and only that user, and no other users. This is
represented by storing a list of "<project>:<pattern>" values in the
REF_STATE_PATTERN field. A change is up to date only if all refs in
each <project> matching each <pattern> are also listed explicitly in
REF_STATE. The pattern format is a slightly extended git refspec
pattern which allows arbitrarily many '*'s, to deal with the tricky
case of "refs/users/UU/UUUU/edit-C/P" => "refs/users/*/edit-C/*".

This change implements the REF_STATE and REF_STATE_PATTERN change
fields, which require some additional ChangeData methods, as well as
structured data types for these fields. These data types live in a new
class StalenessChecker, which will be later extended to use them to
check for staleness.

Change-Id: I622dbbb334b5aca4d58f770ad453e034a5bc9529
This commit is contained in:
Dave Borowitz
2016-11-22 08:49:53 -05:00
parent 47fee2a156
commit 7d0e67c6ec
10 changed files with 509 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
gerrit-elasticsearch/src/main/java/com/google/gerrit/elasticsearch/ElasticChangeIndex.java
View File

@@ -23,6 +23,7 @@ import static org.apache.commons.codec.binary.Base64.decodeBase64;
import com.google.common.collect.FluentIterable;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Iterables;
import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
import com.google.gerrit.elasticsearch.ElasticMapping.MappingProperties;
@@ -398,6 +399,21 @@ class ElasticChangeIndex extends AbstractElasticIndex<Change.Id, ChangeData>
ChangeField.STORED_SUBMIT_RECORD_LENIENT.getName(),
ChangeField.SUBMIT_RULE_OPTIONS_LENIENT, cd);
if (source.get(ChangeField.REF_STATE.getName()) != null) {
JsonArray refStates =
source.get(ChangeField.REF_STATE.getName()).getAsJsonArray();
cd.setRefStates(
Iterables.transform(
refStates, e -> Base64.decodeBase64(e.getAsString())));
}
if (source.get(ChangeField.REF_STATE_PATTERN.getName()) != null) {
JsonArray refStatePatterns = source.get(
ChangeField.REF_STATE_PATTERN.getName()).getAsJsonArray();
cd.setRefStatePatterns(
Iterables.transform(
refStatePatterns, e -> Base64.decodeBase64(e.getAsString())));
}
return cd;
}