Add backend API and implementation for ported comments

When a reviewer leaves comments on a patchset, those comments stay
still attached to that patchset when the change owner uploads further
patchsets. Replies to such comments (even when newer patchsets are
available) are also attached to these older patchsets. This doesn't make
a difference for views which simply list comments like the "Comments"
tab or the "Change log". However, diff views only show comments of the
chosen patchsets. Hence, changes with comments scattered all over
different patchsets require quite some navigation. Sometimes, users
might even miss some of these comments if they don't go to the right
views.

Ported comments represent a solution for the described situation. A
ported comment is a comment which was left on an earlier patchset and
is shown on a later patchset. The idea is that users can simply go to
a diff view and see all comments which were left until that patchset.
The plan is to make the ported comments distinguishable from the actual
comments on the patchset. However, exact details will be left to the
frontend implementation.

To show the comments left until a patchset, the frontend will need to
ask the backend both for the actual comments and the ported comments
of that patchset (-> two separate REST API calls).

The logic which comments are ported is contained in the backend. The
backend always ports a complete comment thread. In the first version
of ported comments, we decided to exclude resolved comment threads
as showing all comments left on earlier patchsets might produce too much
unrelated noise for users. We can improve on that in later versions of
this feature. This change still also ports resolved comments as we'll
only add the relevant filtering in a follow-up change.

Comments on the parent commit of a patchset should also be ported.
Those comments require additional considerations and hence are also
deferred to a follow-up change.

As for other comment endpoints, we distinguish between ported published
comments and ported draft comments. The endpoint for the latter ones
is added in a follow-up change.

This change focuses on adding the new REST endpoint for ported
published comments along with its implementation and extensive tests.
Adding these detailed tests proved useful as we even noticed a bug in
Gerrit's previous code (see I75a65bd57). We also know from past
experience with diffs and edits due to rebase that only such extensive
tests help to prevent regressions for corner cases.

One major part of the implementation of the ported comments endpoint is
the mapping of the comment positions from the original patchset to the
target patchset. For edits due to rebase, we already have an algorithm
to map a diff hunk between two commits to a diff hunk between two other
commits. With I4062e6a22, we generalized this algorithm so that we can
re-use it to map a comment position in one commit to the corresponding
position in another commit. That algorithm currently drops any positions
which run into a mapping conflict. Deleted files are also not properly
supported. We'll adjust those aspects in follow-up changes.

The position transformation algorithm also requires the diff between
patchset commits as input. At the moment, we get that diff from the
existing diff caches. However, the current structure of the diff caches
makes this a very expensive operation as the computed diff encompasses
all files between the patchsets. This should improve when we have a new
diff cache structure in place which is currently in the making and
which allows to only retrieve the diffs of necessary files (= those on
which comments were left).

Summary of open aspects addressed in follow-up changes:
- Filtering of resolved comment threads
- Handling of comments on parent commits (= "left side comments")
  including special considerations for merge commits
- Separate endpoint for ported draft comments (implemented in
If8802a4f9)
- Correct handling for file deletions (implemented in If8ed1e511)
- Adjustment of transformation logic to not exclude transformed
  positions on conflicts and to map those to file comments instead
  (implemented in I99f80bee3)
- Making endpoint more robust by catching diff errors and omitting
  corresponding comments
- REST API documentation (done last when the final state is available)
For some of them, we already added tests which we can enable as soon
as we have the correct implementation in place.

Change-Id: I4e96af5acf428939a3a130c7ca6f6b74d40dab12

java/com/google/gerrit/extensions/api/changes/RevisionApi.java

@@ -106,6 +106,8 @@ public interface RevisionApi {
 
   List<RobotCommentInfo> robotCommentsAsList() throws RestApiException;
 
+  Map<String, List<CommentInfo>> portedComments() throws RestApiException;
+
   /**
    * Applies the indicated fix by creating a new change edit or integrating the fix with the
    * existing change edit. If no change edit exists before this call, the fix must refer to the
@@ -293,6 +295,11 @@ public interface RevisionApi {
       throw new NotImplementedException();
     }
 
+    @Override
+    public Map<String, List<CommentInfo>> portedComments() throws RestApiException {
+      throw new NotImplementedException();
+    }
+
     @Override
     public EditInfo applyFix(String fixId) throws RestApiException {
       throw new NotImplementedException();

java/com/google/gerrit/extensions/common/testing/CommentInfoSubject.java

@@ -27,6 +27,7 @@ import com.google.common.truth.Subject;
 import com.google.gerrit.extensions.client.Side;
 import com.google.gerrit.extensions.common.CommentInfo;
 import com.google.gerrit.truth.ListSubject;
+import java.sql.Timestamp;
 import java.util.List;
 
 public class CommentInfoSubject extends Subject {
@@ -91,6 +92,10 @@ public class CommentInfoSubject extends Subject {
     return check("inReplyTo").that(commentInfo().inReplyTo);
   }
 
+  public StringSubject commitId() {
+    return check("commitId").that(commentInfo().commitId);
+  }
+
   public AccountInfoSubject author() {
     return check("author").about(accounts()).that(commentInfo().author);
   }
@@ -99,6 +104,14 @@ public class CommentInfoSubject extends Subject {
     return check("tag").that(commentInfo().tag);
   }
 
+  public ComparableSubject<Timestamp> updated() {
+    return check("updated").that(commentInfo().updated);
+  }
+
+  public StringSubject changeMessageId() {
+    return check("changeMessageId").that(commentInfo().changeMessageId);
+  }
+
   private CommentInfo commentInfo() {
     isNotNull();
     return commentInfo;

java/com/google/gerrit/server/api/changes/RevisionApiImpl.java

@@ -79,6 +79,7 @@ import com.google.gerrit.server.restapi.change.GetMergeList;
 import com.google.gerrit.server.restapi.change.GetPatch;
 import com.google.gerrit.server.restapi.change.GetRelated;
 import com.google.gerrit.server.restapi.change.GetRevisionActions;
+import com.google.gerrit.server.restapi.change.ListPortedComments;
 import com.google.gerrit.server.restapi.change.ListRevisionComments;
 import com.google.gerrit.server.restapi.change.ListRevisionDrafts;
 import com.google.gerrit.server.restapi.change.ListRobotComments;
@@ -130,6 +131,7 @@ class RevisionApiImpl implements RevisionApi {
   private final FileApiImpl.Factory fileApi;
   private final ListRevisionComments listComments;
   private final ListRobotComments listRobotComments;
+  private final ListPortedComments listPortedComments;
   private final ApplyFix applyFix;
   private final GetFixPreview getFixPreview;
   private final Fixes fixes;
@@ -175,6 +177,7 @@ class RevisionApiImpl implements RevisionApi {
       FileApiImpl.Factory fileApi,
       ListRevisionComments listComments,
       ListRobotComments listRobotComments,
+      ListPortedComments listPortedComments,
       ApplyFix applyFix,
       GetFixPreview getFixPreview,
       Fixes fixes,
@@ -219,6 +222,7 @@ class RevisionApiImpl implements RevisionApi {
     this.listComments = listComments;
     this.robotComments = robotComments;
     this.listRobotComments = listRobotComments;
+    this.listPortedComments = listPortedComments;
     this.applyFix = applyFix;
     this.getFixPreview = getFixPreview;
     this.fixes = fixes;
@@ -453,6 +457,15 @@ class RevisionApiImpl implements RevisionApi {
     }
   }
 
+  @Override
+  public Map<String, List<CommentInfo>> portedComments() throws RestApiException {
+    try {
+      return listPortedComments.apply(revision).value();
+    } catch (Exception e) {
+      throw asRestApiException("Cannot retrieve ported comments", e);
+    }
+  }
+
   @Override
   public EditInfo applyFix(String fixId) throws RestApiException {
     try {

java/com/google/gerrit/server/restapi/change/ListPortedComments.java

@@ -0,0 +1,189 @@
+// Copyright (C) 2020 The Android Open Source Project
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package com.google.gerrit.server.restapi.change;
+
+import static com.google.common.collect.ImmutableList.toImmutableList;
+import static com.google.common.collect.ImmutableSet.toImmutableSet;
+import static java.util.stream.Collectors.groupingBy;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.gerrit.entities.Comment;
+import com.google.gerrit.entities.Comment.Range;
+import com.google.gerrit.entities.HumanComment;
+import com.google.gerrit.entities.PatchSet;
+import com.google.gerrit.entities.Project;
+import com.google.gerrit.extensions.client.DiffPreferencesInfo.Whitespace;
+import com.google.gerrit.extensions.common.CommentInfo;
+import com.google.gerrit.extensions.restapi.Response;
+import com.google.gerrit.extensions.restapi.RestReadView;
+import com.google.gerrit.server.CommentsUtil;
+import com.google.gerrit.server.change.RevisionResource;
+import com.google.gerrit.server.notedb.ChangeNotes;
+import com.google.gerrit.server.patch.DiffMappings;
+import com.google.gerrit.server.patch.GitPositionTransformer;
+import com.google.gerrit.server.patch.GitPositionTransformer.Mapping;
+import com.google.gerrit.server.patch.GitPositionTransformer.Position;
+import com.google.gerrit.server.patch.GitPositionTransformer.PositionedEntity;
+import com.google.gerrit.server.patch.PatchList;
+import com.google.gerrit.server.patch.PatchListCache;
+import com.google.gerrit.server.patch.PatchListKey;
+import com.google.gerrit.server.patch.PatchListNotAvailableException;
+import com.google.gerrit.server.permissions.PermissionBackendException;
+import com.google.inject.Inject;
+import com.google.inject.Provider;
+import com.google.inject.Singleton;
+import java.util.List;
+import java.util.Map;
+
+@Singleton
+public class ListPortedComments implements RestReadView<RevisionResource> {
+
+  private final CommentsUtil commentsUtil;
+  private final Provider<CommentJson> commentJson;
+  private final PatchListCache patchListCache;
+
+  @Inject
+  public ListPortedComments(
+      Provider<CommentJson> commentJson, CommentsUtil commentsUtil, PatchListCache patchListCache) {
+    this.commentJson = commentJson;
+    this.commentsUtil = commentsUtil;
+    this.patchListCache = patchListCache;
+  }
+
+  @Override
+  public Response<Map<String, List<CommentInfo>>> apply(RevisionResource revisionResource)
+      throws PermissionBackendException, PatchListNotAvailableException {
+    PatchSet targetPatchset = revisionResource.getPatchSet();
+
+    List<HumanComment> allComments = loadAllPublishedComments(revisionResource);
+    ImmutableList<HumanComment> relevantComments = filterToRelevant(allComments, targetPatchset);
+    ImmutableList<HumanComment> portedComments =
+        portToTargetPatchset(
+            revisionResource.getChangeResource().getNotes(), targetPatchset, relevantComments);
+    return Response.ok(format(portedComments));
+  }
+
+  private List<HumanComment> loadAllPublishedComments(RevisionResource revisionResource) {
+    return commentsUtil.publishedHumanCommentsByChange(revisionResource.getNotes());
+  }
+
+  private ImmutableList<HumanComment> filterToRelevant(
+      List<HumanComment> allComments, PatchSet targetPatchset) {
+    return allComments.stream()
+        .filter(comment -> comment.key.patchSetId < targetPatchset.number())
+        // TODO(aliceks): Also support comments on parent revisions.
+        .filter(comment -> comment.side > 0)
+        .collect(toImmutableList());
+  }
+
+  private ImmutableList<HumanComment> portToTargetPatchset(
+      ChangeNotes notes, PatchSet targetPatchset, List<HumanComment> comments)
+      throws PatchListNotAvailableException {
+    Map<Integer, ImmutableList<HumanComment>> commentsPerPatchset =
+        comments.stream().collect(groupingBy(comment -> comment.key.patchSetId, toImmutableList()));
+
+    ImmutableList.Builder<HumanComment> portedComments =
+        ImmutableList.builderWithExpectedSize(comments.size());
+    for (Integer originalPatchsetId : commentsPerPatchset.keySet()) {
+      ImmutableList<HumanComment> patchsetComments = commentsPerPatchset.get(originalPatchsetId);
+      PatchSet originalPatchset =
+          notes.getPatchSets().get(PatchSet.id(notes.getChangeId(), originalPatchsetId));
+      portedComments.addAll(
+          portToTargetPatchset(
+              notes.getProjectName(), originalPatchset, targetPatchset, patchsetComments));
+    }
+    return portedComments.build();
+  }
+
+  private ImmutableList<HumanComment> portToTargetPatchset(
+      Project.NameKey project,
+      PatchSet originalPatchset,
+      PatchSet targetPatchset,
+      ImmutableList<HumanComment> comments)
+      throws PatchListNotAvailableException {
+    ImmutableSet<Mapping> mappings =
+        loadPatchsetMappings(project, originalPatchset, targetPatchset);
+    ImmutableList<PositionedEntity<HumanComment>> positionedComments =
+        comments.stream().map(this::toPositionedEntity).collect(toImmutableList());
+    return GitPositionTransformer.transform(positionedComments, mappings).stream()
+        .map(PositionedEntity::getEntityAtUpdatedPosition)
+        .collect(toImmutableList());
+  }
+
+  private ImmutableSet<Mapping> loadPatchsetMappings(
+      Project.NameKey project, PatchSet originalPatchset, PatchSet targetPatchset)
+      throws PatchListNotAvailableException {
+    PatchList patchList =
+        patchListCache.get(
+            PatchListKey.againstCommit(
+                originalPatchset.commitId(), targetPatchset.commitId(), Whitespace.IGNORE_NONE),
+            project);
+    return patchList.getPatches().stream().map(DiffMappings::toMapping).collect(toImmutableSet());
+  }
+
+  private PositionedEntity<HumanComment> toPositionedEntity(HumanComment comment) {
+    return PositionedEntity.create(
+        comment,
+        ListPortedComments::extractPosition,
+        ListPortedComments::createCommentAtNewPosition);
+  }
+
+  private static Position extractPosition(HumanComment comment) {
+    Position.Builder positionBuilder = Position.builder().filePath(comment.key.filename);
+    return positionBuilder.lineRange(extractLineRange(comment)).build();
+  }
+
+  private static GitPositionTransformer.Range extractLineRange(HumanComment comment) {
+    // Line specifications in comment are 1-based. Line specifications in Position are 0-based.
+    if (comment.range != null) {
+      // The combination of (line, charOffset) is exclusive and must be mapped to an exclusive line.
+      int exclusiveEndLine =
+          comment.range.endChar > 0 ? comment.range.endLine : comment.range.endLine - 1;
+      return GitPositionTransformer.Range.create(comment.range.startLine - 1, exclusiveEndLine);
+    }
+    return GitPositionTransformer.Range.create(comment.lineNbr - 1, comment.lineNbr);
+  }
+
+  private static HumanComment createCommentAtNewPosition(
+      HumanComment originalComment, Position newPosition) {
+    HumanComment portedComment = new HumanComment(originalComment);
+    portedComment.key.filename = newPosition.filePath();
+    portedComment.lineNbr = newPosition.lineRange().start() + 1;
+    if (portedComment.range != null) {
+      portedComment.range =
+          toRange(
+              newPosition.lineRange(), portedComment.range.startChar, portedComment.range.endChar);
+      portedComment.lineNbr = portedComment.range.endLine;
+    }
+    return portedComment;
+  }
+
+  private static Comment.Range toRange(
+      GitPositionTransformer.Range lineRange, int originalStartChar, int originalEndChar) {
+    int adjustedEndLine = originalEndChar > 0 ? lineRange.end() : lineRange.end() + 1;
+    return new Range(lineRange.start() + 1, originalStartChar, adjustedEndLine, originalEndChar);
+  }
+
+  private Map<String, List<CommentInfo>> format(List<HumanComment> comments)
+      throws PermissionBackendException {
+    return commentJson
+        .get()
+        .setFillAccounts(true)
+        .setFillPatchSet(true)
+        .newHumanCommentFormatter()
+        .format(comments);
+  }
+}

java/com/google/gerrit/server/restapi/change/Module.java

@@ -172,6 +172,8 @@ public class Module extends RestApiModule {
     post(FIX_KIND, "apply").to(ApplyFix.class);
     get(FIX_KIND, "preview").to(GetFixPreview.class);
 
+    get(REVISION_KIND, "ported_comments").to(ListPortedComments.class);
+
     child(REVISION_KIND, "files").to(Files.class);
     put(FILE_KIND, "reviewed").to(PutReviewed.class);
     delete(FILE_KIND, "reviewed").to(DeleteReviewed.class);