Merge branch 'stable-3.1'

* stable-3.1:
  Update replication plugin to get fixes for CI flakiness
  Use consistent capitalization for term NoteDb in documentation
  Hide refs/meta/config on git protocol v2
  GitProtocolV2IT: Split git client version check to a separate method
  CreateTag: Trim revision that is provided in input
  CreateTag: Allow revision in input to be empty
  RefUtil#parseBaseRevision: Do not log an error if baseRevision is invalid
  InvalidRevisionException: Include invalid revision into message
  CreateBranch: Trim revision that is provided in input
  CreateBranch: Allow revision in input to be empty
  CreateBranchIT: Add tests that set revision in the input
  Upgrade gitiles blame-cache to 0.2-7.1
  GitProtocolV2IT: Fail if git client is too old to support v2
  Document how to backup Gerrit
  Update rules_closure to latest version

Change-Id: Ic2efebe3a1379f5c7c642b152b4fec78e1cbcb47

Documentation/backup.txt

@@ -0,0 +1,270 @@
+= Gerrit Code Review - Backup
+
+A Gerrit Code Review site contains data that needs to be backed up regularly.
+This document describes best practices for backing up review data.
+
+[#mand-backup]
+== Data which must be backed up
+
+[#mand-backup-git]
+Git repositories::
++
+The bare Git repositories managed by Gerrit are typically stored in the
+`${SITE}/git` directory. However, the locations can be customized in
+`${site}/etc/gerrit.config`. They contain the history of the respective
+projects, and since 2.15 if you are using _NoteDb_, and for 3.0 and newer,
+also change and review metadata, user accounts and groups.
++
+
+[#mand-backup-db]
+SQL database::
++
+Gerrit releases in the 2.x series store some data in the database you
+have chosen when installing Gerrit. If you are using 2.16 and have
+migrated to _NoteDb_ only the schema version is stored in the database.
++
+If you are using h2 you need to backup the `.db` files in the folder
+`${SITE}/db`.
++
+For all other database types refer to their backup documentation.
++
+Gerrit release 3.0 and newer store all primary data in _NoteDb_ inside
+the git repositories of the Gerrit site. Only the review flag marking in
+the UI when you have reviewed a changed file is stored in a relational
+database. If you are using h2 this database is named
+`account_patch_reviews.h2.db`.
+
+[#optional-backup]
+== Data optional to be backed up
+
+[#data-optional-backup-index]
+Search index::
++
+The _Lucene_ search index is stored in the `${SITE}/index` folder.
+It can be recomputed from primary data in the git repositories but
+reindexing may take a long time hence backing up the index makes sense
+for production installations.
++
+If you have chosen to use _Elastic Search_ for indexing,
+refer to its
+link:https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html[backup documentation].
+
+[#optional-backup-cache]
+Caches::
++
+Gerrit uses many caches which populate automatically. Some of the caches
+are persisted in the directory `${SITE}/cache` to retain the cached data
+across restarts. Since repopulating persistent caches takes time and server
+resources it makes sense to include them in backups to avoid unnecessary
+higher load and degraded performance when a Gerrit site has been restored
+from backup and caches need to be repopulated.
+
+[#optional-backup-config]
+Configuration::
++
+Gerrit configuration files are located in the directory `${SITE}/etc`
+and should be backed up or versioned in a git repository. The `etc`
+directory also contains secrets which should be handled separately
++
+* `secure.config` contains passwords and `auth.registerEmailPrivateKey`
+* public and private SSH host keys
++
+You may consider to use the
+link:https://gerrit.googlesource.com/plugins/secure-config/[secure-config plugin]
+to encrypt these secrets.
+
+[#optional-backup-plugin-data]
+Plugin Data::
++
+The `${SITE}/data/` directory is used by plugins storing data like e.g.
+the delete-project and the replication plugin.
+
+[#optional-backup-libs]
+Libraries::
++
+The `${SITE}/lib/` directory contains libraries used as statically loaded
+plugin or providing additional dependencies needed by Gerrit plugins.
+
+[#optional-backup-plugins]
+Plugins::
++
+The `${SITE}/plugins/` directory contains the installed Gerrit plugins.
+
+[#optional-backup-static]
+Static Resources::
++
+The `${SITE}/static/` directory contains static resources used to customize the
+Gerrit UI and email templates.
+
+[#optional-backup-logs]
+Logs::
++
+The `${SITE}/logs/` directory contains Gerrit server log files. Logs can still
+be written when the server is in read-only mode.
+
+[#cons-backup]
+== Consistent backups
+
+There are several ways to ensure consistency when backing up primary data.
+
+[#cons-backup-snapshot]
+=== Filesystem snapshots
+
+Gerrit 3.0 or newer::
++
+* all primary data is stored in git
+* Use a file system like lvm, zfs, btrfs or nfs supporting snapshots.
+Create a snapshot and then archive the snapshot.
+
+Gerrit 2.x::
++
+Gerrit 2.16 can use _NoteDb_ to store almost all this data which
+simplifies creating backups since consistency between database and git
+repositories is no longer critical. If you migrated to _NoteDb_ you can
+follow the backup procedure for 3.0 and higher and additionally take
+a backup of the database, which only contains the schema version,
+hence consistency between git and database is no longer critical since
+the schema version only changes during upgrade. If you didn't migrate
+to _NoteDb_ then follow the backup procedure for older 2.x Gerrit versions.
++
+Older 2.x Gerrit versions store change meta data, review comments, votes,
+accounts and group information in a SQL database. Creating consistent backups
+where git repositories and the data stored in the database are backed up
+consistently requires to turn the server read-only or to shut it down
+while creating the backup since there is no integrated transaction handling
+between git repositories and the SQL database. Also crons and currently
+running cron jobs (e.g. repacking repositories) which affect the repositories
+may need to be shut down.
+Use a file system supporting snapshots to keep the period where the gerrit
+server is read-only or down as short as possible.
+
+[#cons-backup-read-only]
+=== Turn master read-only for backup
+
+Make the server read-only before taking the backup. This means read-access
+is still available during backup, because only write operations have to be
+stopped to ensure consistency. This can be implemented using the
+link:https://gerrit.googlesource.com/plugins/readonly/[_readonly_] plugin.
+
+[#cons-backup-replicate]
+=== Replicate data for backup
+
+Replicating the git repositories can backup the most critical repository data
+but does not backup repository meta-data such as the project description
+file, ref-logs, git configs, and alternate configs.
+
+Replicate all git repositories to another file system using
+`git clone --mirror`,
+or the
+link:https://gerrit.googlesource.com/plugins/replication[replication plugin]
+or the
+link:https://gerrit.googlesource.com/plugins/pull-replication[pull-replication plugin].
+Best you use a filesystem supporting snapshots to create a backup archive
+of such a replica.
+
+For 2.x Gerrit versions also set up a database slave for the data stored in the
+SQL database. If you are using 2.16 and migrated to _NoteDb_ you may consider to
+skip setting up a database slave, instead take a backup of the database which only
+contains the current schema version in this case.
+In addition you need to ensure that no write operations are in flight before you
+take the replica offline. Otherwise the database backup might be inconsistent
+with the backup of the git repositories.
+
+Do not skip backing up the replica, the replica alone IS NOT a backup.
+Imagine someone deleted a project by mistake and this deletion got replicated.
+Replication of repository deletions can be switched off using the
+link:https://gerrit.googlesource.com/plugins/replication/+/refs/heads/master/src/main/resources/Documentation/config.md[server option]
+`remote.NAME.replicateProjectDeletions`.
+
+If you are using Gerrit slaves to offload read traffic you can use one of these
+slaves for creating backups.
+
+[#cons-backup-offline]
+=== Take master offline for backup
+
+Shutdown the server before taking a backup. This is simple but means downtime
+for the users. Also crons and currently running cron jobs (e.g. repacking
+repositories) which affect the repositories may need to be shut down.
+
+[#backup-methods]
+== Backup methods
+
+[#backup-methods-snapshots]
+=== Filesystem snapshots
+
+Filesystems supporting copy on write snapshots::
++
+Use a file system supporting copy-on-write snapshots like
+link:https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Snapshots[btrfs]
+or
+https://wiki.debian.org/ZFS#Snapshots[zfs].
+
+
+Other filesystems supporting snapshots::
+https://wiki.archlinux.org/index.php/LVM#Snapshots[lvm] or nfs.
++
+Create a snapshot and then archive the snapshot to another storage.
++
+While snapshots are great for creating high quality backups quickly, they are
+not ideal as a format for storing backup data. Snapshots typically depend and
+reside on the same storage infrastructure as the original disk images.
+Therefore, it’s crucial that you archive these snapshots and store them
+elsewhere.
+
+3.0 or newer::
+Snapshot the complete site directory
+
+2.x::
+Similar, but the data of the database should be stored on the very same volume
+on the same machine, so that the snapshot is taken atomically over both
+the git data and the database data. Because everything should be ACID, it can safely
+crash-recover - as if the power has been plugged and the server got booted up again.
+(Actually more safe than that, because the filesystem knows about taking the snapshot,
+and also about the pending writes it can sync.)
+
+In addition to that, using filesystem snapshots allows to:
+
+* easy and fast roll back without having to access remote backup data (e.g. to restore
+accidental rm -rf git/ back in seconds).
+* incremental transfer of consistent snapshots
+* save a lot of data while still keeping multiple "known consistent states"
+
+[#backup-methods-other]
+=== Other backup methods
+
+To ensure consistent backups these backup methods require to turn the server into
+read-only mode while a backup is running.
+
+* create an archive like `tar.gz` to backup the site
+* `rsync`
+* plain old `cp`
+
+[#backup-methods-test]
+== Test backups
+
+Test backups and fire drill restoring backups to ensure the backups aren't
+corrupt or incomplete and you can restore a backup quickly.
+
+[#backup-dr]
+== Disaster recovery
+
+[#backup-dr-repl]
+=== Replicate backup archives
+
+To enable disaster recovery at least replicate backup archives to another data center.
+And fire drill restoring a new site using the backup.
+
+[#backup-dr-multi-site]
+=== Multi-site setup
+
+Use the https://gerrit.googlesource.com/plugins/multi-site[multi-site plugin]
+to install Gerrit with multiple sites installed in different datacenters
+across different regions. This ensures that in case of a severe problem with
+one of the sites, the other sites can still serve your repositories.
+
+GERRIT
+------
+Part of link:index.html[Gerrit Code Review]
+
+SEARCHBOX
+---------

Documentation/dev-bazel.txt

@@ -304,6 +304,12 @@ To exclude tests that require a Docker host:
   bazel test --test_tag_filters=-docker //...
 ----
 
+To exclude tests that require very recent git client version:
+
+----
+  bazel test --test_tag_filters=-git-protocol-v2 //...
+----
+
 To ignore cached test results:
 
 ----
@@ -324,6 +330,7 @@ The following values are currently supported for the group name:
 * edit
 * elastic
 * git
+* git-protocol-v2
 * notedb
 * pgm
 * rest

Documentation/install.txt

@@ -249,6 +249,11 @@ Place Gerrit plugins in the review_site/plugins directory to have them loaded on
 * http://www.kernel.org/pub/software/scm/git/docs/git-daemon.html[git-daemon]
 
 
+[[backup]]
+== Backup
+
+See the link:backup.html[backup documentation].
+
 GERRIT
 ------
 Part of link:index.html[Gerrit Code Review]

WORKSPACE

@@ -32,9 +32,9 @@ http_archive(
 
 http_archive(
     name = "io_bazel_rules_closure",
-    sha256 = "0409f8bd2a8b6fd1db289cdc0acb394dafd69f60a86d0169bc6495e648e01587",
-    strip_prefix = "rules_closure-18f8acf24ae0d03a9c3ee872ff91dcfbf383d69e",
-    urls = ["https://github.com/bazelbuild/rules_closure/archive/18f8acf24ae0d03a9c3ee872ff91dcfbf383d69e.tar.gz"],
+    sha256 = "03c3b16f205085817fd89cfdcb2220a0138647ee7992be9cef291b069dd90301",
+    strip_prefix = "rules_closure-196a45f0ede2faec11dcc6c60fbc5e7471f4bd58",
+    urls = ["https://github.com/bazelbuild/rules_closure/archive/196a45f0ede2faec11dcc6c60fbc5e7471f4bd58.tar.gz"],
 )
 
 # File is specific to Polymer and copied from the Closure Github -- should be

java/com/google/gerrit/server/git/PermissionAwareReadOnlyRefDatabase.java

@@ -32,6 +32,8 @@ import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
 import org.eclipse.jgit.annotations.NonNull;
 import org.eclipse.jgit.annotations.Nullable;
 import org.eclipse.jgit.lib.ObjectId;
@@ -105,11 +107,25 @@ public class PermissionAwareReadOnlyRefDatabase extends DelegateRefDatabase {
 
     Collection<Ref> result;
     try {
+      // The security filtering assumes to receive the same refMap, independently from the ref
+      // prefix offset
       result = forProject.filter(refs.values(), getDelegate(), RefFilterOptions.defaults());
     } catch (PermissionBackendException e) {
       throw new IOException("");
     }
-    return result.stream().collect(toMap(Ref::getName, r -> r));
+    return buildPrefixRefMap(prefix, result);
+  }
+
+  private Map<String, Ref> buildPrefixRefMap(String prefix, Collection<Ref> refs) {
+    int prefixSlashPos = prefix.lastIndexOf('/') + 1;
+    if (prefixSlashPos > 0) {
+      return refs.stream()
+          .collect(
+              Collectors.toMap(
+                  (Ref ref) -> ref.getName().substring(prefixSlashPos), Function.identity()));
+    }
+
+    return refs.stream().collect(toMap(Ref::getName, r -> r));
   }
 
   @Override

javatests/com/google/gerrit/integration/git/BUILD

@@ -3,5 +3,5 @@ load("//javatests/com/google/gerrit/acceptance:tests.bzl", "acceptance_tests")
 acceptance_tests(
     srcs = glob(["*IT.java"]),
     group = "git",
-    labels = ["git"],
+    labels = ["git-protocol-v2"],
 )

javatests/com/google/gerrit/integration/git/GitProtocolV2IT.java

@@ -15,7 +15,6 @@
 package com.google.gerrit.integration.git;
 
 import static com.google.common.truth.Truth.assertThat;
-import static com.google.common.truth.TruthJUnit.assume;
 import static com.google.gerrit.acceptance.testsuite.project.TestProjectUpdate.allow;
 import static com.google.gerrit.acceptance.testsuite.project.TestProjectUpdate.deny;
 import static java.nio.charset.StandardCharsets.UTF_8;
@@ -30,7 +29,10 @@ import com.google.gerrit.acceptance.TestAccount;
 import com.google.gerrit.acceptance.UseSsh;
 import com.google.gerrit.acceptance.testsuite.project.ProjectOperations;
 import com.google.gerrit.common.data.Permission;
+import com.google.gerrit.entities.Change;
+import com.google.gerrit.entities.PatchSet;
 import com.google.gerrit.entities.Project;
+import com.google.gerrit.entities.RefNames;
 import com.google.gerrit.extensions.api.GerritApi;
 import com.google.gerrit.extensions.common.ChangeInput;
 import com.google.gerrit.server.config.GerritServerConfig;
@@ -40,6 +42,7 @@ import java.io.File;
 import java.net.InetSocketAddress;
 import org.eclipse.jgit.lib.Config;
 import org.eclipse.jgit.lib.Constants;
+import org.junit.BeforeClass;
 import org.junit.Test;
 
 @UseSsh
@@ -48,6 +51,8 @@ public class GitProtocolV2IT extends StandaloneSiteTest {
       new String[] {"ssh-keygen", "-t", "rsa", "-q", "-P", "", "-f"};
   private final String[] GIT_LS_REMOTE =
       new String[] {"git", "-c", "protocol.version=2", "ls-remote", "-o", "trace=12345"};
+  private final String[] GIT_CLONE_MIRROR =
+      new String[] {"git", "-c", "protocol.version=2", "clone", "--mirror"};
   private final String GIT_SSH_COMMAND =
       "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i";
 
@@ -57,15 +62,19 @@ public class GitProtocolV2IT extends StandaloneSiteTest {
   @Inject private @TestSshServerAddress InetSocketAddress sshAddress;
   @Inject private @GerritServerConfig Config config;
 
-  @Test
-  public void testGitWireProtocolV2WithSsh() throws Exception {
+  @BeforeClass
+  public static void assertGitClientVersion() throws Exception {
     // Minimum required git-core version that supports wire protocol v2 is 2.18.0
     GitClientVersion requiredGitVersion = new GitClientVersion(2, 18, 0);
     GitClientVersion actualGitVersion =
-        new GitClientVersion(execute(ImmutableList.of("git", "version")));
-    // If not found, test succeeds with assumption violation
-    assume().that(actualGitVersion).isAtLeast(requiredGitVersion);
+        new GitClientVersion(execute(ImmutableList.of("git", "version"), new File("/")));
+    // If git client version cannot be updated, consider to skip this tests. Due to
+    // an existing issue in bazel, JUnit assumption violation feature cannot be used.
+    assertThat(actualGitVersion).isAtLeast(requiredGitVersion);
+  }
 
+  @Test
+  public void testGitWireProtocolV2WithSsh() throws Exception {
     try (ServerContext ctx = startServer()) {
       ctx.getInjector().injectMembers(this);
 
@@ -193,6 +202,61 @@ public class GitProtocolV2IT extends StandaloneSiteTest {
     }
   }
 
+  @Test
+  public void testGitWireProtocolV2HidesRefMetaConfig() throws Exception {
+    try (ServerContext ctx = startServer()) {
+      ctx.getInjector().injectMembers(this);
+      String url = config.getString("gerrit", null, "canonicalweburl");
+
+      // Create project
+      Project.NameKey allRefsVisibleProject = Project.nameKey("all-refs-visible");
+      gApi.projects().create(allRefsVisibleProject.get());
+
+      // Set protocol.version=2 in target repository
+      execute(
+          ImmutableList.of("git", "config", "protocol.version", "2"),
+          sitePaths
+              .site_path
+              .resolve("git")
+              .resolve(allRefsVisibleProject.get() + Constants.DOT_GIT)
+              .toFile());
+
+      // Set up project permission to allow reading all refs
+      projectOperations
+          .project(allRefsVisibleProject)
+          .forUpdate()
+          .add(allow(Permission.READ).ref("refs/heads/*").group(SystemGroupBackend.ANONYMOUS_USERS))
+          .add(
+              allow(Permission.READ)
+                  .ref("refs/changes/*")
+                  .group(SystemGroupBackend.ANONYMOUS_USERS))
+          .update();
+
+      // Create new change and retrieve refs for the created patch set
+      ChangeInput visibleChangeIn =
+          new ChangeInput(allRefsVisibleProject.get(), "master", "Test public change");
+      visibleChangeIn.newBranch = true;
+      int visibleChangeNumber = gApi.changes().create(visibleChangeIn).info()._number;
+      Change.Id changeId = Change.id(visibleChangeNumber);
+      String visibleChangeNumberRef = RefNames.patchSetRef(PatchSet.id(changeId, 1));
+      String visibleChangeNumberMetaRef = RefNames.changeMetaRef(changeId);
+
+      // Read refs from target repository using git wire protocol v2 over HTTP anonymously
+      String outAnonymousLsRemote =
+          execute(
+              ImmutableList.<String>builder()
+                  .add(GIT_CLONE_MIRROR)
+                  .add(url + "/" + allRefsVisibleProject.get())
+                  .build(),
+              ImmutableMap.of("GIT_TRACE_PACKET", "1"));
+
+      assertThat(outAnonymousLsRemote).contains("git< version 2");
+      assertThat(outAnonymousLsRemote).doesNotContain(RefNames.REFS_CONFIG);
+      assertThat(outAnonymousLsRemote).contains(visibleChangeNumberRef);
+      assertThat(outAnonymousLsRemote).contains(visibleChangeNumberMetaRef);
+    }
+  }
+
   private void setUpUserAuthentication(String username) throws Exception {
     // Assign HTTP password to user
     gApi.accounts().id(username).setHttpPassword("secret");