Migrate external IDs to NoteDb (part 1)

In NoteDb external IDs are stored in the All-Users repository in a Git
Notes branch called refs/meta/external-ids where the sha1 of the
external ID is used as note name. Each note content is a Git config
file that contains an external ID. It has exactly one externalId
subsection with an accountId and optionally email and password:

  [externalId "username:jdoe"]
     accountId = 1003407
     email = jdoe@example.com
     password = bcrypt:4:LCbmSBDivK/hhGVQMfkDpA==:XcWn0pKYSVU/UJgOvhidkEtmqCp6oKB7

Storing the external IDs in a Git Notes branch with using the sha1 of
the external ID as note name ensures that external IDs are unique and
are only assigned to a single account. If it is tried to assign the
same external ID concurrently to different accounts, only one Git
update succeeds while the other Git updates fail with LOCK_FAILURE.
This means assigning external IDs is also safe in a multimaster setup
if a consensus algorithm for updating Git refs is implemented (which
is needed for multimaster in any case). Alternatively it was
considered to store the external IDs per account as Git config file in
the refs/users/<sharded-id> user branches in the All-Users repository
(see abandoned change 9f9f07ef). This approach was given up because in
race conditions it allowed to assign the same external ID to different
accounts by updating different branches in Git.

To support a live migration on a multi-master Gerrit installation, the
migration of external IDs from ReviewDb to NoteDb is done in 2 steps:

- part 1 (this change):
  * always write to both backends (ReviewDb and NoteDb)
  * always read external IDs from ReviewDb
  * upgraded instances write to both backends, old instances only
    write to ReviewDb
  * after upgrading all instances (all still read from ReviewDb)
    run a batch to copy all external IDs from the ReviewDb to NoteDb
- part 2 (next change):
  * bump the database schema version
  * migrate the external IDs from ReviewDb to NoteDb (for single instance
    Gerrit servers)
  * read external IDs from NoteDb
  * delete the database table

With this change reading external IDs from NoteDb is not implemented
yet. This is because the storage format of external IDs in NoteDb
doesn't support efficient lookup of external IDs by account and this
problem is only addressed in the follow-up change (it adds a cache for
external IDs, but this cache uses the revision of the notes branch as
key, and hence can be only implemented once the external IDs are fully
migrated to NoteDb and storing external IDs in ReviewDb is dropped).

The ExternalIdsUpdate class implements updating of external IDs in
both NoteDb and ReviewDb. It provides various methods to update
external IDs (e.g. insert, upsert, delete, replace). For NoteDb each
method invocation leads to one commit in the Git notes branch.
ExternalIdsUpdate has two factories, User and Server. This allows to
record either the calling user or the Gerrit server identity as
committer for an update of the external Ids.

External IDs are now represented by a new AutoValue class called
ExternalId. This class replaces the usage of the old gwtorm entity
AccountExternalId class. For ExternalId scheme names are the same as for
AccountExternalId but no longer include the trailing ':'.

The class ExternalIdsOnInit makes it possible to update external IDs
during the init phase. This is required for inserting external IDs for
the initial admin user which is created by InitAdminUser. We need a
special class for this since not all dependencies of ExternalIdsUpdate
are available during init.

The class ExternalIdsBatchUpdate allows to do batch updates to
external IDs. For NoteDb all updates will result in a single commit to
the refs/meta/external-ids Git notes branch.

LocalUsernamesToLowerCase is now always converting the usernames in a
single thread only. This allows us to get a single commit for the
username convertion in NoteDb (this would not be possible if workers
do updates in parallel). Since LocalUsernamesToLowerCase is rather
light-weight being able to parallelize work is not really needed and
removing the workers simplifies the code significantly.

To protect the refs/meta/external-ids Git notes branch in the All-Users
repository read access for this ref is only allowed to users that have
the 'Access Database' global capability assigned. In addition
there is a commit validator that disallows updating the
refs/meta/external-ids branch by push. This is to prevent that the
external IDs in NoteDb diverge from the external IDs in ReviewDb while
the migration to NoteDb is not fully done yet.

Change-Id: Ic9bd5791e84ee8d332ccb1f709970b59ee66b308
Signed-off-by: Edwin Kempin <ekempin@google.com>

gerrit-httpd/src/main/java/com/google/gerrit/httpd/CacheBasedWebSession.java

@@ -22,12 +22,12 @@ import com.google.gerrit.common.data.HostPageData;
 import com.google.gerrit.httpd.WebSessionManager.Key;
 import com.google.gerrit.httpd.WebSessionManager.Val;
 import com.google.gerrit.reviewdb.client.Account;
-import com.google.gerrit.reviewdb.client.AccountExternalId;
 import com.google.gerrit.server.AccessPath;
 import com.google.gerrit.server.AnonymousUser;
 import com.google.gerrit.server.CurrentUser;
 import com.google.gerrit.server.IdentifiedUser;
 import com.google.gerrit.server.account.AuthResult;
+import com.google.gerrit.server.account.ExternalId;
 import com.google.gerrit.server.config.AuthConfig;
 import com.google.inject.Provider;
 import com.google.inject.servlet.RequestScoped;
@@ -132,7 +132,7 @@ public abstract class CacheBasedWebSession implements WebSession {
   }
 
   @Override
-  public AccountExternalId.Key getLastLoginExternalId() {
+  public ExternalId.Key getLastLoginExternalId() {
     return val != null ? val.getExternalId() : null;
   }
 
@@ -149,9 +149,9 @@ public abstract class CacheBasedWebSession implements WebSession {
   }
 
   @Override
-  public void login(final AuthResult res, final boolean rememberMe) {
-    final Account.Id id = res.getAccountId();
-    final AccountExternalId.Key identity = res.getExternalId();
+  public void login(AuthResult res, boolean rememberMe) {
+    Account.Id id = res.getAccountId();
+    ExternalId.Key identity = res.getExternalId();
 
     if (val != null) {
       manager.destroy(key);

gerrit-httpd/src/main/java/com/google/gerrit/httpd/WebSession.java

@@ -16,10 +16,10 @@ package com.google.gerrit.httpd;
 
 import com.google.gerrit.common.Nullable;
 import com.google.gerrit.reviewdb.client.Account;
-import com.google.gerrit.reviewdb.client.AccountExternalId;
 import com.google.gerrit.server.AccessPath;
 import com.google.gerrit.server.CurrentUser;
 import com.google.gerrit.server.account.AuthResult;
+import com.google.gerrit.server.account.ExternalId;
 
 public interface WebSession {
   boolean isSignedIn();
@@ -29,7 +29,7 @@ public interface WebSession {
 
   boolean isValidXGerritAuth(String keyIn);
 
-  AccountExternalId.Key getLastLoginExternalId();
+  ExternalId.Key getLastLoginExternalId();
 
   CurrentUser getUser();
 

gerrit-httpd/src/main/java/com/google/gerrit/httpd/WebSessionManager.java

@@ -30,7 +30,7 @@ import static java.util.concurrent.TimeUnit.SECONDS;
 
 import com.google.common.cache.Cache;
 import com.google.gerrit.reviewdb.client.Account;
-import com.google.gerrit.reviewdb.client.AccountExternalId;
+import com.google.gerrit.server.account.ExternalId;
 import com.google.gerrit.server.config.ConfigUtil;
 import com.google.gerrit.server.config.GerritServerConfig;
 import com.google.inject.Inject;
@@ -98,18 +98,18 @@ public class WebSessionManager {
     }
   }
 
-  Val createVal(final Key key, final Val val) {
-    final Account.Id who = val.getAccountId();
-    final boolean remember = val.isPersistentCookie();
-    final AccountExternalId.Key lastLogin = val.getExternalId();
+  Val createVal(Key key, Val val) {
+    Account.Id who = val.getAccountId();
+    boolean remember = val.isPersistentCookie();
+    ExternalId.Key lastLogin = val.getExternalId();
     return createVal(key, who, remember, lastLogin, val.sessionId, val.auth);
   }
 
   Val createVal(
-      final Key key,
-      final Account.Id who,
-      final boolean remember,
-      final AccountExternalId.Key lastLogin,
+      Key key,
+      Account.Id who,
+      boolean remember,
+      ExternalId.Key lastLogin,
       String sid,
       String auth) {
     // Refresh the cookie every hour or when it is half-expired.
@@ -191,19 +191,19 @@ public class WebSessionManager {
     private transient Account.Id accountId;
     private transient long refreshCookieAt;
     private transient boolean persistentCookie;
-    private transient AccountExternalId.Key externalId;
+    private transient ExternalId.Key externalId;
     private transient long expiresAt;
     private transient String sessionId;
     private transient String auth;
 
     Val(
-        final Account.Id accountId,
-        final long refreshCookieAt,
-        final boolean persistentCookie,
-        final AccountExternalId.Key externalId,
-        final long expiresAt,
-        final String sessionId,
-        final String auth) {
+        Account.Id accountId,
+        long refreshCookieAt,
+        boolean persistentCookie,
+        ExternalId.Key externalId,
+        long expiresAt,
+        String sessionId,
+        String auth) {
       this.accountId = accountId;
       this.refreshCookieAt = refreshCookieAt;
       this.persistentCookie = persistentCookie;
@@ -221,7 +221,7 @@ public class WebSessionManager {
       return accountId;
     }
 
-    AccountExternalId.Key getExternalId() {
+    ExternalId.Key getExternalId() {
       return externalId;
     }
 
@@ -253,7 +253,7 @@ public class WebSessionManager {
 
       if (externalId != null) {
         writeVarInt32(out, 4);
-        writeString(out, externalId.get());
+        writeString(out, externalId.toString());
       }
 
       if (sessionId != null) {
@@ -289,7 +289,7 @@ public class WebSessionManager {
             persistentCookie = readVarInt32(in) != 0;
             continue;
           case 4:
-            externalId = new AccountExternalId.Key(readString(in));
+            externalId = ExternalId.Key.parse(readString(in));
             continue;
           case 5:
             sessionId = readString(in);

gerrit-httpd/src/main/java/com/google/gerrit/httpd/auth/become/BecomeAnyAccountLoginServlet.java

@@ -14,7 +14,8 @@
 
 package com.google.gerrit.httpd.auth.become;
 
-import static com.google.gerrit.reviewdb.client.AccountExternalId.SCHEME_USERNAME;
+import static com.google.gerrit.server.account.ExternalId.SCHEME_USERNAME;
+import static com.google.gerrit.server.account.ExternalId.SCHEME_UUID;
 
 import com.google.gerrit.common.PageLinks;
 import com.google.gerrit.extensions.registration.DynamicItem;
@@ -23,13 +24,13 @@ import com.google.gerrit.httpd.LoginUrlToken;
 import com.google.gerrit.httpd.WebSession;
 import com.google.gerrit.httpd.template.SiteHeaderFooter;
 import com.google.gerrit.reviewdb.client.Account;
-import com.google.gerrit.reviewdb.client.AccountExternalId;
 import com.google.gerrit.reviewdb.server.ReviewDb;
 import com.google.gerrit.server.account.AccountException;
 import com.google.gerrit.server.account.AccountManager;
 import com.google.gerrit.server.account.AccountState;
 import com.google.gerrit.server.account.AuthRequest;
 import com.google.gerrit.server.account.AuthResult;
+import com.google.gerrit.server.account.ExternalId;
 import com.google.gerrit.server.query.account.InternalAccountQuery;
 import com.google.gwtexpui.server.CacheHeaders;
 import com.google.gwtorm.server.OrmException;
@@ -179,17 +180,16 @@ class BecomeAnyAccountLoginServlet extends HttpServlet {
     return null;
   }
 
-  private AuthResult auth(final AccountExternalId account) {
+  private AuthResult auth(Account.Id account) {
     if (account != null) {
-      return new AuthResult(account.getAccountId(), null, false);
+      return new AuthResult(account, null, false);
     }
     return null;
   }
 
   private AuthResult byUserName(final String userName) {
     try {
-      AccountExternalId.Key extKey = new AccountExternalId.Key(SCHEME_USERNAME, userName);
-      List<AccountState> accountStates = accountQuery.byExternalId(extKey.get());
+      List<AccountState> accountStates = accountQuery.byExternalId(SCHEME_USERNAME, userName);
       if (accountStates.isEmpty()) {
         getServletContext().log("No accounts with username " + userName + " found");
         return null;
@@ -198,7 +198,7 @@ class BecomeAnyAccountLoginServlet extends HttpServlet {
         getServletContext().log("Multiple accounts with username " + userName + " found");
         return null;
       }
-      return auth(new AccountExternalId(accountStates.get(0).getAccount().getId(), extKey));
+      return auth(accountStates.get(0).getAccount().getId());
     } catch (OrmException e) {
       getServletContext().log("cannot query account index", e);
       return null;
@@ -231,9 +231,9 @@ class BecomeAnyAccountLoginServlet extends HttpServlet {
   }
 
   private AuthResult create() throws IOException {
-    String fakeId = AccountExternalId.SCHEME_UUID + UUID.randomUUID();
     try {
-      return accountManager.authenticate(new AuthRequest(fakeId));
+      return accountManager.authenticate(
+          new AuthRequest(ExternalId.Key.create(SCHEME_UUID, UUID.randomUUID().toString())));
     } catch (AccountException e) {
       getServletContext().log("cannot create new account", e);
       return null;

gerrit-httpd/src/main/java/com/google/gerrit/httpd/auth/container/HttpAuthFilter.java

@@ -17,7 +17,7 @@ package com.google.gerrit.httpd.auth.container;
 import static com.google.common.base.MoreObjects.firstNonNull;
 import static com.google.common.base.Strings.emptyToNull;
 import static com.google.common.net.HttpHeaders.AUTHORIZATION;
-import static com.google.gerrit.reviewdb.client.AccountExternalId.SCHEME_GERRIT;
+import static com.google.gerrit.server.account.ExternalId.SCHEME_GERRIT;
 import static java.nio.charset.StandardCharsets.ISO_8859_1;
 import static java.nio.charset.StandardCharsets.UTF_8;
 
@@ -26,7 +26,7 @@ import com.google.gerrit.httpd.HtmlDomUtil;
 import com.google.gerrit.httpd.RemoteUserUtil;
 import com.google.gerrit.httpd.WebSession;
 import com.google.gerrit.httpd.raw.HostPageServlet;
-import com.google.gerrit.reviewdb.client.AccountExternalId;
+import com.google.gerrit.server.account.ExternalId;
 import com.google.gerrit.server.config.AuthConfig;
 import com.google.gwtexpui.server.CacheHeaders;
 import com.google.gwtjsonrpc.server.RPCServletUtils;
@@ -127,8 +127,8 @@ class HttpAuthFilter implements Filter {
   }
 
   private static boolean correctUser(String user, WebSession session) {
-    AccountExternalId.Key id = session.getLastLoginExternalId();
-    return id != null && id.equals(new AccountExternalId.Key(SCHEME_GERRIT, user));
+    ExternalId.Key id = session.getLastLoginExternalId();
+    return id != null && id.equals(ExternalId.Key.create(SCHEME_GERRIT, user));
   }
 
   String getRemoteUser(HttpServletRequest req) {

gerrit-httpd/src/main/java/com/google/gerrit/httpd/auth/container/HttpLoginServlet.java

@@ -14,7 +14,7 @@
 
 package com.google.gerrit.httpd.auth.container;
 
-import static com.google.gerrit.reviewdb.client.AccountExternalId.SCHEME_EXTERNAL;
+import static com.google.gerrit.server.account.ExternalId.SCHEME_EXTERNAL;
 import static java.nio.charset.StandardCharsets.UTF_8;
 
 import com.google.gerrit.common.PageLinks;
@@ -23,11 +23,11 @@ import com.google.gerrit.httpd.CanonicalWebUrl;
 import com.google.gerrit.httpd.HtmlDomUtil;
 import com.google.gerrit.httpd.LoginUrlToken;
 import com.google.gerrit.httpd.WebSession;
-import com.google.gerrit.reviewdb.client.AccountExternalId;
 import com.google.gerrit.server.account.AccountException;
 import com.google.gerrit.server.account.AccountManager;
 import com.google.gerrit.server.account.AuthRequest;
 import com.google.gerrit.server.account.AuthResult;
+import com.google.gerrit.server.account.ExternalId;
 import com.google.gerrit.server.config.AuthConfig;
 import com.google.gwtexpui.server.CacheHeaders;
 import com.google.gwtorm.server.OrmException;
@@ -39,6 +39,7 @@ import javax.servlet.ServletOutputStream;
 import javax.servlet.http.HttpServlet;
 import javax.servlet.http.HttpServletRequest;
 import javax.servlet.http.HttpServletResponse;
+import org.eclipse.jgit.errors.ConfigInvalidException;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.w3c.dom.Document;
@@ -127,7 +128,7 @@ class HttpLoginServlet extends HttpServlet {
       try {
         log.debug("Associating external identity \"{}\" to user \"{}\"", remoteExternalId, user);
         updateRemoteExternalId(arsp, remoteExternalId);
-      } catch (AccountException | OrmException e) {
+      } catch (AccountException | OrmException | ConfigInvalidException e) {
         log.error(
             "Unable to associate external identity \""
                 + remoteExternalId
@@ -156,12 +157,10 @@ class HttpLoginServlet extends HttpServlet {
   }
 
   private void updateRemoteExternalId(AuthResult arsp, String remoteAuthToken)
-      throws AccountException, OrmException, IOException {
-    AccountExternalId remoteAuthExtId =
-        new AccountExternalId(
-            arsp.getAccountId(), new AccountExternalId.Key(SCHEME_EXTERNAL, remoteAuthToken));
+      throws AccountException, OrmException, IOException, ConfigInvalidException {
     accountManager.updateLink(
-        arsp.getAccountId(), new AuthRequest(remoteAuthExtId.getExternalId()));
+        arsp.getAccountId(),
+        new AuthRequest(ExternalId.Key.create(SCHEME_EXTERNAL, remoteAuthToken)));
   }
 
   private void replace(Document doc, String name, String value) {