Allow to update external IDs with BatchRefUpdate

This change adds a VersionedMetaData subclass to update external IDs,
called ExternalIdNotes. Having this class has several advantages:

- External IDs can now be updated as part of a BatchRefUpdate:
  This will allow us to update accounts and external IDs atomically
  (implemented in follow-up changes).
- External ID updates can now easily be batched and committed
  atomically:
  The ExternalIdsBatchUpdate class which implemented replacing of
  external IDs in a batch becomes unneeded and is removed.
- Reading and writing external IDs is now implemented in the same class
  (ExternalIdNotes)
- External ID updates from init, schema migrations, standalone programs
  and tests become easier.
- Using VersionedMetaData to update the external ID notes branch is
  consistent with using VersionedMetaData to update the group names notes
  branch.

Quite a lot of code had to be refactored for this. The most important
changes are:

- The code to parse external IDs is moved from ExternalIdReader to
  ExternalIdNotes:
  ExternalIdReader still takes care of opening the repository and
  loading the external IDs. In addition it updates the metric for
  reading all external IDs and allows tests to disable reading external
  IDs.
- The code to update external IDs is moved from ExternalIdsUpdate to
  ExternalIdNotes:
  ExternalIdsUpdate now only takes care to retry external ID updates on
  LockFailure. When external IDs are updated atomically with account
  updates (follow-up changes) most methods will be removed. Basically
  all external ID updates that touch external IDs of a single account
  will be done through AccountsUpdate. ExternalIdsUpdate will then only
  be used to update external IDs across multiple accounts (if we have
  such updates).
- External ID updates from init, schema migrations, standalone programs
  and tests are now done via ExternalIdNotes:
  This removes the need for public static methods and reduces code
  duplication.

How does ExternalIdNotes work?

- On load the note map from refs/meta/external-ids is read (but the
  external IDs are not parsed yet).
- After loading the note map callers can access single or all external
  IDs. Only now the requested external IDs are parsed.
- After loading the note map callers can stage various external ID
  updates (insert, upsert, delete, replace).
- On save the staged external ID updates are performed.
- After committing the external IDs ExternalIdNotes can be asked to
  update the ExternalIdCache and to evict affected accounts from the
  account cache (which triggers a reindex of the accounts).

There are several modes of instantiating ExternalIdNotes:

- Through Factories
  - ExternalIdNotes.Factory:
    Used to get the caches injected.
  - ExternalIdNotes.FactoryNoReindex:
    Used to get the external IDs cache injected, the account cache is
    not needed since accounts should not be reindex). Used by the
    LocalUsernamesToLowerCase site program and internally at Google.
- Through static methods:
  - loadReadOnly:
    Used when external IDs are only read (since there are no updates the
    caches are not needed)
  - loadNoCacheUpdate:
    Used when updates should be done but the caches don't need to be
    updated (during init and schema migrations where the caches are
    not bound)
- Through package-private constructor:
  Only used by ExternalIdsUpdate that supports external ID updates with
  and without reindex (hence we don't know which of the factories to
  inject).

I tested the schema migrations, init and the LocalUsernamesToLowerCase
site program manually.

Change-Id: Ib643ae8072c61ec467c92f3d1672e595a9cdcbc8
Signed-off-by: Edwin Kempin <ekempin@google.com>
This commit is contained in:
Edwin Kempin
2017-12-14 10:12:14 +01:00
parent d42596d8e9
commit c9cb9a8bd5
15 changed files with 1030 additions and 974 deletions

View File

@@ -718,7 +718,7 @@ public class CommitValidators {
throw new CommitValidationException("invalid external IDs", msgs);
}
return msgs;
} catch (IOException e) {
} catch (IOException | ConfigInvalidException e) {
String m = "error validating external IDs";
log.warn(m, e);
throw new CommitValidationException(m, e);