114 Commits

Author SHA1 Message Date
David Pursehouse
bb61707214 Merge branch 'stable-2.14' into stable-2.15
* stable-2.14:
  AbstractChangeNotes: Never open repo when NoteDb is off
  dev-contributing: Specify buildifier version 0.17.2
  Remove unused dependency on httpcomponents:httpmime
  Fix Elasticsearch dependency on httpcore-nio
  Upgrade Jetty to 9.3.24.v20180605 to fix several CVEs
  [CVE-2018-1000180, CVE-2018-1000613] Upgrade Bouncycastle to 1.60
  Adapt PublicKeyStoreTest to work with BouncyCastle 1.60
  [CVE-2018-10237]: Upgrade guava to 24.1.1-jre
  Stop using CharMatcher.javaLetterOrDigit
  project/Index: Assign and ignore unused future
  [CVE-2017-12629] Upgrade Lucene to 5.5.5
  [CVE-2018-10936] Upgrade postgresql to 42.2.5
  [CVE-2015-1832] Upgrade Apache Derby to 10.12.1.1
  Set version to 2.14.17-SNAPSHOT

Change-Id: Id35c46871597e8005d63379c3fce1d9113b80ed6
2018-11-07 10:18:37 +09:00
David Pursehouse
6f97ef16b3 [CVE-2018-1000180, CVE-2018-1000613] Upgrade Bouncycastle to 1.60
This upgrade fixes CVE-2018-1000180 [1]:

  Bouncy Castle BC 1.54 - 1.59, BC-FJA 1.0.0, BC-FJA 1.0.1 and earlier
  have a flaw in the Low-level interface to RSA key pair generator,
  specifically RSA Key Pairs generated in low-level API with added
  certainty may have less M-R tests than expected. This appears to be
  fixed in versions BC 1.60 beta 4 and later, BC-FJA 1.0.2 and later.

and CVE-2018-1000613 [2]:

  Bouncy Castle Java Cryptography APIs version prior to version 1.60
  contains a CWE-470: Use of Externally-Controlled Input to Select
  Classes or Code ('Unsafe Reflection') vulnerability in XMSS/XMSS^MT
  private key deserialization that can result in Deserializing an
  XMSS/XMSS^MT private key can result in the execution of unexpected
  code. This attack appear to be exploitable via a handcrafted private
  key can include references to unexpected classes which will be picked
  up from the class path for the executing application. This
  vulnerability appears to have been fixed in 1.60 and later.

Bouncycastle 1.57 introduced generics in its APIs. Remove the casts
and @SuppressWarnings("unchecked") annotations that are not necessary
any more.

[1] https://nvd.nist.gov/vuln/detail/CVE-2018-1000180
[2] https://nvd.nist.gov/vuln/detail/CVE-2018-1000613

Bug: Issue 9952
Change-Id: I9b25b1568ac7da555de96d82c597b3dff47966c6
2018-11-05 10:00:40 +09:00
Brandon Weeks
5980f9d707 Adapt PublicKeyStoreTest to work with BouncyCastle 1.60
removePublicKey() operates by removing the chosen key and creating a
new PGPPublicKeyRing with the remainder. However, in BC 1.60[1], there
is new validation logic that requires the key in slot 0 to be a master
key.

In the test case, the master key is removed and reinserted, which
throws with the new validation. Fix it by removing the subkey before
attempting to remove the master key, and then adding it back.

[1] 1c3e644933

Change-Id: Iee072294488bfaaa9ff60b5a1a199a01f4c9800d
2018-11-05 10:00:40 +09:00
David Pursehouse
727b108d2a Merge branch 'stable-2.14' into stable-2.15
* stable-2.14:
  InitSshd: Use correct flag to set empty passphrase
  SshSession: Specify charset in constructor of Scanner
  Specify charset in constructors of InputStreamReader
  Update JGit dependencies to fix building from source

Change-Id: I994d9d26bb7a1b1333a6380fb81126c1e8dc026d
2018-08-31 12:37:23 +09:00
David Pursehouse
ce98c50f38 Specify charset in constructors of InputStreamReader
ErrorProne reports a warning [1] about implicit use of the platform
default charset.

[1] http://errorprone.info/bugpattern/DefaultCharset

Change-Id: Id661781773ca175b52a17cf4d80c37fad17344a8
2018-08-31 10:28:02 +09:00
David Pursehouse
62bd285eb0 Merge branch 'stable-2.14' into stable-2.15
* stable-2.14:
  Fix more comparisons of current user
  Fix permissions checks on Gerrit API on current user
  GroupCacheImpl: Fix log message when UUID is not found

Change-Id: Ida3dd1bd5fc2da5001059a0d5d8b36c83861f00a
2018-05-22 17:44:58 +09:00
Luca Milanesio
5872780a5a Fix permissions checks on Gerrit API on current user
When the current user is using GerritApi, a new user object
with the same user-id of CurrentUser gets created on-the-fly.

Even though the user accountId is the same, the user instances
are different and will fail all the REST API permissions checks.

Turn instance checks with accountId checks to allow user to
execute GerritApi on themselves without the requirement of being
Gerrit administrators.

NOTE: GerritApi are mostly used in plugins, so this change allows
other plugins to function properly.

Change-Id: Iaeb204dda3791eb2757d89fe6bce6994c6305e04
2018-05-21 09:30:09 +01:00
David Pursehouse
b5e620a6a9 Merge branch 'stable-2.14' into stable-2.15
* stable-2.14:
  HttpPluginServlet: Don't trim leading whitespace from about.md content
  ProjectConfig: Don't use JGit's StringUtils to convert to lower case
  Do not abort indexing if < 50% projects failed
  Revert "AllChangesIndexer: Don't abort when failing to open repository"
  VersionedAccountDestinations: Remove unused createSink(String) method
  ProjectBasicAuthFilter: Add comment why cause is not logged
  BazelBuild: Fix exception message when command was interrupted
  GitwebServlet: Write only one log entry for CGI errors
  GitwebServlet: Log unexpected errors on error level
  PostGpgKeys: Remove unneeded use of Joiner
  Remove some logs for errors that are rethrown
  DropWizardMetricMaker: Improve error messages for invalid arguments
  DropWizardMetricMaker: Improve error message when metric name is invalid
  AllChangesIndexer: Don't abort when failing to open repository

Change-Id: I6febb890b7717731fcb5f0653360982668469069
2018-05-17 15:39:34 +09:00
Edwin Kempin
2311ef273e PostGpgKeys: Remove unneeded use of Joiner
We can just rely on the toString formatting of the List. This makes the
code more readable.

Change-Id: I98f0da6b5180198b9fcf0e45271e12e9ab182a52
Signed-off-by: Edwin Kempin <ekempin@google.com>
2018-05-17 10:44:16 +09:00
Edwin Kempin
7d16b055e9 Remove some logs for errors that are rethrown
"log and throw" is considered a poor practice. The logging is uneeded
since the thrown exception will be logged somewhere else.

Change-Id: I82c210fe1e053a4ffd7a58b7fc7de6056f9875a7
Signed-off-by: Edwin Kempin <ekempin@google.com>
2018-05-17 10:44:08 +09:00
David Pursehouse
6ae012e010 Merge branch 'stable-2.14' into stable-2.15
* stable-2.14:
  Update Eclipse compiler settings with Oxygen.3a Release (4.7.3a)
  Introduce mechanism to overload Gerrit core modules
  Move default memory cache implementation out of H2 package
  Introduce CacheImpl annotation
  Untangle persistent/memory cache implementations from each other
  H2CacheFactory: update internal caches list synchronously

Change-Id: I6c0952771b4e630a6628355f8cb2972ab7d12419
2018-05-16 08:15:29 +09:00
Dave Borowitz
a3cc510fc1 Move default memory cache implementation out of H2 package
The memory cache doesn't depend on H2 at all, and this separation of
packages makes it harder to reintroduce the kind of source-level
dependency that was removed in I570fd54adf.

Change-Id: Ie23e39cd7ea9a0813a914e65614b889c248342ae
2018-05-14 13:09:55 +09:00
David Pursehouse
2049f12280 Merge branch 'stable-2.14' into stable-2.15
* stable-2.14:
  PostGpgKeys: Gracefully handle malformed GPG keys input

Change-Id: I72f639ada2db19dd7c33f98954088d4c0836c36c
2017-11-07 17:12:03 +09:00
David Pursehouse
9df753e05e PostGpgKeys: Gracefully handle malformed GPG keys input
Bug: Issue 7647
Change-Id: I226e1d560936db1f1ef0d447ea00b59c66189a96
2017-11-07 16:06:15 +09:00
Edwin Kempin
02e281d9d3 Trigger GitReferenceUpdated event when external IDs are updated
When external IDs are updated the refs/meta/external-ids branch in the
All-Users repository is updated. For this update we must trigger a
GitReferenceUpdated event, so that e.g. this ref update gets replicated
to Gerrit slaves.

PostGpgKeys and GpgKeys are now injected as provider into
GpgApiAdapterImpl to avoid a circular Guice dependency:
GitReferenceUpdated needs EventUtil which needs ChangeJson.Factory.
ChangeJson uses GpgApiAdapter to find out if push certificates are
enabled and if yes to include the push certicates of the patch set
uploaders into RevisionInfo. This doesn't require PostGpgKeys and
GpgKeys. If PostGpgKeys would be injected without provider we would need
ExternalIdsUpdate.User to instantiate it which now needs
GitReferenceUpdated. Hence without injecting PostGpgKeys into
GpgApiAdapterImpl as a provider we would have a dependency circle.

Change-Id: Ia9c71d26669f24f20c659d82ae97ecfca6ab61c3
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-10-05 12:28:15 +02:00
Edwin Kempin
6957747c87 Remove support for writing accounts from ReviewDb
Accounts have been fully migrated to NoteDb. However inside of Google we
still have code that depends on the Accounts table. hence this table is
dropped only later by a follow-up change.

Change-Id: Ia32983e2877e3965561db7d264cf1d2b7b2e40c3
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-08-29 14:49:00 +02:00
Dave Borowitz
769d159701 Make ReceiveCommits non-public
Callers used to have access to both the AsyncReceiveCommits and the
underlying ReceiveCommits. This was confusing at best, and in the
presence of multiple threads (or, eventually, retrying) broken at worst.
Try to simplify the package interface by forcing all callers to go
through AsyncReceiveCommits. It's still somewhat non-obvious, but at
least there is only one choice.

To support tests that want to assert over specific error message
strings, factor out a public ReceiveConstants class.

Change-Id: I1760faed4c2d4d508c38ec8a698f3e5c2aae2c35
2017-08-08 10:50:06 -04:00
Dave Borowitz
798b883b40 Move ReceiveCommits and related classes to their own package
After naively moving the classes, there were almost no incoming
references from the rest of the server packages into the new
server.receive package. This means with only a little more work, it was
possible to create a new java_library target containing just the srcs
in this new package. This is a modest step in the direction of breaking
up the giant //gerrit-server:server package, which will improve compile
times when making modifications that don't change the interface.

Change-Id: I449018a4933a999c688611142dc7ed9c18b4c828
2017-08-08 10:50:06 -04:00
Dave Borowitz
a6985bcfa8 Upgrade JGit to support improved BatchRefUpdates
Now with atomic support for traditional on-disk repos, and finer-grained
control over reflogs.

RefUpdate.Result also grew some new error values, so extend existing
switch statements to cover them.

Change-Id: If685ed7f34d965e82cf11fcf59dd832394f2bb4a
2017-07-28 12:47:26 -04:00
Dave Borowitz
dece981249 Rework NotesMigration hierarchy and get rid of test impl
It used to be that ConfigNotesMigration was the only kind of
NotesMigration in a real server, but it was always immutable, while
TestNotesMigration was the main kind of migration in acceptance tests,
which was mutable. However, now that we support modifying
ConfigNotesMigration at runtime as part of the online NoteDb migration
process, TestNotesMigration is no longer strictly necessary, and
continuing to support it is becoming more trouble than it's worth.

One major problem was that only TestNotesMigration was being populated
via NoteDbMode, and the NoteDbMode was not reflected in the
ConfigNotesMigration at all, so callers that were depending on
ConfigNotesMigration directly would not know about the NoteDb migration
state from the GERRIT_NOTEDB env var in tests.

We could have fixed this (and other) problems directly, but there is a
better solution: get rid of the test implementation entirely, and use
the same implementation of NotesMigration in tests as in a running
server.

The class hierarchy now contains only two classes: NotesMigration and
MutableNotesMigration. Most callers just care about inspecting the
state, so they can inject a NotesMigration. The few callers (migration,
tests) that care about mutating the state at runtime can inject/create
MutableNotesMigrations instead. As an implementation detail, the actual
NotesMigration instance continues to be mutable, containing a reference
to the Snapshot, but the base class does not contain any public methods
to mutate the state. We then ensure with Guice that there is only one
actual NotesMigration instance (the MutableNotesMigration), and callers
just may or may not have access to the mutation methods depending on
what they chose to inject.

Ensuring this gets set up correctly in tests requires a bit of tweaking.
  * Since the NotesMigration is populated in the @UseLocalDisk case from
    reading gerrit.config on disk, we need to prepopulate gerrit.config
    with the right config values at startup time.
  * Since MutableNotesMigration is not in the testutil package, it can't
    have its own setFromEnv() method that depends on NoteDbMode.
    Instead, construct MutableNotesMigrations from the test env by using
    a static factory method in NoteDbMode.

Change-Id: If06db3d025cf3e3c9fe464989d5f38a22ce70b56
2017-07-13 10:03:17 -04:00
Edwin Kempin
b1f59bb6a9 AccountsUpdate: Rename atomicUpdate to update
After the old update method has been renamed to replace, there is no
need to keep 'atomic' in the method name.

Change-Id: I465d29f567f827868928b178037e26313e2b6cec
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-06-27 15:56:38 +02:00
Edwin Kempin
d8c24c67e3 Always update accounts atomically
This prevents that we unintentionally overwrite concurrent updates, e.g.
updates that were done by a racing request or updates we didn't see
because we read from a stale cache.

Change-Id: I4dfc7726c9324f06806919590d3ef83555bd44a4
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-06-27 15:56:38 +02:00
Dave Borowitz
f09980eea7 Update to new generified Bouncy Castle APIs
Change-Id: I6a1b0657467c8890f48e7afac82a13cdf8fdf0e3
2017-06-26 15:48:37 -04:00
Han-Wen Nienhuys
a5c310bf35 Revert "Fix eclipse warnings"
This reverts commit 3175942514602f3c6c0e4441ff37a762af2841b0.

Reason for revert: this change leads to breakages inside google. We are still on BC 1.5.5

gerrit-gpg/test/java/com/google/gerrit/gpg/testutil/TestKey.java:80: error: incompatible types: Object cannot be converted to String
    return getPublicKey().getUserIDs().next();
                                           ^

Change-Id: Idc603ad379f1265da8b851afed3fdea6598bf27c
2017-06-26 12:12:19 +00:00
David Ostrovsky
3175942514 Fix eclipse warnings
In I5fc7f7237c bouncy castle was upgraded to 1.57, that introduced
generics in its APIs. But the casts and @SuppressWarnings("unchecked")
annotations were missed to be removed.

Change-Id: I070119df0e1b61fc4a22dc5394dd413a18d7d6aa
2017-06-23 23:25:22 +00:00
Edwin Kempin
4847c3de4e Let ExternalIdsUpdate take care to evict accounts from the account cache
The account cache holds AccountState instances which contain the
external IDs of the accounts. Hence an account must be evicted from the
account cache when its external IDs are updated. At the moment it's the
responsibility of the caller to do the account eviction, but it can
easily be forgotten and it's more convenient if ExternalIdsUpdate takes
care of it. For some scenarios this may result in a few more cache
evictions (e.g. account creation), but for most operations the number of
account evictions should stay the same.

After updating external IDs the corresponding accounts also need to be
reindexed, but this is automatically done when accounts are evicted from
the account cache.

Change-Id: I1af02c7576eea81bb229a4663cb1e067ab137784
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-06-09 09:56:23 +02:00
Edwin Kempin
f36b3d7a75 Let AccountsUpdate take care to evict accounts from the account cache
Accounts need to be evicted from the account cache when they are
updated. Instead of leaving this responsibility to the caller let
AccountsUpdate take care of this. This is more convenient and ensures
that none of the callers forgets about it.

Accounts also need to be evicted from the account cache on creation,
since the account cache may hold an empty AccountState instance for that
account ID (if it was requested from the account cache before and was
found missing).

After updating accounts they also need to be reindexed, but this is
automatically done when accounts are evicted from the account cache.

Change-Id: I08d5c756eabd43c29bdc217e7b1c3d85ebf15197
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-06-07 16:35:51 +02:00
Edwin Kempin
3ea63124f3 Let ExternalIdsUpdate take care to evict accounts from the account cache
The account cache holds AccountState instances which contain the
external IDs of the accounts. Hence an account must be evicted from the
account cache when its external IDs are updated. At the moment it's the
responsibility of the caller to do the account eviction, but it can
easily be forgotten and it's more convenient if ExternalIdsUpdate takes
care of it. For some scenarios this may result in a few more cache
evictions (e.g. account creation), but for most operations the number of
account evictions should stay the same.

After updating external IDs the corresponding accounts also need to be
reindexed, but this is automatically done when accounts are evicted from
the account cache.

Change-Id: I1af02c7576eea81bb229a4663cb1e067ab137784
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-06-07 16:34:36 +02:00
Edwin Kempin
2803609f8e Update accounts through AccountsUpdate class
The AccountsUpdate class should become the single class that updates
accounts. At the moment it always updates the accounts in ReviewDb, but
in future it will update accounts in NoteDb.

Atomic account updates are not yet moved to AccountsUpdate. This will be
done in a follow-up change.

Change-Id: I2d1b2e054cb47269942b094363824485c4760af2
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-06-07 15:25:11 +02:00
Edwin Kempin
2a3b883511 Get accounts through Accounts class
The Accounts class should become the single class that reads accounts.
At the moment it always reads the accounts from ReviewDb, but in future
it will read accounts from NoteDb.

Change-Id: I7507b734e302ca3953e86fe612aac63ea10c75da
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-06-07 15:24:19 +02:00
Edwin Kempin
276b8a897f Migrate external IDs to NoteDb (part 3)
This is the third part of migrating external IDs from ReviewDb to
NoteDb.

This change:
* changes the code to always read external IDs from NoteDb (the
  user.readExternalIdsFromGit configuration parameter is removed)
* bumps the database schema version
* deletes the database table for external IDs

Pushing to the refs/meta/external-ids branch is still prevented by a
commit validator. Since all external IDs are now in NoteDb only we
could allow pushing to refs/meta/external-ids. However we would still
like to do validation of the branch content and reject invalid content
(e.g. invalid Git config files, usage of non-existing account IDs
etc.) and such a validator is not implemented yet (but can be
implemented in a follow-up change).

Change-Id: Id9e5574a1d8d82f4f48fbb0b6dadc0e27d138a28
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-05-10 08:51:44 +02:00
David Pursehouse
f7d49955cf Merge branch 'stable-2.14'
* stable-2.14:
  Don't ship bouncycastle libraries in plugin API

Change-Id: I2cecc4784c1821a89313bf71b998943df0a449cc
2017-04-10 08:21:56 +09:00
David Ostrovsky
c5f8066629 Don't ship bouncycastle libraries in plugin API
We cannot shade bouncycastle in the plugin API. Still we need it to be
included in the gerrit.war, licenses file and Eclipse classpath.

Expose bouncycastle libraries in PLUGIN_TEST_DEPS constant, so that
the plugins don't need to change anything in tree build mode.

gerrit_api() bazlet in bazlets repository is extended too, so that the
plugins don't need to change anything in standalone build mode.

One side effect of this change, is that bouncycastle libraries are
now listed with neverlink suffix, e.g.:

* bouncycastle:bcprov-neverlink

Bug: Issue 5826
Change-Id: Idb8051e16b14e20c8dd528783ab297ee25707bb3
2017-04-07 07:38:04 +02:00
Edwin Kempin
c07c18a687 Migrate external IDs to NoteDb (part 2)
This is the second part of migrating external IDs from ReviewDb to
NoteDb.

This change:
* migrates the external IDs from ReviewDb to NoteDb (for single
  instance Gerrit servers)
* adds a configuration parameter (user.readExternalIdsFromGit) that
  controls whether external IDs are read from ReviewDb or NoteDb

The new ExternalIds class provides access to external IDs. All code that
needs external IDs is adapted to use this class to retrieve external IDs
(instead of reading directly from the database). ExternalIds gets the
external IDs either directly from the storage backend (via
ExternalIdReader) or from a cache (via ExternalIdCache).

ExternalIdReader reads the external IDs from the storage backend.
Depending on the value of the user.readExternalIdsFromGit parameter the
external IDs are read from ReviewDB or NoteDb.

If reading external IDs from NoteDb is enabled, reading the external IDs
of an account requires parsing all Git notes. This is because external
IDs are keyed by external ID key ('<scheme>:<id>') and the account ID is
only contained in the Git note content. Since parsing all Git notes is
too expensive if it is done frequently, there is a new external ID cache
which makes external IDs accessible by account. This cache is populated
once by reading all external IDs from NoteDb and is then kept up to date
by informing it whenever an external ID is added, updated or deleted.
The external ID cache uses the revision of the refs/meta/external-ids
branch as key, so that all external IDs are reloaded when the
refs/meta/external-ids branch is changed behind Gerrit's back. This
makes it easy to use this cache in a multimaster setup, since an update
of the refs/meta/external-ids branch which is done due to replication
between nodes causes a reload of the external IDs in the receiving node.
The ExternalIdCache is an implementation detail of how external IDs are
read and written, which is why it is package private. Callers should
always use ExternalIds to access external IDs and ExternalIdsUpdate /
ExternalIdsBatchUpdate to update external IDs.

The LocalUsernamesToLowerCase program needs to access all external IDs
only once to update them. After the update they are not accessed again.
Hence the LocalUsernamesToLowerCase program doesn't benefit from caching
external IDs and the external ID cache can be disabled for it.

The external ID cache is defined by the ExternalIdCache interface. It is
implemented by ExternalIdCacheImpl and DisabledExternalIdCache.
DisabledExternalIdCache can be used when an external ID cache is not
needed, e.g. in the LocalUsernamesToLowerCase program or in tests.

Pushing to the refs/meta/external-ids branch, which would only update
the external IDs in NoteDb, is still prevented by a commit validator so
that the external IDs in ReviewDb and NoteDb do not go out of sync.

Change-Id: Ia1dae9306b7ee07388b6c5e1f3dc4a1a5eea4b08
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-03-28 09:10:34 +02:00
Edwin Kempin
c9694da1c2 Move classes that are related to external IDs into an own package
This enables follow-up changes to not expose implementation details of
reading/writing external IDs (since classes can be kept package
private now).

Change-Id: I4b3394592304d8391d7103821d46da7ce475be52
2017-03-27 15:38:01 +02:00
Patrick Hiesel
2b9574c785 Adapt the use of @Inject and @AssistedInject
This change adapts our use of @Inject and @AssistedInject to be in-line
with what the Guice documentation governs [0, 1].

Specifically, @Inject should be used even if the constructor has
@Assisted parameters as long as it is the only injected constructor in
that class. @AssistedInject should be used if there are multiple
constructors with @Assisted parameters.

[0] https://google.github.io/guice/api-docs/4.1/javadoc/index.html?com/google/inject/assistedinject/FactoryModuleBuilder.html
[1] https://google.github.io/guice/api-docs/4.1/javadoc/com/google/inject/assistedinject/AssistedInject.html

Change-Id: I725df1e294d0758f4c27f8b86ec26c21adeb3d8f
2017-03-23 09:20:09 +00:00
ekempin
76553082d3 Merge changes from topic 'http-password'
* changes:
  Migrate external IDs to NoteDb (part 1)
  gerrit-server: use hashed passwords for HTTP.
  AccountByEmailCacheImpl: Consider emails from all external IDs on load


* submodules:
* Update plugins/cookbook-plugin from branch 'master'
  - Merge "Remove for HTTP digest auth from examples."
  - Remove for HTTP digest auth from examples.
    
    Change-Id: I495ee8140cbe2ae12510a4d4cbc2c8360b135b33
2017-02-28 09:03:53 +00:00
Edwin Kempin
744d2b8967 Migrate external IDs to NoteDb (part 1)
In NoteDb external IDs are stored in the All-Users repository in a Git
Notes branch called refs/meta/external-ids where the sha1 of the
external ID is used as note name. Each note content is a Git config
file that contains an external ID. It has exactly one externalId
subsection with an accountId and optionally email and password:

  [externalId "username:jdoe"]
     accountId = 1003407
     email = jdoe@example.com
     password = bcrypt:4:LCbmSBDivK/hhGVQMfkDpA==:XcWn0pKYSVU/UJgOvhidkEtmqCp6oKB7

Storing the external IDs in a Git Notes branch with using the sha1 of
the external ID as note name ensures that external IDs are unique and
are only assigned to a single account. If it is tried to assign the
same external ID concurrently to different accounts, only one Git
update succeeds while the other Git updates fail with LOCK_FAILURE.
This means assigning external IDs is also safe in a multimaster setup
if a consensus algorithm for updating Git refs is implemented (which
is needed for multimaster in any case). Alternatively it was
considered to store the external IDs per account as Git config file in
the refs/users/<sharded-id> user branches in the All-Users repository
(see abandoned change 9f9f07ef). This approach was given up because in
race conditions it allowed to assign the same external ID to different
accounts by updating different branches in Git.

To support a live migration on a multi-master Gerrit installation, the
migration of external IDs from ReviewDb to NoteDb is done in 2 steps:

- part 1 (this change):
  * always write to both backends (ReviewDb and NoteDb)
  * always read external IDs from ReviewDb
  * upgraded instances write to both backends, old instances only
    write to ReviewDb
  * after upgrading all instances (all still read from ReviewDb)
    run a batch to copy all external IDs from the ReviewDb to NoteDb
- part 2 (next change):
  * bump the database schema version
  * migrate the external IDs from ReviewDb to NoteDb (for single instance
    Gerrit servers)
  * read external IDs from NoteDb
  * delete the database table

With this change reading external IDs from NoteDb is not implemented
yet. This is because the storage format of external IDs in NoteDb
doesn't support efficient lookup of external IDs by account and this
problem is only addressed in the follow-up change (it adds a cache for
external IDs, but this cache uses the revision of the notes branch as
key, and hence can be only implemented once the external IDs are fully
migrated to NoteDb and storing external IDs in ReviewDb is dropped).

The ExternalIdsUpdate class implements updating of external IDs in
both NoteDb and ReviewDb. It provides various methods to update
external IDs (e.g. insert, upsert, delete, replace). For NoteDb each
method invocation leads to one commit in the Git notes branch.
ExternalIdsUpdate has two factories, User and Server. This allows to
record either the calling user or the Gerrit server identity as
committer for an update of the external Ids.

External IDs are now represented by a new AutoValue class called
ExternalId. This class replaces the usage of the old gwtorm entity
AccountExternalId class. For ExternalId scheme names are the same as for
AccountExternalId but no longer include the trailing ':'.

The class ExternalIdsOnInit makes it possible to update external IDs
during the init phase. This is required for inserting external IDs for
the initial admin user which is created by InitAdminUser. We need a
special class for this since not all dependencies of ExternalIdsUpdate
are available during init.

The class ExternalIdsBatchUpdate allows to do batch updates to
external IDs. For NoteDb all updates will result in a single commit to
the refs/meta/external-ids Git notes branch.

LocalUsernamesToLowerCase is now always converting the usernames in a
single thread only. This allows us to get a single commit for the
username convertion in NoteDb (this would not be possible if workers
do updates in parallel). Since LocalUsernamesToLowerCase is rather
light-weight being able to parallelize work is not really needed and
removing the workers simplifies the code significantly.

To protect the refs/meta/external-ids Git notes branch in the All-Users
repository read access for this ref is only allowed to users that have
the 'Access Database' global capability assigned. In addition
there is a commit validator that disallows updating the
refs/meta/external-ids branch by push. This is to prevent that the
external IDs in NoteDb diverge from the external IDs in ReviewDb while
the migration to NoteDb is not fully done yet.

Change-Id: Ic9bd5791e84ee8d332ccb1f709970b59ee66b308
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-02-28 09:09:39 +01:00
Edwin Kempin
2208a5f7ce Make static fields in GpgKeys and PreviewSubmit final
Change-Id: Ifaf144224aeeb4339db4e445ab5bf458f89bb151
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-02-27 15:34:26 +01:00
David Pursehouse
1111684821 Squash redundant string concatenations introduced by google-java-format
Reformatting the code with google-java-format tool in change Id5f3c6de9
resulted in wrapped lines being unwrapped to the new 100 columns limit.

In cases where the wrapped line was wrapped on a string concatenation,
the resulting long line now includes redundant concatenations. See the
upstream issue [1] for an example of this.

Squash the redundant concatenations with:

  git ls-files | grep java$ | xargs sed -i '' 's/" + "//g'

(note: this also resulted in a couple of unwanted changes; those are
manually undone.)

[1] https://github.com/google/google-java-format/issues/122

Change-Id: I7348413ae460c8c7a0b0c72dab0a1ae7275a2ec3
2017-02-17 08:57:46 +09:00
Han-Wen Nienhuys
6471e8a94a Ship BouncyCastle in the .war files.
Per
https://www.bis.doc.gov/index.php/forms-documents/encryption/328-flowchart-2/file
open source crypto software can be self-classified as 5D002, and
requires only notification of the U.S. Bureau of Industry and
Security.

This registration has been performed by Google, as of Feb 15, 2017.

This gets rid of the special casing for BouncyCastle, simplifying our
build and deployment process.

Change-Id: I680b0a001e5e2e497ed6e62c90c8b8be30efff48
2017-02-15 13:48:57 +01:00
Dave Borowitz
292fa154c1 Format all Java files with google-java-format
Having a standard tool for formatting saves reviewers' valuable time.
google-java-format is Google's standard formatter and is somewhat
inspired by gofmt[1]. This commit formats everything using
google-java-format version 1.2.

The downside of this one-off formatting is breaking blame. This can be
somewhat hacked around with a tool like git-hyper-blame[2], but it's
definitely not optimal until/unless this kind of feature makes its way
to git core.

Not in this change:
* Tool support, e.g. Eclipse. The command must be run manually [3].
* Documentation of best practice, e.g. new 100-column default.

[1] https://talks.golang.org/2015/gofmt-en.slide#3
[2] https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/git-hyper-blame.html
[3] git ls-files | grep java$ | xargs google-java-format -i

Change-Id: Id5f3c6de95ce0b68b41f0a478b5c99a93675aaa3
Signed-off-by: David Pursehouse <dpursehouse@collab.net>
2017-02-07 10:04:39 +09:00
Edwin Kempin
79e63a39e4 Revert external ID cache
It make more sense to include the commit SHA-1 of the notes branch from
which the external IDs have been read into the cache key. This makes it
easier to implement this cache within a multimaster setup.
However, obviously, we can include the commit SHA-1 of the notes branch
into the cache key only after the external IDs have been migrated to
NoteDb (because before this the notes branch does not exist yet).
Revert the cache for now and re-add an improved version of the cache
later with the change that implements the migration of the external IDs
to NoteDb.

This reverts the following commits:
- 2869caaf70569fcf777676d605f9d7e70c4ac36d:
  Add cache for external ids
- 48d5c9b9aa8aba3de52374328e2a1cc30be64a10:
  Make ExternalIdCacheImpl.AllKey public
- 20e5507d46c2a266e73974878685c74adfd6ff6b:
  ExternalIdCache: Add method to get external IDs by account ID + scheme

Change-Id: I589242aad32a9c70718542ba950c0a351c594e54
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-02-02 17:27:43 +01:00
Edwin Kempin
20e5507d46 ExternalIdCache: Add method to get external IDs by account ID + scheme
Change-Id: I800c41923facbfe9194ce47716ed2d23d0ffd70a
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-01-31 15:58:50 +01:00
David Ostrovsky
fdbfcad77d Remove Buck based build
Bug: Issue 5302
Change-Id: I6e860446ef30ff0ad1c7c49fc0e39d39d921820b
2017-01-23 12:44:58 +00:00
Edwin Kempin
2869caaf70 Add cache for external ids
Introducing a cache for external ids is a preparation step for moving
the external ids from ReviewDb into NoteDb. For NoteDb it is planned
to store the external ids in a git note branch, where the note keys
are the sha1's of the external ids and the note values contain the
external id, the account id and optionally email and password. With
this format we can easily lookup external ids by the external id, but
listing all external ids of an account requires parsing all external
ids. Looking up the external ids of an account is possible from the
account index, however for reindexing an account we would still need
to lookup all external ids of the account from git. Having a cache for
the external ids ensures that the external ids are only loaded once
from git. If there is any update to external ids, we take care to
update the cache as well. For this change it means that all code that
modifies external ids must do an extra call to update the external id
cache. This is not optimal since updating the cache can be easily
forgotten. This is why the follow-up change cleans this up by
introducing a dedicated class that handles all external id updates and
then this is the only class that must take care to update the external
id cache.

Change-Id: I9ea979c646cddb9b39e723de5c061a70a2ce6fd6
Signed-off-by: Edwin Kempin <ekempin@google.com>
2017-01-11 14:32:21 +01:00
Edwin Kempin
10aa4e2bbf Require account index and remove fallbacks
The account data is moved from ReviewDb into git.

Change-Id: I643827179b24601b138f394cfff5890f919b9da9
Signed-off-by: Edwin Kempin <ekempin@google.com>
2016-12-29 11:23:56 +01:00
David Pursehouse
34321931e8 Merge "Remove test prefix from test methods"
* submodules:
* Update plugins/replication from branch 'master'
  - Remove test prefix from test methods in replication plugin
    
    We previously used 'test' to prefix tests but have decided to stop this.
    This change removes the prefix from all test code.
    
    Change-Id: I42e6191ece7872f4647e425e3ca0acf8c6452412
2016-12-07 12:48:37 +00:00
David Ostrovsky
fa18907d7f Bazel: Reformat build files
Reformat the Bazel build files with the buildifier tool [1].

The style is different for Bazel files. Most notably, indentation level
is 4 spaces instead of 2, and " is used instead of '.

[1] https://github.com/bazelbuild/buildifier

Change-Id: I95c0c6f11b6d76572797853b4ebb5cee5ebd3c98
2016-12-07 11:33:07 +00:00
Patrick Hiesel
a02a5c01e9 Remove test prefix from test methods
We previously used 'test' to prefix tests but have decided to stop this.
This change removes the prefix from all test code.

Change-Id: I229a36751adc6a87fbae8d6f373671e141529496
2016-12-06 10:36:42 +01:00