The problem we are facing on the stable-2.14 branch is: we have
intermediate NoteDb migration state for accounts entity due to
merge of: Ic9bd5791e84. That why we are writing to both backends:
ReviewDb and NoteDb. It creates potential risk to be out of sync
between ReviewDb and NoteDb (and secondary index). In addition it
is always bad from performance point of view to unnecessary write
to 2 different backends. The real migration to NoteDb for accounts
entities (phase 2) happens in: Ia1dae9306b7 and Schema_144, that is
migrating the external IDs from ReviewDb to NoteDb, and that change
is not a part of stable-2.14.
In retrospective, we shouldn't include partially migrated code paths
for the production releases. It's error prone and bad for the
performance. Originally, multi-phase upgrade procedure was done on
master only to support multi master and zero downtime upgrades. These
feature is not related to open source gerrit version.
Moreover, now, that we are facing intermittent account corruption
problems: Issue 7652 that is hard to track down, understand and fix,
we are seeing automatic recovery attempt: [1], that is trying to
detect database corruption and synchronize both backends. This change
takes a different approach and avoids two backends where only ReviewDb
is actually used on production release line 2.14.
To avoid fixing too many caller sites the interfaces of ExternalIds,
ExternalIdsOnInit, ExternalIdsBatchUpdate and ExternalIdsUpdate are
mostly preserved, but the code paths for NoteDb mutations is dropped.
This partially reverts commit 744d2b896719e2058539db98443c80eb9368fd77.
[1] https://gerrit-review.googlesource.com/162450
Change-Id: Iec8d0c5639e462d88a7c5d0906febfd6f3337277
In NoteDb external IDs are stored in the All-Users repository in a Git
Notes branch called refs/meta/external-ids where the sha1 of the
external ID is used as note name. Each note content is a Git config
file that contains an external ID. It has exactly one externalId
subsection with an accountId and optionally email and password:
[externalId "username:jdoe"]
accountId = 1003407
email = jdoe@example.com
password = bcrypt:4:LCbmSBDivK/hhGVQMfkDpA==:XcWn0pKYSVU/UJgOvhidkEtmqCp6oKB7
Storing the external IDs in a Git Notes branch with using the sha1 of
the external ID as note name ensures that external IDs are unique and
are only assigned to a single account. If it is tried to assign the
same external ID concurrently to different accounts, only one Git
update succeeds while the other Git updates fail with LOCK_FAILURE.
This means assigning external IDs is also safe in a multimaster setup
if a consensus algorithm for updating Git refs is implemented (which
is needed for multimaster in any case). Alternatively it was
considered to store the external IDs per account as Git config file in
the refs/users/<sharded-id> user branches in the All-Users repository
(see abandoned change 9f9f07ef). This approach was given up because in
race conditions it allowed to assign the same external ID to different
accounts by updating different branches in Git.
To support a live migration on a multi-master Gerrit installation, the
migration of external IDs from ReviewDb to NoteDb is done in 2 steps:
- part 1 (this change):
* always write to both backends (ReviewDb and NoteDb)
* always read external IDs from ReviewDb
* upgraded instances write to both backends, old instances only
write to ReviewDb
* after upgrading all instances (all still read from ReviewDb)
run a batch to copy all external IDs from the ReviewDb to NoteDb
- part 2 (next change):
* bump the database schema version
* migrate the external IDs from ReviewDb to NoteDb (for single instance
Gerrit servers)
* read external IDs from NoteDb
* delete the database table
With this change reading external IDs from NoteDb is not implemented
yet. This is because the storage format of external IDs in NoteDb
doesn't support efficient lookup of external IDs by account and this
problem is only addressed in the follow-up change (it adds a cache for
external IDs, but this cache uses the revision of the notes branch as
key, and hence can be only implemented once the external IDs are fully
migrated to NoteDb and storing external IDs in ReviewDb is dropped).
The ExternalIdsUpdate class implements updating of external IDs in
both NoteDb and ReviewDb. It provides various methods to update
external IDs (e.g. insert, upsert, delete, replace). For NoteDb each
method invocation leads to one commit in the Git notes branch.
ExternalIdsUpdate has two factories, User and Server. This allows to
record either the calling user or the Gerrit server identity as
committer for an update of the external Ids.
External IDs are now represented by a new AutoValue class called
ExternalId. This class replaces the usage of the old gwtorm entity
AccountExternalId class. For ExternalId scheme names are the same as for
AccountExternalId but no longer include the trailing ':'.
The class ExternalIdsOnInit makes it possible to update external IDs
during the init phase. This is required for inserting external IDs for
the initial admin user which is created by InitAdminUser. We need a
special class for this since not all dependencies of ExternalIdsUpdate
are available during init.
The class ExternalIdsBatchUpdate allows to do batch updates to
external IDs. For NoteDb all updates will result in a single commit to
the refs/meta/external-ids Git notes branch.
LocalUsernamesToLowerCase is now always converting the usernames in a
single thread only. This allows us to get a single commit for the
username convertion in NoteDb (this would not be possible if workers
do updates in parallel). Since LocalUsernamesToLowerCase is rather
light-weight being able to parallelize work is not really needed and
removing the workers simplifies the code significantly.
To protect the refs/meta/external-ids Git notes branch in the All-Users
repository read access for this ref is only allowed to users that have
the 'Access Database' global capability assigned. In addition
there is a commit validator that disallows updating the
refs/meta/external-ids branch by push. This is to prevent that the
external IDs in NoteDb diverge from the external IDs in ReviewDb while
the migration to NoteDb is not fully done yet.
Change-Id: Ic9bd5791e84ee8d332ccb1f709970b59ee66b308
Signed-off-by: Edwin Kempin <ekempin@google.com>
Having a standard tool for formatting saves reviewers' valuable time.
google-java-format is Google's standard formatter and is somewhat
inspired by gofmt[1]. This commit formats everything using
google-java-format version 1.2.
The downside of this one-off formatting is breaking blame. This can be
somewhat hacked around with a tool like git-hyper-blame[2], but it's
definitely not optimal until/unless this kind of feature makes its way
to git core.
Not in this change:
* Tool support, e.g. Eclipse. The command must be run manually [3].
* Documentation of best practice, e.g. new 100-column default.
[1] https://talks.golang.org/2015/gofmt-en.slide#3
[2] https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/git-hyper-blame.html
[3] git ls-files | grep java$ | xargs google-java-format -i
Change-Id: Id5f3c6de95ce0b68b41f0a478b5c99a93675aaa3
Signed-off-by: David Pursehouse <dpursehouse@collab.net>
This makes it more clear that a lookup may find no account.
Change-Id: Ia0ffe7547c365be1526eb959ff4b5e25cfa1d37e
Signed-off-by: Edwin Kempin <ekempin@google.com>
Reformat the Bazel build files with the buildifier tool [1].
The style is different for Bazel files. Most notably, indentation level
is 4 spaces instead of 2, and " is used instead of '.
[1] https://github.com/bazelbuild/buildifier
Change-Id: I95c0c6f11b6d76572797853b4ebb5cee5ebd3c98
Current implementation assumes that OAuth provider always exposes
user name by using it for key in OAuth token cache. For some OAuth
providers (most notably Google OAuth provider) this is not the case.
Move from using user name to account id for token cache key. Postpone
populating the cache to the point when the authentication already took
place and use returned account id as cache key.
Bug: Issue 4627
Change-Id: I59f15b7c5ca8be6d52b59d21fac58cba88ba7fe3
To run the tests:
bazel test //...
To build the Gerrit plugin API, run:
bazel build gerrit-plugin-api:plugin-api_deploy.jar
To build the Gerrit extension API, run:
bazel build gerrit-extension-api:extension-api_deploy.jar
TODOs:
Licenses
Reduce visibility (all public for now)
Generate HTML Documentation
Core plugins
gerrit_plugin() rule to build plugins in tree and standalone modes
GWT UI (only gwt_module() skylark rule is provided, no gwt_binary())
PolyGerrit UI
WAR
Publish artifacts to Maven Central
Ask Bazel team to add Gerrit to their CI on ci.bazel.io
Contributed-By: Han-Wen Nienhuys <hanwen@google.com>
Change-Id: I9a86e670882a44a5c966579cdeb8ed79b1590de3
OAuth access tokens retrieved during login in the web UI are stored
privately in OAuthSession. There is no possibility for a user to
obtain that token, e.g. to authenticate with a native Git client.
This patch adds a persistent cache for OAuth tokens and modifies
OAuthSession to store tokens received during the login handshake
with the OAuth provider in this cache.
Since access tokens must be kept secret, the cache defines a new
extension point OAuthTokenEncrypter. If an encrypter is provided,
access tokens are encrypted before storing them in the cache, and
decrypted when reading from the cache. By default, no encryption
is applied.
In subsequent patches a REST API for retrieving OAuth tokens will
be added as well as a corresponding settings page.
Change-Id: I751dd5f70dd30823bd2f531e1ac1da0759f98976
Signed-off-by: Michael Ochmann <michael.ochmann@sap.com>
* stable-2.10:
OAuth: Simplify protocol implementation
Allow to link user identity to another OAuth provider
Update JGit to the 4.0.0.201505050340-m2 version
Hybrid OpenID/OAuth: Support switching identities
Hybrid OpenID/OAuth: Allow to link identity accross protocols
OAuth: Check for session validity during logout
Change-Id: I9da0073a72d8c4327313405b11c66cd253ff640b
When web_sessions cache is expired, OAuth session preserves it
logged in state. This makes new sign-in impossible.
Rectify it by checking the states mismatch and invalidating OAuth
session when web_sessions cache was expired.
GitHub-Bug: https://github.com/davido/gerrit-oauth-provider/issues/5
Change-Id: I3d57193c5af29561fd1fac0804dd19c08a0e9dbe
* stable-2.10:
Update version to 2.10.2
Release notes for Gerrit 2.10.2
Do not return 403 when clicking on Gitweb breadcrumb
Add log messages to troubleshoot OAuth/OpenID linking
Remove unused OAuthToken in authorisation URL
OnlineReindexer: log the success/failure numbers on exit
Update replication plugin
OAuth: Allow to link claimed identity to existing accounts
OAuth: Allow to change username
Change-Id: Ia9fc371b9f957c8e0fc3e215084baa3d31dadd41
When the user needs to be redirected to the OAuth authentication URL
for entering their credentials, the session is not active yet and
there is no OAuthToken available. There is no value then in having
a RequestToken parameter that will always be null anyway.
Change-Id: I00fdbd32923a51e0c92e6bc0efff551936ec344f
One of use cases OAuth plugin based authentication scheme is aiming
to support is switch from deprecated OpenID provider to OAuth scheme
offered by the same povider. In this specific case the database is
already pre-populated with OpenID accounts. After switching the auth
scheme to OAuth all existing accounts must be linked to the new OAuth
identity.
To support linking new OAuth identity to existing accounts, user info
extension point is extended with claimed identity attribute. When
passed, the account for this identity is looked up and when found new
OAuth identity is linked to it.
Change-Id: Ia6489762dd370bfbbaa16a7418cd3106d2d1112a
* stable-2.10:
Set version to 2.10.1
Release notes for Gerrit 2.10.1
Expose extension point for generic OAuth providers
Change-Id: I33006b0ea12611590979f376816d066883177d4f
Further development of OAuth authentication scheme support suggested in
I86fb8fab3 is to restrict the core to expose only the OAuth extension
point and use Gerrit plugin concept for OAuth provider implementations.
When multiple OAuth providers are deployed on Gerrit site (from one or
multiple plugins) selection page is shown to select OAuth provider per
user base (as it's known for OpenID authentication scheme). The only
difference is that the user can only select between deployed providers.
OAuth logo was borrowed from:
http://en.wikipedia.org/wiki/OAuth and
http://en.wikipedia.org/wiki/File:Oauth_logo.svg
The OAuth logo, designed by Chris Messina
Creative Commons Attribution-Share Alike 3.0 Unported license
Converted as base64 using: http://www.base64-image.de
Source for OAuth protocol description: the same link as above.
Bug: issue 2677
Bug: issue 2715
Contributed-by: Luca Milanesio <luca.milanesio@gmail.com>
Change-Id: I7da0a6b3f2a99b6188bd14cf2818f673a3ddd680
