On slaves the test-submit SSH command was bound to
NotSupportedInSlaveModeFailureCommand. While this resulted in a nice
error message, confusingly this caused the test-submit command to appear
in the list of available commands when running 'gerrit --help' on
slaves. Remove the binding on slaves so that the test-submit command is
not listed as available command. This is consistent with how all other
SSH commands that are not available on slaves are handled.
Change-Id: Icb1e1ed79c0f0472d93a133516ec50b19c353d06
Signed-off-by: Edwin Kempin <ekempin@google.com>
Gerrit slaves are supposed to be read-only but the ban-commit command
creates a Git note and hence writes to the repository.
Change-Id: Ic6e43d88cfc22877256e8a6f8d8ee3673c0664e8
Signed-off-by: Edwin Kempin <ekempin@google.com>
* stable-2.13:
Fix typo in waitTimeout configuration and clarify its use
SshDaemon: introduce sshd.waitTimeout to set WAIT_FOR_SPACE_TIMEOUT
Change-Id: I9d8b69a5dbbf3a5f173285b463e6c4c92b1bdbe5
sshd introduced a new channel property
'channel-output-wait-for-space-timeout' [1] set by default to 30s.
The property isn't exposed at the ChannelSessionFactory level and thus
the default value remained hardcoded in the properties.
Without this config the consequence is that clones that requires the
server process to spend over 30s are failing.
Allow administrators to configure this via a new setting sshd.waitTimout.
Default to 30 seconds if it is not set.
[1] https://issues.apache.org/jira/browse/SSHD-565
Bug: Issue 7425
Change-Id: Ib3fd9a25d7eaaa87a15d5c159995e09a9581dadb
When executing the command:
ssh user@host gerrit stream-events --help
the onExit and destroy methods attempt to remove the event listener
registration which has not been initialized, resulting in NPE and
the command hanging.
Add null checks to prevent this.
Change-Id: I1b9ccd41d017e62f0a4206114ab15faa6ed8e000
As described in SSHD-715 [1] the NIO2_READ_TIMEOUT parameter was
introduced in sshd version 1.3.
[1] https://issues.apache.org/jira/browse/SSHD-715
Bug: Issue 6173
Change-Id: I6f930cafef9583a83aed2e6d05ff2a9f27c33cb8
FileInputStream and FileOutputStream rely on finalize() method to ensure
resources are closed. This implies they are added to the finalizer queue
which causes additional work for the JVM GC process.
This is an open bug on the OpenJDK [1] and the recommended workaround is
to use the Files.newInputStream and Files.newOutputStream static methods
instead.
[1] https://bugs.openjdk.java.net/browse/JDK-8080225
Change-Id: I3cef6fcf198dde2be7cd15bded8d2fa247177654
Since we now ship BouncyCastle in the .war file, this will always
return true. Remove the code that is now redundant.
Change-Id: I35d6191b6f5e4cea40a022236cbc848eb01d7ba1
Include the key string, which will help to track down which one
failed, and omit the entire stack trace, which doesn't actually
provide any useful information.
Change-Id: I67c1fbe75c99f8cda6dbebe27c050e338e571315
Implement online reindexing for ElasticSearch based on the code for
Lucene online reindex.
Testing scenario:
1. Start fresh Gerrit site with this patch
2. Create account
3. Verify data in ElasticSearch:
curl http://localhost:9200/gerritaccounts_0004/
curl http://localhost:9200/gerritaccounts_0004/_search
4. Stop Gerrit
5. Cherry pick change I77e1643cd1a7fbef9f4d2fa214823759188e9592
6. Start Gerrit
6. Wait for log message:
Starting online reindex from schema version 4 to 5
7. Verify state in ElasticSearch:
curl http://localhost:9200/gerritaccounts_0005/
curl http://localhost:9200/gerritaccounts_0005/_search
Entry for user account created in step 2 should have "elastic_online"
property with value "reindex work".
Change-Id: I9efcf5735e65b4f2dc2a97914d398f81656fc12a
REST API and UI allow project owner to change the project settings so
fix inconsistency by allowing the same in the ssh command.
Change-Id: I123007629db87c1df6162cb1e56fc51bacff9631
We cannot shade bouncycastle in the plugin API. Still we need it to be
included in the gerrit.war, licenses file and Eclipse classpath.
Expose bouncycastle libraries in PLUGIN_TEST_DEPS constant, so that
the plugins don't need to change anything in tree build mode.
gerrit_api() bazlet in bazlets repository is extended too, so that the
plugins don't need to change anything in standalone build mode.
One side effect of this change, is that bouncycastle libraries are
now listed with neverlink suffix, e.g.:
* bouncycastle:bcprov-neverlink
Bug: Issue 5826
Change-Id: Idb8051e16b14e20c8dd528783ab297ee25707bb3
When Gerrit SSH Daemon is stopped, there is no value in keeping its
executors threads alive as it would only consume precious resources
we do need during our Integration Tests suite.
Apache SSHD does not manage the shutdown of the internal executors
by himself, so we need to close them manually.
Change-Id: I09a62759769bbb222abd4a3ea60be8b8c5571ac9
The LfsPluginAuthCommand is only installed when lfs.plugin is set in the
gerrit.config file. However, if the corresponding plugin is not actually
installed, or it does not provide the command implementation, Failure is
thrown which results in a stack trace being emitted to the log on every
ssh push operation.
Change it to throw UnloggedFailure, and write a less verbose message to
the log at warn level. This will still be emitted to the log on every
ssh push, but it's less noisy than before.
Change-Id: Ie638bda96ddbac173eccd74f9f6a25ed87c778c8
Gerrit users who are delegated to maintain the server may be highly
restricted to access projects and branches.
There are maintenance operations that can still be allowed
on change-ids even on non-visible projects.
By allowing users delegated to maintain the server to find
non-visible change-ids in the CLI we can enable features such as
the ad-hoc change indexing.
Change-Id: Ia2ec07517c70f2cc4aed1bee832ea04a45fc6466
The Gerrit SSH command "index changes" may receive change numbers that
cannot be found or refer to repositories that have problems.
The assumption of "DB is down" when receiving an OrmException was wrong
because the JGit errors were simply wrapped as OrmException and
do not necessarily mean that you cannot continue to process the other
ones in the list.
Change-Id: I1098be16b682dce8d4e63120e7e50f937b19a7b8
(cherry picked from commit 9a52dd1dce8a79b90e8bfad4f718659aafe1a3e8)
Display a warning when a change is not found but then
continue to try to reindex the rest of the changes, consistently
with the current behavior of the command.
Change-Id: I51b98a8bb28362456ac7a5df5efa74c1fa1de284
In NoteDb external IDs are stored in the All-Users repository in a Git
Notes branch called refs/meta/external-ids where the sha1 of the
external ID is used as note name. Each note content is a Git config
file that contains an external ID. It has exactly one externalId
subsection with an accountId and optionally email and password:
[externalId "username:jdoe"]
accountId = 1003407
email = jdoe@example.com
password = bcrypt:4:LCbmSBDivK/hhGVQMfkDpA==:XcWn0pKYSVU/UJgOvhidkEtmqCp6oKB7
Storing the external IDs in a Git Notes branch with using the sha1 of
the external ID as note name ensures that external IDs are unique and
are only assigned to a single account. If it is tried to assign the
same external ID concurrently to different accounts, only one Git
update succeeds while the other Git updates fail with LOCK_FAILURE.
This means assigning external IDs is also safe in a multimaster setup
if a consensus algorithm for updating Git refs is implemented (which
is needed for multimaster in any case). Alternatively it was
considered to store the external IDs per account as Git config file in
the refs/users/<sharded-id> user branches in the All-Users repository
(see abandoned change 9f9f07ef). This approach was given up because in
race conditions it allowed to assign the same external ID to different
accounts by updating different branches in Git.
To support a live migration on a multi-master Gerrit installation, the
migration of external IDs from ReviewDb to NoteDb is done in 2 steps:
- part 1 (this change):
* always write to both backends (ReviewDb and NoteDb)
* always read external IDs from ReviewDb
* upgraded instances write to both backends, old instances only
write to ReviewDb
* after upgrading all instances (all still read from ReviewDb)
run a batch to copy all external IDs from the ReviewDb to NoteDb
- part 2 (next change):
* bump the database schema version
* migrate the external IDs from ReviewDb to NoteDb (for single instance
Gerrit servers)
* read external IDs from NoteDb
* delete the database table
With this change reading external IDs from NoteDb is not implemented
yet. This is because the storage format of external IDs in NoteDb
doesn't support efficient lookup of external IDs by account and this
problem is only addressed in the follow-up change (it adds a cache for
external IDs, but this cache uses the revision of the notes branch as
key, and hence can be only implemented once the external IDs are fully
migrated to NoteDb and storing external IDs in ReviewDb is dropped).
The ExternalIdsUpdate class implements updating of external IDs in
both NoteDb and ReviewDb. It provides various methods to update
external IDs (e.g. insert, upsert, delete, replace). For NoteDb each
method invocation leads to one commit in the Git notes branch.
ExternalIdsUpdate has two factories, User and Server. This allows to
record either the calling user or the Gerrit server identity as
committer for an update of the external Ids.
External IDs are now represented by a new AutoValue class called
ExternalId. This class replaces the usage of the old gwtorm entity
AccountExternalId class. For ExternalId scheme names are the same as for
AccountExternalId but no longer include the trailing ':'.
The class ExternalIdsOnInit makes it possible to update external IDs
during the init phase. This is required for inserting external IDs for
the initial admin user which is created by InitAdminUser. We need a
special class for this since not all dependencies of ExternalIdsUpdate
are available during init.
The class ExternalIdsBatchUpdate allows to do batch updates to
external IDs. For NoteDb all updates will result in a single commit to
the refs/meta/external-ids Git notes branch.
LocalUsernamesToLowerCase is now always converting the usernames in a
single thread only. This allows us to get a single commit for the
username convertion in NoteDb (this would not be possible if workers
do updates in parallel). Since LocalUsernamesToLowerCase is rather
light-weight being able to parallelize work is not really needed and
removing the workers simplifies the code significantly.
To protect the refs/meta/external-ids Git notes branch in the All-Users
repository read access for this ref is only allowed to users that have
the 'Access Database' global capability assigned. In addition
there is a commit validator that disallows updating the
refs/meta/external-ids branch by push. This is to prevent that the
external IDs in NoteDb diverge from the external IDs in ReviewDb while
the migration to NoteDb is not fully done yet.
Change-Id: Ic9bd5791e84ee8d332ccb1f709970b59ee66b308
Signed-off-by: Edwin Kempin <ekempin@google.com>
So far, indexing a series of changes had to be done by listing them one
after the other which can be cumbersome when the number of changes to
re-index is high.
Add the possibility of indexing all the changes belonging to a project.
Change-Id: Ib47f6d65ca49c85713d30806d7579f9d2d9cf823
This differs from the behavior of %r= in the magic branch or even
the new push option support of "-o r=". Drop the visibility check
here and just pass down the identities to ReceiveCommits so that
behavior is consistent.
Change-Id: Iabec4d5c48afd4e820d6bd824c6ae24b0a1e37b5
Reformatting the code with google-java-format tool in change Id5f3c6de9
resulted in wrapped lines being unwrapped to the new 100 columns limit.
In cases where the wrapped line was wrapped on a string concatenation,
the resulting long line now includes redundant concatenations. See the
upstream issue [1] for an example of this.
Squash the redundant concatenations with:
git ls-files | grep java$ | xargs sed -i '' 's/" + "//g'
(note: this also resulted in a couple of unwanted changes; those are
manually undone.)
[1] https://github.com/google/google-java-format/issues/122
Change-Id: I7348413ae460c8c7a0b0c72dab0a1ae7275a2ec3
Also add a test for email deletion and add asserts for the addEmail()
test.
Bug: Issue 5385
Change-Id: I3e192638fdaac58d0cdaa535430bb968d7d32abb
Signed-off-by: Edwin Kempin <ekempin@google.com>
Having a standard tool for formatting saves reviewers' valuable time.
google-java-format is Google's standard formatter and is somewhat
inspired by gofmt[1]. This commit formats everything using
google-java-format version 1.2.
The downside of this one-off formatting is breaking blame. This can be
somewhat hacked around with a tool like git-hyper-blame[2], but it's
definitely not optimal until/unless this kind of feature makes its way
to git core.
Not in this change:
* Tool support, e.g. Eclipse. The command must be run manually [3].
* Documentation of best practice, e.g. new 100-column default.
[1] https://talks.golang.org/2015/gofmt-en.slide#3
[2] https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/git-hyper-blame.html
[3] git ls-files | grep java$ | xargs google-java-format -i
Change-Id: Id5f3c6de95ce0b68b41f0a478b5c99a93675aaa3
Signed-off-by: David Pursehouse <dpursehouse@collab.net>
Guava team recommends MultimapBuilder over specific Multimap
implementations, so callers don't have to know the specific key/value
behaviors of the individual implementations. The static factory
methods in the implementations will be removed in a later version.
LinkedListMultimap and LinkedHashMultimap are not affected, since
MultimapBuilder lacks support for specifying linked entries, and thus
the factory methods are not in immediate danger of deletion.
Change-Id: I7744db687da84a7beae31d1cb8953e782ed23c1d
Guava team recommends using the subinterfaces of Multimap, for the
same reasons they recommend using Set and List rather than Collection:
it documents expectations about ordering, uniqueness, and behavior of
equals. Do this across the board in Gerrit.
Mostly this is straightforward and I tried to exactly match existing
behavior where possible. However, there were a few wrinkles, where
different callers passed different subtypes to the same method.
The main one is arguments to ParameterParser#parse and
splitQueryString, where some callers used SetMultimaps (perhaps
semi-intentionally, or perhaps misunderstanding the nature of
HashMultimap). For the purposes of parameter parsing, a ListMultimap
makes more sense, because it preserves argument order and repetition.
Another instance is a couple places in ReceiveCommits and downstream
where there were SetMultimap<?, Ref>. Since Refs do not implement
equals, this is effectively the same thing as a ListMultimap, and
changing the interface no longer misleads readers into thinking there
might be some deduplication happening.
Finally, this change includes a breaking API change to the return
type of ExternalIncludedIn#getIncludedIn.
Change-Id: I5f1d15e27a32e534a6aaefe204e7a31815f4c8d7
SSH command registration was extended so that command gets registered
only in case when plugin is configured in gerrit.config.
Change-Id: Iffaf00775bdf1242e1fbe1dea15e5ce0bf912079
Signed-off-by: Jacek Centkowski <geminica.programs@gmail.com>
According to [1] Git LFS can use SSH protocol to either obtain Git LFS
endpoint or authorize the following Git LFS request (upload/download).
This patchset introduces Git LFS SSH command stub that forwards call to
implementation provided by Git LFS plugin. In case no plugin is loaded
it exits with 1.
Different attempts to solve this problem were submitted for review ([2],
[3]) but there were either too generic or simply to controversial to
address the problem correctly. I believe that this attempt is superior
as:
1. it is specific to the problem - doesn't open any Pandora's box
2. it comes along with existing solution that adds Git LFS HTTP Servlet
endpoint that transfers Git LFS HTTP calls to plugin
[1]
https://github.com/git-lfs/git-lfs/blob/master/docs/api/server-discovery.md
[2] https://gerrit-review.googlesource.com/93496
[3] https://gerrit-review.googlesource.com/93373
Change-Id: I5f9b95c4b29cd8c0d64053e986851500f147eb5a
Signed-off-by: Jacek Centkowski <geminica.programs@gmail.com>
According to [1] Git LFS can use SSH protocol to either obtain Git LFS
endpoint or authorize the following Git LFS request (upload/download).
This patchset introduces Git LFS SSH command stub that forwards call to
implementation provided by Git LFS plugin. In case no plugin is loaded
it exits with 1.
Different attempts to solve this problem were submitted for review ([2],
[3]) but there were either too generic or simply to controversial to
address the problem correctly. I believe that this attempt is superior
as:
1. it is specific to the problem - doesn't open any Pandora's box
2. it comes along with existing solution that adds Git LFS HTTP Servlet
endpoint that transfers Git LFS HTTP calls to plugin
[1]
https://github.com/git-lfs/git-lfs/blob/master/docs/api/server-discovery.md
[2] https://gerrit-review.googlesource.com/93496
[3] https://gerrit-review.googlesource.com/93373
Change-Id: I5f9b95c4b29cd8c0d64053e986851500f147eb5a
Signed-off-by: Jacek Centkowski <geminica.programs@gmail.com>
The Gerrit SSH command "index changes" may receive change numbers that
cannot be found or refer to repositories that have problems.
The assumption of "DB is down" when receiving an OrmException was wrong
because the JGit errors were simply wrapped as OrmException and
do not necessarily mean that you cannot continue to process the other
ones in the list.
Change-Id: I1098be16b682dce8d4e63120e7e50f937b19a7b8
Reformat the Bazel build files with the buildifier tool [1].
The style is different for Bazel files. Most notably, indentation level
is 4 spaces instead of 2, and " is used instead of '.
[1] https://github.com/bazelbuild/buildifier
Change-Id: I95c0c6f11b6d76572797853b4ebb5cee5ebd3c98
Display a warning when a change is not found but then
continue to try to reindex the rest of the changes, consistently
with the current behavior of the command.
Change-Id: Idbd3d450c34788a9f5dd304b052e5810593088fe
When an invalid Change-Id containing a comma ',' is provided to the
SSH gerrit index changes command, it does not blow up and instead
returns an "Invalid change ID" error and exit code 2.
Change-Id: I97e888e372eb886b9ac9815791a7a725802d736f
Bug: Issue 4911
