Now the elasticsearch implementation is considered stable enough for
experimental usage, and is mentioned as such in the 2.14 release notes,
we should remove the "flaky" tag so that the tests get run on CI.
Change-Id: I225a902b74730afbc763a0989b4ce7d7481fed42
Recently we changed how Elasticsearch is configured but forgot to update
configuration for integration tests.
Change-Id: I0f17188d2dc2eedf5e05484e98345886c0dc9621
* stable-2.14:
Index: Extract common parts into AbstractVersionManager
ES: Adds configuration for the JEST client
Change-Id: Idcd918450d6ab51fb71f68d949f77e35494043ee
Most of code in LuceneVersionManager and ElasticVersionManager is
similar. The only difference is in the way how both implementation get
current version of schema. This patch extract common parts into
AbstracttVersionManager and refactor both classes to use common parent.
Change-Id: I170e16073b03f648e9904401b0b42b807891d471
Adds new gerrit.config parameters that allows configure JEST client.
Additionally it also moves all Elasticsearch related configuration
options into dedicated section.
Change-Id: I4c6e68783a64273a3de20ec8392d875c28ae7e80
* stable-2.14:
Move systemctl files to /init/ rather than /systemd/
SitePathInitializer: Fix destination file name for gerrit.socket
Clarify documentation for accountPatchReviewDb.url
Add the new gerrit systemctl file to init
Remove unneeded output in MigrateAccountPatchReviewDb
ES: Implement online reindex for ElasticSearch
JdbcAccountPatchReviewStore: Fix copyright year
Fix documentation nits in pgm-MigrateAccountPatchReviewDb.txt
Support Jdbc implementation of AccountPatchReviewStore
ES: Temporarily disable server discovery
Change-Id: I1034869bc6146db929f6242610c9eb13a7b092f7
Add possibility to configure more than one ES servers in the
gerrit.config file.
We also permanently disable auto-discovery feature of JEST since
administrators can configure multiple ES servers it is no longer
required to relay on it. Especially that we would need to write
complicated code to ensure that we don't use not fully initialized JEST
client.
Change-Id: I8277150ee6210183cb41a10ef179d7800471844b
Implement online reindexing for ElasticSearch based on the code for
Lucene online reindex.
Testing scenario:
1. Start fresh Gerrit site with this patch
2. Create account
3. Verify data in ElasticSearch:
curl http://localhost:9200/gerritaccounts_0004/
curl http://localhost:9200/gerritaccounts_0004/_search
4. Stop Gerrit
5. Cherry pick change I77e1643cd1a7fbef9f4d2fa214823759188e9592
6. Start Gerrit
6. Wait for log message:
Starting online reindex from schema version 4 to 5
7. Verify state in ElasticSearch:
curl http://localhost:9200/gerritaccounts_0005/
curl http://localhost:9200/gerritaccounts_0005/_search
Entry for user account created in step 2 should have "elastic_online"
property with value "reindex work".
Change-Id: I9efcf5735e65b4f2dc2a97914d398f81656fc12a
When server discovery is enabled Gerrit will fail to run off-line reindex
and start up. The reason behind it is that we are trying to use JEST
client without assigned servers - the discovery process has not yet
finished.
Therefore for now we should disable this feature and enable it again
when we are able to wait until this discovery process finishes.
Change-Id: Ia89b7c072a5a4e898c29fd0d18d6ce4385fee4e7
Extract jest client builder from AbstractElasticIndex into separate
class. This way we would be able to create other elastic client for
online reindexing purposes like eg. listing current index versios.
Change-Id: I3ab4e94e9f8e24cf0f3085370d3e32cf18eb570b
Documentation points at default values, which is not implemented.
Provide these default configuration values.
Change-Id: Ie727391855d89567405e9817e5b7a7ecef15133f
Extract jest client builder from AbstractElasticIndex into separate
class. This way we would be able to create other elastic client for
online reindexing purposes like eg. listing current index versios.
Change-Id: I3ab4e94e9f8e24cf0f3085370d3e32cf18eb570b
Some secondary index implementations store changes in nonoverlapping
subindexes. Due to the non-atomic nature of these subindexes, we might
fleetingly observe a change as present twice. This is not an error that
the user can do anything about, so relax ChangeFinder a bit in this
specific case.
Change-Id: Ie434937ad6e05c063bae04fc0affe3eaef6a51e0
Documentation points at default values, which is not implemented.
Provide these default configuration values.
Change-Id: Ie727391855d89567405e9817e5b7a7ecef15133f
This change adds reviewers by email to the ChangeIndex and adds tests
for the new code.
It also expands the 'reviewer' and 'cc' queries to match on
reviewerByEmail as well.
Bug: Issue 4134
Change-Id: Iae9ada56e2ab9a03b6d5c20de4bca53ec27a4767
The ref_states and ref_state_patterns fields are being set only when
they exist in the index.
In the case where the fields are requested, but are not in the index,
this results null values exceptions when attempting to populate the
change data.
Set the fields when they are requested. If the field does not exist
in the index, set it to an empty list.
Bug: Issue 5783
Change-Id: I35326bfe175cdd94b43c30f2875e83417be253ba
Currently, the JSON passed to ES when updating a document represents
repeated fields with an array-of-arrays containing exactly one outer
element:
"submit_record": [
[
"NOT_READY",
"NEED,code-review"
]
],
This was not intended. The correct representation should be a single
array:
"submit_record": [
"NOT_READY",
"NEED,code-review"
],
Convert the values to a single array. Also filter out strings with
empty values.
Change-Id: I72b56428ed620c34bf110c57cdda450c493be2f8
Inspired by Han-Wen's entirely reasonable annotation of the "stored"
argument to the FieldDef constructor[1], I thought this smells like it
could use the builder pattern to improve readability. Passing a
functional interface to the build method also means we can use lambdas,
for an overall significant boilerplate reduction: many FieldDefs can now
be one-liners.
[1] https://gerrit-review.googlesource.com/c/98014/9/gerrit-server/src/main/java/com/google/gerrit/server/index/change/ChangeField.java#606
Change-Id: I845a7d9a28dda7f3e0cc0c049b94372118eb4480
When changes are queried, we want to return the number of unresolved
comments with each change. This number should be pre-counted rather than
loading all the comments and counting each time.
We also add new search operators, which allow user to query by
"has:unresolved" and "unresolved:<RELATION><NUMBER>".
The 'unresolvedCommentCount' field of 'ChangeData' will be null if
'lazyLoad' is false.
Change-Id: I97c6aeb47db48f56f94c5d184dfa36c8d7868cc8
Having a standard tool for formatting saves reviewers' valuable time.
google-java-format is Google's standard formatter and is somewhat
inspired by gofmt[1]. This commit formats everything using
google-java-format version 1.2.
The downside of this one-off formatting is breaking blame. This can be
somewhat hacked around with a tool like git-hyper-blame[2], but it's
definitely not optimal until/unless this kind of feature makes its way
to git core.
Not in this change:
* Tool support, e.g. Eclipse. The command must be run manually [3].
* Documentation of best practice, e.g. new 100-column default.
[1] https://talks.golang.org/2015/gofmt-en.slide#3
[2] https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/git-hyper-blame.html
[3] git ls-files | grep java$ | xargs google-java-format -i
Change-Id: Id5f3c6de95ce0b68b41f0a478b5c99a93675aaa3
Signed-off-by: David Pursehouse <dpursehouse@collab.net>
FillArgs are currently not used for indexing accounts and groups,
hence we can just provide null for the FillArgs. This makes the
handling of FillArgs consistent with the Lucene account and group
index, which also use null for FillArgs.
Change-Id: Ib66b9c6a2d7db5ee9841c14670d0da8086d40f34
Signed-off-by: Edwin Kempin <ekempin@google.com>
The Gerrit build of the Bazel team fails with:
ERROR: /home/ci/workspace/gerrit/BAZEL_VERSION/HEAD/PLATFORM_NAME/linux-x86_64/gerrit-elasticsearch/BUILD:32:1:
Java compilation in rule '//gerrit-elasticsearch:elasticsearch_test_utils' failed:
Worker process sent response with exit code: 1..
gerrit-elasticsearch/src/test/java/com/google/gerrit/elasticsearch/ElasticTestUtils.java:162:
error: [strict] Using type com.google.gerrit.reviewdb.client.AccountGroup from an indirect
dependency (TOOL_INFO: "//gerrit-reviewdb:client"). See command below **
Schema<AccountGroup> groupSchema =
^
1 error
Change-Id: I6e3a9988becf087d9b0f9bca6c04721daa4468fc
Signed-off-by: Edwin Kempin <ekempin@google.com>
* changes:
Index groups that are created during init
Implement group index for ElasticSearch
Add REST endpoint to reindex a single group
Support default query for groups
Add query operator that matches groups that are visible to all users
Add group query operator to match groups by owner
Add group query operator to match groups by description
Support querying groups by name and name part
Set '_more_groups' on last group of query result
Add tests for group index
Groups API: Add query methods
Support arbitrary group queries via REST
Add new optional interface for REST collections to get request params
Add GroupQueryProcessor
IsVisibleToPredicate: Move describe method into base class
Index group on creation and whenever a group is evicted from cache
Initial implementation of group index in Lucene
Add initial GroupQueryBuilder and define key predicate for group index
Add schema and index definitions for groups index
During the initialization of a new site the InitAdminUser init step
may create an account for the initial admin user. This new account
must be added to the index so that it becomes queryable. For this init
must have the account index available so that it can write the new
account to it. Reading from the index during init is not needed and
not supported.
Alternatively we could require users to run the Reindex program after
the initial site initialization, but likely many people would forget
about it and then wonder why the initial admin user is not working.
Change-Id: Iebee152fa0b28df3fe59da74ad688c4373488c53
Signed-off-by: Edwin Kempin <ekempin@google.com>
During the initialization of a new site some default groups are
automatically created ('Administrators' and 'Non-Interactive Users').
These new groups must be added to the index so that they become
queryable. For this init must have the group index available so that
it can write the new groups to it. Reading from the index during init
is not needed and not supported.
Alternatively we could require users to run the Reindex program after
the initial site initialization, but likely many people would forget
about it and then wonder why the default groups are not available from
the group index.
Change-Id: I274b142a6efde224bba68f8b5c459736d7a1f985
Signed-off-by: Edwin Kempin <ekempin@google.com>
Most index implementations have an empty stop() method. Make use of
Java 8 and make the empty stop() method the default implementation in
the Index interface, so that most indexes don't need to implement it.
Change-Id: I3889b4a13f7253e4129254f946040f8714c3a570
Signed-off-by: Edwin Kempin <ekempin@google.com>
Fixed the tests, so skipping them is no longer required.
This reverts commit dd7f60e6c942c314f0e8667edad99c025925d73d.
Change-Id: Ib9e3bb56745dbeeb43f9ecf2184b9dfbe55c245b
Change query tests subtly depend on the account index for the account
setup step: omitting the account index caused
AccountManager#authenticate to throw an exception if an account
external ID was repeated, instead of returning the account with that
ID.
Always set up all indexes, using a helper in ElasticTestUtils. The
delete code can be simplified as well using the special string "_all"
to delete all indexes.
Change-Id: I7ab7be37a8cf953b6873745014e781e6b5e910f8
ES test were disabled in both Buck and Bazel driven builds on
GerritForge CI (labels flaky).
Since I4206f725e ES tests are inherently broken and we temporarily
disable them on Bazel CI as well.
Change-Id: Iaa6eee03c0204b20cb77402ccfe89b632b3fc7f4
Reformat the Bazel build files with the buildifier tool [1].
The style is different for Bazel files. Most notably, indentation level
is 4 spaces instead of 2, and " is used instead of '.
[1] https://github.com/bazelbuild/buildifier
Change-Id: I95c0c6f11b6d76572797853b4ebb5cee5ebd3c98
* changes:
ChangeIndexer: Reindex if stale after every index update
ChangeIndexer: Add method to reindex a change if it is stale
Implement staleness checker for index changes
Store in index the state of all refs that contribute to a change
Add method on Index to look up a single document by key
Initial implementation of the account index for ElasticSearch.
TODO: Define schema mappings when Gerrit starts up
TODO: Add support for online reindex
Change-Id: I21034daa71c36a1e462e3107b0c9a7ba6392f379
Signed-off-by: Dariusz Luksza <dluksza@collab.net>
Signed-off-by: David Pursehouse <dpursehouse@collab.net>
As a preparatory step before adding the implementation of the account
index in Elastic Search, split out code that can be reused into utility
classes.
Change-Id: I46e0fe4e4e2ef969565cd2f481ab45c46397016e
Signed-off-by: Dariusz Luksza <dluksza@collab.net>
Signed-off-by: David Pursehouse <dpursehouse@collab.net>
