opendev/gerrit
Code Issues Proposed changes
1531f6101a
gerrit/Documentation/dev-bazel.txt
David Ostrovsky f55b69f5c9 Bazel: Add support for Java 10 
Bazel@HEAD doesn't support Java 10 any more. The tool chain is extended
to support building with Java 10 using host_javabase option and vanilla
java builder. To use --host_javabase option we have to provide absolute
path to the JDK 10. To keep that non-portable part out of the build file
we use variable that is substitued during build invocation. Say, the
location of the JDK 10 is: /usr/lib64/jvm/java-10, then the build is
invoked with:

  $ bazel build --host_javabase=:absolute_javabase \
    --define=ABSOLUTE_JAVABASE=/usr/lib64/jvm/java-10 ...

Given that absolute_javabase rule is not a part of released Bazel yet,
but will be only included in future Bazel releases, we have to add it in
our own build file:

  java_runtime(
      name = "absolute_javabase",
      java_home = "$(ABSOLUTE_JAVABASE)",
      visibility = ["//visibility:public"],
  )

While this works, it fails the Gerrit-CI, because recently it was
changed to exercise //... rule, and cannot resolve ABSOLUTE_JAVABASE
variable, because it wasn't provided in the CI environment. CI is still
using Java 8.

One approach to address that problem would be to use "manual" tag for a
rule that would exclude it from generic targets like //..., but
unfortunately, java_runtime rule doesn't expose tag attribute.

The next workaround is to hide that variable behind a select statement.
This is a harmless hack:

  config_setting(
      name = "use_absolute_javabase",
      values = {"define": "USE_ABSOLUTE_JAVABASE=true"},
  )

  java_runtime(
      name = "absolute_javabase",
      java_home = select({
          ":use_absolute_javabase": "$(ABSOLUTE_JAVABASE)",
          "//conditions:default": "",
      }),
      visibility = ["//visibility:public"],
  )

Now from the CI the rule: //:absolute_javabase would produce non working
java_runtime, because java_home wasn't specified, but given that this
java_runtime is not used during the build, it doesn't matter. Add also
a TODO comment to replace that interim solution when absolute_javabase
is supported in released Bazel version. As the consequence for hiding
that rule behind a condition, we need to provide additional variable so
that Bazel can correctly resolve java_home in java_runtime rule:

  $ bazel build --host_javabase=:absolute_javabase \
    --define=ABSOLUTE_JAVABASE=/usr/lib64/jvm/java-10 \
    --define=USE_ABSOLUTE_JAVABASE=true [...]

Also extend tools/eclipse/project.py to accept Java 10 specific options.
There is already --java <jdk number> option that was added recently to
support Eclipse .classpath generation when JDK 9 is used, but this can
not be used, because for Java 10 we have to provide absolute path to the
Java 10 location. Add new option --edge_java where all Java 10 specific
options can be passed:

  $ tools/eclipse/project.py --edge_java \
    "--host_javabase=:absolute_javabase \
    --define=ABSOLUTE_JAVABASE=/usr/lib64/jvm/java-10 \
    --define=USE_ABSOLUTE_JAVABASE=true \
    --host_java_toolchain=//:toolchain_vanilla \
    --java_toolchain=//:toolchain_vanilla"

Alternative approach would be to add those options to Bazel resource
file.

Test Plan:

  $ bazel build --host_javabase=:absolute_javabase \
    --define=ABSOLUTE_JAVABASE=/usr/lib64/jvm/java-10 \
    --define=USE_ABSOLUTE_JAVABASE=true \
    --host_java_toolchain=//:toolchain_vanilla \
    --java_toolchain=//:toolchain_vanilla \
    :release

and replace ABSOLUTE_JAVABASE variable with the location of JDK 10.

To verify that major version of generated classes is 54, use:

  $ $JAVA_HOME/bin/javap -verbose \
    -cp bazel-bin/java/com/google/gerrit/common/libserver.jar \
     com.google.gerrit.common.data.ProjectAccess | grep "major version"
  54

[1] https://github.com/bazelbuild/bazel/issues/6012
Change-Id: I5e652f7a19afbcf3607febd9a7a165dc07918bd0
2018-09-25 07:31:33 +00:00

541 lines
13 KiB
Plaintext
Raw Blame History

= Gerrit Code Review - Building with Bazel
[[installation]]
== Prerequisites
To build Gerrit from source, you need:
* A Linux or macOS system (Windows is not supported at this time)
* A JDK for Java 8|9|10
* Python 2 or 3
* Node.js
* link:https://www.bazel.io/versions/master/docs/install.html[Bazel]
* Maven
* zip, unzip
* gcc
[[Java 10 support]]
Java 10 is supported through vanilla java toolchain
link:https://docs.bazel.build/versions/master/toolchains.html[Bazel option].
To build Gerrit with Java 10, specify vanilla java toolchain and provide
path to Java 10 home:
```
$ bazel build --host_javabase=:absolute_javabase \
--define=ABSOLUTE_JAVABASE=<path-to-java-10> \
--define=USE_ABSOLUTE_JAVABASE=true \
--host_java_toolchain=//:toolchain_vanilla \
--java_toolchain=//:toolchain_vanilla \
:release
```
Note that the following options must be added to `container.javaOptions`
in `$gerrit_site/etc/gerrit.config` to run Gerrit with Java 10:
```
[container]
javaOptions = --add-modules java.activation
javaOptions = --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
```
[[Java 9 support]]
Java 9 is supported through alternative java toolchain
link:https://docs.bazel.build/versions/master/toolchains.html[Bazel option].
The Java 9 support is backwards compatible. Java 8 is still the default.
To build Gerrit with Java 9, specify JDK 9 java toolchain:
```
$ bazel build \
--host_java_toolchain=@bazel_tools//tools/jdk:toolchain_java9 \
--java_toolchain=@bazel_tools//tools/jdk:toolchain_java9 \
:release
```
Note that the following option must be added to `container.javaOptions`
in `$gerrit_site/etc/gerrit.config` to run Gerrit with Java 9:
```
[container]
javaOptions = --add-modules java.activation \
--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
```
[[build]]
== Building on the Command Line
=== Gerrit Development WAR File
To build the Gerrit web application that includes the PolyGerrit UI:
----
bazel build gerrit
----
[NOTE]
PolyGerrit UI may require additional tools (such as npm). Please read
the polygerrit-ui/README.md for more info.
The output executable WAR will be placed in:
----
bazel-bin/gerrit.war
----
[[release]]
=== Gerrit Release WAR File
To build the Gerrit web application that includes the GWT UI, the
PolyGerrit UI, core plugins and documentation:
----
bazel build release
----
The output executable WAR will be placed in:
----
bazel-bin/release.war
----
=== Headless Mode
To build Gerrit in headless mode, i.e. without the PolyGerrit and GWT
Web UI:
----
bazel build headless
----
The output executable WAR will be placed in:
----
bazel-bin/headless.war
----
=== Extension and Plugin API JAR Files
To build the extension, plugin and GWT API JAR files:
----
bazel build api
----
The output archive that contains Java binaries, Java sources and
Java docs will be placed in:
----
bazel-genfiles/api.zip
----
Install {extension,plugin,gwt}-api to the local maven repository:
----
tools/maven/api.sh install
----
Install gerrit.war to the local maven repository:
----
tools/maven/api.sh war_install
----
=== Plugins
----
bazel build plugins:core
----
The output JAR files for individual plugins will be placed in:
----
bazel-genfiles/plugins/<name>/<name>.jar
----
The JAR files will also be packaged in:
----
bazel-genfiles/plugins/core.zip
----
To build a specific plugin:
----
bazel build plugins/<name>
----
The output JAR file will be be placed in:
----
bazel-genfiles/plugins/<name>/<name>.jar
----
Note that when building an individual plugin, the `core.zip` package
is not regenerated.
To build with all Error Prone warnings activated, run:
----
bazel build --java_toolchain //tools:error_prone_warnings_toolchain //...
----
[[IDEs]]
== Using an IDE.
=== IntelliJ
The Gerrit build works with Bazel's link:https://ij.bazel.io[IntelliJ plugin].
Please follow the instructions on <<dev-intellij#,IntelliJ Setup>>.
=== Eclipse
==== Generating the Eclipse Project
Create the Eclipse project:
----
tools/eclipse/project.py
----
and then follow the link:dev-eclipse.html#setup[setup instructions].
==== Refreshing the Classpath
If an updated classpath is needed, the Eclipse project can be
refreshed and missing dependency JARs can be downloaded by running
`project.py` again. For IntelliJ, you need to click the `Sync Project
with BUILD Files` button of link:https://ij.bazel.io[IntelliJ plugin].
[[documentation]]
=== Documentation
To build only the documentation for testing or static hosting:
----
bazel build Documentation:searchfree
----
The html files will be bundled into `searchfree.zip` in this location:
----
bazel-bin/Documentation/searchfree.zip
----
To build the executable WAR with the documentation included:
----
bazel build withdocs
----
The WAR file will be placed in:
----
bazel-bin/withdocs.war
----
[[tests]]
== Running Unit Tests
----
bazel test --build_tests_only //...
----
Debugging tests:
----
bazel test --test_output=streamed --test_filter=com.gerrit.TestClass.testMethod testTarget
----
Debug test example:
----
bazel test --test_output=streamed --test_filter=com.google.gerrit.acceptance.api.change.ChangeIT.getAmbiguous //javatests/com/google/gerrit/acceptance/api/change:api_change
----
To run a specific test group, e.g. the rest-account test group:
----
bazel test //javatests/com/google/gerrit/acceptance/rest/account:rest_account
----
To run the tests against NoteDb backend with write
to NoteDb, but not read from it:
----
bazel test --test_env=GERRIT_NOTEDB=WRITE //...
----
Write and read from NoteDb:
----
bazel test --test_env=GERRIT_NOTEDB=READ_WRITE //...
----
Primary storage NoteDb:
----
bazel test --test_env=GERRIT_NOTEDB=PRIMARY //...
----
Primary storage NoteDb and ReviewDb disabled:
----
bazel test --test_env=GERRIT_NOTEDB=ON //...
----
To run only tests that do not use SSH:
----
bazel test --test_env=GERRIT_USE_SSH=NO //...
----
To exclude tests that have been marked as flaky:
----
bazel test --test_tag_filters=-flaky //...
----
To exclude tests that require a Docker host:
----
bazel test --test_tag_filters=-docker //...
----
To ignore cached test results:
----
bazel test --cache_test_results=NO //...
----
To run one or more specific groups of tests:
----
bazel test --test_tag_filters=api,git //...
----
The following values are currently supported for the group name:
* annotation
* api
* docker
* edit
* elastic
* git
* notedb
* pgm
* rest
* server
* ssh
[[elasticsearch]]
=== Elasticsearch
Successfully running the Elasticsearch tests requires Docker, and
may require setting the local
link:https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html[virtual memory].
If Docker is not available, the Elasticsearch tests will be skipped.
Note that Bazel currently does not show
link:https://github.com/bazelbuild/bazel/issues/3476[the skipped tests].
== Dependencies
Dependency JARs are normally downloaded as needed, but you can
download everything upfront. This is useful to enable
subsequent builds to run without network access:
----
bazel fetch //...
----
When downloading from behind a proxy (which is common in some corporate
environments), it might be necessary to explicitly specify the proxy that
is then used by `curl`:
----
export http_proxy=http://<proxy_user_id>:<proxy_password>@<proxy_server>:<proxy_port>
----
Redirection to local mirrors of Maven Central and the Gerrit storage
bucket is supported by defining specific properties in
`local.properties`, a file that is not tracked by Git:
----
echo download.GERRIT = http://nexus.my-company.com/ >>local.properties
echo download.MAVEN_CENTRAL = http://nexus.my-company.com/ >>local.properties
----
The `local.properties` file may be placed in the root of the gerrit repository
being built, or in `~/.gerritcodereview/`. The file in the root of the gerrit
repository has precedence.
== Building against unpublished Maven JARs
To build against unpublished Maven JARs, like gwtorm or PrologCafe, the custom
JARs must be installed in the local Maven repository (`mvn clean install`) and
`maven_jar()` must be updated to point to the `MAVEN_LOCAL` Maven repository for
that artifact:
[source,python]
----
maven_jar(
name = 'gwtorm',
artifact = 'gwtorm:gwtorm:42',
repository = MAVEN_LOCAL,
)
----
== Building against artifacts from custom Maven repositories
To build against custom Maven repositories, two modes of operations are
supported: with rewrite in local.properties and without.
Without rewrite the URL of custom Maven repository can be directly passed
to the maven_jar() function:
[source,python]
----
GERRIT_FORGE = 'http://gerritforge.com/snapshot'
maven_jar(
name = 'gitblit',
artifact = 'com.gitblit:gitblit:1.4.0',
sha1 = '1b130dbf5578ace37507430a4a523f6594bf34fa',
repository = GERRIT_FORGE,
)
----
When the custom URL has to be rewritten, then the same logic as with Gerrit
known Maven repository is used: Repo name must be defined that matches an entry
in local.properties file:
----
download.GERRIT_FORGE = http://my.company.mirror/gerrit-forge
----
And corresponding WORKSPACE excerpt:
[source,python]
----
GERRIT_FORGE = 'GERRIT_FORGE:'
maven_jar(
name = 'gitblit',
artifact = 'com.gitblit:gitblit:1.4.0',
sha1 = '1b130dbf5578ace37507430a4a523f6594bf34fa',
repository = GERRIT_FORGE,
)
----
[[consume-jgit-from-development-tree]]
To consume the JGit dependency from the development tree, edit
`lib/jgit/jgit.bzl` setting LOCAL_JGIT_REPO to a directory holding a
JGit repository.
[[bazel-local-caches]]
To accelerate builds, several caches are activated per default:
* ~/.gerritcodereview/bazel-cache/downloaded-artifacts
* ~/.gerritcodereview/bazel-cache/repository
* ~/.gerritcodereview/bazel-cache/cas
Currently none of these caches have a maximum size limit. See
link:https://github.com/bazelbuild/bazel/issues/5139[this bazel issue] for
details. Users should watch the cache sizes and clean them manually if
necessary.
[[npm-binary]]
== NPM Binaries
Parts of the PolyGerrit build require running NPM-based JavaScript programs as
"binaries". We don't attempt to resolve and download NPM dependencies at build
time, but instead use pre-built bundles of the NPM binary along with all its
dependencies. Some packages on
link:https://docs.npmjs.com/misc/registry[registry.npmjs.org] come with their
dependencies bundled, but this is the exception rather than the rule. More
commonly, to add a new binary to this list, you will need to bundle the binary
yourself.
[NOTE]
We can only use binaries that meet certain licensing requirements, and that do
not include any native code.
Start by checking that the license and file types of the bundle are acceptable:
[source,bash]
----
gerrit_repo=/path/to/gerrit
package=some-npm-package
version=1.2.3
npm install -g license-checker && \
rm -rf /tmp/$package-$version && mkdir -p /tmp/$package-$version && \
cd /tmp/$package-$version && \
npm install $package@$version && \
license-checker | grep licenses: | sort -u
----
This will output a list of the different licenses used by the package and all
its transitive dependencies. We can only legally distribute a bundle via our
storage bucket if the licenses allow us to do so. As long as all of the listed
license are allowed by
link:https://opensource.google.com/docs/thirdparty/licenses/[Google's
standards]. Any `by_exception_only`, commercial, prohibited, or unlisted
licenses are not allowed; otherwise, it is ok to distribute the source. If in
doubt, contact a maintainer who is a Googler.
Next, check the file types:
[source,bash]
----
cd /tmp/$package-$version
find . -type f | xargs file | grep -v 'ASCII\|UTF-8\|empty$'
----
If you see anything that looks like a native library or binary, then we can't
use the bundle.
If everything looks good, create the bundle, and note the SHA-1:
[source,bash]
----
$gerrit_repo/tools/js/npm_pack.py $package $version && \
sha1sum $package-$version.tgz
----
This creates a file named `$package-$version.tgz` in your working directory.
Any project maintainer can upload this file to the
link:https://console.cloud.google.com/storage/browser/gerrit-maven/npm-packages[storage
bucket].
Finally, add the new binary to the build process:
----
# WORKSPACE
npm_binary(
name = "some-npm-package",
repository = GERRIT,
)
# lib/js/npm.bzl
NPM_VERSIONS = {
...
"some-npm-package": "1.2.3",
}
NPM_SHA1S = {
...
"some-npm-package": "<sha1>",
}
----
To use the binary from the Bazel build, you need to use the `run_npm_binary.py`
wrapper script. For an example, see the use of `crisper` in `tools/bzl/js.bzl`.
GERRIT
------
Part of link:index.html[Gerrit Code Review]
SEARCHBOX
---------
View Git Blame Copy Permalink