ebeb15460c
Asciidoc may fail to render link attributes for external links supposed to open in a new window correctly. This change adds :linkattrs: to the beginning of such files to force parsing link attributes correctly. Bug: Issue 12068 Change-Id: If18be60de646ff78f672239dd4fa435fd4fd92ab
152 lines
6.6 KiB
Plaintext
152 lines
6.6 KiB
Plaintext
:linkattrs:
|
|
= Request Tracing
|
|
|
|
[[on-demand]]
|
|
== On-demand Request Tracing
|
|
|
|
Gerrit supports on-demand tracing of single requests that results in
|
|
additional logs with debug information that are written to the
|
|
`error_log`. The logs that correspond to a traced request are
|
|
associated with a unique trace ID. This trace ID is returned with the
|
|
response and can be used by an administrator to find the matching log
|
|
entries.
|
|
|
|
How tracing is enabled and how the trace ID is returned depends on the
|
|
request type:
|
|
|
|
* REST API: For REST calls tracing can be enabled by setting the
|
|
`trace` request parameter or the `X-Gerrit-Trace` header, the trace
|
|
ID is returned as `X-Gerrit-Trace` header. More information about
|
|
this can be found in the link:rest-api.html#tracing[Request Tracing]
|
|
section of the link:rest-api.html[REST API documentation].
|
|
* SSH API: For SSH calls tracing can be enabled by setting the
|
|
`--trace` option. More information about this can be found in
|
|
the link:cmd-index.html#trace[Trace] section of the
|
|
link:cmd-index.html[SSH command documentation].
|
|
* Git Push (requires usage of git protocol v2): For Git pushes tracing
|
|
can be enabled by setting the `trace` push option, the trace ID is
|
|
returned in the command output. More information about this can be
|
|
found in the link:user-upload.html#trace[Trace] section of the
|
|
link:user-upload.html[upload documentation].
|
|
* Git Clone/Fetch/Ls-Remote (requires usage of git protocol v2): For
|
|
Git clone/fetch/ls-remote tracing can be enabled by setting the
|
|
`trace` server option. Use '-o trace=<TRACE-ID>' for `git fetch` and
|
|
`git ls-remote`, and '--server-option trace=<TRACE-ID>' for
|
|
`git clone`. If the `trace` server option is set without a value
|
|
(without trace ID) a trace ID is generated but the generated trace ID
|
|
is not returned to the client (hence a trace ID should always be
|
|
set).
|
|
|
|
When request tracing is enabled it is possible to provide an ID that
|
|
should be used as trace ID. If a trace ID is not provided a trace ID is
|
|
automatically generated. The trace ID must be provided to the support
|
|
team (administrator of the server) so that they can find the trace.
|
|
|
|
When doing traces it is recommended to specify the ID of the issue
|
|
that is being investigated as trace ID so that the traces of the issue
|
|
can be found more easily. When the issue ID is used as trace ID there
|
|
is no need to find the generated trace ID and report it in the issue.
|
|
|
|
Since tracing consumes additional server resources tracing should only
|
|
be enabled for single requests if there is a concrete need for
|
|
debugging. In particular bots should never enable tracing for all their
|
|
requests by default.
|
|
|
|
[[auto-retry]]
|
|
== Automatic Request Tracing
|
|
|
|
Gerrit can be link:config-gerrit.html#retry.retryWithTraceOnFailure[
|
|
configured] to automatically retry requests on non-recoverable failures
|
|
with tracing enabled. This allows to automatically captures traces of
|
|
these failures for further analysis by the Gerrit administrators.
|
|
|
|
The auto-retry on failure behaves the same way as if the calling user
|
|
would retry the failed operation with tracing enabled.
|
|
|
|
It is expected that the auto-retry fails with the same exception that
|
|
triggered the auto-retry, however this is not guaranteed:
|
|
|
|
* Not all Gerrit operations are fully atomic and it can happen that
|
|
some parts of the operation have been successfully performed before
|
|
the failure happened. In this case the auto-retry may fail with a
|
|
different exception.
|
|
* Some exceptions may mistakenly be considered as non-recoverable and
|
|
the auto-retry actually succeeds.
|
|
|
|
[[auto-retry-succeeded]]
|
|
If an auto-retry succeeds you may consider filing this as
|
|
link:https://bugs.chromium.org/p/gerrit/issues/entry?template=GoogleSource+Issue[
|
|
Gerrit issue,role=external,window=_blank] so that the Gerrit developers can fix this and treat this
|
|
exception as recoverable.
|
|
|
|
The trace IDs for auto-retries are generated and start with
|
|
`retry-on-failure-`. For REST requests they are returned to the client
|
|
as `X-Gerrit-Trace` header.
|
|
|
|
The best way to search for auto-retries in logs is to do a grep by
|
|
`AutoRetry`. For each auto-retry that happened this should match 1 or 2
|
|
log entries:
|
|
|
|
* one `ERROR` log entry with the exception that triggered the
|
|
auto-retry
|
|
* one `FINE` log entry with the exception that happened on auto-retry
|
|
(if this log entry is not present the operation succeeded on
|
|
auto-retry)
|
|
|
|
To inspect single auto-retry occurrences in detail you can do a
|
|
link:#find-trace[grep by the trace ID]. The trace ID is part of the log
|
|
entries which have been found by the previous grep (watch out for
|
|
something like: `retry-on-failure-1534166888910-3985dfba`).
|
|
|
|
[TIP]
|
|
Auto-retrying on failures is only supported by some of the REST
|
|
endpoints (change REST endpoints that perform updates).
|
|
|
|
[[auto-retry-metrics]]
|
|
=== Metrics
|
|
|
|
If auto-retry is link:config-gerrit.html#retry.retryWithTraceOnFailure[
|
|
enabled] the following metrics are reported:
|
|
|
|
* `action/auto_retry_count`: Number of automatic retries with tracing
|
|
* `action/failures_on_auto_retry_count`: Number of failures on auto retry
|
|
|
|
By comparing the values of these counters one can see how often the
|
|
auto-retry succeeds. As explained link:#auto-retry-succeeded[above] if
|
|
auto-retries succeed that's an issue with Gerrit that you may want to
|
|
report.
|
|
|
|
[[find-trace]]
|
|
== Find log entries for a trace ID
|
|
|
|
If tracing is enabled all log messages that correspond to the traced
|
|
request have a `TRACE_ID` tag set, e.g.:
|
|
|
|
----
|
|
[2018-08-13 15:28:08,913] [HTTP-76] TRACE com.google.gerrit.httpd.restapi.RestApiServlet : Received REST request: GET /a/accounts/self (parameters: [trace]) [CONTEXT forced=true TRACE_ID="1534166888910-3985dfba" ]
|
|
[2018-08-13 15:28:08,914] [HTTP-76] TRACE com.google.gerrit.httpd.restapi.RestApiServlet : Calling user: admin [CONTEXT forced=true TRACE_ID="1534166888910-3985dfba" ]
|
|
[2018-08-13 15:28:08,942] [HTTP-76] TRACE com.google.gerrit.httpd.restapi.RestApiServlet : REST call succeeded: 200 [CONTEXT forced=true TRACE_ID="1534166888910-3985dfba" ]
|
|
----
|
|
|
|
By doing a grep with the trace ID over the error log the log entries
|
|
that correspond to the request can be found.
|
|
|
|
[TIP]
|
|
Usually only server administrators have access to the logs.
|
|
|
|
== Which information is captured in a trace?
|
|
|
|
* request details
|
|
** REST API: request URL, request parameter names, calling user,
|
|
response code, response body on errors
|
|
** SSH API: parameter names
|
|
** Git API: push options, magic branch parameter names
|
|
* cache misses, cache evictions
|
|
* reads from NoteDb, writes to NoteDb
|
|
* reads of meta data files, writes of meta data files
|
|
* index queries (with parameters and matches)
|
|
* reindex events
|
|
* permission checks (e.g. which rule is responsible for a deny)
|
|
* timer metrics
|
|
* all other logs
|