From 4a143160ab46356494868ca17bb8c65dff7da561 Mon Sep 17 00:00:00 2001 From: Edwin Kempin Date: Wed, 28 Aug 2019 15:45:21 +0200 Subject: [PATCH 1/2] Request tracing docs: Add heading for first section to fix formatting Without this heading the first paragraph is rendered with a larger font than the rest of the text. Having this heading will anyway be needed since we also want to describe automatic request tracing on this page, and the documentation for on-demand tracing should be separate from it. Change-Id: I455811809b4d82ff190b80b1728ae05a7b2ba38c Signed-off-by: Edwin Kempin --- Documentation/user-request-tracing.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Documentation/user-request-tracing.txt b/Documentation/user-request-tracing.txt index bc54c4a6bc..59641104fb 100644 --- a/Documentation/user-request-tracing.txt +++ b/Documentation/user-request-tracing.txt @@ -1,5 +1,8 @@ = Request Tracing +[[on-demand]] +== On-demand Request Tracing + Gerrit supports on-demand tracing of single requests that results in additional logs with debug information that are written to the `error_log`. The logs that correspond to a traced request are From b49263356668e052b5c9ed5cd61a2ffd2dd33281 Mon Sep 17 00:00:00 2001 From: Edwin Kempin Date: Wed, 28 Aug 2019 17:07:17 +0200 Subject: [PATCH 2/2] Document details about automatic request tracing Change-Id: Ic694b36cf95277435f71d520051cfe86765421ca Signed-off-by: Edwin Kempin --- Documentation/user-request-tracing.txt | 64 ++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) diff --git a/Documentation/user-request-tracing.txt b/Documentation/user-request-tracing.txt index 59641104fb..1b9a6e5c6e 100644 --- a/Documentation/user-request-tracing.txt +++ b/Documentation/user-request-tracing.txt @@ -44,6 +44,70 @@ be enabled for single requests if there is a concrete need for debugging. In particular bots should never enable tracing for all their requests by default. +[[auto-retry]] +== Automatic Request Tracing + +Gerrit can be link:config-gerrit.html#retry.retryWithTraceOnFailure[ +configured] to automatically retry requests on non-recoverable failures +with tracing enabled. This allows to automatically captures traces of +these failures for further analysis by the Gerrit administrators. + +The auto-retry on failure behaves the same way as if the calling user +would retry the failed operation with tracing enabled. + +It is expected that the auto-retry fails with the same exception that +triggered the auto-retry, however this is not guaranteed: + +* Not all Gerrit operations are fully atomic and it can happen that + some parts of the operation have been successfully performed before + the failure happened. In this case the auto-retry may fail with a + different exception. +* Some exceptions may mistakenly be considered as non-recoverable and + the auto-retry actually succeeds. + +[[auto-retry-succeeded]] +If an auto-retry succeeds you may consider filing this as +link:https://bugs.chromium.org/p/gerrit/issues/entry?template=GoogleSource+Issue[ +Gerrit issue] so that the Gerrit developers can fix this and treat this +exception as recoverable. + +The trace IDs for auto-retries are generated and start with +`retry-on-failure-`. + +The best way to search for auto-retries in logs is to do a grep by +`AutoRetry`. For each auto-retry that happened this should match 1 or 2 +log entries: + +* one `ERROR` log entry with the exception that triggered the + auto-retry +* one `FINE` log entry with the exception that happened on auto-retry + (if this log entry is not present the operation succeeded on + auto-retry) + +To inspect single auto-retry occurrences in detail you can do a +link:#find-trace[grep by the trace ID]. The trace ID is part of the log +entries which have been found by the previous grep (watch out for +something like: `retry-on-failure-1534166888910-3985dfba`). + +[TIP] +Auto-retrying on failures is only supported by some of the REST +endpoints (change REST endpoints that perform updates). + +[[auto-retry-metrics]] +=== Metrics + +If auto-retry is link:config-gerrit.html#retry.retryWithTraceOnFailure[ +enabled] the following metrics are reported: + +* `action/auto_retry_count`: Number of automatic retries with tracing +* `action/failures_on_auto_retry_count`: Number of failures on auto retry + +By comparing the values of these counters one can see how often the +auto-retry succeeds. As explained link:#auto-retry-succeeded[above] if +auto-retries succeed that's an issue with Gerrit that you may want to +report. + +[[find-trace]] == Find log entries for a trace ID If tracing is enabled all log messages that correspond to the traced