Add documentation for request tracing

Change-Id: If2d57ae451abeb0f0040b5960c7f7a930247328a
Signed-off-by: Edwin Kempin <ekempin@google.com>

Documentation/cmd-index.txt

@@ -205,6 +205,27 @@ link:cmd-show-queue.html[ps]::
 link:cmd-suexec.html[suexec]::
 	Execute a command as any registered user account.
 
+=== [[trace]]Trace
+
+For executing SSH commands tracing can be enabled by setting the
+`--trace` option.
+
+----
+  $ ssh -p 29418 review.example.com gerrit create-project --trace foo/bar
+----
+
+Enabling tracing results in additional logs with debug information that
+are written to the `error_log`. All logs that correspond to the traced
+request are associated with a unique trace ID. This trace ID is printed
+to the stderr command output:
+
+----
+  TRACE_ID: 1534174322774-7edf2a7b
+----
+
+Given the trace ID an administrator can find the corresponding logs and
+investigate issues more easily.
+
 GERRIT
 ------
 Part of link:index.html[Gerrit Code Review]

Documentation/index.txt

@@ -67,6 +67,7 @@
 . link:config-reverseproxy.html[Reverse Proxy]
 . link:config-auto-site-initialization.html[Automatic Site Initialization on Startup]
 . link:pgm-index.html[Server Side Administrative Tools]
+. link:user-request-tracing.html[Request Tracing]
 . link:note-db.html[NoteDb]
 . link:config-accounts.html[Accounts on NoteDb]
 . link:config-groups.html[Groups on NoteDb]

Documentation/rest-api.txt

@@ -191,6 +191,35 @@ Preconditions] section.
 "`422 Unprocessable Entity`" is returned if the ID of a resource that is
 specified in the request body cannot be resolved.
 
+[[tracing]]
+=== Request Tracing
+For each REST endpoint tracing can be enabled by setting the `trace`
+request parameter.
+
+.Example Request
+----
+  GET /changes/myProject~master~I8473b95934b5732ac55d26311a706c9c2bde9940/suggest_reviewers?trace&q=J
+----
+
+Enabling tracing results in additional logs with debug information that
+are written to the `error_log`. All logs that correspond to the traced
+request are associated with a unique trace ID. This trace ID is
+returned with the REST response in the `X-Gerrit-Trace` header.
+
+.Example Response
+----
+HTTP/1.1 200 OK
+  Content-Disposition: attachment
+  Content-Type: application/json; charset=UTF-8
+  X-Gerrit-Trace: 1533885943749-8257c498
+
+  )]}'
+  ... <json> ...
+----
+
+Given the trace ID an administrator can find the corresponding logs and
+investigate issues more easily.
+
 GERRIT
 ------
 Part of link:index.html[Gerrit Code Review]

Documentation/user-request-tracing.txt

@@ -0,0 +1,41 @@
+= Request Tracing
+
+Gerrit supports on-demand tracing of single requests that results in
+additional logs with debug information that are written to the
+`error_log`. The logs that correspond to a traced request are
+associated with a unique trace ID. This trace ID is returned with the
+response and can be used by an administrator to find the matching log
+entries.
+
+How tracing is enabled and how the trace ID is returned depends on the
+request type:
+
+* REST API: For REST calls tracing can be enabled by setting the
+  `trace` request parameter, the trace ID is returned as
+  `X-Gerrit-Trace` header. More information about this can be found in
+  the link:rest-api.html#tracing[Request Tracing] section of the
+  link:rest-api.html[REST API documentation].
+* SSH API: For SSH calls tracing can be enabled by setting the
+  `--trace` option. More information about this can be found in
+  the link:cmd-index.html#trace[Trace] section of the
+  link:cmd-index.html[SSH command documentation].
+* Git: For Git pushes tracing can be enabled by setting the
+  `trace` push option, the trace ID is returned in the command output.
+  More information about this can be found in
+  the link:user-upload.html#trace[Trace] section of the
+  link:user-upload.html[upload documentation]. Tracing for Git requests
+  other than Git push is not supported.
+
+== Find log entries for a trace ID
+
+If tracing is enabled all log messages that correspond to the traced
+request have a `TRACE_ID` tag set, e.g.:
+
+----
+[2018-08-13 15:28:08,913] [HTTP-76] TRACE com.google.gerrit.httpd.restapi.RestApiServlet : Received REST request: GET /a/accounts/self (parameters: [trace]) [CONTEXT forced=true TRACE_ID="1534166888910-3985dfba" ]
+[2018-08-13 15:28:08,914] [HTTP-76] TRACE com.google.gerrit.httpd.restapi.RestApiServlet : Calling user: admin [CONTEXT forced=true TRACE_ID="1534166888910-3985dfba" ]
+[2018-08-13 15:28:08,942] [HTTP-76] TRACE com.google.gerrit.httpd.restapi.RestApiServlet : REST call succeeded: 200 [CONTEXT forced=true TRACE_ID="1534166888910-3985dfba" ]
+----
+
+By doing a grep with the trace ID over the error log the log entries
+that correspond to the request can be found.

Documentation/user-upload.txt

@@ -421,6 +421,28 @@ branches, consider adding a custom remote block to your project's
   $ git push exp
 ----
 
+[[trace]]
+==== Trace
+
+When pushing to Gerrit tracing can be enabled by setting the `trace`
+push option.
+
+----
+  git push -o trace=true ssh://john.doe@git.example.com:29418/kernel/common HEAD:refs/for/master
+----
+
+Enabling tracing results in additional logs with debug information that
+are written to the `error_log`. All logs that correspond to the traced
+request are associated with a unique trace ID. This trace ID is
+returned in the command output:
+
+----
+  remote: TRACE_ID: 1534174322774-7edf2a7b
+----
+
+Given the trace ID an administrator can find the corresponding logs and
+investigate issues more easily.
+
 [[push_replace]]
 === Replace Changes