From f8b1d914aa37f8048cf983303d8ef9ad25ce4d34 Mon Sep 17 00:00:00 2001 From: Clark Boylan Date: Fri, 22 Oct 2021 09:25:45 -0700 Subject: [PATCH] Document Zuul's SIGUSR2 handler This is more zuul debugging documentation. Change-Id: I5298f62658cd68f2bd19ec02fb2c1970d855bf84 --- doc/source/zuul.rst | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/doc/source/zuul.rst b/doc/source/zuul.rst index 682f77b961..4c684220e9 100644 --- a/doc/source/zuul.rst +++ b/doc/source/zuul.rst @@ -275,6 +275,31 @@ the problem. Note that ``Traceback`` messages are not prefixed with the event ID which means you'll have to grep with additional context, for example using ``grep -B20 -A20``. +Another useful debugging tool is Zuul's SIGUSR2 handler. This signal handler +produces a thread dump in the debug log and toggles the yappi python profiler. +Each Zuul service supports the signal handler and it can be triggered via:: + + you@zuul02$ sudo kill -USR2 $ZUUL_PID + +To determine ``$ZUUL_PID`` you can run ``ps`` against the ``zuul-*`` service +that you are interested in getting information from. For example:: + + you@zuul02$ ps -ef | grep zuul-scheduler + zuuld 1893030 1893010 0 08:33 ? 00:00:00 /usr/bin/dumb-init -- /usr/local/bin/zuul-scheduler -f + zuuld 1893052 1893030 69 08:33 ? 07:57:42 /usr/local/bin/python /usr/local/bin/zuul-scheduler -f + zuuld 1893198 1893052 0 08:33 ? 00:03:22 /usr/local/bin/python /usr/local/bin/zuul-scheduler -f + +All of the zuul services are run under ``dumb-init``. The process to send +SIGUSR2 to is the child of the ``dumb-init`` process. In the example above +``$ZUUL_PID`` would be ``1893052``. + +The first time you run it you will turn on the yappi profiler. This profiler +does incur a runtime cost which can significantly slow down Zuul's processing +of pipelines. Be sure to resend the signal once you have let Zuul run long +enough to collect a representative set of profiler data. In most cases a minute +or two should be sufficient. Slow memory leaks may require hours, but running +Zuul under yappi for hours isn't practical. + .. _zuul_github_projects: GitHub Projects