api: enable oslo.reports when using uWSGI

At the moment, oslo.reports is enabled when running nova-api
standalone, but not when using uWSGI.

We're now updating the uwsgi entry point as well to include the
oslo.reports hook, which is extremely helpful when debugging
deadlocks.

Change-Id: I605f0e40417fe9b0a383cc8b3fefa1325f9690d9
This commit is contained in:
Lucian Petrut
2021-09-24 14:20:43 +00:00
parent 62406b5728
commit 46401ef666
3 changed files with 41 additions and 6 deletions

View File

@@ -23,7 +23,23 @@ Generating a GMR
A *GMR* can be generated by sending the *USR2* signal to any Nova process with support (see below). The *GMR* will then be outputted standard error for that particular process. A *GMR* can be generated by sending the *USR2* signal to any Nova process with support (see below). The *GMR* will then be outputted standard error for that particular process.
For example, suppose that ``nova-api`` has process id ``8675``, and was run with ``2>/var/log/nova/nova-api-err.log``. Then, ``kill -USR2 8675`` will trigger the Guru Meditation report to be printed to ``/var/log/nova/nova-api-err.log``. For example, suppose that ``nova-compute`` has process id ``8675``, and was run with ``2>/var/log/nova/nova-compute-err.log``. Then, ``kill -USR2 8675`` will trigger the Guru Meditation report to be printed to ``/var/log/nova/nova-compute-err.log``.
Nova API is commonly run under uWSGI, which intercepts ``SIGUSR2`` signals. In this case, a file trigger may be used instead:
.. code-block:: ini
[oslo_reports]
log_dir = /var/log/nova
file_event_handler = /var/log/nova/gmr_trigger
Whenever the trigger file is modified, a *GMR* will be generated. To get a
report, one may use ``touch /var/log/nova/gmr_trigger``.
Note that the configured file trigger must exist when Nova starts.
If a log dir is specified, the report will be written to a file within that
directory instead of ``stderr``. The report file will be named
``${serviceName}_gurumeditation_${timestamp}``.
Structure of a GMR Structure of a GMR
------------------ ------------------
@@ -52,20 +68,27 @@ First import the module, as well as the Nova version module:
.. code-block:: python .. code-block:: python
from oslo_reports import guru_meditation_report as gmr from oslo_reports import guru_meditation_report as gmr
from oslo_reports import opts as gmr_opts
from nova import version from nova import version
Then, register any additional sections (optional): Then, register any additional sections (optional):
.. code-block:: python .. code-block:: python
TextGuruMeditation.register_section('Some Special Section', gmr.TextGuruMeditation.register_section('Some Special Section',
some_section_generator) some_section_generator)
Finally (under main), before running the "main loop" of the executable (usually ``service.server(server)`` or something similar), register the *GMR* hook: Finally (under main), before running the "main loop" of the executable (usually ``service.server(server)`` or something similar), register the *GMR* hook:
.. code-block:: python .. code-block:: python
TextGuruMeditation.setup_autorun(version) gmr_opts.set_defaults(CONF)
gmr.TextGuruMeditation.setup_autorun(
version, conf=CONF, service_name=service_name)
The service name is used when generating report files. If unspecified, *GMR*
tries to automatically detect the binary name using the stack trace but usually
ends up with ``thread.py``.
Extending the GMR Extending the GMR
----------------- -----------------

View File

@@ -16,6 +16,8 @@ import sys
from oslo_config import cfg from oslo_config import cfg
from oslo_log import log as logging from oslo_log import log as logging
from oslo_reports import guru_meditation_report as gmr
from oslo_reports import opts as gmr_opts
from oslo_service import _options as service_opts from oslo_service import _options as service_opts
from paste import deploy from paste import deploy
@@ -25,6 +27,7 @@ from nova import exception
from nova import objects from nova import objects
from nova import service from nova import service
from nova import utils from nova import utils
from nova import version
CONF = cfg.CONF CONF = cfg.CONF
@@ -79,7 +82,7 @@ def error_application(exc, name):
@utils.run_once('Global data already initialized, not re-initializing.', @utils.run_once('Global data already initialized, not re-initializing.',
LOG.info) LOG.info)
def init_global_data(conf_files): def init_global_data(conf_files, service_name):
# NOTE(melwitt): parse_args initializes logging and calls global rpc.init() # NOTE(melwitt): parse_args initializes logging and calls global rpc.init()
# and db_api.configure(). The db_api.configure() call does not initiate any # and db_api.configure(). The db_api.configure() call does not initiate any
# connection to the database. # connection to the database.
@@ -89,6 +92,9 @@ def init_global_data(conf_files):
config.parse_args(sys.argv, default_config_files=conf_files) config.parse_args(sys.argv, default_config_files=conf_files)
logging.setup(CONF, "nova") logging.setup(CONF, "nova")
gmr_opts.set_defaults(CONF)
gmr.TextGuruMeditation.setup_autorun(
version, conf=CONF, service_name=service_name)
# dump conf at debug (log_options option comes from oslo.service) # dump conf at debug (log_options option comes from oslo.service)
# FIXME(mriedem): This is gross but we don't have a public hook into # FIXME(mriedem): This is gross but we don't have a public hook into
@@ -110,7 +116,7 @@ def init_application(name):
# apache/mod_wsgi reloads the init_application script. So, we initialize # apache/mod_wsgi reloads the init_application script. So, we initialize
# global data separately and decorate the method to run only once in a # global data separately and decorate the method to run only once in a
# python interpreter instance. # python interpreter instance.
init_global_data(conf_files) init_global_data(conf_files, name)
try: try:
_setup_service(CONF.host, name) _setup_service(CONF.host, name)

View File

@@ -0,0 +1,6 @@
---
features:
- |
Guru Meditation Reports can now be generated for the Nova API service
when running under uWSGI. Note that uWSGI intercepts SIGUSR2 signals,
so a file trigger should be used instead.