46401ef666
At the moment, oslo.reports is enabled when running nova-api standalone, but not when using uWSGI. We're now updating the uwsgi entry point as well to include the oslo.reports hook, which is extremely helpful when debugging deadlocks. Change-Id: I605f0e40417fe9b0a383cc8b3fefa1325f9690d9
97 lines
3.8 KiB
ReStructuredText
97 lines
3.8 KiB
ReStructuredText
..
|
|
Copyright (c) 2014 OpenStack Foundation
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may
|
|
not use this file except in compliance with the License. You may obtain
|
|
a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
|
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
|
License for the specific language governing permissions and limitations
|
|
under the License.
|
|
|
|
Guru Meditation Reports
|
|
=======================
|
|
|
|
Nova contains a mechanism whereby developers and system administrators can generate a report about the state of a running Nova executable. This report is called a *Guru Meditation Report* (*GMR* for short).
|
|
|
|
Generating a GMR
|
|
----------------
|
|
|
|
A *GMR* can be generated by sending the *USR2* signal to any Nova process with support (see below). The *GMR* will then be outputted standard error for that particular process.
|
|
|
|
For example, suppose that ``nova-compute`` has process id ``8675``, and was run with ``2>/var/log/nova/nova-compute-err.log``. Then, ``kill -USR2 8675`` will trigger the Guru Meditation report to be printed to ``/var/log/nova/nova-compute-err.log``.
|
|
|
|
Nova API is commonly run under uWSGI, which intercepts ``SIGUSR2`` signals. In this case, a file trigger may be used instead:
|
|
|
|
.. code-block:: ini
|
|
|
|
[oslo_reports]
|
|
log_dir = /var/log/nova
|
|
file_event_handler = /var/log/nova/gmr_trigger
|
|
|
|
Whenever the trigger file is modified, a *GMR* will be generated. To get a
|
|
report, one may use ``touch /var/log/nova/gmr_trigger``.
|
|
Note that the configured file trigger must exist when Nova starts.
|
|
|
|
If a log dir is specified, the report will be written to a file within that
|
|
directory instead of ``stderr``. The report file will be named
|
|
``${serviceName}_gurumeditation_${timestamp}``.
|
|
|
|
Structure of a GMR
|
|
------------------
|
|
|
|
The *GMR* is designed to be extensible; any particular executable may add its own sections. However, the base *GMR* consists of several sections:
|
|
|
|
Package
|
|
Shows information about the package to which this process belongs, including version information
|
|
|
|
Threads
|
|
Shows stack traces and thread ids for each of the threads within this process
|
|
|
|
Green Threads
|
|
Shows stack traces for each of the green threads within this process (green threads don't have thread ids)
|
|
|
|
Configuration
|
|
Lists all the configuration options currently accessible via the CONF object for the current process
|
|
|
|
Adding Support for GMRs to New Executables
|
|
------------------------------------------
|
|
|
|
Adding support for a *GMR* to a given executable is fairly easy.
|
|
|
|
First import the module, as well as the Nova version module:
|
|
|
|
.. code-block:: python
|
|
|
|
from oslo_reports import guru_meditation_report as gmr
|
|
from oslo_reports import opts as gmr_opts
|
|
from nova import version
|
|
|
|
Then, register any additional sections (optional):
|
|
|
|
.. code-block:: python
|
|
|
|
gmr.TextGuruMeditation.register_section('Some Special Section',
|
|
some_section_generator)
|
|
|
|
Finally (under main), before running the "main loop" of the executable (usually ``service.server(server)`` or something similar), register the *GMR* hook:
|
|
|
|
.. code-block:: python
|
|
|
|
gmr_opts.set_defaults(CONF)
|
|
gmr.TextGuruMeditation.setup_autorun(
|
|
version, conf=CONF, service_name=service_name)
|
|
|
|
The service name is used when generating report files. If unspecified, *GMR*
|
|
tries to automatically detect the binary name using the stack trace but usually
|
|
ends up with ``thread.py``.
|
|
|
|
Extending the GMR
|
|
-----------------
|
|
|
|
As mentioned above, additional sections can be added to the GMR for a particular executable. For more information, see the inline documentation under :mod:`oslo.reports`
|