Introduce Guru to Mogan

Guru is a mechanism whereby developers
and system administrators can generate
a report about the state of a running
Zaqar executable.

This report is called a *Guru Meditation Report*

This mechanism will help developer or operator
to fix issues in (production) deployments without
stopping Mogan service.

Change-Id: Ib0dafa160dc1f4684e17ec30918e1decb4f2c906
Implements: blueprint introduce-guru-to-mogan

doc/source/dev/gmr.rst

@@ -0,0 +1,85 @@
+..
+      Copyright (c) 2017 OpenStack Foundation
+      All Rights Reserved.
+
+      Licensed under the Apache License, Version 2.0 (the "License"); you may
+      not use this file except in compliance with the License. You may obtain
+      a copy of the License at
+
+          http://www.apache.org/licenses/LICENSE-2.0
+
+      Unless required by applicable law or agreed to in writing, software
+      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+      License for the specific language governing permissions and limitations
+      under the License.
+
+=======================
+Guru Meditation Reports
+=======================
+
+Mogan contains a mechanism whereby developers and system administrators can
+generate a report about the state of a running Mogan executable.
+This report is called a *Guru Meditation Report* (*GMR* for short).
+
+Generating a GMR
+----------------
+
+A *GMR* can be generated by sending the *USR2* signal to any Mogan process
+with support (see below).
+The *GMR* will then output to standard error for that particular process.
+
+For example, suppose that ``mogan-api`` has process id ``8675``, and was run
+with ``2>/var/log/mogan/mogan-api-err.log``.
+Then, ``kill -USR2 8675`` will trigger the Guru Meditation report to be printed
+to ``/var/log/mogan/mogan-api-err.log``.
+
+There is other way to trigger a generation of report, user should add
+a configuration in Mogan's conf file::
+
+    [oslo_reports]
+    file_event_handler=['The path to a file to watch for changes to trigger '
+                        'the reports, instead of signals. Setting this option '
+                        'disables the signal trigger for the reports.']
+    file_event_handler_interval=['How many seconds to wait between polls when '
+                                 'file_event_handler is set, default value '
+                                 'is 1']
+
+a *GMR* can be generated by "touch"ing the file which was specified in
+file_event_handler. The *GMR* will then output to standard error for
+that particular process.
+
+For example, suppose that ``mogan-api`` was run with
+``2>/var/log/mogan/mogan-api-err.log``, and the file path is
+``/tmp/guru_report``.
+Then, ``touch /tmp/guru_report`` will trigger the Guru Meditation report to be
+printed to ``/var/log/mogan/mogan-api-err.log``.
+
+Structure of a GMR
+------------------
+
+The *GMR* is designed to be extensible; any particular executable may add
+its own sections.  However, the base *GMR* consists of several sections:
+
+Package
+  Shows information about the package to which this process belongs,
+  including version information
+
+Threads
+  Shows stack traces and thread ids for each of the threads within this process
+
+Green Threads
+  Shows stack traces for each of the green threads within this process
+  (green threads don't have thread ids)
+
+Configuration
+  Lists all the configuration options currently accessible via the CONF object
+  for the current process
+
+Extending the GMR
+-----------------
+
+As mentioned above, additional sections can be added to the GMR for a
+particular executable.  For more information, see the inline documentation
+about oslo.reports:
+`oslo.reports <http://docs.openstack.org/developer/oslo.reports/>`_

doc/source/index.rst

@@ -65,3 +65,11 @@ Indices and tables
 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`
+
+Guru Meditation Reports
+=======================
+
+.. toctree::
+   :maxdepth: 1
+
+   dev/gmr

mogan/cmd/api.py

@@ -18,16 +18,22 @@
 import sys
 
 from oslo_config import cfg
+from oslo_reports import guru_meditation_report as gmr
+from oslo_reports import opts as gmr_opts
 
 from mogan.common import service as mogan_service
+from mogan import version
 
 CONF = cfg.CONF
 
 
 def main():
+    gmr_opts.set_defaults(CONF)
     # Parse config file and command line options, then start logging
     mogan_service.prepare_service(sys.argv)
 
+    gmr.TextGuruMeditation.setup_autorun(version, conf=CONF)
+
     # Build and start the WSGI app
     launcher = mogan_service.process_launcher()
     server = mogan_service.WSGIService('mogan_api', CONF.api.enable_ssl_api)

mogan/cmd/consoleauth.py

@@ -20,18 +20,24 @@ The Mogan console auth Service
 import sys
 
 from oslo_config import cfg
+from oslo_reports import guru_meditation_report as gmr
+from oslo_reports import opts as gmr_opts
 from oslo_service import service
 
 from mogan.common import constants
 from mogan.common import service as mogan_service
+from mogan import version
 
 CONF = cfg.CONF
 
 
 def main():
+    gmr_opts.set_defaults(CONF)
     # Parse config file and command line options, then start logging
     mogan_service.prepare_service(sys.argv)
 
+    gmr.TextGuruMeditation.setup_autorun(version, conf=CONF)
+
     mgr = mogan_service.RPCService('mogan.consoleauth.manager',
                                    'ConsoleAuthManager',
                                    constants.MANAGER_CONSOLEAUTH_TOPIC)

mogan/cmd/engine.py

@@ -20,18 +20,24 @@ The Mogan Management Service
 import sys
 
 from oslo_config import cfg
+from oslo_reports import guru_meditation_report as gmr
+from oslo_reports import opts as gmr_opts
 from oslo_service import service
 
 from mogan.common import constants
 from mogan.common import service as mogan_service
+from mogan import version
 
 CONF = cfg.CONF
 
 
 def main():
+    gmr_opts.set_defaults(CONF)
     # Parse config file and command line options, then start logging
     mogan_service.prepare_service(sys.argv)
 
+    gmr.TextGuruMeditation.setup_autorun(version, conf=CONF)
+
     mgr = mogan_service.RPCService('mogan.engine.manager',
                                    'EngineManager',
                                    constants.ENGINE_TOPIC)

mogan/cmd/scheduler.py

@@ -20,18 +20,24 @@ The Mogan Scheduler Service
 import sys
 
 from oslo_config import cfg
+from oslo_reports import guru_meditation_report as gmr
+from oslo_reports import opts as gmr_opts
 from oslo_service import service
 
 from mogan.common import constants
 from mogan.common import service as mogan_service
+from mogan import version
 
 CONF = cfg.CONF
 
 
 def main():
+    gmr_opts.set_defaults(CONF)
     # Parse config file and command line options, then start logging
     mogan_service.prepare_service(sys.argv)
 
+    gmr.TextGuruMeditation.setup_autorun(version, conf=CONF)
+
     mgr = mogan_service.RPCService('mogan.scheduler.manager',
                                    'SchedulerManager',
                                    constants.SCHEDULER_TOPIC)