Address issues with Trove eventlet monkey-patching

There have been several issues with the way in which Trove does its
monkey patching including the fact that oslo.messaging gets rather
annoyed if eventlet is incorrectly patched.

So, this change follows the recommendations in (note multi-line URL)

http://specs.openstack.org/openstack/\
openstack-specs/specs/eventlet-best-practices.html

Other things to keep in mind are that the old way in which Trove was
doing monkey patching was flawed. What Trove did was to first call

eventlet.monkey_patch(all=True, thread=False)

with the fond hope that this meant monkey patching of "all but thread"
would be done. However, the documentation and the code for the
function are pretty clear that this is not the case.

See: http://eventlet.net/doc/basic_usage.html

"If all is True, then all modules are patched regardless of the other
arguments. If it’s False, then the rest of the keyword arguments
control patching of specific subsections of the standard library."

This also explains why debugger didn't quite work well with Trove, the
whole purpose of the shenanigans that Trove was attempting to do.

Then there's also the wonderful fact that you can't reliably check
(with is_monkey_patched()) whether the 'thread' module is in fact
monkey patched. See (note the multi-line URL)

http://stackoverflow.com/questions/32452110/\
does-eventlet-do-monkey-patch-for-threading-module

Therefore the code actually attempts to issue a meaningful warning if
debugging is attempted with eventlet monkey patched.

This is related to a number of other bugs I've found that indict the
way in which projects have monkey_patched eventlet. See, for example,
https://review.openstack.org/#/c/156942/,
https://review.openstack.org/#/c/153699/, and
https://bugs.launchpad.net/neutron/+bug/1417386.

Change-Id: I92ab95f9113eae862e0cd56e7d65cdd48944615f
Closes-Bug: 1365736
Closes-Bug: 1510542

trove/cmd/__init__.py

@@ -0,0 +1,28 @@
+# Copyright 2015 Tesora, Inc.
+#
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+# This file implements eventlet monkey patching according to the OpenStack
+# guidelines and best practices found at (note the multi-line URL)
+# http://specs.openstack.org/openstack/
+#     openstack-specs/specs/eventlet-best-practices.html
+#
+# It is not safe to leave monkey patching till later.
+
+import os
+
+if not os.environ.get('NO_EVENTLET_MONKEYPATCH'):
+    import eventlet
+    eventlet.monkey_patch(all=True)

trove/cmd/common.py

@@ -19,12 +19,6 @@ def initialize(extra_opts=None, pre_logging=None):
     import gettext
     gettext.install('trove', unicode=1)
 
-    # Apply whole eventlet.monkey_patch excluding 'thread' module.
-    # Decision for 'thread' module patching will be made
-    # after debug_utils is set up.
-    import eventlet
-    eventlet.monkey_patch(all=True, thread=False)
-
     # Import only the modules necessary to initialize logging and determine if
     # debug_utils are enabled.
     import sys
@@ -45,10 +39,6 @@ def initialize(extra_opts=None, pre_logging=None):
     logging.setup(conf, None)
     debug_utils.setup()
 
-    # Patch 'thread' module if debug is disabled.
-    if not debug_utils.enabled():
-        eventlet.monkey_patch(thread=True)
-
     # rpc module must be loaded after decision about thread monkeypatching
     # because if thread module is not monkeypatched we can't use eventlet
     # executor from oslo_messaging library.

trove/cmd/guest.py

@@ -13,12 +13,6 @@
 #    License for the specific language governing permissions and limitations
 #    under the License.
 
-import eventlet
-# Apply whole eventlet.monkey_patch excluding 'thread' module.
-# Decision for 'thread' module patching will be made
-# after debug_utils setting up
-eventlet.monkey_patch(all=True, thread=False)
-
 import gettext
 gettext.install('trove', unicode=1)
 
@@ -43,10 +37,6 @@ def main():
 
     debug_utils.setup()
 
-    # Patch 'thread' module if debug is disabled
-    if not debug_utils.enabled():
-        eventlet.monkey_patch(thread=True)
-
     from trove.guestagent import dbaas
     manager = dbaas.datastore_registry().get(CONF.datastore_manager)
     if not manager:

trove/common/debug_utils.py

@@ -83,6 +83,23 @@ def enabled():
     """
     assert __debug_state is not None, ("debug_utils are not initialized. "
                                        "Please call setup() method first")
+
+    # if __debug_state is set and we have monkey patched
+    # eventlet.thread, issue a warning.
+    # You can't safely use eventlet.is_monkey_patched() on the
+    # threading module so you have to do this little dance.
+    # Discovered after much head scratching, see also
+    #
+    # http://stackoverflow.com/questions/32452110/
+    #     does-eventlet-do-monkey-patch-for-threading-module
+    #
+    # note multi-line URL
+    if __debug_state:
+        import threading
+        if threading.current_thread.__module__ == 'eventlet.green.threading':
+            LOG.warn(_("Enabling debugging with eventlet monkey patched "
+                       "could produce unexpected behavior."))
+
     return __debug_state