Merge "Eventlet Best Practices"
This commit is contained in:
commit
69a3e2889f
196
specs/eventlet-best-practices.rst
Normal file
196
specs/eventlet-best-practices.rst
Normal file
@ -0,0 +1,196 @@
|
||||
=========================
|
||||
Eventlet Best Practices
|
||||
=========================
|
||||
|
||||
No blueprint, this is intended as a reference document.
|
||||
|
||||
Eventlet is used in many of the OpenStack projects as the default concurrency
|
||||
model, and there are some things we've learned about it over the years that
|
||||
currently exist only as tribal knowledge. This is an attempt to codify those
|
||||
in a central location for everyone's benefit.
|
||||
|
||||
It is worth noting that while there has been a push from some members of the
|
||||
community to move away from eventlet entirely, there is currently no approved
|
||||
plan to do so. Even if there were, it will likely take a long time to
|
||||
implement, so eventlet will be something we have to care about for at least
|
||||
the short and medium term.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
In some ways eventlet behaves much differently from other concurrency models
|
||||
and can even change the behavior of the Python standard library. This means
|
||||
that scenarios exist where a bad interaction between eventlet and some other
|
||||
code, often code that is not eventlet-aware, can cause problems. We need some
|
||||
best practices that will minimize the potential for these issues to occur.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
Guidelines for using eventlet:
|
||||
|
||||
Monkey Patching
|
||||
---------------
|
||||
|
||||
* When using eventlet.monkey_patch, do it first or not at all. In practice,
|
||||
this means monkey patching in a top-level __init__.py which is guaranteed
|
||||
to be run before any other project code. As an example, Nova monkey patches
|
||||
in nova/cmd/__init__.py and nova/tests/unit/__init__.py so that in both the
|
||||
runtime and test scenarios the monkey patching happens before any Nova code
|
||||
executes.
|
||||
|
||||
The reasoning behind this is that unpatched stdlib modules may not play
|
||||
nicely with eventlet monkey patched ones. For example, if thread A is
|
||||
started, the application monkey patches, then starts thread B, now you've
|
||||
mixed native threads and green threads and the results are undefined but
|
||||
most likely bad.
|
||||
|
||||
It is not practical to expect developers to recognize all such
|
||||
possible race conditions during development or review, and in fact it is
|
||||
impossible because the race condition could be introduced by code we
|
||||
consume from another library. Because of this, it is safest to
|
||||
simply eliminate the races by monkey patching before any other code is run.
|
||||
|
||||
* Monkey patching should also be done in a way that allows services to run
|
||||
without it, such as when an API service runs under Apache. This is the
|
||||
reason for Nova not simply monkey patching in nova/__init__.py.
|
||||
|
||||
Another example is Keystone, which recommends running under Apache but also
|
||||
supports eventlet. They have a separate eventlet binary 'keystone-all' which
|
||||
handles monkey patching before running any other code. Note that
|
||||
`eventlet is deprecated`_ in Keystone as of the Kilo cycle.
|
||||
|
||||
.. _`eventlet is deprecated`: http://lists.openstack.org/pipermail/openstack-dev/2015-February/057359.html
|
||||
|
||||
* Monkey patching with thread=False is likely to cause problems. This is done
|
||||
conditionally in many services due to `problems running under a debugger`_
|
||||
with the threading module monkey patched. Unfortunately, even simple
|
||||
concurrency scenarios can result in deadlocks with this sort of setup. For
|
||||
example, the following code provided by Josh Harlow will cause hangs::
|
||||
|
||||
import eventlet
|
||||
|
||||
eventlet.monkey_patch(os=False, thread=False)
|
||||
|
||||
|
||||
import threading
|
||||
|
||||
import time
|
||||
|
||||
|
||||
thingy_lock = threading.Lock()
|
||||
|
||||
|
||||
def do_it():
|
||||
with thingy_lock:
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
threads = []
|
||||
for i in range(0, 5):
|
||||
threads.append(eventlet.spawn(do_it))
|
||||
while threads:
|
||||
t = threads.pop()
|
||||
t.wait()
|
||||
|
||||
It is unclear at this time whether there is a way to enable debuggers and
|
||||
also have a sane monkey patched environment. The `eventlet backdoor`_ was
|
||||
mentioned as a possible alternative.
|
||||
|
||||
.. _`problems running under a debugger`: http://lists.openstack.org/pipermail/openstack-dev/2012-August/000693.html
|
||||
.. _`eventlet backdoor`: http://lists.openstack.org/pipermail/openstack-dev/2012-August/000873.html
|
||||
|
||||
* Monkey patching can cause problems running flake8 with multiple workers.
|
||||
If it does, the monkey patching can be made conditional based on an
|
||||
environment variable that can be set during flake8 test runs. This should
|
||||
not be a problem as monkey patching is not needed for flake8.
|
||||
|
||||
For example::
|
||||
|
||||
import os
|
||||
|
||||
if not os.environ.get('DISABLE_EVENTLET_PATCHING'):
|
||||
import eventlet
|
||||
eventlet.monkey_patch()
|
||||
|
||||
Even though os is being imported before monkey patching, this should be safe
|
||||
as long as no other code is run before monkey patching occurs.
|
||||
|
||||
Greenthread-aware Modules
|
||||
-------------------------
|
||||
|
||||
* There is a greenthread-aware subprocess module in eventlet, but it does
|
||||
*not* get patched in by eventlet.monkey_patch. Code that has interactions
|
||||
between green threads and the subprocess module must be sure to use the
|
||||
green subprocess module explicitly. A simpler alternative is to use
|
||||
processutils from oslo.concurrency, which selects the appropriate module
|
||||
depending on the status of eventlet's monkey patching.
|
||||
|
||||
Database Drivers
|
||||
----------------
|
||||
|
||||
* Eventlet can cause deadlocks_ in some Python database drivers. The current
|
||||
plan is to move our recommended and default driver_ to something that is more
|
||||
eventlet-friendly.
|
||||
|
||||
.. _deadlocks: https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad
|
||||
.. _driver: https://wiki.openstack.org/wiki/PyMySQL_evaluation#MySQL_DB_Drivers_Comparison
|
||||
|
||||
Tools for Ensuring Monkey Patch Sanity
|
||||
--------------------------------------
|
||||
|
||||
* The oslo.utils project has an eventletutils_ module that can help ensure
|
||||
proper monkey patching for code that knows what it needs patched. This
|
||||
could, for example, be used to raise a warning when a service is run under
|
||||
a debugger without threading patched. At least that way the user will have
|
||||
a clue what is wrong if deadlocks occur.
|
||||
|
||||
.. _eventletutils: http://docs.openstack.org/developer/oslo.utils/api/eventletutils.html
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
* Continue to have each project implement eventlet in its own way. This is
|
||||
undesirable because it will result in projects hitting bugs that may have
|
||||
been solved in another project.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee: bnemec
|
||||
|
||||
Additional contributors: harlowja, ihrachyshka
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
* Audit the use of eventlet in OpenStack projects and make any changes
|
||||
necessary to abide by these guidelines.
|
||||
|
||||
* Follow up with the eventlet team on whether the green subprocess module
|
||||
not being included in monkey patching is intentional.
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
|
||||
History
|
||||
=======
|
||||
|
||||
.. list-table:: Revisions
|
||||
:header-rows: 1
|
||||
|
||||
* - Release Name
|
||||
- Description
|
||||
* - Kilo
|
||||
- Introduced
|
||||
|
||||
.. note::
|
||||
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
Loading…
Reference in New Issue
Block a user