Always run the rabbitmq heartbeat inside a standard pthread.

Propose changes to always run the rabbitmq health check heartbeat in a
standard python thread.

With the described specs we want to isolate the heartbeat execution model
from the parent process inherited execution model.

We want to force to use the python stdlib threading module to run
heartbeat.

The proposed changes will fix related issues who run heartbeat under
mod_wsgi in a monkey patched environment.

Change-Id: Ia6d873c3fc5dcc6b1c181b983b9d0d2787348163
This commit is contained in:
Hervé Beraud 2019-05-24 16:45:32 +02:00
parent ae0fc25995
commit a26df0f3ce
1 changed files with 166 additions and 0 deletions

View File

@ -0,0 +1,166 @@
======================================================
Run the rabbitmq heartbeat in a python standard thread
======================================================
https://blueprints.launchpad.net/oslo.messaging/+spec/rabbitmq-driver-threads
oslo.messaging RabbitMq driver heartbeat is designed to health check
the connections with the rabbitmq server.
olso.messaging rabbitmq heartbeat doesn't take care about the execution
context and parent execution model and all the time keep the parent context
applied.
If the users need async on their side and use eventlet to
monkey patch the python stdlib then the heartbeat will use greenthreads too.
This specification proposes changes to adapt the rabbitmq driver to
isolate heartbeat execution model from the parent execution model and stop
to inherit the parent execution model.
The proposed changes will force to run the heartbeat mechanism inside a
standard python thread.
The heartbeat execution model will now designed by the oslo team to avoid
issues and confusion between the parent context and the heartbeat needs.
In other words we want to isolate the heartbeat execution model from the
parent execution problem to avoid possible issues related to the context
(cf. eventlet/mod_wsgi).
Problem description
===================
The rabbitmq driver is commonly used to health check the AMQP connections with
rabbitmq to ensure connections by using the driver heartbeat mechanism.
Nowadays the heartbeat is designed to be reproduce the same execution than
the parent process who launch him.
Parent process can need async on its side. To obtain async some user can use
eventlet. Eventlet can be used to monkey patch the stdlib and to patch the
python threading module and so obtain greenthreads and async execution model.
In many use cases (nova, neutron) the AMQP heartbeat can be used inside an
execution evironment under mod_wsgi or uwsgi.
Actually using greenthreads and eventlet monkey patched python stdlib in the
parent process force the heartbeat to inherit this context and run itself
inside similar execution model.
The heartbeat health check is running in a dedicated thread and doesn't need
to become async from the parent process point of view.
The main issue here is that we kept parent execution based on eventlet and
async for the heartbeat dedicated heartbeat who can be implicitely run under
mod_wsgi/uwsgi. As is described below mod_wsgi and eventlet greenthreads are
not compatible.
Indeed, mod_wsgi stops the execution of its embedded interpreter, the
AMQP heartbeat thread can't be resumed until there's a message to be
processed in the mod_wsgi queue, which would resume the python
interpreter and make eventlet resume the thread.
Force the heartbeat to use a standard pthreads and disabling eventlet will fix
this kind of issues in problematic environment (mod_wsgi).
Proposed change
===============
The spec simply proposes to force the heartbeat to use pthreads instead of
let's the user execution model come inherited.
In all the case we will explicitely initialize a pthread who
do not take care about the possible parent model who have monkey patch the
python stdlib and the threading module.
Alternatives
------------
Let's the choice to the users to continue to inherit from the parent execution
model or to run heartbeat by using the python standard threading module.
This alernative is not really safe and can introduce possible issues related to
their contexts where the heartbeat will fail.
Impact on Existing usages
-------------------------
No impact on the existing usages since the major use case of the heartbeat do
not take care about the async execution model.
If users use heartbeat inside a wsgi environment and also use a monkey patched
environment then we will fix the heartbeat issue in this context.
Security impact
---------------
No significant security impact is expected.
Performance Impact
------------------
No significant performance impact is expected.
Configuration Impact
--------------------
No significant configuration impact is expected.
Developer Impact
----------------
No significant developer impact is expected.
Testing Impact
--------------
No significant testing impact is expected.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
hberaud
Other contributors:
volunteers?
Milestones
----------
..TODO(hberaud): figure this out
Work Items
----------
* Implement the described changes in the rabbitmq driver.
* Update documentation to specify that we force the usage of pthreads.
Incubation
==========
N/A
Documentation Impact
====================
The documentation will need to be updated to indicate that we force the
usage of pthreads.
Dependencies
============
N/A
References
==========
* Eventlet Best Practices: https://review.opendev.org/#/c/154642/
* Related discussion on the ML about mod_wsgi and monkey patch: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005305.html
* Python asynchronous IO support - PEP 3156: https://www.python.org/dev/peps/pep-3156/
.. note::
This work is licensed under a Creative Commons Attribution 3.0
Unported License.
http://creativecommons.org/licenses/by/3.0/legalcode