167 lines
5.1 KiB
ReStructuredText
167 lines
5.1 KiB
ReStructuredText
======================================================
|
|
Run the rabbitmq heartbeat in a python standard thread
|
|
======================================================
|
|
|
|
https://blueprints.launchpad.net/oslo.messaging/+spec/rabbitmq-driver-threads
|
|
|
|
oslo.messaging RabbitMq driver heartbeat is designed to health check
|
|
the connections with the rabbitmq server.
|
|
olso.messaging rabbitmq heartbeat doesn't take care about the execution
|
|
context and parent execution model and all the time keep the parent context
|
|
applied.
|
|
If the users need async on their side and use eventlet to
|
|
monkey patch the python stdlib then the heartbeat will use greenthreads too.
|
|
|
|
This specification proposes changes to adapt the rabbitmq driver to
|
|
isolate heartbeat execution model from the parent execution model and stop
|
|
to inherit the parent execution model.
|
|
The proposed changes will force to run the heartbeat mechanism inside a
|
|
standard python thread.
|
|
The heartbeat execution model will now designed by the oslo team to avoid
|
|
issues and confusion between the parent context and the heartbeat needs.
|
|
|
|
In other words we want to isolate the heartbeat execution model from the
|
|
parent execution problem to avoid possible issues related to the context
|
|
(cf. eventlet/mod_wsgi).
|
|
|
|
Problem description
|
|
===================
|
|
|
|
The rabbitmq driver is commonly used to health check the AMQP connections with
|
|
rabbitmq to ensure connections by using the driver heartbeat mechanism.
|
|
|
|
Nowadays the heartbeat is designed to be reproduce the same execution than
|
|
the parent process who launch him.
|
|
|
|
Parent process can need async on its side. To obtain async some user can use
|
|
eventlet. Eventlet can be used to monkey patch the stdlib and to patch the
|
|
python threading module and so obtain greenthreads and async execution model.
|
|
|
|
In many use cases (nova, neutron) the AMQP heartbeat can be used inside an
|
|
execution evironment under mod_wsgi or uwsgi.
|
|
|
|
Actually using greenthreads and eventlet monkey patched python stdlib in the
|
|
parent process force the heartbeat to inherit this context and run itself
|
|
inside similar execution model.
|
|
|
|
The heartbeat health check is running in a dedicated thread and doesn't need
|
|
to become async from the parent process point of view.
|
|
|
|
The main issue here is that we kept parent execution based on eventlet and
|
|
async for the heartbeat dedicated heartbeat who can be implicitely run under
|
|
mod_wsgi/uwsgi. As is described below mod_wsgi and eventlet greenthreads are
|
|
not compatible.
|
|
|
|
Indeed, mod_wsgi stops the execution of its embedded interpreter, the
|
|
AMQP heartbeat thread can't be resumed until there's a message to be
|
|
processed in the mod_wsgi queue, which would resume the python
|
|
interpreter and make eventlet resume the thread.
|
|
|
|
Force the heartbeat to use a standard pthreads and disabling eventlet will fix
|
|
this kind of issues in problematic environment (mod_wsgi).
|
|
|
|
Proposed change
|
|
===============
|
|
|
|
The spec simply proposes to force the heartbeat to use pthreads instead of
|
|
let's the user execution model come inherited.
|
|
|
|
In all the case we will explicitely initialize a pthread who
|
|
do not take care about the possible parent model who have monkey patch the
|
|
python stdlib and the threading module.
|
|
|
|
Alternatives
|
|
------------
|
|
|
|
Let's the choice to the users to continue to inherit from the parent execution
|
|
model or to run heartbeat by using the python standard threading module.
|
|
|
|
This alernative is not really safe and can introduce possible issues related to
|
|
their contexts where the heartbeat will fail.
|
|
|
|
Impact on Existing usages
|
|
-------------------------
|
|
|
|
No impact on the existing usages since the major use case of the heartbeat do
|
|
not take care about the async execution model.
|
|
|
|
If users use heartbeat inside a wsgi environment and also use a monkey patched
|
|
environment then we will fix the heartbeat issue in this context.
|
|
|
|
Security impact
|
|
---------------
|
|
|
|
No significant security impact is expected.
|
|
|
|
Performance Impact
|
|
------------------
|
|
|
|
No significant performance impact is expected.
|
|
|
|
Configuration Impact
|
|
--------------------
|
|
|
|
No significant configuration impact is expected.
|
|
|
|
Developer Impact
|
|
----------------
|
|
|
|
No significant developer impact is expected.
|
|
|
|
Testing Impact
|
|
--------------
|
|
|
|
No significant testing impact is expected.
|
|
|
|
Implementation
|
|
==============
|
|
|
|
Assignee(s)
|
|
-----------
|
|
|
|
Primary assignee:
|
|
hberaud
|
|
|
|
Other contributors:
|
|
volunteers?
|
|
|
|
Milestones
|
|
----------
|
|
|
|
..TODO(hberaud): figure this out
|
|
|
|
Work Items
|
|
----------
|
|
|
|
* Implement the described changes in the rabbitmq driver.
|
|
* Update documentation to specify that we force the usage of pthreads.
|
|
|
|
Incubation
|
|
==========
|
|
|
|
N/A
|
|
|
|
Documentation Impact
|
|
====================
|
|
|
|
The documentation will need to be updated to indicate that we force the
|
|
usage of pthreads.
|
|
|
|
Dependencies
|
|
============
|
|
|
|
N/A
|
|
|
|
References
|
|
==========
|
|
|
|
* Eventlet Best Practices: https://review.opendev.org/#/c/154642/
|
|
* Related discussion on the ML about mod_wsgi and monkey patch: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005305.html
|
|
* Python asynchronous IO support - PEP 3156: https://www.python.org/dev/peps/pep-3156/
|
|
|
|
.. note::
|
|
|
|
This work is licensed under a Creative Commons Attribution 3.0
|
|
Unported License.
|
|
http://creativecommons.org/licenses/by/3.0/legalcode
|