5.1 KiB
Run the rabbitmq heartbeat in a python standard thread
https://blueprints.launchpad.net/oslo.messaging/+spec/rabbitmq-driver-threads
oslo.messaging RabbitMq driver heartbeat is designed to health check the connections with the rabbitmq server. olso.messaging rabbitmq heartbeat doesn't take care about the execution context and parent execution model and all the time keep the parent context applied. If the users need async on their side and use eventlet to monkey patch the python stdlib then the heartbeat will use greenthreads too.
This specification proposes changes to adapt the rabbitmq driver to isolate heartbeat execution model from the parent execution model and stop to inherit the parent execution model. The proposed changes will force to run the heartbeat mechanism inside a standard python thread. The heartbeat execution model will now designed by the oslo team to avoid issues and confusion between the parent context and the heartbeat needs.
In other words we want to isolate the heartbeat execution model from the parent execution problem to avoid possible issues related to the context (cf. eventlet/mod_wsgi).
Problem description
The rabbitmq driver is commonly used to health check the AMQP connections with rabbitmq to ensure connections by using the driver heartbeat mechanism.
Nowadays the heartbeat is designed to be reproduce the same execution than the parent process who launch him.
Parent process can need async on its side. To obtain async some user can use eventlet. Eventlet can be used to monkey patch the stdlib and to patch the python threading module and so obtain greenthreads and async execution model.
In many use cases (nova, neutron) the AMQP heartbeat can be used inside an execution evironment under mod_wsgi or uwsgi.
Actually using greenthreads and eventlet monkey patched python stdlib in the parent process force the heartbeat to inherit this context and run itself inside similar execution model.
The heartbeat health check is running in a dedicated thread and doesn't need to become async from the parent process point of view.
The main issue here is that we kept parent execution based on eventlet and async for the heartbeat dedicated heartbeat who can be implicitely run under mod_wsgi/uwsgi. As is described below mod_wsgi and eventlet greenthreads are not compatible.
Indeed, mod_wsgi stops the execution of its embedded interpreter, the AMQP heartbeat thread can't be resumed until there's a message to be processed in the mod_wsgi queue, which would resume the python interpreter and make eventlet resume the thread.
Force the heartbeat to use a standard pthreads and disabling eventlet will fix this kind of issues in problematic environment (mod_wsgi).
Proposed change
The spec simply proposes to force the heartbeat to use pthreads instead of let's the user execution model come inherited.
In all the case we will explicitely initialize a pthread who do not take care about the possible parent model who have monkey patch the python stdlib and the threading module.
Alternatives
Let's the choice to the users to continue to inherit from the parent execution model or to run heartbeat by using the python standard threading module.
This alernative is not really safe and can introduce possible issues related to their contexts where the heartbeat will fail.
Impact on Existing usages
No impact on the existing usages since the major use case of the heartbeat do not take care about the async execution model.
If users use heartbeat inside a wsgi environment and also use a monkey patched environment then we will fix the heartbeat issue in this context.
Security impact
No significant security impact is expected.
Performance Impact
No significant performance impact is expected.
Configuration Impact
No significant configuration impact is expected.
Developer Impact
No significant developer impact is expected.
Testing Impact
No significant testing impact is expected.
Implementation
Assignee(s)
- Primary assignee:
-
hberaud
- Other contributors:
-
volunteers?
Milestones
..TODO(hberaud): figure this out
Work Items
- Implement the described changes in the rabbitmq driver.
- Update documentation to specify that we force the usage of pthreads.
Incubation
N/A
Documentation Impact
The documentation will need to be updated to indicate that we force the usage of pthreads.
Dependencies
N/A
References
- Eventlet Best Practices: https://review.opendev.org/#/c/154642/
- Related discussion on the ML about mod_wsgi and monkey patch: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005305.html
- Python asynchronous IO support - PEP 3156: https://www.python.org/dev/peps/pep-3156/
Note
This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode