diff --git a/specs/train/amqp-rabbitmq-heartbeat-isolated-pthread.rst b/specs/train/amqp-rabbitmq-heartbeat-isolated-pthread.rst new file mode 100644 index 0000000..f2b535e --- /dev/null +++ b/specs/train/amqp-rabbitmq-heartbeat-isolated-pthread.rst @@ -0,0 +1,166 @@ +====================================================== +Run the rabbitmq heartbeat in a python standard thread +====================================================== + +https://blueprints.launchpad.net/oslo.messaging/+spec/rabbitmq-driver-threads + +oslo.messaging RabbitMq driver heartbeat is designed to health check +the connections with the rabbitmq server. +olso.messaging rabbitmq heartbeat doesn't take care about the execution +context and parent execution model and all the time keep the parent context +applied. +If the users need async on their side and use eventlet to +monkey patch the python stdlib then the heartbeat will use greenthreads too. + +This specification proposes changes to adapt the rabbitmq driver to +isolate heartbeat execution model from the parent execution model and stop +to inherit the parent execution model. +The proposed changes will force to run the heartbeat mechanism inside a +standard python thread. +The heartbeat execution model will now designed by the oslo team to avoid +issues and confusion between the parent context and the heartbeat needs. + +In other words we want to isolate the heartbeat execution model from the +parent execution problem to avoid possible issues related to the context +(cf. eventlet/mod_wsgi). + +Problem description +=================== + +The rabbitmq driver is commonly used to health check the AMQP connections with +rabbitmq to ensure connections by using the driver heartbeat mechanism. + +Nowadays the heartbeat is designed to be reproduce the same execution than +the parent process who launch him. + +Parent process can need async on its side. To obtain async some user can use +eventlet. Eventlet can be used to monkey patch the stdlib and to patch the +python threading module and so obtain greenthreads and async execution model. + +In many use cases (nova, neutron) the AMQP heartbeat can be used inside an +execution evironment under mod_wsgi or uwsgi. + +Actually using greenthreads and eventlet monkey patched python stdlib in the +parent process force the heartbeat to inherit this context and run itself +inside similar execution model. + +The heartbeat health check is running in a dedicated thread and doesn't need +to become async from the parent process point of view. + +The main issue here is that we kept parent execution based on eventlet and +async for the heartbeat dedicated heartbeat who can be implicitely run under +mod_wsgi/uwsgi. As is described below mod_wsgi and eventlet greenthreads are +not compatible. + +Indeed, mod_wsgi stops the execution of its embedded interpreter, the +AMQP heartbeat thread can't be resumed until there's a message to be +processed in the mod_wsgi queue, which would resume the python +interpreter and make eventlet resume the thread. + +Force the heartbeat to use a standard pthreads and disabling eventlet will fix +this kind of issues in problematic environment (mod_wsgi). + +Proposed change +=============== + +The spec simply proposes to force the heartbeat to use pthreads instead of +let's the user execution model come inherited. + +In all the case we will explicitely initialize a pthread who +do not take care about the possible parent model who have monkey patch the +python stdlib and the threading module. + +Alternatives +------------ + +Let's the choice to the users to continue to inherit from the parent execution +model or to run heartbeat by using the python standard threading module. + +This alernative is not really safe and can introduce possible issues related to +their contexts where the heartbeat will fail. + +Impact on Existing usages +------------------------- + +No impact on the existing usages since the major use case of the heartbeat do +not take care about the async execution model. + +If users use heartbeat inside a wsgi environment and also use a monkey patched +environment then we will fix the heartbeat issue in this context. + +Security impact +--------------- + +No significant security impact is expected. + +Performance Impact +------------------ + +No significant performance impact is expected. + +Configuration Impact +-------------------- + +No significant configuration impact is expected. + +Developer Impact +---------------- + +No significant developer impact is expected. + +Testing Impact +-------------- + +No significant testing impact is expected. + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + hberaud + +Other contributors: + volunteers? + +Milestones +---------- + +..TODO(hberaud): figure this out + +Work Items +---------- + +* Implement the described changes in the rabbitmq driver. +* Update documentation to specify that we force the usage of pthreads. + +Incubation +========== + +N/A + +Documentation Impact +==================== + +The documentation will need to be updated to indicate that we force the +usage of pthreads. + +Dependencies +============ + +N/A + +References +========== + +* Eventlet Best Practices: https://review.opendev.org/#/c/154642/ +* Related discussion on the ML about mod_wsgi and monkey patch: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005305.html +* Python asynchronous IO support - PEP 3156: https://www.python.org/dev/peps/pep-3156/ + +.. note:: + + This work is licensed under a Creative Commons Attribution 3.0 + Unported License. + http://creativecommons.org/licenses/by/3.0/legalcode