Every in tree driver that implements RPC send uses jsonutils.dumps to
serialize the message, except FakeDriver. FakeDriver uses json.dumps.
However json.dumps is a lot more strict than jsonutils. This caused
nova to introduce test specific changes in the rpc handling .
This patch makes sure that each driver uses the same json serialization.
I've tried to dig in to the history of the strictness of the
FakeDriver. That driver with the json.dumps() call was added back in
2013 with e2b74cc9e6. (I cannot link to
that commit in any online way but it is in my local git clone.)
Checking out that commit I don't see any other drivers present in the
repo but the code does mention drivers like RabbitDriver and ZmqDriver
in oslo.messaging/openstack/common/messaging/drivers.py but only there.
Today the oslo_messaging._drivers.common.serialize_msg() call is used
to do the final serialization of the message. It uses jsonutils.dumps
since Icd54ee8e3f5c976dfd50b4b62c7f51288649e112 which is a revert of
I0e0f6b715ffc4a9ad82be52e55696d032b6d0976 that changed from
jsonutils.dumps to jsonutils.dump_as_bytes by mistake. And before this
back and forth it was jsonutils.dumps since the code was imported from
oslo-incubator by I38507382b1ce68c7f8f697522f9a1bf00e76532d. Here
I lost the trail. Honestly I don't know the reason why the fake driver
was made stricter than the real drivers. Still I think today the
strictness is unnecessary as every driver uses jsonutils and even
counterproductive as in .
 09bf71407f/nova/compute/rpcapi.py (L820)
Add file to the reno documentation build to show release notes for
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
Introduce a RabbitMQ driver documentation for admin.
- some RabbitMQ and AMQP specifications (exchanges, queues, routing-key)
- the heartbeat specification and the type of used threads
- the driver options
This is an experimental feature.
The proposed changes will fix related issues when we run
heartbeat under apache/httpd enviornment with the apache MPM `prefork`
 engine and mod_wsgi or uwsgi in a monkey patched environment.
Propose changes to allow user to choose to run the rabbitmq health check
heartbeat in a standard python thread.
We facing an issue with the rabbitmq driver heartbeat
under apache MPM `prefork` module and mod_wsgi when nova_api monkey
patched the stdlib by using eventlet.
nova_api calling eventlet.monkey_patch()  when it runs under mod_wsgi.
This impacts the AMQP heartbeat thread,
which is meant to be a native thread. Instead of checking AMQP sockets
every 15s, It is now suspended and resumed by eventlet. However,
resuming greenthreads can take a very long time if mod_wsgi isn't
processing traffic regularly, which can cause rabbitmq to close the AMQP
The oslo.messaging RabbitMQ driver and especially the heartbeat
suffer to inherit the execution model of the service which consume him.
In this scenario nova_api need green threads to manage cells and edge
features so nova_api monkey patch the stdlib to obtain async features,
and the oslo.messaging rabbitmq driver endure these changes.
I think the main issue here is that nova_api want async and use eventlet green
threads to obtain it.
We want to allow user to isolate the heartbeat execution model
from the parent process inherited execution model by passing the
`heartbeat_in_pthread` option through the driver config.
While we use MPM `prefork` we want to avoid to use libevent and epoll.
If the `heartbeat_in_pthread` option is given we want to force to use the
python stdlib threading module to run the
rabbitmq heartbeat to avoid issue related to a non "standard"
environment. I mean "standard" because async features isn't the default
config in mostly case, starting by apache which define `prefork` is the
This is an experimental feature, we can help us to ensure to run heartbeat
through a classical python thread
There is no need to explicitly list the choices in the help text.
The oslo.config sample generator will include the choices automatically
Also tweaks the wording of text to make it clear that it is the allowed
values which vary based on kafka version.
0: 2488c1e1ce/oslo_config/generator.py (L263)
Some options are now automatically configured by the version 1.20:
Lots of exchanges create problems during failover under high
load. Please see bug report for details.
This is a step 2 patch.
Step 1 was: only using default exchange
Step 2 is to update consumers to only listen on default exchange,
happening now in T release.
With this feature, it is possible to specialize the parameters to send.
`options = oslo_messaging.TransportOptions(at_least_once=True)`
TransportOptions is used in every single driver,
for example in RabbitMQ driver is used to handle the mandatory flag.
- The idea of creating a new class TransportOptions is because I'd like
to have an abstract class not related only to the RPCClient
- at_least_once is the first parameter, when needed we can add the
Implements: blueprint transport-options (second point)
The blueprint link is 
To test it you can use 
When the message is large, in order to improve the efficiency of
kafka, we need to compress the message before send it, so we need to
support kafka message compression.
Implements: blueprint support-kafka-compression
With this new parameter is possible to pass other parameters
from the client to the drivers.
So it is possible to tune the driver behavior.
For example can be used to send the mandatory flag in RabbitMQ
- The transport_options parameter is not actually used (yet).
- This part of blueprint transport-options (first part)
Implements: blueprint transport-options
The blueprint link is
It seems that versions are deleted from www.apache.org pretty quickly.
They stick around longer on archive.apache.org so we won't have to
be constantly chasing the latest version in our functional tests.
Bandit 1.6.0 accidentally changed how the exclusion list option is
handled and breaks our use of it. Cap to the previous version until
Bandit has fixed the problem.
Sphinx 2.0 no longer works on python 2.7, so we need to start
capping it there as well.
In a clustered rabbitmq when a node disappears, we get a
ConnectionRefusedError because the socket get disconnected.
The socket access yields a OSError because the heartbeat
tries to reach an unreachable host (No route to host).
Catch these exceptions to ensure that we call ensure_connection for switching
the connection destination.
POC is available at github.com:4383/rabbitmq-oslo_messging-error-poc
$ git clone firstname.lastname@example.org:4383/rabbitmq-oslo_messging-error-poc
$ cd rabbitmq-oslo_messging-error-poc
$ python -m virtualenv .
$ source bin/activate
$ pip install -r requirements.txt
$ sudo podman run -d --hostname my-rabbit --name rabbit rabbitmq:3
$ python poc.py $(sudo podman inspect rabbit | niet '..NetworkSettings.IPAddress')
And in parallele in an another shell|tmux
$ podman stop rabbit
$ # observe the output of the poc.py script we now call ensure_connection
Now you can observe some output relative to the connection who is
modified and not catched before these changes.
Related to: https://bugzilla.redhat.com/show_bug.cgi?id=1665399
Log messages are no longer being translated. This removes all use of
the _LE, _LI, and _LW translation markers to simplify logging and to
avoid confusion with new contributions.