openstack
/
large-scale
Code Issues Proposed changes

rabbitmq: add erlang scheduler recommendation 

The official rabbitmq  dockerimage sets `+stbt db` as a launch option
of erlang. This option defines how the erlang scheduler threads (the
threads actually running the erlang code) are distributed over the
phyiscal cpu cores. Per default erlang spawns one scheduler thread per
logical core (so hyperthread or phyiscal core depending on if you have
hyperthreading enabled).

The setting `db` translates to the setting
`thread_no_node_processor_spread` which basically means that each
scheduler thread is pinned to one specific logical core in a
predetermined order. There are some more details to this specific
setting, but they are not relevant for this issue (if interested please
see https://www.erlang.org/doc/man/erl.html#+sbt).

When possible the erlang runtime tries to run processes on the lowest
possible scheduler id. As the scheduler id consistently mapping to a
logical core this means that if possible all rabbitmq processes on a
given host try to run code on the same logical core.

The erlang documentation already metions that this can cause
performance issues:
> If the Erlang runtime system is the only operating system process
  that binds threads to logical processors, this improves the
  performance of the runtime system. However, if other operating
  system processes (for example another Erlang runtime system) also
  bind threads to logical processors, there can be a performance
  penalty instead. This performance penalty can sometimes be severe.
  If so, you are advised not to bind the schedulers.

As this recommendation is quite hidden we add it to our config
recommendations as well.

The above is a summary of https://www.erlang.org/doc/man/erl.html#+sbt.

Change-Id: Iee95287911097274c96c630fa912e7f994339b6a
This commit is contained in:
Felix Huettner 2022-09-01 14:02:50 +02:00
parent bdf1c7102c
commit 280ed67c35
1 changed files with 9 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
doc/source/other/rabbitmq.rst
View File

@ -75,6 +75,14 @@ When running the rabbit software on a node, you can configure some parameters fo
The most important configuration are the following
Erlang scheduler configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you run multiple rabbitmq clusters it is common to run these clusters on the same 3 hosts.
If you do this then you must ensure that all of the rabbitmq/erlang processes are started with `-stbt u` or `-stb u`.
This ensures that erlang does not compete over using the same cpu cores.
For details see https://www.erlang.org/doc/man/erl.html#+sbt
net_ticktime and heartbeat
^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -116,7 +124,7 @@ pattern
Policies are applied based on a regex pattern. The pattern we agreed on (from the mailing list discussion) is the following:
.. code-block:: console
'^(?!(amq\.)|(.*_fanout_)|(reply_)).*'
which will set HA on all queues, except the one that: