RabbitMQ: document handling stream replicas

Closes-Bug: #2138290
Change-Id: Ic5fc1b7e801b28a2bb20b4a60146ac443ffe5ef0
Signed-off-by: Matt Crees <mattc@stackhpc.com>

contrib/ops/rabbitmq/rabbitmq-repair-stream-replicas.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+running_nodes=$(docker exec rabbitmq rabbitmqctl cluster_status \
+    | awk '/Running Nodes/{f=1;next}/Versions/{f=0}f{print $1}')
+
+streams=$(docker exec rabbitmq rabbitmqctl list_queues name type --silent | awk '/stream/ {print $1}')
+
+for stream in $streams; do
+    stream_nodes=$(docker exec rabbitmq rabbitmq-streams stream_status -s $stream \
+        | awk 'NR>3 {print $4}')
+
+    missing=0
+    for node in $running_nodes; do
+        if ! grep -qx "$node" <<< "$stream_nodes"; then
+            echo "MISSING replica of $stream on $node"
+            missing=1
+            docker exec rabbitmq rabbitmq-streams add_replica $stream $node
+        fi
+    done
+
+    if [ $missing -eq 0 ]; then
+        echo "OK: All running nodes have a replica of $stream"
+    fi
+done

doc/source/reference/message-queues/rabbitmq.rst

@@ -159,3 +159,22 @@ different type, the follow procedure will be needed.
    .. code-block:: console
 
       kolla-ansible deploy --tags <service-tags>
+
+Handling Stream Replicas
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+RabbitMQ streams are expected to be replicated across the nodes in the
+cluster. However, RabbitMQ itself will only create replicas of a stream when
+the stream is initially declared. This means that any streams declared when a
+RabbitMQ node is out of service must be explicitly managed by an operator.
+RabbitMQ documents how to manage stream replicas here:
+https://www.rabbitmq.com/docs/streams#member-management
+
+An example script to create any missing stream replicas can be found under
+`kolla-ansible/contrib/ops/rabbitmq/rabbitmq-repair-stream-replicas.sh
+<https://opendev.org/openstack/kolla-ansible/src/branch/master/contrib/ops/rabbitmq/rabbitmq-repair-stream-replicas.sh>`__.
+This should be executed from a host running the RabbitMQ container.
+Currently, membership changes for streams `is not entirely safe
+<https://github.com/rabbitmq/rabbitmq-server/discussions/14246>`__, so this
+script should only be used when the RabbitMQ cluster is in a known healthy
+state.