RabbitMQ is the default AMQP server used by many OpenStack services. Making the RabbitMQ service highly available involves: configuring a DRBD device for use by RabbitMQ, configuring RabbitMQ to use a data directory residing on that DRBD device, selecting and assigning a virtual IP address (VIP) that can freely float between cluster nodes, configuring RabbitMQ to listen on that IP address, managing all resources, including the RabbitMQ daemon itself, with the Pacemaker cluster manager. There is an alternative method of configuring RabbitMQ for high availability. That approach, known as active-active mirrored queues, happens to be the one preferred by the RabbitMQ developers — however it has shown less than ideal consistency and reliability in OpenStack clusters. Thus, at the time of writing, the Pacemaker/DRBD based approach remains the recommended one for OpenStack environments, although this may change in the near future as RabbitMQ active-active mirrored queues mature.

The Pacemaker based RabbitMQ server requires a DRBD resource from which it mounts the /var/lib/rabbitmq directory. In this example, the DRBD resource is simply named rabbitmq: resource rabbitmq { device minor 1; disk "/dev/data/rabbitmq"; meta-disk internal; on node1 { address ipv4 10.0.42.100:7701; } on node2 { address ipv4 10.0.42.254:7701; } } This resource uses an underlying local disk (in DRBD terminology, a backing device) named /dev/data/rabbitmq on both cluster nodes, node1 and node2. Normally, this would be an LVM Logical Volume specifically set aside for this purpose. The DRBD meta-disk is internal, meaning DRBD-specific metadata is being stored at the end of the disk device itself. The device is configured to communicate between IPv4 addresses 10.0.42.100 and 10.0.42.254, using TCP port 7701. Once enabled, it will map to a local DRBD block device with the device minor number 1, that is, /dev/drbd1. Enabling a DRBD resource is explained in detail in the DRBD User’s Guide. In brief, the proper sequence of commands is this: drbdadm create-md rabbitmq drbdadm up rabbitmq drbdadm -- --force primary rabbitmq Initializes DRBD metadata and writes the initial set of metadata to /dev/data/rabbitmq. Must be completed on both nodes. Creates the /dev/drbd1 device node, attaches the DRBD device to its backing store, and connects the DRBD node to its peer. Must be completed on both nodes. Kicks off the initial device synchronization, and puts the device into the primary (readable and writable) role. See Resource roles (from the DRBD User’s Guide) for a more detailed description of the primary and secondary roles in DRBD. Must be completed on one node only, namely the one where you are about to continue with creating your filesystem.

Once the DRBD resource is running and in the primary role (and potentially still in the process of running the initial device synchronization), you may proceed with creating the filesystem for RabbitMQ data. XFS is generally the recommended filesystem: mkfs -t xfs /dev/drbd1 You may also use the alternate device path for the DRBD device, which may be easier to remember as it includes the self-explanatory resource name: mkfs -t xfs /dev/drbd/by-res/rabbitmq Once completed, you may safely return the device to the secondary role. Any ongoing device synchronization will continue in the background: drbdadm secondary rabbitmq

In order for Pacemaker monitoring to function properly, you must ensure that RabbitMQ’s .erlang.cookie files are identical on all nodes, regardless of whether DRBD is mounted there or not. The simplest way of doing so is to take an existing .erlang.cookie from one of your nodes, copying it to the RabbitMQ data directory on the other node, and also copying it to the DRBD-backed filesystem. node1:# scp -a /var/lib/rabbitmq/.erlang.cookie node2:/var/lib/rabbitmq/ node1:# mount /dev/drbd/by-res/rabbitmq /mnt node1:# cp -a /var/lib/rabbitmq/.erlang.cookie /mnt node1:# umount /mnt

You may now proceed with adding the Pacemaker configuration for RabbitMQ resources. Connect to the Pacemaker cluster with crm configure, and add the following cluster resources: primitive p_ip_rabbitmp ocf:heartbeat:IPaddr2 \ params ip="192.168.42.100" cidr_netmask="24" \ op monitor interval="10s" primitive p_drbd_rabbitmq ocf:linbit:drbd \ params drbd_resource="rabbitmq" \ op start timeout="90s" \ op stop timeout="180s" \ op promote timeout="180s" \ op demote timeout="180s" \ op monitor interval="30s" role="Slave" \ op monitor interval="29s" role="Master" primitive p_fs_rabbitmq ocf:heartbeat:Filesystem \ params device="/dev/drbd/by-res/rabbitmq" \ directory="/var/lib/rabbitmq" \ fstype="xfs" options="relatime" \ op start timeout="60s" \ op stop timeout="180s" \ op monitor interval="60s" timeout="60s" primitive p_rabbitmq ocf:rabbitmq:rabbitmq-server \ params nodename="rabbit@localhost" \ mnesia_base="/var/lib/rabbitmq" \ op monitor interval="20s" timeout="10s" group g_rabbitmq p_ip_rabbitmq p_fs_rabbitmq p_rabbitmq ms ms_drbd_rabbitmq p_drbd_rabbitmq \ meta notify="true" master-max="1" clone-max="2" colocation c_rabbitmq_on_drbd inf: g_rabbitmq ms_drbd_rabbitmq:Master order o_drbd_before_rabbitmq inf: ms_drbd_rabbitmq:promote g_rabbitmq:start This configuration creates p_ip_rabbitmp, a virtual IP address for use by RabbitMQ (192.168.42.100), p_fs_rabbitmq, a Pacemaker managed filesystem mounted to /var/lib/rabbitmq on whatever node currently runs the RabbitMQ service, ms_drbd_rabbitmq, the master/slave set managing the rabbitmq DRBD resource, a service group and order and colocation constraints to ensure resources are started on the correct nodes, and in the correct sequence. crm configure supports batch input, so you may copy and paste the above into your live pacemaker configuration, and then make changes as required. For example, you may enter edit p_ip_rabbitmq from the crm configure menu and edit the resource to match your preferred virtual IP address. Once completed, commit your configuration changes by entering commit from the crm configure menu. Pacemaker will then start the RabbitMQ service, and its dependent resources, on one of your nodes.

Your OpenStack services must now point their RabbitMQ configuration to the highly available, virtual cluster IP address — rather than a RabbitMQ server’s physical IP address as you normally would. For OpenStack Image, for example, if your RabbitMQ service IP address is 192.168.42.100 as in the configuration explained here, you would use the following line in your OpenStack Image API configuration file (glance-api.conf): rabbit_host = 192.168.42.100 No other changes are necessary to your OpenStack configuration. If the node currently hosting your RabbitMQ experiences a problem necessitating service failover, your OpenStack services may experience a brief RabbitMQ interruption, as they would in the event of a network hiccup, and then continue to run normally.