tripleo-specs/specs/queens/tripleo-messaging.rst

Tripleo RPC and Notification Messaging Support

https://blueprints.launchpad.net/tripleo

This specification proposes changes to tripleo to enable the selection and configuration of separate messaging backends for oslo.messaging RPC and Notification communications. This proposal is a derivative of the work associated with the original blueprint¹ and specification ² to enable dual backends for oslo.messaging in tripleo.

Most of the groundwork to enable dual backends was implemented during the pike release and the introduction of an alternative messaging backend (qdrouterd) service was made. Presently, the deployment of this alternative messaging backend is accomplished by aliasing the rabbitmq service as the tripleo implementation does not model separate messaging backends.

Problem Description

The oslo.messaging library supports the deployment of dual messaging system backends for RPC and Notification communications. However, tripleo currently deploys a single rabbitmq server (cluster) that serves as a single messaging backend for both RPC and Notifications.

+------------+         +----------+
| RPC Caller |         | Notifier |
+-----+------+         +----+-----+
      |                     |
      +--+               +--+
         |               |
         v               v
       +-+---------------+-+
       |  RabbitMQ Service |
       | Messaging Backend |
       |                   |
       +-+---------------+-+
         ^               ^
         |               |
      +--+               +--+
      |                     |
      v                     v
+------+-----+        +------+-------+
|    RPC     |        | Notification |
|   Server   |        |    Server    |
+------------+        +--------------+

To support two separate and distinct messaging backends, tripleo needs to "duplicate" the set of parameters needed to specify each messaging system. The oslo.messaging library in OpenStack provides the API to the messaging services. It is proposed that the implementation model the RPC and Notification messaging services in place of the backend messaging server (e.g. rabbitmq).

+------------+          +----------+
| RPC Caller |          | Notifier |
+-----+------+          +----+-----+
      |                      |
      |                      |
      v                      v
+-------------------+  +-------------------+
|       RPC         |  |    Notification   |
| Messaging Service |  | Messaging Service |
|                   |  |                   |
+--------+----------+  +--------+----------+
      |                      |
      |                      |
      v                      v
+------------+        +------+-------+
|    RPC     |        | Notification |
|   Server   |        |    Server    |
+------------+        +--------------+

Introducing the separate messaging services and associated parameters in place of the rabbitmq server is not a major rework but special consideration must be made to upgrade paths and capabilities to ensure that existing configurations are not impacted.

Having separate messaging backends for RPC and Notification communications provides a number of benefits. These benefits include:

• tuning the backend to the messaging patterns
• increased aggregate message capacity
• reduced applied load to messaging servers
• increased message throughput
• reduced message latency
• etc.

Proposed Change

A number of issues need to be resolved in order to express RPC and Notification messaging services on top of the backend messaging systems.

Overview

The proposed change is similar to the concept of a service "backend" that is configured by tripleo. A number of existing services support such a backend (or plugin) model. The implementation of a messaging service backend model should account for the following requirements:

• deploy a single messaging backend for both RPC and Notifications
• deploy a messaging backend twice, once for RPC and once for Notifications
• deploy a messaging backend for RPC and a different messaging backend for Notifications
• deploy an external messaging backend for RPC
• deploy an external messaging backend for Notifications

Generally, the parameters that were required for deployment of the rabbitmq service should be duplicated and renamed to "RPC Messaging" and "Notify Messaging" backend service definitions. Individual backend files would exist for each possible backend type (e.g. rabbitmq, qdrouterd, zeromq, kafka or external). The backend selected will correspondingly define the messaging transport for the messaging system.

• transport specifier
• username
• password (and generation)
• host
• port
• virtual host(s)
• ssl (enabled)
• ssl configuration
• health checks

Tripleo should continue to have a default configuration that deploys RPC and Notifications messaging services on top of a single rabbitmq backend server (cluster). Tripleo upgrades should map the legacy rabbitmq service deployment onto the RPC and Notification messaging services model.

Alternatives

The configuration of separate messaging backends could be post overcloud deployment (e.g. external to tripleo framework). This would be problematic over the lifecycle of deployments e.g. during upgrades etc.

Security Impact

The deployment of dual messaging backends for RPC and Notification communications should be the same from a security standpoint. This assumes the backends have parity from a security feature perspective, e.g authentication and encryption.

Other End User Impact

Depending on the configuration of the messaging backend deployment, there could be a number of end user impacts including the following:

• monitoring of separated messaging backend services
• understanding differences in functionality/behaviors between different messaging backends (e.g. broker versus router, etc.)
• handling exceptions (e.g. different places for logs, etc.)

Performance Impact

Using separate messaging systems for RPC and Notifications should have a positive impact on performance and scalability by:

• separating RPC and Notification messaging loads
• increased parallelism in message processing
• increased aggregate message transfer capacity
• tuned backend configuration aligned to messaging patterns

Other Deployer Impact

The deployment of hybrid messaging will be new to OpenStack operators. Operators will need to learn the architectural differences as compared to a single backend deployment. This will include capacity planning, monitoring, troubleshooting and maintenance best practices.

Developer Impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Primary assignee:

• Andy Smith <ansmith@redhat.com>
• John Eckersberg <jeckersb@redhat.com>

Work Items

tripleo-heat-templates:

• Modify puppet/services/<service>base.yaml to introduce separate RPC and Notification Messaging parameters (e.g. replace 'rabbit' parameters)
• Support two ssl environments (e.g. one for RPC and one for Notification when separate backends are deployed)
• Consider example backend model such as the following:

tripleo-heat-templates
|
+--+ /environments
|  |
|  +--+ /messaging
|     |
|     +--+ messaging-(rpc/notify)-rabbitmq.yaml
|     +--+ messaging-(rpc/notify)-qdrouterd.yaml
|     +--+ messaging-(rpc/notify)-zmq.yaml
|     +--+ messaging-(rpc/notify)-kafka.yaml
+--+ /puppet
|  |
|  +--+ /services
|     |
|     +--+ messaging-(rpc/notify)-backend-rabbitmq.yaml
|     +--+ messaging-(rpc/notify)-backend-qdrouterd.yaml
|     +--+ messaging-(rpc/notify)-backend-zmq.yaml
|     +--+ messaging-(rpc/notify)-backend-kafka.yaml
|
+--+ /roles

puppet_tripleo:

• Replace rabbitmq_node_names with messaging_rpc_node_names and messaging_notify_node_names or similar
• Add vhost support
• Consider example backend model such as the following:

puppet-tripleo
|
+--+ /manifests
   |
   +--+ /profile
      |
      +--+ /base
         |
         +--+ /messaging
            |
            +--+ backend.pp
            +--+ rpc.pp
            +--+ notify.pp
               |
               +--+ /backend
                  |
                  +--+ rabbitmq.pp
                  +--+ qdrouterd.pp
                  +--+ zmq.pp
                  +--+ kafka.pp

tripleo_common:

• Add user and password management for RPC and Messaging services
• Support distinct health checks for separated messaging backends

packemaker:

• Determine what should happen when two separate rabbitmq clusters are deployed. Does this result in two pacemaker services or one? Some experimentation may be required.

Dependencies

None.

Testing

In order to test this in CI, an environment will be needed where separate messaging system backends (e.g. rabbitMQ server and dispatch-router server) are deployed. Any existing hardware configuration should be appropriate for the dual backend deployment.

Documentation Impact

The deployment documentation will need to be updated to cover the configuration of the separate messaging (RPC and Notify) services.

References

---

1. https://blueprints.launchpad.net/tripleo/+spec/om-dual-backends↩︎

2. https://review.openstack.org/#/c/396740/↩︎