opendev/infra-specs
Code Issues Proposed changes
cab1a48a1a
infra-specs/specs/firehose.rst
Matthew Treinish 05b0c53133 Firehose: A unified message bus for Infra services 
This commit adds a spec for implementing a unified Infra message
bus, the "firehose."

Change-Id: I6295aec6025600a9ef02de5249a3a5c834dcba8c
Story: #2000643
2016-07-12 20:50:59 +00:00

5.4 KiB
Raw Blame History

::

This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode

Firehose: A unified message bus for Infra services

https://storyboard.openstack.org/#!/story/2000643

Using MQTT start running a unified message bus that community infrastructure services can emit events on. This will enable efficient external consumption of infrastructure events for experimentation and rapid innovation of tooling within the community.

Problem Description

There are a lot of services running in openstack-infra doing lots of different things all the time. Following events generated by these services can be tricky, and we don't have a consistent solution to making arbitrary state change details externally visible (so we fall back to relying on those with access to the machines letting us know what operations are being performed). What event data does exist is inconsistently presented in a variety of places over different protocols and requires significant effort to correlate cross-service.

Proposed Change

The goal of this spec is to create a unified message bus that Infra services can publish event streams to. This will enable external public consumption of Infra events in a consistent manner.

To accomplish this we will deploy a new server and run the mosquitto MQTT broker on it. MQTT protocol was designed in the late '90s to be lightweight pub-sub protocol that would work with connections to remote locations that needed a "small code footprint" and where there was limited network bandwith. More recently this has made it quite popular in the IoT space. However, it is also a good fit for the distributed cloud model that a lot of Infra services use. We will also deploy mosquitto using websockets this should also provide a benefit to international community members where things like the Gerrit event stream are problematic to access over Gerrit's SSH port.

That will provide the base infrastructure for publishing and subscribing to events but nothing will be using MQTT after that. To start the event reporting we'll leverage germqtt (and make it an Infra project). This will be the initial usage for the unified message bus and make the Gerrit event stream available over MQTT. While the scope of this spec ends with getting Gerrit into MQTT, once we have an example we can more easily add more sources reporting directly into MQTT.

Alternatives

Use fedmsg

The fedora infrastructure team had a similar need/desire to have a unified message bus and they started the fedmsg project to address this need. The primary issue with fedmsg is that it depends on AMQP or ZMQ both of which we have had previous issues with. The choice to go with MQTT was partially a reaction to that.

Use a different mqtt broker

There are several open source MQTT brokers besides Mosquitto out there including ones like RabbitMQ, VerneMQ, and mosca.

Mosquitto was chosen because it's a lightweight C program which will need less resources to run it. It also should be simpler to deploy since we don't have to worry about deploying a JavaVM, nodejs, or erlang (which seems to be quite popular for writing message brokers).

Implementation

Assignee(s)

Primary assignee:

  • mtreinish
  • fungi

Gerrit Topic

Use Gerrit topic "firehose" for all patches related to this spec.

git-review -t firehose

Work Items

  1. Write puppet-mosquitto module
  2. Deploy firehose01 server with puppet-mosquitto module and a

    firehose.openstack.org CNAME to it

  3. Add germqtt to openstack-infra
  4. Write puppet-germqtt
  5. Deploy puppet-germqtt on firehose01

Repositories

2 New repos will be created in the initial implemention, puppet-mosquitto and puppet-germqtt.

Servers

1 new server will need to be created for running mosquitto, firehose01. Since mosquitto and MQTT is fairly lightweight we probably can get away with a smaller server (2GB flavor should be plenty).

DNS Entries

Forward/reverse DNS entries will be added for firehose01.openstack.org, as well as a CNAME of firehose.openstack.org.

Documentation

Initial service documentation will be added to the openstack-infra/system-config repo as usual. An availability announcement will be sent to the openstack-dev mailing list.

Security

Sensitive data should not be published over the message bus. It is public and anonymously accessible.

Testing

The new Puppet modules will run our usual Puppet test and style jobs, and the germqtt repo will have typical Python unit testing and style checks as well.

Dependencies

  • This will be the first use of germqtt in infra (which will be added as an infra project)
  • 2 new puppet modules will be created, 1 to deploy mosquitto and the other to run germqtt