Remove firehose.openstack.org
Once we are satisfied that we have disabled the inputs to firehose we can land this change to stop managing it in config management. Once that is complete the server can be removed. Change-Id: I7ebd54f566f8d6f940a921b38139b54a9c4569d8
This commit is contained in:
Clark Boylan
parent
2e222a5b48
commit
2eebb858af
|
|
@ -1,366 +0,0 @@
|
||||||
:title: Firehose
|
|
||||||
|
|
||||||
.. _firehose:
|
|
||||||
|
|
||||||
Firehose
|
|
||||||
########
|
|
||||||
|
|
||||||
The unified message bus for Infra services.
|
|
||||||
|
|
||||||
At a Glance
|
|
||||||
===========
|
|
||||||
|
|
||||||
:Hosts:
|
|
||||||
* firehose*.openstack.org
|
|
||||||
:Puppet:
|
|
||||||
* https://opendev.org/opendev/puppet-mosquitto
|
|
||||||
* https://opendev.org/opendev/puppet-germqtt
|
|
||||||
* https://opendev.org/opendev/puppet-lpmqtt
|
|
||||||
* :git_file:`modules/openstack_project/manifests/firehose.pp`
|
|
||||||
:Projects:
|
|
||||||
* https://mosquitto.org/
|
|
||||||
* http://opendev.org/opendev/germqtt/
|
|
||||||
* http://opendev.org/opendev/lpmqtt/
|
|
||||||
|
|
||||||
Overview
|
|
||||||
========
|
|
||||||
|
|
||||||
The firehose is an infra run MQTT broker that is a place for any infra run
|
|
||||||
service to publish events to. The concept behind it is that if anything needs
|
|
||||||
to consume an event from an infra run service we should have a single place
|
|
||||||
to go for consuming them.
|
|
||||||
|
|
||||||
firehose.openstack.org hosts an instance of Mosquitto to be the MQTT broker
|
|
||||||
and also locally runs an instance of germqtt to publish the gerrit event
|
|
||||||
stream over MQTT and lpmqtt to publish a launchpad event stream over MQTT.
|
|
||||||
|
|
||||||
Connection Info
|
|
||||||
---------------
|
|
||||||
|
|
||||||
firehose.openstack.org has 2 open ports for MQTT traffic:
|
|
||||||
|
|
||||||
* **1883** - The default MQTT port
|
|
||||||
* **80** - Uses websockets for the MQTT communication
|
|
||||||
* **8883** - The default SSL/TLS MQTT port
|
|
||||||
* **443** - The SSL/TLS websockets port
|
|
||||||
|
|
||||||
|
|
||||||
Topics
|
|
||||||
------
|
|
||||||
|
|
||||||
Topics at a top level are set based on the name of the service publishing the
|
|
||||||
messages. The higher levels are specified by the publisher. For example::
|
|
||||||
|
|
||||||
gerrit/openstack-infra/germqtt/comment-added
|
|
||||||
|
|
||||||
is a typical message topic on firehose. The top level 'gerrit' specifies the
|
|
||||||
service the message is from, and the rest of the message comes from germqtt
|
|
||||||
(the daemon used for publishing the gerrit events)
|
|
||||||
|
|
||||||
MQTT topics are hierarchical and you can filter your subscription on part of the
|
|
||||||
hierarchy. `[1]`_
|
|
||||||
|
|
||||||
.. _[1]: https://mosquitto.org/man/mqtt-7.html
|
|
||||||
|
|
||||||
Services Publishing to firehose
|
|
||||||
-------------------------------
|
|
||||||
|
|
||||||
As of right now the following services publish messages to the firehose:
|
|
||||||
|
|
||||||
+-----------------+------------------+----------------------------+
|
|
||||||
| Service | Base Topic | Source of Messages |
|
|
||||||
+=================+==================+============================+
|
|
||||||
| ansible | ansible | `ansible_mqtt_plugin`_ |
|
|
||||||
+-----------------+------------------+----------------------------+
|
|
||||||
| gerrit | gerrit | `germqtt`_ |
|
|
||||||
+-----------------+------------------+----------------------------+
|
|
||||||
| launchpad | launchpad | `lpmqtt`_ |
|
|
||||||
+-----------------+------------------+----------------------------+
|
|
||||||
| subunit worker | gearman-subunit | `subunit-gearman-worker`_ |
|
|
||||||
+-----------------+------------------+----------------------------+
|
|
||||||
| logstash worker | gearman-logstash | `logstash-gearman-worker`_ |
|
|
||||||
+-----------------+------------------+----------------------------+
|
|
||||||
|
|
||||||
.. _germqtt: http://opendev.org/opendev/germqtt/
|
|
||||||
.. _lpmqtt: http://opendev.org/opendev/lpmqtt/
|
|
||||||
.. _subunit-gearman-worker: https://opendev.org/opendev/puppet-subunit2sql/src/branch/master/files/subunit-gearman-worker.py
|
|
||||||
.. _ansible_mqtt_plugin: https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/puppetmaster/mqtt.py
|
|
||||||
.. _logstash-gearman-worker: https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-worker.py
|
|
||||||
|
|
||||||
For a full schema description see :ref:`firehose_schema`
|
|
||||||
|
|
||||||
Client Usage
|
|
||||||
============
|
|
||||||
There is no outside access to publishing messages to the firehose available,
|
|
||||||
however anyone is able to subscribe to any topic services publish to. To
|
|
||||||
interact with the firehose you need to use the MQTT protocol. The specific
|
|
||||||
contents of the payload are dictated by the service publishing the
|
|
||||||
messages. So this section only covers how to subscribe and receive the messages
|
|
||||||
not how to consume the content received.
|
|
||||||
|
|
||||||
Available Clients
|
|
||||||
-----------------
|
|
||||||
The MQTT community wiki maintains a page that lists available client bindings
|
|
||||||
for many languages here: https://github.com/mqtt/mqtt.github.io/wiki/libraries
|
|
||||||
For python using the `paho-mqtt`_ library is recommended
|
|
||||||
|
|
||||||
.. _paho-mqtt: https://pypi.org/project/paho-mqtt/
|
|
||||||
|
|
||||||
CLI Example
|
|
||||||
-----------
|
|
||||||
The mosquitto project also provides both a CLI publisher and subscriber client
|
|
||||||
that can be used to easily subscribe to any topic and receive the messages. On
|
|
||||||
debian based distributions these are included in the mosquitto-clients package.
|
|
||||||
For example, to subscribe to every topic on the firehose you would run::
|
|
||||||
|
|
||||||
mosquitto_sub -h firehose.openstack.org --topic '#'
|
|
||||||
|
|
||||||
You can adjust the value of the topic parameter to make what you're subscribing
|
|
||||||
to more specific.
|
|
||||||
|
|
||||||
MQTT Protocol Example
|
|
||||||
---------------------
|
|
||||||
Interacting with firehose on the unencrypted MQTT port is normally pretty easy in
|
|
||||||
most language bindings. Here are some examples that will have the same behavior
|
|
||||||
as the CLI example above and will subscribe to all topics on the firehose and
|
|
||||||
print it to STDOUT.
|
|
||||||
|
|
||||||
|
|
||||||
Python
|
|
||||||
''''''
|
|
||||||
.. code-block:: python
|
|
||||||
|
|
||||||
import paho.mqtt.client as mqtt
|
|
||||||
|
|
||||||
|
|
||||||
def on_connect(client, userdata, flags, rc):
|
|
||||||
print("Connected with result code " + str(rc))
|
|
||||||
client.subscribe('#')
|
|
||||||
|
|
||||||
def on_message(client, userdata, msg):
|
|
||||||
print(msg.topic+" "+str(msg.payload))
|
|
||||||
|
|
||||||
# Create a websockets client
|
|
||||||
client = mqtt.Client()
|
|
||||||
client.on_connect = on_connect
|
|
||||||
client.on_message = on_message
|
|
||||||
|
|
||||||
# Connect to the firehose
|
|
||||||
client.connect('firehose.openstack.org')
|
|
||||||
# Listen forever
|
|
||||||
client.loop_forever()
|
|
||||||
|
|
||||||
Haskell
|
|
||||||
'''''''
|
|
||||||
This requires the `mqtt-hs`_ library to be installed.
|
|
||||||
|
|
||||||
.. _mqtt-hs: https://hackage.haskell.org/package/mqtt-hs
|
|
||||||
|
|
||||||
.. code-block:: haskell
|
|
||||||
|
|
||||||
{-# Language DataKinds, OverloadedStrings #-}
|
|
||||||
|
|
||||||
module Subscribe where
|
|
||||||
|
|
||||||
import Control.Concurrent
|
|
||||||
import Control.Concurrent.STM
|
|
||||||
import Control.Monad (unless, forever)
|
|
||||||
import System.Exit (exitFailure)
|
|
||||||
import System.IO (hPutStrLn, stderr)
|
|
||||||
|
|
||||||
import qualified Network.MQTT as MQTT
|
|
||||||
|
|
||||||
topic :: MQTT.Topic
|
|
||||||
topic = "#"
|
|
||||||
|
|
||||||
handleMsg :: MQTT.Message MQTT.PUBLISH -> IO ()
|
|
||||||
handleMsg msg = do
|
|
||||||
let t = MQTT.topic $ MQTT.body msg
|
|
||||||
p = MQTT.payload $ MQTT.body msg
|
|
||||||
print t
|
|
||||||
print p
|
|
||||||
|
|
||||||
main :: IO ()
|
|
||||||
main = do
|
|
||||||
cmds <- MQTT.mkCommands
|
|
||||||
pubChan <- newTChanIO
|
|
||||||
let conf = (MQTT.defaultConfig cmds pubChan)
|
|
||||||
{ MQTT.cHost = "firehose.openstack.org" }
|
|
||||||
_ <- forkIO $ do
|
|
||||||
qosGranted <- MQTT.subscribe conf [(topic, MQTT.Handshake)]
|
|
||||||
forever $ atomically (readTChan pubChan) >>= handleMsg
|
|
||||||
terminated <- MQTT.run conf
|
|
||||||
print terminated
|
|
||||||
|
|
||||||
Go
|
|
||||||
''
|
|
||||||
.. code-block:: go
|
|
||||||
|
|
||||||
package main
|
|
||||||
import (
|
|
||||||
"fmt"
|
|
||||||
MQTT "github.com/eclipse/paho.mqtt.golang"
|
|
||||||
"os"
|
|
||||||
"strconv"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
func onMessageReceived(client MQTT.Client, msg MQTT.Message) {
|
|
||||||
fmt.Printf("TOPIC: %s\n", msg.Topic())
|
|
||||||
fmt.Printf("MSG: %s\n", msg.Payload())
|
|
||||||
}
|
|
||||||
func main() {
|
|
||||||
hostname, _ := os.Hostname()
|
|
||||||
opts := &MQTT.ClientOptions{
|
|
||||||
ClientID: hostname+strconv.Itoa(time.Now().Second()),
|
|
||||||
}
|
|
||||||
opts.AddBroker("tcp://firehose.openstack.org:1883")
|
|
||||||
opts.OnConnect = func(c MQTT.Client) {
|
|
||||||
if token := c.Subscribe("#", 0, onMessageReceived); token.Wait() && token.Error() != nil {
|
|
||||||
fmt.Println(token.Error())
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
client := MQTT.NewClient(opts)
|
|
||||||
if token := client.Connect(); token.Wait() && token.Error() != nil {
|
|
||||||
panic(token.Error())
|
|
||||||
}
|
|
||||||
for {
|
|
||||||
time.Sleep(1 * time.Second)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
Websocket Example
|
|
||||||
-----------------
|
|
||||||
In addition to using the raw MQTT protocol firehose.o.o provides a websocket
|
|
||||||
interface on port 80 that MQTT traffic can go through. This is especially useful
|
|
||||||
for web applications that intend to consume any events from MQTT. To see an
|
|
||||||
example of this in action you can try: http://mitsuruog.github.io/what-mqtt/
|
|
||||||
(the source is available here: https://github.com/mitsuruog/what-mqtt) and use
|
|
||||||
that to subscribe to any topics on firehose.openstack.org.
|
|
||||||
|
|
||||||
Another advantage of using websockets over port 80 is that it's much more
|
|
||||||
firewall friendly, especially in environments that are more locked down. If you
|
|
||||||
would like to consume events from the firehose and are concerned about a
|
|
||||||
firewall blocking your access, the websocket interface is a good choice.
|
|
||||||
|
|
||||||
You can also use the paho-mqtt python library to subscribe to mqtt over
|
|
||||||
websockets fairly easily. For example this script will subscribe to all topics
|
|
||||||
on the firehose and print it to STDOUT
|
|
||||||
|
|
||||||
.. code-block:: python
|
|
||||||
:emphasize-lines: 12,17
|
|
||||||
|
|
||||||
import paho.mqtt.client as mqtt
|
|
||||||
|
|
||||||
|
|
||||||
def on_connect(client, userdata, flags, rc):
|
|
||||||
print("Connected with result code " + str(rc))
|
|
||||||
client.subscribe('#')
|
|
||||||
|
|
||||||
def on_message(client, userdata, msg):
|
|
||||||
print(msg.topic+" "+str(msg.payload))
|
|
||||||
|
|
||||||
# Create a websockets client
|
|
||||||
client = mqtt.Client(transport="websockets")
|
|
||||||
client.on_connect = on_connect
|
|
||||||
client.on_message = on_message
|
|
||||||
|
|
||||||
# Connect to the firehose
|
|
||||||
client.connect('firehose.openstack.org', port=80)
|
|
||||||
# Listen forever
|
|
||||||
client.loop_forever()
|
|
||||||
|
|
||||||
Using SSL/TLS
|
|
||||||
-------------
|
|
||||||
If you would like to connect to the firehose using ssl to encrypt the events you
|
|
||||||
receive from MQTT you just need to connect with ssl enabled via either of the
|
|
||||||
encrypted ports. If you'd like to verify the server ssl certificate when
|
|
||||||
connecting you'll need to provide a CA bundle to use as most MQTT clients do
|
|
||||||
not know how to use the system trusted CA bundle like most http clients.
|
|
||||||
|
|
||||||
To connect to the firehose and subscribe to all topics you can use the
|
|
||||||
mosquitto CLI client::
|
|
||||||
|
|
||||||
mosquitto_sub --topic '#' -h firehose.openstack.org --cafile /etc/ca-certificates/extracted/tls-ca-bundle.pem -p 8883
|
|
||||||
|
|
||||||
You can use python:
|
|
||||||
|
|
||||||
.. code-block:: python
|
|
||||||
:emphasize-lines: 15,20
|
|
||||||
|
|
||||||
import paho.mqtt.client as mqtt
|
|
||||||
|
|
||||||
|
|
||||||
def on_connect(client, userdata, flags, rc):
|
|
||||||
print("Connected with result code " + str(rc))
|
|
||||||
client.subscribe('#')
|
|
||||||
|
|
||||||
|
|
||||||
def on_message(client, userdata, msg):
|
|
||||||
print(msg.topic+" "+str(msg.payload))
|
|
||||||
|
|
||||||
|
|
||||||
# Create an SSL encrypted websockets client
|
|
||||||
client = mqtt.Client()
|
|
||||||
client.tls_set(ca_certs='/etc/ca-certificates/extracted/tls-ca-bundle.pem')
|
|
||||||
client.on_connect = on_connect
|
|
||||||
client.on_message = on_message
|
|
||||||
|
|
||||||
# Connect to the firehose
|
|
||||||
client.connect('firehose.openstack.org', port=8883)
|
|
||||||
client.loop_forever()
|
|
||||||
|
|
||||||
|
|
||||||
Or with ruby:
|
|
||||||
|
|
||||||
.. code-block:: ruby
|
|
||||||
:emphasize-lines: 6,7,8
|
|
||||||
|
|
||||||
require 'rubygems'
|
|
||||||
require 'mqtt'
|
|
||||||
|
|
||||||
client = MQTT::Client.new
|
|
||||||
client.host = 'firehose.openstack.org'
|
|
||||||
client.ssl = true
|
|
||||||
client.cert_file = '/etc/ca-certificates/extracted/tls-ca-bundle.pem'
|
|
||||||
client.port = 8883
|
|
||||||
client.connect()
|
|
||||||
client.subscribe('#')
|
|
||||||
|
|
||||||
client.get do |topic,message|
|
|
||||||
puts message
|
|
||||||
end
|
|
||||||
|
|
||||||
Example Use Cases
|
|
||||||
=================
|
|
||||||
|
|
||||||
Event Notifications
|
|
||||||
-------------------
|
|
||||||
|
|
||||||
A common use case for the event bus is to get a notification when an event
|
|
||||||
occurs. There is an open source tool, `mqttwarn`_ that makes setting this up
|
|
||||||
off the firehose (or any other mqtt broker) very straightforward.
|
|
||||||
|
|
||||||
.. _mqttwarn: https://github.com/jpmens/mqttwarn
|
|
||||||
|
|
||||||
You can use mqttwarn to setup custom notifications to a large number of tools
|
|
||||||
and services. (both local and remote). You can read the full docs on how to
|
|
||||||
configure and use mqttwarn at https://github.com/jpmens/mqttwarn/wiki and
|
|
||||||
https://github.com/jpmens/mqttwarn/blob/master/README.md
|
|
||||||
|
|
||||||
|
|
||||||
IMAP and MX
|
|
||||||
===========
|
|
||||||
|
|
||||||
We're using Cyrus as an IMAP server in order to consume launchpad bug
|
|
||||||
events via email. The configuration of the admin password account and
|
|
||||||
creation of the lpmqtt user for Cyrus were completed using the
|
|
||||||
following::
|
|
||||||
|
|
||||||
$ sudo saslpasswd2 cyrus
|
|
||||||
$ cyradm --user=cyrus --server=localhost
|
|
||||||
Password:
|
|
||||||
localhost> create user.lpmqtt
|
|
||||||
|
|
||||||
An MX record has also been set up to point to the firehose server.
|
|
||||||
|
|
@ -1,434 +0,0 @@
|
||||||
:title: Firehose Schema
|
|
||||||
|
|
||||||
.. _firehose_schema:
|
|
||||||
|
|
||||||
Firehose Schema
|
|
||||||
###############
|
|
||||||
|
|
||||||
This attempts to document the topic and payload schema for all the services
|
|
||||||
reporting to the firehose. However since much of what is reported to firehose
|
|
||||||
is dynamically generated it is possible this document misses a case.
|
|
||||||
|
|
||||||
Gerrit
|
|
||||||
======
|
|
||||||
|
|
||||||
Messages on firehose for gerrit are generated using the `germqtt`_ project. For
|
|
||||||
the most part these are basically identical to what gerrit returns on it's
|
|
||||||
native event stream except over MQTT.
|
|
||||||
|
|
||||||
.. _germqtt: http://opendev.org/opendev/germqtt/
|
|
||||||
|
|
||||||
Topics
|
|
||||||
------
|
|
||||||
|
|
||||||
The topics for gerrit are generated dynamically. However, they follow a fairly
|
|
||||||
straightforward pattern. The basic formula for this is::
|
|
||||||
|
|
||||||
gerrit/<git namespace>/<repo name>/<gerrit event>
|
|
||||||
|
|
||||||
So for example a typical topic would be::
|
|
||||||
|
|
||||||
gerrit/openstack/nova/comment-added
|
|
||||||
|
|
||||||
The ``git namespace`` and ``repo name`` are pretty self explanatory and are just
|
|
||||||
from the git repository the change in gerrit is for. The event is defined in the gerrit event stream. You can see the full reference for topics in the Gerrit
|
|
||||||
docs for `Gerrit events`_. However, for simplicity the possible values are:
|
|
||||||
|
|
||||||
* change-abandoned
|
|
||||||
* change-merged
|
|
||||||
* change-restored
|
|
||||||
* comment-added
|
|
||||||
* draft-published
|
|
||||||
* hashtags-changed
|
|
||||||
* merge-failed
|
|
||||||
* patchset-created
|
|
||||||
* ref-updated
|
|
||||||
* reviewer-added
|
|
||||||
* topic-changed
|
|
||||||
|
|
||||||
Payload
|
|
||||||
-------
|
|
||||||
The payload for gerrit messages are basically the same JSON that gets returned
|
|
||||||
by gerrit's event stream command. Instead of repeating the entire gerrit schema
|
|
||||||
doc here just refer to gerrit's docs on the `JSON payload`_ which documents the
|
|
||||||
contents of each JSON object and refer to the doc on `Gerrit events`_ for which
|
|
||||||
JSON objects are included with which event type.
|
|
||||||
|
|
||||||
.. _JSON payload: https://review.opendev.org/Documentation/json.html
|
|
||||||
.. _Gerrit events: https://review.opendev.org/Documentation/cmd-stream-events.html#events
|
|
||||||
|
|
||||||
Launchpad
|
|
||||||
=========
|
|
||||||
The messages sent to firehose for launchpad are generated using `lpmqtt`_
|
|
||||||
|
|
||||||
.. _lpmqtt: http://opendev.org/opendev/lpmqtt/
|
|
||||||
|
|
||||||
Topics
|
|
||||||
------
|
|
||||||
|
|
||||||
The topics for lpmqtt follow a pretty simple formula::
|
|
||||||
|
|
||||||
launchpad/<project>/<event type>/<bug number>
|
|
||||||
|
|
||||||
the ``project`` is the launchpad project name, ``event type`` will always be
|
|
||||||
"bug" (or not present). The intent of this was to be "bug" or "blueprint", but
|
|
||||||
due to limitations in launchpad getting notifications from blueprints is not
|
|
||||||
possible. The flexibility was left in the schema just in case this ever changes.
|
|
||||||
The ``bug number`` is obviously the bug number from launchpad.
|
|
||||||
|
|
||||||
It's also worth noting that only the base topic is a guaranteed field. Depending
|
|
||||||
on the notification email from launchpad some of the other fields may not be
|
|
||||||
present. In those cases the topic will be populated left to right until a
|
|
||||||
missing field is encountered.
|
|
||||||
|
|
||||||
Payload
|
|
||||||
-------
|
|
||||||
|
|
||||||
The payload of messages is dynamically generated and dependent on the
|
|
||||||
notification received from launchpad, and launchpad isn't always consistent in
|
|
||||||
what fields are present in those notifications.
|
|
||||||
|
|
||||||
However, for bug event types there is a standard format. The fields which
|
|
||||||
are always present for bugs (which should normally be the only message for
|
|
||||||
firehose) are:
|
|
||||||
|
|
||||||
* commenters
|
|
||||||
* bug-reporter
|
|
||||||
* bug-modifier
|
|
||||||
* bug-number
|
|
||||||
* event-type
|
|
||||||
|
|
||||||
The remaining fields are dynamic and depend on launchpad. An example message
|
|
||||||
payload (with the body trimmed) for a bug is::
|
|
||||||
|
|
||||||
{
|
|
||||||
"status": "Triaged",
|
|
||||||
"project": "octavia",
|
|
||||||
"assignee": "email@fakedomain.com",
|
|
||||||
"bug-reporter": "Full name (username)",
|
|
||||||
"event-type": "bug",
|
|
||||||
"bug-number": "1680938",
|
|
||||||
"commenters": ["username"]
|
|
||||||
"tags": ["rfe"],
|
|
||||||
"importance": "Medium",
|
|
||||||
"bug-modifier": "Full Name (username)",
|
|
||||||
"body": "notification body, often is just bug comment or summary",
|
|
||||||
}
|
|
||||||
|
|
||||||
Subunit Workers
|
|
||||||
===============
|
|
||||||
|
|
||||||
The messages for the subunit workers are generated directly in the
|
|
||||||
`subunit gearman worker scripts`_.
|
|
||||||
|
|
||||||
.. _subunit gearman worker scripts: https://opendev.org/opendev/puppet-subunit2sql/src/branch/master/files/subunit-gearman-worker.py
|
|
||||||
|
|
||||||
Topics
|
|
||||||
------
|
|
||||||
|
|
||||||
The topics for the subunit workers follow a simple pattern::
|
|
||||||
|
|
||||||
gearman-subunit/<worker hostname>/<git namespace/<repo name>/<change number>
|
|
||||||
|
|
||||||
Where ``worker hostname`` is the host which processed the subunit file, as
|
|
||||||
of right now there are 2, subunit-worker01 and subunit-worker02, but there may
|
|
||||||
be more (or fewer) in the future. The ``git namespace`` and ``repo name`` are
|
|
||||||
pretty self explanatory, and are just for the git repo under test that the
|
|
||||||
subunit was emitted from. ``change number`` is the gerrit change number for the
|
|
||||||
job that launched the tests the subunit is for.
|
|
||||||
|
|
||||||
Payload
|
|
||||||
-------
|
|
||||||
The payload for the messages from the subunit workers is pretty straightforward
|
|
||||||
json that contains 3 fields: ``status``, ``build_uuid``, and ``source_url``.
|
|
||||||
|
|
||||||
An example is::
|
|
||||||
|
|
||||||
{
|
|
||||||
'status': 'success',
|
|
||||||
'build_uuid': '45f7c1ddbfd74c6aba94662623bd61b8'
|
|
||||||
'source_url': 'A url',
|
|
||||||
}
|
|
||||||
|
|
||||||
Ansible
|
|
||||||
=======
|
|
||||||
|
|
||||||
We have mqtt events emitted from ansible being run on :ref:`bridge`.
|
|
||||||
These events are generated using a `MQTT Ansible Callback Plugin`_.
|
|
||||||
|
|
||||||
.. _MQTT Ansible Callback Plugin: https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/puppetmaster/mqtt.py
|
|
||||||
|
|
||||||
Topics
|
|
||||||
------
|
|
||||||
|
|
||||||
The topics for ansible are a bit more involved than some of the other services
|
|
||||||
publishing to firehose. It depends on the type of event that ansible just
|
|
||||||
finished. There are 3 categories of events which have slightly different topic
|
|
||||||
formulas (and payloads).
|
|
||||||
|
|
||||||
Playbook Events
|
|
||||||
'''''''''''''''
|
|
||||||
Whenever a playbook action occurs the callback plugin will emit an event for
|
|
||||||
it. The topics for playbook events fall into this pattern::
|
|
||||||
|
|
||||||
ansible/playbook/<playbook uuid>/action/<playbook action>/<status>
|
|
||||||
|
|
||||||
``playbook uuid`` is pretty self explanatory here, it's the uuid ansible uses
|
|
||||||
to uniquely identify the playbook being run. ``playbook action`` is the action
|
|
||||||
that the event is for, this is either going to be ``start`` or ``finish``.
|
|
||||||
``status`` is only set on ``finish`` and will be one of the following:
|
|
||||||
|
|
||||||
* ``OK``
|
|
||||||
* ``FAILED``
|
|
||||||
|
|
||||||
to indicate whether the playbook successfully executed or not.
|
|
||||||
|
|
||||||
Playbook Stats Events
|
|
||||||
'''''''''''''''''''''
|
|
||||||
|
|
||||||
At the end of a playbook these events are emitted for each host that tasks were
|
|
||||||
run on. The topics for these events fall into the following pattern::
|
|
||||||
|
|
||||||
ansible/playbook/<playbook uuid>/stats/<hostname>
|
|
||||||
|
|
||||||
In this case ``playbook uuid`` is the same as above and the internal ansible
|
|
||||||
unique playbook identifier. ``hostname`` here is the host that ansible was
|
|
||||||
running tasks on as part of the playbook.
|
|
||||||
|
|
||||||
Task Events
|
|
||||||
'''''''''''
|
|
||||||
|
|
||||||
At the end of each individual task the callback plugin will emit an event. Those
|
|
||||||
events' topics fall into the following pattern::
|
|
||||||
|
|
||||||
ansible/playbook/<playbook uuid>/task/<hostname>/<status>
|
|
||||||
|
|
||||||
``playbook uuid`` is the same as in the previous 2 event types. ``hostname`` is
|
|
||||||
the hostname the task was executed on. ``status`` is the result of the task
|
|
||||||
and will be one of the following:
|
|
||||||
|
|
||||||
* ``OK``
|
|
||||||
* ``FAILED``
|
|
||||||
* ``UNREACHABLE``
|
|
||||||
|
|
||||||
Payload
|
|
||||||
-------
|
|
||||||
|
|
||||||
Just as with the topics the message payloads depend on the event type. Each
|
|
||||||
event uses a JSON payload with slightly different fields.
|
|
||||||
|
|
||||||
Playbook Events
|
|
||||||
'''''''''''''''
|
|
||||||
|
|
||||||
For playbook events the payload falls into this schema on playbook starts::
|
|
||||||
|
|
||||||
{
|
|
||||||
"status": "OK",
|
|
||||||
"host": <hostname>
|
|
||||||
"session": <session id>,
|
|
||||||
"playbook_name": <playbook name>,
|
|
||||||
"playbook_id": <playbook uuid>,
|
|
||||||
"ansible_type": "start",
|
|
||||||
}
|
|
||||||
|
|
||||||
When a playbook finishes the payload is slightly smaller and the schema is::
|
|
||||||
|
|
||||||
{
|
|
||||||
"playbook_id": <playbook uuid>,
|
|
||||||
"playbook_name": <playbook name>,
|
|
||||||
"status": <status>,
|
|
||||||
}
|
|
||||||
|
|
||||||
In both cases ``playbook uuid`` is the same field from the topic.
|
|
||||||
``playbook name`` is the human readable name for the playbook. If one is
|
|
||||||
set in the playbook this will be that. ``status`` will be whether the
|
|
||||||
playbook was successfully executed or not. It will always be ``OK`` on starts
|
|
||||||
(otherwise the event isn't emitted) but on failures, just like in the topic,
|
|
||||||
this will be one of the following:
|
|
||||||
|
|
||||||
* ``OK``
|
|
||||||
* ``FAILED``
|
|
||||||
|
|
||||||
``session id`` is a UUID generated by the callback plugin to uniquely identify
|
|
||||||
the execution of the playbook. ``hostname`` is the hostname where the ansible
|
|
||||||
playbook was launched. (which is not necessarily where tasks are being run)
|
|
||||||
|
|
||||||
|
|
||||||
An example of this from the system is for a start event::
|
|
||||||
|
|
||||||
{
|
|
||||||
"status": "OK",
|
|
||||||
"playbook_name": "localhost:!disabled",
|
|
||||||
"ansible_type": "start",
|
|
||||||
"host": "puppetmaster.openstack.org",
|
|
||||||
"session": "14d6e568-2c75-11e7-bd24-bc764e048db9",
|
|
||||||
"playbook_id": "5a95e9da-8d33-4dbb-a8b3-a77affc065d0"
|
|
||||||
}
|
|
||||||
|
|
||||||
and for a finish::
|
|
||||||
|
|
||||||
{
|
|
||||||
"status": "FAILED",
|
|
||||||
"playbook_name": "compute*.ic.openstack.org:!disabled",
|
|
||||||
"playbook_id": "b259ac6d-6cb5-4403-bb8d-0ff2131c3d7a"
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
Playbook Stats Events
|
|
||||||
'''''''''''''''''''''
|
|
||||||
|
|
||||||
The schema for stats events is::
|
|
||||||
|
|
||||||
{
|
|
||||||
"host": <hostname>,
|
|
||||||
"ansible_host": <execute hostname>,
|
|
||||||
"playbook_id": <playbook uuid>,
|
|
||||||
"playbook_name": <playbook name>,
|
|
||||||
"stats": {
|
|
||||||
"unreachable": int,
|
|
||||||
"skipped": int,
|
|
||||||
"ok": int,
|
|
||||||
"changed": int,
|
|
||||||
"failures": int,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
``playbook uuid`` is the same field from the topic. ``playbook name`` is the
|
|
||||||
human readable name for the playbook. If one is set in the playbook this will be
|
|
||||||
that. ``execute hostname`` is the hostname where the tasks were being executed,
|
|
||||||
while ``hostname`` is the hostname where ansible launched the playbook. The
|
|
||||||
``stats`` subdict contains the task status counts where the key is the tasks
|
|
||||||
statuses.
|
|
||||||
|
|
||||||
An example from the running system is::
|
|
||||||
|
|
||||||
{
|
|
||||||
"playbook_name": "compute*.ic.openstack.org:!disabled",
|
|
||||||
"host": "puppetmaster.openstack.org",
|
|
||||||
"stats": {
|
|
||||||
"unreachable": 0,
|
|
||||||
"skipped": 5,
|
|
||||||
"ok": 13,
|
|
||||||
"changed": 1,
|
|
||||||
"failures": 0
|
|
||||||
},
|
|
||||||
"playbook_id": "b259ac6d-6cb5-4403-bb8d-0ff2131c3d7a",
|
|
||||||
"ansible_host": "controller00.vanilla.ic.openstack.org"
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
Task Events
|
|
||||||
'''''''''''
|
|
||||||
|
|
||||||
The schema for tasks events is::
|
|
||||||
|
|
||||||
{
|
|
||||||
"status": <status>,
|
|
||||||
"host": <hostname>,
|
|
||||||
"ansible_host": <execute hostname>,
|
|
||||||
"session": <session id>,
|
|
||||||
"ansible_type": "task",
|
|
||||||
"playbook_name": <playbook name>,
|
|
||||||
"playbook_id": <playbook uuid>,
|
|
||||||
"ansible_task": <task name>,
|
|
||||||
"ansible_result": <ansible result>
|
|
||||||
}
|
|
||||||
|
|
||||||
``playbook uuid`` is the same field from the topic. ``playbook name`` is the
|
|
||||||
human readable name for the playbook. If one is set in the playbook this will be
|
|
||||||
that. ``execute hostname`` is the hostname where the tasks were being executed,
|
|
||||||
while ``hostname`` is the hostname where ansible launched the playbook. ``task
|
|
||||||
name``, like the name implies, is the human readable name of the task executed.
|
|
||||||
If one was specified in the playbook that will be the value. ``status`` is the
|
|
||||||
result of the task and will be one of the following:
|
|
||||||
|
|
||||||
* ``OK``
|
|
||||||
* ``FAILED``
|
|
||||||
* ``UNREACHABLE``
|
|
||||||
|
|
||||||
``session id`` is a UUID generated by the callback plugin to uniquely identify
|
|
||||||
the execution of the playbook.
|
|
||||||
|
|
||||||
``ansible result`` is a free form subdict that comes directly from ansible to
|
|
||||||
completely describe the task that just finished. The structure here is fully
|
|
||||||
dependent on ansible internals and the way that the task was invoked in the
|
|
||||||
playbook. Note, that sometimes this can be quite large in size depending on the
|
|
||||||
task and whether facts were enabled or not.
|
|
||||||
|
|
||||||
An example of a task event from the running system is::
|
|
||||||
|
|
||||||
{
|
|
||||||
"status": "OK",
|
|
||||||
"host": "puppetmaster.openstack.org",
|
|
||||||
"session": "092aa3fa-2c73-11e7-bd24-bc764e048db9",
|
|
||||||
"playbook_name": "compute*.ic.openstack.org:!disabled",
|
|
||||||
"ansible_result": {
|
|
||||||
"_ansible_parsed": true,
|
|
||||||
"_ansible_no_log": false,
|
|
||||||
"stdout": "",
|
|
||||||
"changed": false,
|
|
||||||
"stderr": "",
|
|
||||||
"rc": 0,
|
|
||||||
"invocation": {
|
|
||||||
"module_name": "puppet",
|
|
||||||
"module_args": {
|
|
||||||
"logdest": "syslog",
|
|
||||||
"execute": null,
|
|
||||||
"facter_basename": "ansible",
|
|
||||||
"tags": null,
|
|
||||||
"puppetmaster": null,
|
|
||||||
"show_diff": false,
|
|
||||||
"certname": null,
|
|
||||||
"manifest": "/opt/system-config/manifests/site.pp",
|
|
||||||
"environment": "production",
|
|
||||||
"debug": false,
|
|
||||||
"noop": false,
|
|
||||||
"timeout": "30m",
|
|
||||||
"facts": null
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"stdout_lines": []
|
|
||||||
},
|
|
||||||
"ansible_type": "task",
|
|
||||||
"ansible_task": "TASK: puppet : run puppet",
|
|
||||||
"playbook_id": "b259ac6d-6cb5-4403-bb8d-0ff2131c3d7a",
|
|
||||||
"ansible_host": "compute014.chocolate.ic.openstack.org"
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
Logstash Workers
|
|
||||||
================
|
|
||||||
|
|
||||||
The messages for the subunit workers are generated directly in the
|
|
||||||
`logstash gearman worker scripts`_.
|
|
||||||
|
|
||||||
.. _logstash gearman worker scripts: https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-worker.py
|
|
||||||
|
|
||||||
Topics
|
|
||||||
------
|
|
||||||
|
|
||||||
The topics for the subunit workers follow a simple pattern::
|
|
||||||
|
|
||||||
gearman-logstash/<worker hostname>/<git namespace>/<repo name>/<change number>/<action>
|
|
||||||
|
|
||||||
Where ``worker hostname`` is the host which processed the log file. The
|
|
||||||
``git namespace`` and ``repo name`` are pretty self explanatory, and are just
|
|
||||||
for the git repo under test that the log was generated. ``change number`` is
|
|
||||||
the gerrit change number for the job that launched the tests the subunit is for.
|
|
||||||
In the case of periodic or post queue jobs, this will either say ``periodic`` or
|
|
||||||
``post`` because there isn't an associated change number. The action field is
|
|
||||||
the phase of the log processing that just completed. Right now the only
|
|
||||||
possible value is ``retrieve_logs`` but there may be others in the future.
|
|
||||||
|
|
||||||
Payload
|
|
||||||
-------
|
|
||||||
The payload for the messages from the logstash workers is pretty straightforward
|
|
||||||
json that contains 3 fields: ``status``, ``build_uuid``, and ``source_url``.
|
|
||||||
|
|
||||||
An example is::
|
|
||||||
|
|
||||||
{
|
|
||||||
'status': 'success',
|
|
||||||
'build_uuid': '45f7c1ddbfd74c6aba94662623bd61b8'
|
|
||||||
'source_url': 'A url',
|
|
||||||
}
|
|
||||||
|
|
@ -38,8 +38,6 @@ Major Systems
|
||||||
refstack
|
refstack
|
||||||
codesearch
|
codesearch
|
||||||
signing
|
signing
|
||||||
firehose
|
|
||||||
firehose_schema
|
|
||||||
github
|
github
|
||||||
activity
|
activity
|
||||||
asterisk
|
asterisk
|
||||||
|
|
|
||||||
|
|
@ -222,7 +222,6 @@ cacti_hosts:
|
||||||
- elasticsearch07.openstack.org
|
- elasticsearch07.openstack.org
|
||||||
- ethercalc02.openstack.org
|
- ethercalc02.openstack.org
|
||||||
- etherpad01.opendev.org
|
- etherpad01.opendev.org
|
||||||
- firehose01.openstack.org
|
|
||||||
- gitea-lb01.opendev.org
|
- gitea-lb01.opendev.org
|
||||||
- gitea01.opendev.org
|
- gitea01.opendev.org
|
||||||
- gitea02.opendev.org
|
- gitea02.opendev.org
|
||||||
|
|
|
||||||
|
|
@ -154,13 +154,6 @@ all:
|
||||||
region_name: DFW
|
region_name: DFW
|
||||||
public_v4: 104.130.124.120
|
public_v4: 104.130.124.120
|
||||||
public_v6: 2001:4800:7818:104:be76:4eff:fe02:b0ff
|
public_v6: 2001:4800:7818:104:be76:4eff:fe02:b0ff
|
||||||
firehose01.openstack.org:
|
|
||||||
ansible_host: 104.130.155.115
|
|
||||||
location:
|
|
||||||
cloud: openstackci-rax
|
|
||||||
region_name: DFW
|
|
||||||
public_v4: 104.130.155.115
|
|
||||||
public_v6: 2001:4800:7818:103:be76:4eff:fe04:40aa
|
|
||||||
gitea-lb01.opendev.org:
|
gitea-lb01.opendev.org:
|
||||||
ansible_host: 38.108.68.124
|
ansible_host: 38.108.68.124
|
||||||
location:
|
location:
|
||||||
|
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
exim_local_domains: "@:firehose.openstack.org"
|
|
||||||
# TODO(jeblair): have the cyrus router check to see if there is a
|
|
||||||
# cyrus account.
|
|
||||||
exim_routers:
|
|
||||||
- dnslookup: '{{ exim_dnslookup_router }}'
|
|
||||||
- system_aliases: '{{ exim_system_aliases_router }}'
|
|
||||||
- cyrus: |
|
|
||||||
driver = accept
|
|
||||||
domains = +local_domains
|
|
||||||
local_part_suffix = +*
|
|
||||||
local_part_suffix_optional
|
|
||||||
transport = cyrus
|
|
||||||
- localuser: '{{ exim_localuser_router }}'
|
|
||||||
exim_transports:
|
|
||||||
- cyrus: |
|
|
||||||
driver = lmtp
|
|
||||||
socket = /var/run/cyrus/socket/lmtp
|
|
||||||
user = cyrus
|
|
||||||
batch_max = 35
|
|
||||||
iptables_extra_public_tcp_ports:
|
|
||||||
- 25
|
|
||||||
- 80
|
|
||||||
- 443
|
|
||||||
- 1883
|
|
||||||
- 8883
|
|
||||||
|
|
@ -7,7 +7,6 @@ iptables_extra_allowed_hosts:
|
||||||
protocol: udp
|
protocol: udp
|
||||||
|
|
||||||
iptables_extra_allowed_groups:
|
iptables_extra_allowed_groups:
|
||||||
- {'protocol': 'udp', 'port': '8125', 'group': 'firehose'}
|
|
||||||
- {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'}
|
- {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'}
|
||||||
- {'protocol': 'udp', 'port': '8125', 'group': 'logstash'}
|
- {'protocol': 'udp', 'port': '8125', 'group': 'logstash'}
|
||||||
- {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'}
|
- {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'}
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,6 @@ iptables_extra_allowed_hosts:
|
||||||
protocol: udp
|
protocol: udp
|
||||||
|
|
||||||
iptables_extra_allowed_groups:
|
iptables_extra_allowed_groups:
|
||||||
- {'protocol': 'udp', 'port': '8125', 'group': 'firehose'}
|
|
||||||
- {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'}
|
- {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'}
|
||||||
- {'protocol': 'udp', 'port': '8125', 'group': 'logstash'}
|
- {'protocol': 'udp', 'port': '8125', 'group': 'logstash'}
|
||||||
- {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'}
|
- {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'}
|
||||||
|
|
|
||||||
|
|
@ -55,7 +55,6 @@ groups:
|
||||||
elasticsearch: elasticsearch[0-9]*.open*.org
|
elasticsearch: elasticsearch[0-9]*.open*.org
|
||||||
ethercalc: ethercalc*.open*.org
|
ethercalc: ethercalc*.open*.org
|
||||||
etherpad: etherpad[0-9]*.open*.org
|
etherpad: etherpad[0-9]*.open*.org
|
||||||
firehose: firehose[0-9]*.open*.org
|
|
||||||
gerrit:
|
gerrit:
|
||||||
- review[0-9]*.open*.org
|
- review[0-9]*.open*.org
|
||||||
gitea:
|
gitea:
|
||||||
|
|
@ -138,7 +137,6 @@ groups:
|
||||||
- eavesdrop[0-9]*.open*.org
|
- eavesdrop[0-9]*.open*.org
|
||||||
- elasticsearch[0-9]*.open*.org
|
- elasticsearch[0-9]*.open*.org
|
||||||
- ethercalc[0-9]*.open*.org
|
- ethercalc[0-9]*.open*.org
|
||||||
- firehose[0-9]*.open*.org
|
|
||||||
- health[0-9]*.openstack.org
|
- health[0-9]*.openstack.org
|
||||||
- lists*.katacontainers.io
|
- lists*.katacontainers.io
|
||||||
- lists*.open*.org
|
- lists*.open*.org
|
||||||
|
|
@ -167,7 +165,6 @@ groups:
|
||||||
- eavesdrop[0-9]*.open*.org
|
- eavesdrop[0-9]*.open*.org
|
||||||
- elasticsearch[0-9]*.open*.org
|
- elasticsearch[0-9]*.open*.org
|
||||||
- ethercalc[0-9]*.open*.org
|
- ethercalc[0-9]*.open*.org
|
||||||
- firehose[0-9]*.open*.org
|
|
||||||
- health[0-9]*.openstack.org
|
- health[0-9]*.openstack.org
|
||||||
- lists*.katacontainers.io
|
- lists*.katacontainers.io
|
||||||
- lists*.open*.org
|
- lists*.open*.org
|
||||||
|
|
|
||||||
|
|
@ -181,24 +181,6 @@ node /^elasticsearch\d+\.open.*\.org$/ {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
# Node-OS: xenial
|
|
||||||
node /^firehose\d+\.open.*\.org$/ {
|
|
||||||
class { 'openstack_project::server': }
|
|
||||||
class { 'openstack_project::firehose':
|
|
||||||
gerrit_ssh_host_key => hiera('gerrit_ssh_rsa_pubkey_contents'),
|
|
||||||
gerrit_public_key => hiera('germqtt_gerrit_ssh_public_key'),
|
|
||||||
gerrit_private_key => hiera('germqtt_gerrit_ssh_private_key'),
|
|
||||||
mqtt_password => hiera('mqtt_service_user_password'),
|
|
||||||
ca_file => hiera('mosquitto_tls_ca_file'),
|
|
||||||
cert_file => hiera('mosquitto_tls_server_cert_file'),
|
|
||||||
key_file => hiera('mosquitto_tls_server_key_file'),
|
|
||||||
imap_hostname => hiera('lpmqtt_imap_server'),
|
|
||||||
imap_username => hiera('lpmqtt_imap_username'),
|
|
||||||
imap_password => hiera('lpmqtt_imap_password'),
|
|
||||||
statsd_host => 'graphite.opendev.org',
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
# A machine to run Storyboard
|
# A machine to run Storyboard
|
||||||
# Node-OS: xenial
|
# Node-OS: xenial
|
||||||
node /^storyboard\d+\.opendev\.org$/ {
|
node /^storyboard\d+\.opendev\.org$/ {
|
||||||
|
|
|
||||||
|
|
@ -9,7 +9,6 @@ docs.starlingx.io 443
|
||||||
ethercalc.openstack.org 443
|
ethercalc.openstack.org 443
|
||||||
etherpad.openstack.org 443
|
etherpad.openstack.org 443
|
||||||
files.openstack.org 443
|
files.openstack.org 443
|
||||||
firehose.openstack.org 8883
|
|
||||||
git.airshipit.org 443
|
git.airshipit.org 443
|
||||||
git.openstack.org 443
|
git.openstack.org 443
|
||||||
git.starlingx.io 443
|
git.starlingx.io 443
|
||||||
|
|
|
||||||
|
|
@ -1,87 +0,0 @@
|
||||||
# Copyright 2016 Hewlett-Packard Development Company, L.P.
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
|
||||||
# not use this file except in compliance with the License. You may obtain
|
|
||||||
# a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
|
||||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
|
||||||
# License for the specific language governing permissions and limitations
|
|
||||||
# under the License.
|
|
||||||
#
|
|
||||||
# firehose glue class.
|
|
||||||
#
|
|
||||||
class openstack_project::firehose (
|
|
||||||
$gerrit_username = 'germqtt',
|
|
||||||
$gerrit_public_key,
|
|
||||||
$gerrit_private_key,
|
|
||||||
$gerrit_ssh_host_key,
|
|
||||||
$imap_username,
|
|
||||||
$imap_hostname,
|
|
||||||
$imap_password,
|
|
||||||
$mqtt_hostname = 'firehose.openstack.org',
|
|
||||||
$mqtt_password,
|
|
||||||
$mqtt_username = 'infra',
|
|
||||||
$statsd_host,
|
|
||||||
$ca_file,
|
|
||||||
$cert_file,
|
|
||||||
$key_file,
|
|
||||||
) {
|
|
||||||
include mosquitto
|
|
||||||
class {'mosquitto::server':
|
|
||||||
infra_service_username => $mqtt_username,
|
|
||||||
infra_service_password => $mqtt_password,
|
|
||||||
enable_tls => true,
|
|
||||||
enable_tls_websocket => true,
|
|
||||||
ca_file => $ca_file,
|
|
||||||
cert_file => $cert_file,
|
|
||||||
key_file => $key_file,
|
|
||||||
websocket_tls_port => 443,
|
|
||||||
}
|
|
||||||
|
|
||||||
include germqtt
|
|
||||||
class {'germqtt::server':
|
|
||||||
gerrit_username => $gerrit_username,
|
|
||||||
gerrit_public_key => $gerrit_public_key,
|
|
||||||
gerrit_private_key => $gerrit_private_key,
|
|
||||||
gerrit_ssh_host_key => $gerrit_ssh_host_key,
|
|
||||||
mqtt_username => $mqtt_username,
|
|
||||||
mqtt_password => $mqtt_password,
|
|
||||||
}
|
|
||||||
|
|
||||||
package {'cyrus-imapd':
|
|
||||||
ensure => latest,
|
|
||||||
}
|
|
||||||
|
|
||||||
package {'sasl2-bin':
|
|
||||||
ensure => latest,
|
|
||||||
}
|
|
||||||
|
|
||||||
package {'cyrus-admin':
|
|
||||||
ensure => latest,
|
|
||||||
}
|
|
||||||
|
|
||||||
service {'cyrus-imapd':
|
|
||||||
ensure => running,
|
|
||||||
}
|
|
||||||
|
|
||||||
include lpmqtt
|
|
||||||
class {'lpmqtt::server':
|
|
||||||
mqtt_username => $mqtt_username,
|
|
||||||
mqtt_password => $mqtt_password,
|
|
||||||
imap_hostname => $imap_hostname,
|
|
||||||
imap_username => $imap_username,
|
|
||||||
imap_password => $imap_password,
|
|
||||||
imap_use_ssl => false,
|
|
||||||
imap_delete_old => true,
|
|
||||||
}
|
|
||||||
|
|
||||||
include mqtt_statsd
|
|
||||||
class {'mqtt_statsd::server':
|
|
||||||
mqtt_hostname => $mqtt_hostname,
|
|
||||||
statsd_hostname => $statsd_host,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -2,7 +2,6 @@ letsencrypt_certcheck_additional_domains:
|
||||||
- ask.openstack.org 443
|
- ask.openstack.org 443
|
||||||
- ethercalc.openstack.org 443
|
- ethercalc.openstack.org 443
|
||||||
- etherpad.openstack.org 443
|
- etherpad.openstack.org 443
|
||||||
- firehose.openstack.org 8883
|
|
||||||
- git.openstack.org 443
|
- git.openstack.org 443
|
||||||
- openstackid-dev.openstack.org 443
|
- openstackid-dev.openstack.org 443
|
||||||
- openstackid.org 443
|
- openstackid.org 443
|
||||||
|
|
|
||||||
|
|
@ -13,11 +13,6 @@ results:
|
||||||
- afs-client
|
- afs-client
|
||||||
- kerberos-client
|
- kerberos-client
|
||||||
|
|
||||||
firehose01.openstack.org:
|
|
||||||
- firehose
|
|
||||||
- puppet
|
|
||||||
- puppet4
|
|
||||||
|
|
||||||
graphite02.opendev.org:
|
graphite02.opendev.org:
|
||||||
- graphite
|
- graphite
|
||||||
- letsencrypt
|
- letsencrypt
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue