diff --git a/doc/source/firehose.rst b/doc/source/firehose.rst deleted file mode 100644 index 56f3aa1a91..0000000000 --- a/doc/source/firehose.rst +++ /dev/null @@ -1,366 +0,0 @@ -:title: Firehose - -.. _firehose: - -Firehose -######## - -The unified message bus for Infra services. - -At a Glance -=========== - -:Hosts: - * firehose*.openstack.org -:Puppet: - * https://opendev.org/opendev/puppet-mosquitto - * https://opendev.org/opendev/puppet-germqtt - * https://opendev.org/opendev/puppet-lpmqtt - * :git_file:`modules/openstack_project/manifests/firehose.pp` -:Projects: - * https://mosquitto.org/ - * http://opendev.org/opendev/germqtt/ - * http://opendev.org/opendev/lpmqtt/ - -Overview -======== - -The firehose is an infra run MQTT broker that is a place for any infra run -service to publish events to. The concept behind it is that if anything needs -to consume an event from an infra run service we should have a single place -to go for consuming them. - -firehose.openstack.org hosts an instance of Mosquitto to be the MQTT broker -and also locally runs an instance of germqtt to publish the gerrit event -stream over MQTT and lpmqtt to publish a launchpad event stream over MQTT. - -Connection Info ---------------- - -firehose.openstack.org has 2 open ports for MQTT traffic: - -* **1883** - The default MQTT port -* **80** - Uses websockets for the MQTT communication -* **8883** - The default SSL/TLS MQTT port -* **443** - The SSL/TLS websockets port - - -Topics ------- - -Topics at a top level are set based on the name of the service publishing the -messages. The higher levels are specified by the publisher. For example:: - - gerrit/openstack-infra/germqtt/comment-added - -is a typical message topic on firehose. The top level 'gerrit' specifies the -service the message is from, and the rest of the message comes from germqtt -(the daemon used for publishing the gerrit events) - -MQTT topics are hierarchical and you can filter your subscription on part of the -hierarchy. `[1]`_ - -.. _[1]: https://mosquitto.org/man/mqtt-7.html - -Services Publishing to firehose -------------------------------- - -As of right now the following services publish messages to the firehose: - -+-----------------+------------------+----------------------------+ -| Service | Base Topic | Source of Messages | -+=================+==================+============================+ -| ansible | ansible | `ansible_mqtt_plugin`_ | -+-----------------+------------------+----------------------------+ -| gerrit | gerrit | `germqtt`_ | -+-----------------+------------------+----------------------------+ -| launchpad | launchpad | `lpmqtt`_ | -+-----------------+------------------+----------------------------+ -| subunit worker | gearman-subunit | `subunit-gearman-worker`_ | -+-----------------+------------------+----------------------------+ -| logstash worker | gearman-logstash | `logstash-gearman-worker`_ | -+-----------------+------------------+----------------------------+ - -.. _germqtt: http://opendev.org/opendev/germqtt/ -.. _lpmqtt: http://opendev.org/opendev/lpmqtt/ -.. _subunit-gearman-worker: https://opendev.org/opendev/puppet-subunit2sql/src/branch/master/files/subunit-gearman-worker.py -.. _ansible_mqtt_plugin: https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/puppetmaster/mqtt.py -.. _logstash-gearman-worker: https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-worker.py - -For a full schema description see :ref:`firehose_schema` - -Client Usage -============ -There is no outside access to publishing messages to the firehose available, -however anyone is able to subscribe to any topic services publish to. To -interact with the firehose you need to use the MQTT protocol. The specific -contents of the payload are dictated by the service publishing the -messages. So this section only covers how to subscribe and receive the messages -not how to consume the content received. - -Available Clients ------------------ -The MQTT community wiki maintains a page that lists available client bindings -for many languages here: https://github.com/mqtt/mqtt.github.io/wiki/libraries -For python using the `paho-mqtt`_ library is recommended - -.. _paho-mqtt: https://pypi.org/project/paho-mqtt/ - -CLI Example ------------ -The mosquitto project also provides both a CLI publisher and subscriber client -that can be used to easily subscribe to any topic and receive the messages. On -debian based distributions these are included in the mosquitto-clients package. -For example, to subscribe to every topic on the firehose you would run:: - - mosquitto_sub -h firehose.openstack.org --topic '#' - -You can adjust the value of the topic parameter to make what you're subscribing -to more specific. - -MQTT Protocol Example ---------------------- -Interacting with firehose on the unencrypted MQTT port is normally pretty easy in -most language bindings. Here are some examples that will have the same behavior -as the CLI example above and will subscribe to all topics on the firehose and -print it to STDOUT. - - -Python -'''''' -.. code-block:: python - - import paho.mqtt.client as mqtt - - - def on_connect(client, userdata, flags, rc): - print("Connected with result code " + str(rc)) - client.subscribe('#') - - def on_message(client, userdata, msg): - print(msg.topic+" "+str(msg.payload)) - - # Create a websockets client - client = mqtt.Client() - client.on_connect = on_connect - client.on_message = on_message - - # Connect to the firehose - client.connect('firehose.openstack.org') - # Listen forever - client.loop_forever() - -Haskell -''''''' -This requires the `mqtt-hs`_ library to be installed. - -.. _mqtt-hs: https://hackage.haskell.org/package/mqtt-hs - -.. code-block:: haskell - - {-# Language DataKinds, OverloadedStrings #-} - - module Subscribe where - - import Control.Concurrent - import Control.Concurrent.STM - import Control.Monad (unless, forever) - import System.Exit (exitFailure) - import System.IO (hPutStrLn, stderr) - - import qualified Network.MQTT as MQTT - - topic :: MQTT.Topic - topic = "#" - - handleMsg :: MQTT.Message MQTT.PUBLISH -> IO () - handleMsg msg = do - let t = MQTT.topic $ MQTT.body msg - p = MQTT.payload $ MQTT.body msg - print t - print p - - main :: IO () - main = do - cmds <- MQTT.mkCommands - pubChan <- newTChanIO - let conf = (MQTT.defaultConfig cmds pubChan) - { MQTT.cHost = "firehose.openstack.org" } - _ <- forkIO $ do - qosGranted <- MQTT.subscribe conf [(topic, MQTT.Handshake)] - forever $ atomically (readTChan pubChan) >>= handleMsg - terminated <- MQTT.run conf - print terminated - -Go -'' -.. code-block:: go - - package main - import ( - "fmt" - MQTT "github.com/eclipse/paho.mqtt.golang" - "os" - "strconv" - "time" - ) - func onMessageReceived(client MQTT.Client, msg MQTT.Message) { - fmt.Printf("TOPIC: %s\n", msg.Topic()) - fmt.Printf("MSG: %s\n", msg.Payload()) - } - func main() { - hostname, _ := os.Hostname() - opts := &MQTT.ClientOptions{ - ClientID: hostname+strconv.Itoa(time.Now().Second()), - } - opts.AddBroker("tcp://firehose.openstack.org:1883") - opts.OnConnect = func(c MQTT.Client) { - if token := c.Subscribe("#", 0, onMessageReceived); token.Wait() && token.Error() != nil { - fmt.Println(token.Error()) - os.Exit(1) - } - } - client := MQTT.NewClient(opts) - if token := client.Connect(); token.Wait() && token.Error() != nil { - panic(token.Error()) - } - for { - time.Sleep(1 * time.Second) - } - } - - -Websocket Example ------------------ -In addition to using the raw MQTT protocol firehose.o.o provides a websocket -interface on port 80 that MQTT traffic can go through. This is especially useful -for web applications that intend to consume any events from MQTT. To see an -example of this in action you can try: http://mitsuruog.github.io/what-mqtt/ -(the source is available here: https://github.com/mitsuruog/what-mqtt) and use -that to subscribe to any topics on firehose.openstack.org. - -Another advantage of using websockets over port 80 is that it's much more -firewall friendly, especially in environments that are more locked down. If you -would like to consume events from the firehose and are concerned about a -firewall blocking your access, the websocket interface is a good choice. - -You can also use the paho-mqtt python library to subscribe to mqtt over -websockets fairly easily. For example this script will subscribe to all topics -on the firehose and print it to STDOUT - -.. code-block:: python - :emphasize-lines: 12,17 - - import paho.mqtt.client as mqtt - - - def on_connect(client, userdata, flags, rc): - print("Connected with result code " + str(rc)) - client.subscribe('#') - - def on_message(client, userdata, msg): - print(msg.topic+" "+str(msg.payload)) - - # Create a websockets client - client = mqtt.Client(transport="websockets") - client.on_connect = on_connect - client.on_message = on_message - - # Connect to the firehose - client.connect('firehose.openstack.org', port=80) - # Listen forever - client.loop_forever() - -Using SSL/TLS -------------- -If you would like to connect to the firehose using ssl to encrypt the events you -receive from MQTT you just need to connect with ssl enabled via either of the -encrypted ports. If you'd like to verify the server ssl certificate when -connecting you'll need to provide a CA bundle to use as most MQTT clients do -not know how to use the system trusted CA bundle like most http clients. - -To connect to the firehose and subscribe to all topics you can use the -mosquitto CLI client:: - - mosquitto_sub --topic '#' -h firehose.openstack.org --cafile /etc/ca-certificates/extracted/tls-ca-bundle.pem -p 8883 - -You can use python: - -.. code-block:: python - :emphasize-lines: 15,20 - - import paho.mqtt.client as mqtt - - - def on_connect(client, userdata, flags, rc): - print("Connected with result code " + str(rc)) - client.subscribe('#') - - - def on_message(client, userdata, msg): - print(msg.topic+" "+str(msg.payload)) - - - # Create an SSL encrypted websockets client - client = mqtt.Client() - client.tls_set(ca_certs='/etc/ca-certificates/extracted/tls-ca-bundle.pem') - client.on_connect = on_connect - client.on_message = on_message - - # Connect to the firehose - client.connect('firehose.openstack.org', port=8883) - client.loop_forever() - - -Or with ruby: - -.. code-block:: ruby - :emphasize-lines: 6,7,8 - - require 'rubygems' - require 'mqtt' - - client = MQTT::Client.new - client.host = 'firehose.openstack.org' - client.ssl = true - client.cert_file = '/etc/ca-certificates/extracted/tls-ca-bundle.pem' - client.port = 8883 - client.connect() - client.subscribe('#') - - client.get do |topic,message| - puts message - end - -Example Use Cases -================= - -Event Notifications -------------------- - -A common use case for the event bus is to get a notification when an event -occurs. There is an open source tool, `mqttwarn`_ that makes setting this up -off the firehose (or any other mqtt broker) very straightforward. - -.. _mqttwarn: https://github.com/jpmens/mqttwarn - -You can use mqttwarn to setup custom notifications to a large number of tools -and services. (both local and remote). You can read the full docs on how to -configure and use mqttwarn at https://github.com/jpmens/mqttwarn/wiki and -https://github.com/jpmens/mqttwarn/blob/master/README.md - - -IMAP and MX -=========== - -We're using Cyrus as an IMAP server in order to consume launchpad bug -events via email. The configuration of the admin password account and -creation of the lpmqtt user for Cyrus were completed using the -following:: - - $ sudo saslpasswd2 cyrus - $ cyradm --user=cyrus --server=localhost - Password: - localhost> create user.lpmqtt - -An MX record has also been set up to point to the firehose server. diff --git a/doc/source/firehose_schema.rst b/doc/source/firehose_schema.rst deleted file mode 100644 index 7cb7dce097..0000000000 --- a/doc/source/firehose_schema.rst +++ /dev/null @@ -1,434 +0,0 @@ -:title: Firehose Schema - -.. _firehose_schema: - -Firehose Schema -############### - -This attempts to document the topic and payload schema for all the services -reporting to the firehose. However since much of what is reported to firehose -is dynamically generated it is possible this document misses a case. - -Gerrit -====== - -Messages on firehose for gerrit are generated using the `germqtt`_ project. For -the most part these are basically identical to what gerrit returns on it's -native event stream except over MQTT. - -.. _germqtt: http://opendev.org/opendev/germqtt/ - -Topics ------- - -The topics for gerrit are generated dynamically. However, they follow a fairly -straightforward pattern. The basic formula for this is:: - - gerrit/// - -So for example a typical topic would be:: - - gerrit/openstack/nova/comment-added - -The ``git namespace`` and ``repo name`` are pretty self explanatory and are just -from the git repository the change in gerrit is for. The event is defined in the gerrit event stream. You can see the full reference for topics in the Gerrit -docs for `Gerrit events`_. However, for simplicity the possible values are: - -* change-abandoned -* change-merged -* change-restored -* comment-added -* draft-published -* hashtags-changed -* merge-failed -* patchset-created -* ref-updated -* reviewer-added -* topic-changed - -Payload -------- -The payload for gerrit messages are basically the same JSON that gets returned -by gerrit's event stream command. Instead of repeating the entire gerrit schema -doc here just refer to gerrit's docs on the `JSON payload`_ which documents the -contents of each JSON object and refer to the doc on `Gerrit events`_ for which -JSON objects are included with which event type. - -.. _JSON payload: https://review.opendev.org/Documentation/json.html -.. _Gerrit events: https://review.opendev.org/Documentation/cmd-stream-events.html#events - -Launchpad -========= -The messages sent to firehose for launchpad are generated using `lpmqtt`_ - -.. _lpmqtt: http://opendev.org/opendev/lpmqtt/ - -Topics ------- - -The topics for lpmqtt follow a pretty simple formula:: - - launchpad/// - -the ``project`` is the launchpad project name, ``event type`` will always be -"bug" (or not present). The intent of this was to be "bug" or "blueprint", but -due to limitations in launchpad getting notifications from blueprints is not -possible. The flexibility was left in the schema just in case this ever changes. -The ``bug number`` is obviously the bug number from launchpad. - -It's also worth noting that only the base topic is a guaranteed field. Depending -on the notification email from launchpad some of the other fields may not be -present. In those cases the topic will be populated left to right until a -missing field is encountered. - -Payload -------- - -The payload of messages is dynamically generated and dependent on the -notification received from launchpad, and launchpad isn't always consistent in -what fields are present in those notifications. - -However, for bug event types there is a standard format. The fields which -are always present for bugs (which should normally be the only message for -firehose) are: - - * commenters - * bug-reporter - * bug-modifier - * bug-number - * event-type - -The remaining fields are dynamic and depend on launchpad. An example message -payload (with the body trimmed) for a bug is:: - - { - "status": "Triaged", - "project": "octavia", - "assignee": "email@fakedomain.com", - "bug-reporter": "Full name (username)", - "event-type": "bug", - "bug-number": "1680938", - "commenters": ["username"] - "tags": ["rfe"], - "importance": "Medium", - "bug-modifier": "Full Name (username)", - "body": "notification body, often is just bug comment or summary", - } - -Subunit Workers -=============== - -The messages for the subunit workers are generated directly in the -`subunit gearman worker scripts`_. - -.. _subunit gearman worker scripts: https://opendev.org/opendev/puppet-subunit2sql/src/branch/master/files/subunit-gearman-worker.py - -Topics ------- - -The topics for the subunit workers follow a simple pattern:: - - gearman-subunit/// - -Where ``worker hostname`` is the host which processed the subunit file, as -of right now there are 2, subunit-worker01 and subunit-worker02, but there may -be more (or fewer) in the future. The ``git namespace`` and ``repo name`` are -pretty self explanatory, and are just for the git repo under test that the -subunit was emitted from. ``change number`` is the gerrit change number for the -job that launched the tests the subunit is for. - -Payload -------- -The payload for the messages from the subunit workers is pretty straightforward -json that contains 3 fields: ``status``, ``build_uuid``, and ``source_url``. - -An example is:: - - { - 'status': 'success', - 'build_uuid': '45f7c1ddbfd74c6aba94662623bd61b8' - 'source_url': 'A url', - } - -Ansible -======= - -We have mqtt events emitted from ansible being run on :ref:`bridge`. -These events are generated using a `MQTT Ansible Callback Plugin`_. - -.. _MQTT Ansible Callback Plugin: https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/puppetmaster/mqtt.py - -Topics ------- - -The topics for ansible are a bit more involved than some of the other services -publishing to firehose. It depends on the type of event that ansible just -finished. There are 3 categories of events which have slightly different topic -formulas (and payloads). - -Playbook Events -''''''''''''''' -Whenever a playbook action occurs the callback plugin will emit an event for -it. The topics for playbook events fall into this pattern:: - - ansible/playbook//action// - -``playbook uuid`` is pretty self explanatory here, it's the uuid ansible uses -to uniquely identify the playbook being run. ``playbook action`` is the action -that the event is for, this is either going to be ``start`` or ``finish``. -``status`` is only set on ``finish`` and will be one of the following: - - * ``OK`` - * ``FAILED`` - -to indicate whether the playbook successfully executed or not. - -Playbook Stats Events -''''''''''''''''''''' - -At the end of a playbook these events are emitted for each host that tasks were -run on. The topics for these events fall into the following pattern:: - - ansible/playbook//stats/ - -In this case ``playbook uuid`` is the same as above and the internal ansible -unique playbook identifier. ``hostname`` here is the host that ansible was -running tasks on as part of the playbook. - -Task Events -''''''''''' - -At the end of each individual task the callback plugin will emit an event. Those -events' topics fall into the following pattern:: - - ansible/playbook//task// - -``playbook uuid`` is the same as in the previous 2 event types. ``hostname`` is -the hostname the task was executed on. ``status`` is the result of the task -and will be one of the following: - - * ``OK`` - * ``FAILED`` - * ``UNREACHABLE`` - -Payload -------- - -Just as with the topics the message payloads depend on the event type. Each -event uses a JSON payload with slightly different fields. - -Playbook Events -''''''''''''''' - -For playbook events the payload falls into this schema on playbook starts:: - - { - "status": "OK", - "host": - "session": , - "playbook_name": , - "playbook_id": , - "ansible_type": "start", - } - -When a playbook finishes the payload is slightly smaller and the schema is:: - - { - "playbook_id": , - "playbook_name": , - "status": , - } - -In both cases ``playbook uuid`` is the same field from the topic. -``playbook name`` is the human readable name for the playbook. If one is -set in the playbook this will be that. ``status`` will be whether the -playbook was successfully executed or not. It will always be ``OK`` on starts -(otherwise the event isn't emitted) but on failures, just like in the topic, -this will be one of the following: - - * ``OK`` - * ``FAILED`` - -``session id`` is a UUID generated by the callback plugin to uniquely identify -the execution of the playbook. ``hostname`` is the hostname where the ansible -playbook was launched. (which is not necessarily where tasks are being run) - - -An example of this from the system is for a start event:: - - { - "status": "OK", - "playbook_name": "localhost:!disabled", - "ansible_type": "start", - "host": "puppetmaster.openstack.org", - "session": "14d6e568-2c75-11e7-bd24-bc764e048db9", - "playbook_id": "5a95e9da-8d33-4dbb-a8b3-a77affc065d0" - } - -and for a finish:: - - { - "status": "FAILED", - "playbook_name": "compute*.ic.openstack.org:!disabled", - "playbook_id": "b259ac6d-6cb5-4403-bb8d-0ff2131c3d7a" - } - - -Playbook Stats Events -''''''''''''''''''''' - -The schema for stats events is:: - - { - "host": , - "ansible_host": , - "playbook_id": , - "playbook_name": , - "stats": { - "unreachable": int, - "skipped": int, - "ok": int, - "changed": int, - "failures": int, - } - } - -``playbook uuid`` is the same field from the topic. ``playbook name`` is the -human readable name for the playbook. If one is set in the playbook this will be -that. ``execute hostname`` is the hostname where the tasks were being executed, -while ``hostname`` is the hostname where ansible launched the playbook. The -``stats`` subdict contains the task status counts where the key is the tasks -statuses. - -An example from the running system is:: - - { - "playbook_name": "compute*.ic.openstack.org:!disabled", - "host": "puppetmaster.openstack.org", - "stats": { - "unreachable": 0, - "skipped": 5, - "ok": 13, - "changed": 1, - "failures": 0 - }, - "playbook_id": "b259ac6d-6cb5-4403-bb8d-0ff2131c3d7a", - "ansible_host": "controller00.vanilla.ic.openstack.org" - } - - -Task Events -''''''''''' - -The schema for tasks events is:: - - { - "status": , - "host": , - "ansible_host": , - "session": , - "ansible_type": "task", - "playbook_name": , - "playbook_id": , - "ansible_task": , - "ansible_result": - } - -``playbook uuid`` is the same field from the topic. ``playbook name`` is the -human readable name for the playbook. If one is set in the playbook this will be -that. ``execute hostname`` is the hostname where the tasks were being executed, -while ``hostname`` is the hostname where ansible launched the playbook. ``task -name``, like the name implies, is the human readable name of the task executed. -If one was specified in the playbook that will be the value. ``status`` is the -result of the task and will be one of the following: - - * ``OK`` - * ``FAILED`` - * ``UNREACHABLE`` - -``session id`` is a UUID generated by the callback plugin to uniquely identify -the execution of the playbook. - -``ansible result`` is a free form subdict that comes directly from ansible to -completely describe the task that just finished. The structure here is fully -dependent on ansible internals and the way that the task was invoked in the -playbook. Note, that sometimes this can be quite large in size depending on the -task and whether facts were enabled or not. - -An example of a task event from the running system is:: - - { - "status": "OK", - "host": "puppetmaster.openstack.org", - "session": "092aa3fa-2c73-11e7-bd24-bc764e048db9", - "playbook_name": "compute*.ic.openstack.org:!disabled", - "ansible_result": { - "_ansible_parsed": true, - "_ansible_no_log": false, - "stdout": "", - "changed": false, - "stderr": "", - "rc": 0, - "invocation": { - "module_name": "puppet", - "module_args": { - "logdest": "syslog", - "execute": null, - "facter_basename": "ansible", - "tags": null, - "puppetmaster": null, - "show_diff": false, - "certname": null, - "manifest": "/opt/system-config/manifests/site.pp", - "environment": "production", - "debug": false, - "noop": false, - "timeout": "30m", - "facts": null - } - }, - "stdout_lines": [] - }, - "ansible_type": "task", - "ansible_task": "TASK: puppet : run puppet", - "playbook_id": "b259ac6d-6cb5-4403-bb8d-0ff2131c3d7a", - "ansible_host": "compute014.chocolate.ic.openstack.org" - } - - -Logstash Workers -================ - -The messages for the subunit workers are generated directly in the -`logstash gearman worker scripts`_. - -.. _logstash gearman worker scripts: https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-worker.py - -Topics ------- - -The topics for the subunit workers follow a simple pattern:: - - gearman-logstash///// - -Where ``worker hostname`` is the host which processed the log file. The -``git namespace`` and ``repo name`` are pretty self explanatory, and are just -for the git repo under test that the log was generated. ``change number`` is -the gerrit change number for the job that launched the tests the subunit is for. -In the case of periodic or post queue jobs, this will either say ``periodic`` or -``post`` because there isn't an associated change number. The action field is -the phase of the log processing that just completed. Right now the only -possible value is ``retrieve_logs`` but there may be others in the future. - -Payload -------- -The payload for the messages from the logstash workers is pretty straightforward -json that contains 3 fields: ``status``, ``build_uuid``, and ``source_url``. - -An example is:: - - { - 'status': 'success', - 'build_uuid': '45f7c1ddbfd74c6aba94662623bd61b8' - 'source_url': 'A url', - } diff --git a/doc/source/systems.rst b/doc/source/systems.rst index a4bb1a7799..b85e40f443 100644 --- a/doc/source/systems.rst +++ b/doc/source/systems.rst @@ -38,8 +38,6 @@ Major Systems refstack codesearch signing - firehose - firehose_schema github activity asterisk diff --git a/hiera/common.yaml b/hiera/common.yaml index 1b8bcea211..8107a55fbf 100644 --- a/hiera/common.yaml +++ b/hiera/common.yaml @@ -222,7 +222,6 @@ cacti_hosts: - elasticsearch07.openstack.org - ethercalc02.openstack.org - etherpad01.opendev.org -- firehose01.openstack.org - gitea-lb01.opendev.org - gitea01.opendev.org - gitea02.opendev.org diff --git a/inventory/base/hosts.yaml b/inventory/base/hosts.yaml index 5d38e4ae27..95ff6a4851 100644 --- a/inventory/base/hosts.yaml +++ b/inventory/base/hosts.yaml @@ -154,13 +154,6 @@ all: region_name: DFW public_v4: 104.130.124.120 public_v6: 2001:4800:7818:104:be76:4eff:fe02:b0ff - firehose01.openstack.org: - ansible_host: 104.130.155.115 - location: - cloud: openstackci-rax - region_name: DFW - public_v4: 104.130.155.115 - public_v6: 2001:4800:7818:103:be76:4eff:fe04:40aa gitea-lb01.opendev.org: ansible_host: 38.108.68.124 location: diff --git a/inventory/service/group_vars/firehose.yaml b/inventory/service/group_vars/firehose.yaml deleted file mode 100644 index 4bfb2382b1..0000000000 --- a/inventory/service/group_vars/firehose.yaml +++ /dev/null @@ -1,25 +0,0 @@ -exim_local_domains: "@:firehose.openstack.org" -# TODO(jeblair): have the cyrus router check to see if there is a -# cyrus account. -exim_routers: - - dnslookup: '{{ exim_dnslookup_router }}' - - system_aliases: '{{ exim_system_aliases_router }}' - - cyrus: | - driver = accept - domains = +local_domains - local_part_suffix = +* - local_part_suffix_optional - transport = cyrus - - localuser: '{{ exim_localuser_router }}' -exim_transports: - - cyrus: | - driver = lmtp - socket = /var/run/cyrus/socket/lmtp - user = cyrus - batch_max = 35 -iptables_extra_public_tcp_ports: - - 25 - - 80 - - 443 - - 1883 - - 8883 diff --git a/inventory/service/group_vars/graphite.yaml b/inventory/service/group_vars/graphite.yaml index f3df914739..25753b7eed 100644 --- a/inventory/service/group_vars/graphite.yaml +++ b/inventory/service/group_vars/graphite.yaml @@ -7,7 +7,6 @@ iptables_extra_allowed_hosts: protocol: udp iptables_extra_allowed_groups: - - {'protocol': 'udp', 'port': '8125', 'group': 'firehose'} - {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'} - {'protocol': 'udp', 'port': '8125', 'group': 'logstash'} - {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'} diff --git a/inventory/service/group_vars/graphite_opendev.org b/inventory/service/group_vars/graphite_opendev.org index f3df914739..25753b7eed 100644 --- a/inventory/service/group_vars/graphite_opendev.org +++ b/inventory/service/group_vars/graphite_opendev.org @@ -7,7 +7,6 @@ iptables_extra_allowed_hosts: protocol: udp iptables_extra_allowed_groups: - - {'protocol': 'udp', 'port': '8125', 'group': 'firehose'} - {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'} - {'protocol': 'udp', 'port': '8125', 'group': 'logstash'} - {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'} diff --git a/inventory/service/groups.yaml b/inventory/service/groups.yaml index a0c4108a04..ec94a7980c 100644 --- a/inventory/service/groups.yaml +++ b/inventory/service/groups.yaml @@ -55,7 +55,6 @@ groups: elasticsearch: elasticsearch[0-9]*.open*.org ethercalc: ethercalc*.open*.org etherpad: etherpad[0-9]*.open*.org - firehose: firehose[0-9]*.open*.org gerrit: - review[0-9]*.open*.org gitea: @@ -138,7 +137,6 @@ groups: - eavesdrop[0-9]*.open*.org - elasticsearch[0-9]*.open*.org - ethercalc[0-9]*.open*.org - - firehose[0-9]*.open*.org - health[0-9]*.openstack.org - lists*.katacontainers.io - lists*.open*.org @@ -167,7 +165,6 @@ groups: - eavesdrop[0-9]*.open*.org - elasticsearch[0-9]*.open*.org - ethercalc[0-9]*.open*.org - - firehose[0-9]*.open*.org - health[0-9]*.openstack.org - lists*.katacontainers.io - lists*.open*.org diff --git a/manifests/site.pp b/manifests/site.pp index f8902aa622..79eba60dcb 100644 --- a/manifests/site.pp +++ b/manifests/site.pp @@ -181,24 +181,6 @@ node /^elasticsearch\d+\.open.*\.org$/ { } } -# Node-OS: xenial -node /^firehose\d+\.open.*\.org$/ { - class { 'openstack_project::server': } - class { 'openstack_project::firehose': - gerrit_ssh_host_key => hiera('gerrit_ssh_rsa_pubkey_contents'), - gerrit_public_key => hiera('germqtt_gerrit_ssh_public_key'), - gerrit_private_key => hiera('germqtt_gerrit_ssh_private_key'), - mqtt_password => hiera('mqtt_service_user_password'), - ca_file => hiera('mosquitto_tls_ca_file'), - cert_file => hiera('mosquitto_tls_server_cert_file'), - key_file => hiera('mosquitto_tls_server_key_file'), - imap_hostname => hiera('lpmqtt_imap_server'), - imap_username => hiera('lpmqtt_imap_username'), - imap_password => hiera('lpmqtt_imap_password'), - statsd_host => 'graphite.opendev.org', - } -} - # A machine to run Storyboard # Node-OS: xenial node /^storyboard\d+\.opendev\.org$/ { diff --git a/modules/openstack_project/files/ssl_cert_check/ssldomains b/modules/openstack_project/files/ssl_cert_check/ssldomains index 3561aff134..f3a6c208f8 100644 --- a/modules/openstack_project/files/ssl_cert_check/ssldomains +++ b/modules/openstack_project/files/ssl_cert_check/ssldomains @@ -9,7 +9,6 @@ docs.starlingx.io 443 ethercalc.openstack.org 443 etherpad.openstack.org 443 files.openstack.org 443 -firehose.openstack.org 8883 git.airshipit.org 443 git.openstack.org 443 git.starlingx.io 443 diff --git a/modules/openstack_project/manifests/firehose.pp b/modules/openstack_project/manifests/firehose.pp deleted file mode 100644 index a9a5b4a74e..0000000000 --- a/modules/openstack_project/manifests/firehose.pp +++ /dev/null @@ -1,87 +0,0 @@ -# Copyright 2016 Hewlett-Packard Development Company, L.P. -# -# Licensed under the Apache License, Version 2.0 (the "License"); you may -# not use this file except in compliance with the License. You may obtain -# a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT -# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the -# License for the specific language governing permissions and limitations -# under the License. -# -# firehose glue class. -# -class openstack_project::firehose ( - $gerrit_username = 'germqtt', - $gerrit_public_key, - $gerrit_private_key, - $gerrit_ssh_host_key, - $imap_username, - $imap_hostname, - $imap_password, - $mqtt_hostname = 'firehose.openstack.org', - $mqtt_password, - $mqtt_username = 'infra', - $statsd_host, - $ca_file, - $cert_file, - $key_file, -) { - include mosquitto - class {'mosquitto::server': - infra_service_username => $mqtt_username, - infra_service_password => $mqtt_password, - enable_tls => true, - enable_tls_websocket => true, - ca_file => $ca_file, - cert_file => $cert_file, - key_file => $key_file, - websocket_tls_port => 443, - } - - include germqtt - class {'germqtt::server': - gerrit_username => $gerrit_username, - gerrit_public_key => $gerrit_public_key, - gerrit_private_key => $gerrit_private_key, - gerrit_ssh_host_key => $gerrit_ssh_host_key, - mqtt_username => $mqtt_username, - mqtt_password => $mqtt_password, - } - - package {'cyrus-imapd': - ensure => latest, - } - - package {'sasl2-bin': - ensure => latest, - } - - package {'cyrus-admin': - ensure => latest, - } - - service {'cyrus-imapd': - ensure => running, - } - - include lpmqtt - class {'lpmqtt::server': - mqtt_username => $mqtt_username, - mqtt_password => $mqtt_password, - imap_hostname => $imap_hostname, - imap_username => $imap_username, - imap_password => $imap_password, - imap_use_ssl => false, - imap_delete_old => true, - } - - include mqtt_statsd - class {'mqtt_statsd::server': - mqtt_hostname => $mqtt_hostname, - statsd_hostname => $statsd_host, - } -} diff --git a/playbooks/group_vars/certcheck.yaml b/playbooks/group_vars/certcheck.yaml index d077aac519..b2e647db77 100644 --- a/playbooks/group_vars/certcheck.yaml +++ b/playbooks/group_vars/certcheck.yaml @@ -2,7 +2,6 @@ letsencrypt_certcheck_additional_domains: - ask.openstack.org 443 - ethercalc.openstack.org 443 - etherpad.openstack.org 443 - - firehose.openstack.org 8883 - git.openstack.org 443 - openstackid-dev.openstack.org 443 - openstackid.org 443 diff --git a/playbooks/roles/install-ansible/files/inventory_plugins/test-fixtures/results.yaml b/playbooks/roles/install-ansible/files/inventory_plugins/test-fixtures/results.yaml index bf60bd977f..3ab5c701be 100644 --- a/playbooks/roles/install-ansible/files/inventory_plugins/test-fixtures/results.yaml +++ b/playbooks/roles/install-ansible/files/inventory_plugins/test-fixtures/results.yaml @@ -13,11 +13,6 @@ results: - afs-client - kerberos-client - firehose01.openstack.org: - - firehose - - puppet - - puppet4 - graphite02.opendev.org: - graphite - letsencrypt