Merge "Remove configuration management for ELK stack"
This commit is contained in:
Zuul
Gerrit Code Review
commit
7b09f7baab
@ -1,293 +0,0 @@
|
||||
:title: Logstash
|
||||
|
||||
.. _logstash:
|
||||
|
||||
Logstash
|
||||
########
|
||||
|
||||
Logstash is a high-performance indexing and search engine for logs.
|
||||
|
||||
At a Glance
|
||||
===========
|
||||
|
||||
:Hosts:
|
||||
* http://logstash.openstack.org
|
||||
* logstash-worker\*.openstack.org
|
||||
* elasticsearch\*.openstack.org
|
||||
:Puppet:
|
||||
* https://opendev.org/opendev/puppet-logstash
|
||||
* :git_file:`modules/openstack_project/manifests/logstash.pp`
|
||||
* :git_file:`modules/openstack_project/manifests/logstash_worker.pp`
|
||||
* :git_file:`modules/openstack_project/manifests/elasticsearch.pp`
|
||||
:Configuration:
|
||||
* :git_file:`modules/openstack_project/files/logstash`
|
||||
* :git_file:`modules/openstack_project/templates/logstash`
|
||||
* `submit-logstash-jobs defaults`_
|
||||
:Projects:
|
||||
* http://logstash.net/
|
||||
* http://kibana.org/
|
||||
* http://www.elasticsearch.org/
|
||||
* http://crm114.sourceforge.net/
|
||||
:Bugs:
|
||||
* https://storyboard.openstack.org/#!/project/748
|
||||
* https://logstash.jira.com/secure/Dashboard.jspa
|
||||
* https://github.com/rashidkpc/Kibana/issues
|
||||
* https://github.com/elasticsearch/elasticsearch/issues
|
||||
|
||||
Overview
|
||||
========
|
||||
|
||||
Logs from Zuul test runs are sent to logstash where they are
|
||||
indexed and stored. Logstash facilitates reviewing logs from multiple
|
||||
sources in a single test run, searching for errors or particular
|
||||
events within a test run, as well as searching for log event trends
|
||||
across test runs.
|
||||
|
||||
System Architecture
|
||||
===================
|
||||
|
||||
There are four major layers in our Logstash setup.
|
||||
|
||||
1. Submit Logstash Jobs.
|
||||
The `logs post-playbook`_ in the Zuul ``base`` job submit logs defined
|
||||
in the `submit-logstash-jobs defaults`_ to a Logstash Indexer.
|
||||
2. Logstash Indexer.
|
||||
Reads these log events from the log pusher, filters them to remove
|
||||
unwanted lines, collapses multiline events together, and parses
|
||||
useful information out of the events before shipping them to
|
||||
ElasticSearch for storage and indexing.
|
||||
3. ElasticSearch.
|
||||
Provides log storage, indexing, and search.
|
||||
4. Kibana.
|
||||
A Logstash oriented web client for ElasticSearch. You can perform
|
||||
queries on your Logstash logs in ElasticSearch through Kibana using
|
||||
the Lucene query language.
|
||||
|
||||
Each layer scales horizontally. As the number of logs grows we can add
|
||||
more log pushers, more Logstash indexers, and more ElasticSearch nodes.
|
||||
Currently we have multiple Logstash worker nodes that pair a log pusher
|
||||
with a Logstash indexer. We did this as each Logstash process can only
|
||||
dedicate a single thread to filtering log events which turns into a
|
||||
bottleneck very quickly. This looks something like:
|
||||
|
||||
::
|
||||
|
||||
zuul post-logs playbook
|
||||
|
|
||||
|
|
||||
gearman-client
|
||||
/ | \
|
||||
/ | \
|
||||
gearman gearman gearman
|
||||
worker1 worker2 worker3
|
||||
| | |
|
||||
logstash logstash logstash
|
||||
indexer1 indexer2 indexer3
|
||||
\ | /
|
||||
\ | /
|
||||
elasticsearch
|
||||
cluster
|
||||
|
|
||||
|
|
||||
kibana
|
||||
|
||||
Log Pusher
|
||||
----------
|
||||
|
||||
This is an ansible module in the `submit-log-processor-jobs role`_. It
|
||||
submits Gearman jobs to push log files into logstash.
|
||||
|
||||
Log pushing looks like this:
|
||||
|
||||
* Zuul runs post-playbook on job completion.
|
||||
* Using info in the Gearman job log files are retrieved.
|
||||
* Log files are processed then shipped to Logstash.
|
||||
|
||||
Using Gearman allows us to scale the number of log pushers
|
||||
horizontally. It is as simple as adding another process that talks to
|
||||
the Gearman server.
|
||||
|
||||
If you are interested in technical details the source of these scripts
|
||||
can be found at
|
||||
|
||||
* https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-client.py
|
||||
* https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/log-gearman-worker.py
|
||||
|
||||
Logstash
|
||||
--------
|
||||
|
||||
Logstash does the heavy lifting of squashing all of our log lines into
|
||||
events with a common format. It reads the JSON log events from the log
|
||||
pusher connected to it, deletes events we don't want, parses log lines
|
||||
to set the timestamp, message, and other fields for the event, then
|
||||
ships these processed events off to ElasticSearch where they are stored
|
||||
and made queryable.
|
||||
|
||||
At a high level Logstash takes:
|
||||
|
||||
::
|
||||
|
||||
{
|
||||
"fields" {
|
||||
"build_name": "gate-foo",
|
||||
"build_numer": "10",
|
||||
"event_message": "2013-05-31T17:31:39.113 DEBUG Something happened",
|
||||
},
|
||||
}
|
||||
|
||||
And turns that into:
|
||||
|
||||
::
|
||||
|
||||
{
|
||||
"fields" {
|
||||
"build_name": "gate-foo",
|
||||
"build_numer": "10",
|
||||
"loglevel": "DEBUG"
|
||||
},
|
||||
"@message": "Something happened",
|
||||
"@timestamp": "2013-05-31T17:31:39.113Z",
|
||||
}
|
||||
|
||||
It flattens each log line into something that looks very much like
|
||||
all of the other events regardless of the source log line format. This
|
||||
makes querying your logs for lines from a specific build that failed
|
||||
between two timestamps with specific message content very easy. You
|
||||
don't need to write complicated greps instead you query against a
|
||||
schema.
|
||||
|
||||
The config file that tells Logstash how to do this flattening can be
|
||||
found at
|
||||
https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf
|
||||
|
||||
This works via the tags that are associated with a given message.
|
||||
|
||||
The tags in
|
||||
https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf
|
||||
are used to tell logstash how to parse a given file's messages, based
|
||||
on the file's message format.
|
||||
|
||||
When adding a new file to be indexed to
|
||||
https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-logstash-jobs/defaults/main.yaml
|
||||
at least one tag from the openstack-filters.conf file should be associated
|
||||
with the new file. One can expect to see '{%logmessage%}' instead of
|
||||
actual message data if indexing is not working properly.
|
||||
|
||||
In the event a new file's format is not covered, a patch for
|
||||
https://opendev.org/openstack/logstash-filters/src/branch/master/filters/openstack-filters.conf
|
||||
should be submitted with an appropriate parsing pattern.
|
||||
|
||||
ElasticSearch
|
||||
-------------
|
||||
|
||||
ElasticSearch is basically a REST API layer for Lucene. It provides
|
||||
the storage and search engine for Logstash. It scales horizontally and
|
||||
loves it when you give it more memory. Currently we run a multi-node
|
||||
cluster on large VMs to give ElasticSearch both memory and disk space.
|
||||
Per index (Logstash creates one index per day) we have N+1 replica
|
||||
redundancy to distribute disk utilization and provide high availability.
|
||||
Each replica is broken into multiple shards providing increased indexing
|
||||
and search throughput as each shard is essentially a valid mini index.
|
||||
|
||||
To check on the cluster health, run this command on any es.* node::
|
||||
|
||||
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
|
||||
|
||||
Kibana
|
||||
------
|
||||
|
||||
Kibana is a ruby app sitting behind Apache that provides a nice web UI
|
||||
for querying Logstash events stored in ElasticSearch. Our install can
|
||||
be reached at http://logstash.openstack.org. See
|
||||
:ref:`query-logstash` for more info on using Kibana to perform
|
||||
queries.
|
||||
|
||||
simpleproxy
|
||||
-----------
|
||||
Simpleproxy is a simple tcp proxy which allows forwarding tcp connections from
|
||||
one host to another. We use it to forward mysql traffic from a publicly
|
||||
accessible host to the trove instance running the subunit2sql MySQL DB. This
|
||||
allows for public access to the data on the database through the host
|
||||
logstash.openstack.org.
|
||||
|
||||
.. _query-logstash:
|
||||
|
||||
Querying Logstash
|
||||
=================
|
||||
|
||||
Hop on over to http://logstash.openstack.org and by default you get the
|
||||
last 15 minutes of everything Logstash knows about in chunks of 100.
|
||||
We run a lot of tests but it is possible no logs have come in over the
|
||||
last 15 minutes, change the dropdown in the top left from ``Last 15m``
|
||||
to ``Last 60m`` to get a better window on the logs. At this point you
|
||||
should see a list of logs, if you click on a log event it will expand
|
||||
and show you all of the fields associated with that event and their
|
||||
values (note Chromium and Kibana seem to have trouble with this at times
|
||||
and some fields end up without values, use Firefox if this happens).
|
||||
You can search based on all of these fields and if you click the
|
||||
magnifying glass next to a field in the expanded event view it will add
|
||||
that field and value to your search. This is a good way of refining
|
||||
searches without a lot of typing.
|
||||
|
||||
The above is good info for poking around in the Logstash logs, but
|
||||
one of your changes has a failing test and you want to know why. We
|
||||
can jumpstart the refining process with a simple query.
|
||||
|
||||
``@fields.build_change:"$FAILING_CHANGE" AND @fields.build_patchset:"$FAILING_PATCHSET" AND @fields.build_name:"$FAILING_BUILD_NAME" AND @fields.build_number:"$FAILING_BUILD_NUMBER"``
|
||||
|
||||
This will show you all logs available from the patchset and build pair
|
||||
that failed. Chances are that this is still a significant number of
|
||||
logs and you will want to do more filtering. You can add more filters
|
||||
to the query using ``AND`` and ``OR`` and parentheses can be used to
|
||||
group sections of the query. Potential additions to the above query
|
||||
might be
|
||||
|
||||
* ``AND @fields.filename:"logs/syslog.txt"`` to get syslog events.
|
||||
* ``AND @fields.filename:"logs/screen-n-api.txt"`` to get Nova API events.
|
||||
* ``AND @fields.loglevel:"ERROR"`` to get ERROR level events.
|
||||
* ``AND @message"error"`` to get events with error in their message.
|
||||
and so on.
|
||||
|
||||
General query tips:
|
||||
|
||||
* Don't search ``All time``. ElasticSearch is bad at trying to find all
|
||||
the things it ever knew about. Give it a window of time to look
|
||||
through. You can use the presets in the dropdown to select a window or
|
||||
use the ``foo`` to ``bar`` boxes above the frequency graph.
|
||||
* Only the @message field can have fuzzy searches performed on it. Other
|
||||
fields require specific information.
|
||||
* This system is growing fast and may not always keep up with the load.
|
||||
Be patient. If expected logs do not show up immediately after the
|
||||
Zuul job completes wait a few minutes.
|
||||
|
||||
crm114
|
||||
=======
|
||||
|
||||
In an effort to assist with automated failure detection, the infra team
|
||||
has started leveraging crm114 to classify and analyze the messages stored
|
||||
by logstash.
|
||||
|
||||
The tool utilizes a statistical approach for classifying data, and is
|
||||
frequently used as an email spam detector. For logstash data, the idea
|
||||
is to flag those log entries that are not in passing runs and only in
|
||||
failing ones, which should be useful in pinpointing what caused the
|
||||
failures.
|
||||
|
||||
In the OpenStack logstash system, crm114 attaches an error_pr attribute
|
||||
to all indexed entries. Values from -1000.00 to -10.00 should be considered
|
||||
sufficient to get all potential errors as identified by the program.
|
||||
Used in a kibana query, it would be structured like this:
|
||||
|
||||
::
|
||||
|
||||
error_pr:["-1000.0" TO "-10.0"]
|
||||
|
||||
|
||||
This is still an early effort and additional tuning and refinement should
|
||||
be expected. Should the crm114 settings need to be tuned or expanded,
|
||||
a patch may be submitted for this file, which controls the process:
|
||||
https://opendev.org/opendev/puppet-log_processor/src/branch/master/files/classify-log.crm
|
||||
|
||||
.. _logs post-playbook: https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/post-logs.yaml
|
||||
.. _submit-logstash-jobs defaults: https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-logstash-jobs/defaults/main.yaml
|
||||
.. _submit-log-processor-jobs role: https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-log-processor-jobs
|
||||
@ -16,7 +16,6 @@ Major Systems
|
||||
grafyaml
|
||||
keycloak
|
||||
zuul
|
||||
logstash
|
||||
elastic-recheck
|
||||
devstack-gate
|
||||
nodepool
|
||||
|
||||
@ -14,12 +14,6 @@ cacti_hosts:
|
||||
- bridge.openstack.org
|
||||
- cacti.openstack.org
|
||||
- eavesdrop01.opendev.org
|
||||
- elasticsearch02.openstack.org
|
||||
- elasticsearch03.openstack.org
|
||||
- elasticsearch04.openstack.org
|
||||
- elasticsearch05.openstack.org
|
||||
- elasticsearch06.openstack.org
|
||||
- elasticsearch07.openstack.org
|
||||
- ethercalc02.openstack.org
|
||||
- etherpad01.opendev.org
|
||||
- gitea-lb01.opendev.org
|
||||
@ -39,27 +33,6 @@ cacti_hosts:
|
||||
- kdc04.openstack.org
|
||||
- keycloak01.opendev.org
|
||||
- lists.openstack.org
|
||||
- logstash-worker01.openstack.org
|
||||
- logstash-worker02.openstack.org
|
||||
- logstash-worker03.openstack.org
|
||||
- logstash-worker04.openstack.org
|
||||
- logstash-worker05.openstack.org
|
||||
- logstash-worker06.openstack.org
|
||||
- logstash-worker07.openstack.org
|
||||
- logstash-worker08.openstack.org
|
||||
- logstash-worker09.openstack.org
|
||||
- logstash-worker10.openstack.org
|
||||
- logstash-worker11.openstack.org
|
||||
- logstash-worker12.openstack.org
|
||||
- logstash-worker13.openstack.org
|
||||
- logstash-worker14.openstack.org
|
||||
- logstash-worker15.openstack.org
|
||||
- logstash-worker16.openstack.org
|
||||
- logstash-worker17.openstack.org
|
||||
- logstash-worker18.openstack.org
|
||||
- logstash-worker19.openstack.org
|
||||
- logstash-worker20.openstack.org
|
||||
- logstash.openstack.org
|
||||
- nb01.opendev.org
|
||||
- nb02.opendev.org
|
||||
- nb03.opendev.org
|
||||
|
||||
@ -91,48 +91,6 @@ all:
|
||||
region_name: DFW
|
||||
public_v4: 104.239.144.232
|
||||
public_v6: 2001:4800:7818:104:be76:4eff:fe04:46c8
|
||||
elasticsearch02.openstack.org:
|
||||
ansible_host: 104.130.159.19
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 104.130.159.19
|
||||
public_v6: 2001:4800:7818:102:be76:4eff:fe05:40b
|
||||
elasticsearch03.openstack.org:
|
||||
ansible_host: 104.130.246.141
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 104.130.246.141
|
||||
public_v6: 2001:4800:7819:103:be76:4eff:fe05:f26
|
||||
elasticsearch04.openstack.org:
|
||||
ansible_host: 104.130.246.138
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 104.130.246.138
|
||||
public_v6: 2001:4800:7819:103:be76:4eff:fe02:3042
|
||||
elasticsearch05.openstack.org:
|
||||
ansible_host: 104.130.159.15
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 104.130.159.15
|
||||
public_v6: 2001:4800:7818:102:be76:4eff:fe04:55b5
|
||||
elasticsearch06.openstack.org:
|
||||
ansible_host: 104.130.246.111
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 104.130.246.111
|
||||
public_v6: 2001:4800:7819:103:be76:4eff:fe04:b9d7
|
||||
elasticsearch07.openstack.org:
|
||||
ansible_host: 104.130.246.44
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 104.130.246.44
|
||||
public_v6: 2001:4800:7819:103:be76:4eff:fe04:35bb
|
||||
ethercalc02.openstack.org:
|
||||
ansible_host: 162.242.144.125
|
||||
location:
|
||||
@ -279,153 +237,6 @@ all:
|
||||
region_name: DFW
|
||||
public_v4: 50.56.173.222
|
||||
public_v6: 2001:4800:780e:510:3bc3:d7f6:ff04:b736
|
||||
logstash-worker01.openstack.org:
|
||||
ansible_host: 172.99.116.137
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 172.99.116.137
|
||||
public_v6: 2001:4800:7821:105:be76:4eff:fe04:b9e7
|
||||
logstash-worker02.openstack.org:
|
||||
ansible_host: 23.253.253.66
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.253.66
|
||||
public_v6: 2001:4800:7817:104:be76:4eff:fe04:aee1
|
||||
logstash-worker03.openstack.org:
|
||||
ansible_host: 23.253.242.246
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.242.246
|
||||
public_v6: 2001:4800:7817:104:be76:4eff:fe04:81e6
|
||||
logstash-worker04.openstack.org:
|
||||
ansible_host: 166.78.47.27
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 166.78.47.27
|
||||
public_v6: 2001:4800:7817:101:be76:4eff:fe04:46ec
|
||||
logstash-worker05.openstack.org:
|
||||
ansible_host: 23.253.100.127
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.100.127
|
||||
public_v6: 2001:4800:7817:101:be76:4eff:fe04:3b27
|
||||
logstash-worker06.openstack.org:
|
||||
ansible_host: 23.253.100.168
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.100.168
|
||||
public_v6: 2001:4800:7817:101:be76:4eff:fe04:5e21
|
||||
logstash-worker07.openstack.org:
|
||||
ansible_host: 172.99.116.187
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 172.99.116.187
|
||||
public_v6: 2001:4800:7821:105:be76:4eff:fe04:bca2
|
||||
logstash-worker08.openstack.org:
|
||||
ansible_host: 166.78.47.61
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 166.78.47.61
|
||||
public_v6: 2001:4800:7817:101:be76:4eff:fe04:432c
|
||||
logstash-worker09.openstack.org:
|
||||
ansible_host: 23.253.119.179
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.119.179
|
||||
public_v6: 2001:4800:7817:103:be76:4eff:fe04:38d1
|
||||
logstash-worker10.openstack.org:
|
||||
ansible_host: 172.99.116.111
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 172.99.116.111
|
||||
public_v6: 2001:4800:7821:105:be76:4eff:fe04:3a04
|
||||
logstash-worker11.openstack.org:
|
||||
ansible_host: 166.78.47.86
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 166.78.47.86
|
||||
public_v6: 2001:4800:7817:101:be76:4eff:fe05:3e1
|
||||
logstash-worker12.openstack.org:
|
||||
ansible_host: 166.78.174.249
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 166.78.174.249
|
||||
public_v6: 2001:4800:7815:105:be76:4eff:fe01:61e4
|
||||
logstash-worker13.openstack.org:
|
||||
ansible_host: 23.253.230.28
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.230.28
|
||||
public_v6: 2001:4800:7817:103:be76:4eff:fe04:6ff1
|
||||
logstash-worker14.openstack.org:
|
||||
ansible_host: 172.99.116.170
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 172.99.116.170
|
||||
public_v6: 2001:4800:7821:105:be76:4eff:fe04:4a4a
|
||||
logstash-worker15.openstack.org:
|
||||
ansible_host: 23.253.242.14
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.242.14
|
||||
public_v6: 2001:4800:7817:104:be76:4eff:fe04:829b
|
||||
logstash-worker16.openstack.org:
|
||||
ansible_host: 23.253.230.58
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.230.58
|
||||
public_v6: 2001:4800:7817:103:be76:4eff:fe03:5f45
|
||||
logstash-worker17.openstack.org:
|
||||
ansible_host: 172.99.117.120
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 172.99.117.120
|
||||
public_v6: 2001:4800:7821:105:be76:4eff:fe04:6173
|
||||
logstash-worker18.openstack.org:
|
||||
ansible_host: 172.99.116.203
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 172.99.116.203
|
||||
public_v6: 2001:4800:7821:105:be76:4eff:fe05:435
|
||||
logstash-worker19.openstack.org:
|
||||
ansible_host: 166.78.47.173
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 166.78.47.173
|
||||
public_v6: 2001:4800:7817:101:be76:4eff:fe03:a5ec
|
||||
logstash-worker20.openstack.org:
|
||||
ansible_host: 23.253.76.170
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 23.253.76.170
|
||||
public_v6: 2001:4800:7815:105:be76:4eff:fe04:59a6
|
||||
logstash01.openstack.org:
|
||||
ansible_host: 104.239.141.90
|
||||
location:
|
||||
cloud: openstackci-rax
|
||||
region_name: DFW
|
||||
public_v4: 104.239.141.90
|
||||
public_v6: 2001:4800:7819:104:be76:4eff:fe04:935b
|
||||
meetpad01.opendev.org:
|
||||
ansible_host: 104.239.240.194
|
||||
location:
|
||||
|
||||
@ -1,4 +0,0 @@
|
||||
iptables_extra_allowed_groups:
|
||||
- {'protocol': 'tcp', 'port': '9200:9400', 'group': 'elasticsearch'}
|
||||
- {'protocol': 'tcp', 'port': '9200:9400', 'group': 'logstash'}
|
||||
- {'protocol': 'tcp', 'port': '9200:9400', 'group': 'logstash-worker'}
|
||||
@ -8,7 +8,6 @@ iptables_extra_allowed_hosts:
|
||||
|
||||
iptables_extra_allowed_groups:
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'logstash'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'zookeeper'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'zuul'}
|
||||
|
||||
@ -8,7 +8,6 @@ iptables_extra_allowed_hosts:
|
||||
|
||||
iptables_extra_allowed_groups:
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'mirror-update'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'logstash'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'nodepool'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'zookeeper'}
|
||||
- {'protocol': 'udp', 'port': '8125', 'group': 'zuul'}
|
||||
|
||||
@ -1,6 +0,0 @@
|
||||
iptables_extra_public_tcp_ports:
|
||||
- 80
|
||||
- 3306
|
||||
iptables_extra_allowed_groups:
|
||||
- {'protocol': 'tcp', 'port': '4730', 'group': 'logstash-worker'}
|
||||
- {'protocol': 'tcp', 'port': '4730', 'group': 'zuul-executor'}
|
||||
@ -52,7 +52,6 @@ groups:
|
||||
- adns*.opendev.org
|
||||
- ns*.opendev.org
|
||||
eavesdrop: eavesdrop[0-9]*.opendev.org
|
||||
elasticsearch: elasticsearch[0-9]*.open*.org
|
||||
ethercalc: ethercalc*.open*.org
|
||||
etherpad: etherpad[0-9]*.open*.org
|
||||
gitea:
|
||||
@ -105,10 +104,6 @@ groups:
|
||||
- storyboard[0-9]*.opendev.org
|
||||
- translate[0-9]*.open*.org
|
||||
- zuul[0-9]*.opendev.org
|
||||
logstash:
|
||||
- logstash[0-9]*.open*.org
|
||||
logstash-worker:
|
||||
- logstash-worker[0-9]*.open*.org
|
||||
mailman:
|
||||
- lists*.katacontainers.io
|
||||
- lists*.open*.org
|
||||
@ -131,11 +126,8 @@ groups:
|
||||
- paste[0-9]*.opendev.org
|
||||
puppet:
|
||||
- cacti[0-9]*.open*.org
|
||||
- elasticsearch[0-9]*.open*.org
|
||||
- ethercalc[0-9]*.open*.org
|
||||
- health[0-9]*.openstack.org
|
||||
- logstash-worker[0-9]*.open*.org
|
||||
- logstash[0-9]*.open*.org
|
||||
- status*.open*.org
|
||||
- storyboard-dev[0-9]*.opendev.org
|
||||
- storyboard[0-9]*.opendev.org
|
||||
@ -143,11 +135,8 @@ groups:
|
||||
- translate[0-9]*.open*.org
|
||||
puppet4:
|
||||
- cacti[0-9]*.open*.org
|
||||
- elasticsearch[0-9]*.open*.org
|
||||
- ethercalc[0-9]*.open*.org
|
||||
- health[0-9]*.openstack.org
|
||||
- logstash-worker[0-9]*.open*.org
|
||||
- logstash[0-9]*.open*.org
|
||||
- status*.open*.org
|
||||
- storyboard[0-9]*.opendev.org
|
||||
- storyboard-dev[0-9]*.opendev.org
|
||||
|
||||
@ -20,66 +20,6 @@ node /^ethercalc\d+\.open.*\.org$/ {
|
||||
}
|
||||
}
|
||||
|
||||
# Node-OS: xenial
|
||||
node /^logstash\d*\.open.*\.org$/ {
|
||||
class { 'openstack_project::server': }
|
||||
|
||||
class { 'openstack_project::logstash':
|
||||
discover_nodes => [
|
||||
'elasticsearch03.openstack.org:9200',
|
||||
'elasticsearch04.openstack.org:9200',
|
||||
'elasticsearch05.openstack.org:9200',
|
||||
'elasticsearch06.openstack.org:9200',
|
||||
'elasticsearch07.openstack.org:9200',
|
||||
'elasticsearch02.openstack.org:9200',
|
||||
],
|
||||
subunit2sql_db_host => hiera('subunit2sql_db_host', ''),
|
||||
subunit2sql_db_pass => hiera('subunit2sql_db_password', ''),
|
||||
}
|
||||
}
|
||||
|
||||
# Node-OS: xenial
|
||||
node /^logstash-worker\d+\.open.*\.org$/ {
|
||||
$group = 'logstash-worker'
|
||||
|
||||
$elasticsearch_nodes = [
|
||||
'elasticsearch02.openstack.org',
|
||||
'elasticsearch03.openstack.org',
|
||||
'elasticsearch04.openstack.org',
|
||||
'elasticsearch05.openstack.org',
|
||||
'elasticsearch06.openstack.org',
|
||||
'elasticsearch07.openstack.org',
|
||||
]
|
||||
|
||||
class { 'openstack_project::server': }
|
||||
|
||||
class { 'openstack_project::logstash_worker':
|
||||
discover_node => 'elasticsearch03.openstack.org',
|
||||
enable_mqtt => false,
|
||||
mqtt_password => hiera('mqtt_service_user_password'),
|
||||
mqtt_ca_cert_contents => hiera('mosquitto_tls_ca_file'),
|
||||
}
|
||||
}
|
||||
|
||||
# Node-OS: xenial
|
||||
node /^elasticsearch\d+\.open.*\.org$/ {
|
||||
$group = "elasticsearch"
|
||||
|
||||
$elasticsearch_nodes = [
|
||||
'elasticsearch02.openstack.org',
|
||||
'elasticsearch03.openstack.org',
|
||||
'elasticsearch04.openstack.org',
|
||||
'elasticsearch05.openstack.org',
|
||||
'elasticsearch06.openstack.org',
|
||||
'elasticsearch07.openstack.org',
|
||||
]
|
||||
|
||||
class { 'openstack_project::server': }
|
||||
class { 'openstack_project::elasticsearch_node':
|
||||
discover_nodes => $elasticsearch_nodes,
|
||||
}
|
||||
}
|
||||
|
||||
# A machine to run Storyboard
|
||||
# Node-OS: xenial
|
||||
node /^storyboard\d+\.opendev\.org$/ {
|
||||
|
||||
@ -58,15 +58,12 @@ SOURCE_MODULES["https://github.com/voxpupuli/puppet-nodejs"]="v2.3.0"
|
||||
# Please keep sorted
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-bup"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-elastic_recheck"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-elasticsearch"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-ethercalc"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-httpd"]="origin/master"
|
||||
# Storyboard and translate use the jeepyb module
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-jeepyb"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-kibana"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-log_processor"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-logrotate"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-logstash"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-mysql_backup"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-pip"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-project_config"]="origin/master"
|
||||
@ -76,7 +73,6 @@ INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-reviewday"]="origin/mast
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-simpleproxy"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-ssh"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-storyboard"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-subunit2sql"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-tmpreaper"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-ulimit"]="origin/master"
|
||||
INTEGRATION_MODULES["$OPENSTACK_GIT_ROOT/opendev/puppet-user"]="origin/master"
|
||||
|
||||
@ -1,3 +0,0 @@
|
||||
# Increase the max heap size to twice the default.
|
||||
# Default is 25% of memory or 1g whichever is less.
|
||||
JAVA_ARGS='-Xmx2g'
|
||||
@ -1,46 +0,0 @@
|
||||
# Copyright 2013 Hewlett-Packard Development Company, L.P.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
# Elasticsearch server glue class.
|
||||
#
|
||||
class openstack_project::elasticsearch_node (
|
||||
$discover_nodes = ['localhost'],
|
||||
$heap_size = '30g',
|
||||
) {
|
||||
class { 'logstash::elasticsearch': }
|
||||
|
||||
class { '::elasticsearch':
|
||||
es_template_config => {
|
||||
'index.store.compress.stored' => true,
|
||||
'index.store.compress.tv' => true,
|
||||
'indices.memory.index_buffer_size' => '33%',
|
||||
'indices.breaker.fielddata.limit' => '70%',
|
||||
'bootstrap.mlockall' => true,
|
||||
'gateway.recover_after_nodes' => '5',
|
||||
'gateway.recover_after_time' => '5m',
|
||||
'gateway.expected_nodes' => '6',
|
||||
'discovery.zen.minimum_master_nodes' => '4',
|
||||
'discovery.zen.ping.multicast.enabled' => false,
|
||||
'discovery.zen.ping.unicast.hosts' => $discover_nodes,
|
||||
'http.cors.enabled' => true,
|
||||
'http.cors.allow-origin' => "'*'", # lint:ignore:double_quoted_strings
|
||||
},
|
||||
heap_size => $heap_size,
|
||||
version => '1.7.6',
|
||||
}
|
||||
|
||||
class { 'logstash::curator':
|
||||
keep_for_days => '7',
|
||||
}
|
||||
}
|
||||
@ -1,47 +0,0 @@
|
||||
# Copyright 2013 Hewlett-Packard Development Company, L.P.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
# Logstash web frontend glue class.
|
||||
#
|
||||
class openstack_project::logstash (
|
||||
$discover_nodes = ['elasticsearch02.openstack.org:9200'],
|
||||
$statsd_host = 'graphite.opendev.org',
|
||||
$subunit2sql_db_host,
|
||||
$subunit2sql_db_pass,
|
||||
) {
|
||||
class { 'logstash::web':
|
||||
frontend => 'kibana',
|
||||
discover_nodes => $discover_nodes,
|
||||
proxy_elasticsearch => true,
|
||||
}
|
||||
|
||||
class { 'log_processor': }
|
||||
|
||||
class { 'log_processor::geard':
|
||||
statsd_host => $statsd_host,
|
||||
}
|
||||
|
||||
include 'subunit2sql'
|
||||
|
||||
class { 'subunit2sql::server':
|
||||
db_host => $subunit2sql_db_host,
|
||||
db_pass => $subunit2sql_db_pass,
|
||||
}
|
||||
|
||||
include 'simpleproxy'
|
||||
|
||||
class { 'simpleproxy::server':
|
||||
db_host => $subunit2sql_db_host,
|
||||
}
|
||||
}
|
||||
@ -1,100 +0,0 @@
|
||||
# Copyright 2013 Hewlett-Packard Development Company, L.P.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
#
|
||||
# Logstash indexer worker glue class.
|
||||
#
|
||||
class openstack_project::logstash_worker (
|
||||
$discover_node = 'elasticsearch02.openstack.org',
|
||||
$filter_rev = 'master',
|
||||
$filter_source = 'https://opendev.org/openstack/logstash-filters',
|
||||
$enable_mqtt = true,
|
||||
$mqtt_hostname = 'firehose.openstack.org',
|
||||
$mqtt_port = 8883,
|
||||
$mqtt_topic = "logstash/${::hostname}",
|
||||
$mqtt_username = 'infra',
|
||||
$mqtt_password = undef,
|
||||
$mqtt_ca_cert_contents = undef,
|
||||
) {
|
||||
|
||||
file { '/etc/logprocessor/worker.yaml':
|
||||
ensure => present,
|
||||
owner => 'root',
|
||||
group => 'root',
|
||||
mode => '0644',
|
||||
content => template('openstack_project/logstash/jenkins-log-worker.yaml.erb'),
|
||||
require => Class['::log_processor'],
|
||||
}
|
||||
|
||||
file { '/etc/default/logstash-indexer':
|
||||
ensure => present,
|
||||
owner => 'root',
|
||||
group => 'root',
|
||||
mode => '0644',
|
||||
source => 'puppet:///modules/openstack_project/logstash/logstash-indexer.default',
|
||||
}
|
||||
|
||||
vcsrepo { '/opt/logstash-filters':
|
||||
ensure => latest,
|
||||
provider => git,
|
||||
revision => $filter_rev,
|
||||
source => $filter_source,
|
||||
}
|
||||
|
||||
include ::logstash
|
||||
|
||||
logstash::filter { 'openstack-logstash-filters':
|
||||
level => '50',
|
||||
target => '/opt/logstash-filters/filters/openstack-filters.conf',
|
||||
require => [
|
||||
Class['::logstash'],
|
||||
Vcsrepo['/opt/logstash-filters'],
|
||||
],
|
||||
notify => Service['logstash'],
|
||||
}
|
||||
|
||||
file { '/etc/logstash/mqtt-root-CA.pem.crt':
|
||||
ensure => present,
|
||||
content => $mqtt_ca_cert_contents,
|
||||
replace => true,
|
||||
owner => 'root',
|
||||
group => 'root',
|
||||
mode => '0555',
|
||||
require => Class['::logstash'],
|
||||
}
|
||||
|
||||
validate_array($elasticsearch_nodes) # needed by output.conf.erb
|
||||
class { '::logstash::indexer':
|
||||
input_template => 'openstack_project/logstash/input.conf.erb',
|
||||
output_template => 'openstack_project/logstash/output.conf.erb',
|
||||
require => Logstash::Filter['openstack-logstash-filters'],
|
||||
}
|
||||
|
||||
include ::log_processor
|
||||
log_processor::worker { 'A':
|
||||
config_file => '/etc/logprocessor/worker.yaml',
|
||||
require => File['/etc/logprocessor/worker.yaml'],
|
||||
}
|
||||
log_processor::worker { 'B':
|
||||
config_file => '/etc/logprocessor/worker.yaml',
|
||||
require => File['/etc/logprocessor/worker.yaml'],
|
||||
}
|
||||
log_processor::worker { 'C':
|
||||
config_file => '/etc/logprocessor/worker.yaml',
|
||||
require => File['/etc/logprocessor/worker.yaml'],
|
||||
}
|
||||
log_processor::worker { 'D':
|
||||
config_file => '/etc/logprocessor/worker.yaml',
|
||||
require => File['/etc/logprocessor/worker.yaml'],
|
||||
}
|
||||
}
|
||||
@ -1,8 +0,0 @@
|
||||
input {
|
||||
tcp {
|
||||
host => "localhost"
|
||||
port => 9999
|
||||
codec => json_lines {}
|
||||
type => "jenkins"
|
||||
}
|
||||
}
|
||||
@ -1,13 +0,0 @@
|
||||
gearman-host: logstash.openstack.org
|
||||
gearman-port: 4730
|
||||
output-host: localhost
|
||||
output-port: 9999
|
||||
output-mode: tcp
|
||||
crm114-script: /usr/local/bin/classify-log.crm
|
||||
crm114-data: /var/lib/crm114
|
||||
mqtt-host: <%= @mqtt_hostname %>
|
||||
mqtt-port: <%= @mqtt_port %>
|
||||
mqtt-topic: gearman-logstash/<%= @hostname %>
|
||||
mqtt-user: <%= @mqtt_username %>
|
||||
mqtt-pass: <%= @mqtt_password %>
|
||||
mqtt-ca-certs: /etc/logstash/mqtt-root-CA.pem.crt
|
||||
@ -1,9 +0,0 @@
|
||||
output {
|
||||
elasticsearch {
|
||||
hosts => <%= @elasticsearch_nodes.map { |node| node + ":9200" }.inspect %>
|
||||
manage_template => false
|
||||
flush_size => 1024
|
||||
timeout => 300
|
||||
}
|
||||
|
||||
}
|
||||
@ -1,3 +0,0 @@
|
||||
output {
|
||||
redis { host => "127.0.0.1" data_type => "list" key => "logstash" }
|
||||
}
|
||||
@ -22,11 +22,6 @@ results:
|
||||
- letsencrypt
|
||||
- mailman
|
||||
|
||||
logstash-worker02.openstack.org:
|
||||
- logstash-worker
|
||||
- puppet
|
||||
- puppet4
|
||||
|
||||
mirror02.regionone.linaro-us.opendev.org:
|
||||
- afs-client
|
||||
- kerberos-client
|
||||
|
||||
@ -11,14 +11,11 @@
|
||||
- opendev/ansible-role-puppet
|
||||
- opendev/puppet-bup
|
||||
- opendev/puppet-elastic_recheck
|
||||
- opendev/puppet-elasticsearch
|
||||
- opendev/puppet-ethercalc
|
||||
- opendev/puppet-httpd
|
||||
- opendev/puppet-jeepyb
|
||||
- opendev/puppet-kibana
|
||||
- opendev/puppet-log_processor
|
||||
- opendev/puppet-logrotate
|
||||
- opendev/puppet-logstash
|
||||
- opendev/puppet-mysql_backup
|
||||
- opendev/puppet-openstack_infra_spec_helper
|
||||
- opendev/puppet-pip
|
||||
@ -28,7 +25,6 @@
|
||||
- opendev/puppet-simpleproxy
|
||||
- opendev/puppet-ssh
|
||||
- opendev/puppet-storyboard
|
||||
- opendev/puppet-subunit2sql
|
||||
- opendev/puppet-tmpreaper
|
||||
- opendev/puppet-ulimit
|
||||
- opendev/puppet-user
|
||||
@ -87,19 +83,15 @@
|
||||
- opendev/puppet-project_config
|
||||
- opendev/puppet-ethercalc
|
||||
- opendev/puppet-httpd
|
||||
- opendev/puppet-subunit2sql
|
||||
- opendev/puppet-reviewday
|
||||
- opendev/puppet-kibana
|
||||
- opendev/puppet-redis
|
||||
- opendev/puppet-zanata
|
||||
- opendev/puppet-logstash
|
||||
- opendev/puppet-tmpreaper
|
||||
- opendev/puppet-elastic_recheck
|
||||
- opendev/puppet-ulimit
|
||||
- opendev/puppet-logrotate
|
||||
- opendev/puppet-elasticsearch
|
||||
- opendev/puppet-storyboard
|
||||
- opendev/puppet-log_processor
|
||||
- opendev/puppet-simpleproxy
|
||||
- opendev/puppet-bup
|
||||
- opendev/puppet-ssh
|
||||
|
||||
Loading…
Reference in New Issue
Block a user