32 Commits

Author SHA1 Message Date
Spencer Krum
cfd2031844 Fixup logstash vhost template to use @global vars
Change-Id: Ie1564aa338bf5f9c2d155da68cdb6cb8a43a9b55
2014-03-13 21:02:58 -07:00
Clark Boylan
f32c5808d8 Actually restart services in logstash watchdog
Give full rooted paths to stop and start so that the executables are
found resulting in service restarts.

Change-Id: I4f2ded0a57c9f73b7c5afcbaab5827500cd4467b
2014-03-01 08:47:16 -08:00
Clark Boylan
e6efff22f6 Log logstash watchdog.
Log the activity of the logstash watchdog script so that we can get
numbers on how often this bug occurs and to help debug the script if it
doesn't do what we expect.

Change-Id: Ib12be4e5212c311744f51f0d191323651752731f
2014-02-28 17:01:15 -08:00
Clark Boylan
e6d78af997 Add a simple watchdog for logstash-indexer service
The logstash-indexer services sometimes fall out of the elasticsearch
cluster when it is under heavy load and unable to respond to pings.
Logstash doesn't do anything to reconnect :( so restart the service if a
node detects that it has fallen out of the cluster.

Upstream bug submitted at https://logstash.jira.com/browse/LOGSTASH-1951

Change-Id: I2e7767c5fe20cff279366fec2ddadd7710dbb4a9
2014-02-28 13:53:53 -08:00
Clark Boylan
1986685e29 Fix logstash indexer upstart conf.
A line which should've been a comment in the upstart config for
logstash-indexers lost its '#' make it not a comment and a syntax error.
Make this line a comment again.

Change-Id: Idc4853db68a3ae25bea6ebee80f047cc087457ac
2014-02-27 11:06:52 -08:00
Clark Boylan
4454aac98f Incrase heap size for logstash indexers.
The default java heap size is too small for our logstash indexers.
Double it to 2g. Do this by adding an /etc/default/logstash-indexer file
that the upstart configs source if it is present. This required some
tweaks to the upstart configuration to load the defaults properly.

Co-Authored-By: K Jonathan Harker <k.jonathan.harker@hp.com>
Change-Id: I63447f59f3fa6d466a7d275476121fe8339479dc
2014-02-27 10:30:28 -08:00
Clark Boylan
5a8351635d Upgrade to logstash 1.3.3 and ElasticSearch 0.90.9
Upgrade to logstash 1.3.3 and ElasticSearch 0.90.9. Logstash 1.3.3 comes
with many performance improvements and bug fixes and needs ElasticSearch
0.90.9 to use the ElasticSearch output so upgrade ES as well. These
upgrades come with config changes for logstash as well to use new
features and disable others that are not wanted.

Change-Id: Ia180c5edfa755214a801f2cad06f091125814b0e
2014-02-11 17:07:34 -08:00
Clark Boylan
1f07114346 Upgrade Logstash to 1.2.1.
* modules/logstash/manifests/init.pp: Download and install Logstash
1.2.1.

* modules/openstack_project/files/logstash/log-gearman-client.py:
Logstash 1.2.1 comes with a new schema. Update the job data sent to log
push workers to better accomodate the new schema.

* modules/openstack_project/files/logstash/log-gearman-worker.py: Push
Logstash 1.2.1 schema compliant JSON to the Logstash TCP input.

* modules/openstack_project/templates/logstash/indexer.conf.erb:
Logstash 1.2.1 comes with a new schema and many input and filter
changes. Use the newly supported features like conditionals to keep the
config up to date.

* modules/kibana/templates/config.rb.erb: Change the default field for
kibana to 'message'. It was @message which is deprecated in the new
logstash schema.

Change-Id: Id19fc05bcce8d42c5c0cf33df3da7e95f5794107
2013-10-21 10:17:48 -07:00
Clark Boylan
2487dbfafa Fix ElasticSearch reverse proxy settings.
* modules/logstash/templates/kibana.vhost.erb: Need to allow POST to
_aliases and _search and OPTIONS to _search. Also fix the ProxySet
settings directive by placing it in a <Proxy> section.

Change-Id: I0bb11b72eec9e8753ed8cf1d5324f411bb574f49
2013-10-16 15:35:38 -07:00
Clark Boylan
48a6fdcb15 Proxy ElasticSearch with mod_rewrite.
* modules/logstash/templates/kibana.vhost.erb: Use mod_rewrite to proxy
elasticsearch so that we can safely proxy _mapping. To do this safely we
need to restrict requests to GETs. Also, add _nodes to the list of
proxies URIs.

Change-Id: Ibe86dc104c429c144b31fed547034ee14f7f2e1d
2013-10-16 12:35:13 -07:00
Ryan Lane
3870a5a2fa Separate logstash/elasticsearch and verify hash
To allow for reuse of elasticsearch I'm splitting it away from
logstash. Also, I'm doing a hash check of the elasticsearch wget
for better security.

Change-Id: Iff42d538cd941abd50b000879ea4a237ea48d40e
2013-10-15 11:57:51 -07:00
Clark Boylan
ff9e6449c4 Increase connection timeout to ES servers.
* modules/logstash/templates/kibana.vhost.erb: Increase the connection
timeout to ElasticSearch when proxying to the limited ES API. This
prevents errors when ElasticSearch is slow due to cold caches.

Change-Id: I066a7c7822a0ae75bda52b9b37ad27bc07bd7e0a
2013-09-13 16:17:04 -07:00
K Jonathan Harker
5c141b464a Poke a hole through the proxy for cluster health
Interesting read-only data on the elasticsearch cluster can be found at
_cluster/health, this opens that url in the apache reverse proxy so that
we can see the health of the cluster.

Change-Id: I8d7a2eeec1f02aaa78af5f348b1ae91af76f522d
2013-09-06 21:38:48 +00:00
Clark Boylan
2820835473 Add sixth elasticsearch node.
* manifests/site.pp: Put elasticsearch nodes in a list variable that is
reused.

* modules/logstash/templates/elasticsearch.yml.erb: Update cluster
settings to be more appropriate for the larger cluster size, including
changing the minimum number of master eligible nodes that must be seen
before becoming operational in a cluster. Should help prevent split
brained clusters.

* modules/openstack_project/manifests/cacti.pp: Add new node to cacti
node list.

Change-Id: I9a18422aa56a56b9ad030892a312772d9b6b2a79
2013-08-02 13:22:47 -07:00
Clark Boylan
2e6d20c87c Tune ElasticSearch settings.
* modules/logstash/templates/elasticsearch.yml.erb:
Give 33% of memory to indexing instead of 40%.

* modules/logstash/files/es-logstash-template.json:
Flush the translog every 50k operations instead of the default 5k
(we can do 5k ops every second or two so the flushing was happening too
often). Limit the total number of shards per index per node to 3. We
have 5 nodes and 10 shards per index, 5*3 > 10 so we should be fine.
Set the field cache to soft type so that its entries can be garbage
collected in memory constricted situations.

* modules/logstash/manifests/elasticsearch.pp:
Apply es-logstash-template.json to the elasticsearch servers.

Change-Id: I2337fc41998fd00e090b0acfd29f007dfb6ec8df
2013-07-19 16:11:49 -07:00
Clark Boylan
76d3e632d4 Add two more elasticsearch nodes.
* manifests/site.pp: List the two new elasticsearch nodes in the
appropriate lists.

* modules/logstash/manifests/elasticsearch.pp: Do not restart
elasticsearch when config files change. Service restarts are costly and
should be manually performed when necessary. Otherwise puppet should
simply update the config files.

* modules/logstash/templates/elasticsearch.yml.erb: Update elasticsearch
config with new cluster topology. Increase memory available for
indexing.

* modules/openstack_project/manifests/cacti.pp: Add new nodes to cacti
monitoring list.

Adding two more elasticsearch nodes to relieve memory pressure (more
nodes means fewer indexes per nodes which requires less memory to
manage). And two more nodes gives us more disk to retain older indexes
in. These new nodes should allow us to retain at least 3 weeks of
indexed logs.

Change-Id: I3a5a02311e939c8147e401110c7b96d085eb8274
Reviewed-on: https://review.openstack.org/36305
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-07-09 18:40:21 +00:00
Clark Boylan
3d2d1772fe Better elasticsearch proxy settings.
* modules/logstash/templates/kibana.vhost.erb: Set proxy timeouts and
allow /_status and /_search.

Change-Id: I67f6152bcda5999a27fef07c8428ac12d3a0201c
Reviewed-on: https://review.openstack.org/34252
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-07-01 17:49:25 +00:00
Clark Boylan
443539b7fa Better elasticsearch cluster settings.
Use mutliple discover nodes to determine elasticsearch cluster
membership. Put a timeout on recovery starting instead of the default
to recovery immediately. Describe cluster topology in elasticsearch yaml
config so that it can make smarter decisions. Round robin kibana
requests across each discover node.

Change-Id: I08ef9dd158ddf6a6ce01dfb2050626f543d45b10
Reviewed-on: https://review.openstack.org/34106
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
2013-07-01 17:49:09 +00:00
Clark Boylan
33367c88de Add support for elasticsearch cluster.
We need to expand our elasticsearch install base. Update puppet to make
this possible.

Change-Id: Id0dae839b12ebf47715cf40a363832e0f661a94f
Reviewed-on: https://review.openstack.org/33910
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-07-01 17:48:16 +00:00
Clark Boylan
f51d974205 Reverse proxy subset of elasticsearch API.
It has been requested that we provide programmatic access to the
elasticsearch API in addition to the Kibana web UI. Do this by reverse
proxying http://logstash.openstack.org/elasticearch/$API_REQUEST to
http://elasticsearch.openstack.org:9200/$API_REQUEST.

The only values for $API_REQUEST that will be passed through are
_aliases, _status, and _search.

Change-Id: Ib41f6d91e2e59d493218074a67155af450ec8c93
Reviewed-on: https://review.openstack.org/33316
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-06-18 19:01:18 +00:00
Clark Boylan
c140bec6d2 Install java7 runtime with logstash root manifest.
Logstash depends on java but the runtime wasn't being installed by
the Logstash manifest. Fix this.

Change-Id: I63f2c9831a9c9eae63f4eebd2493800dd606f8fb
Reviewed-on: https://review.openstack.org/30919
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-29 18:20:14 +00:00
Clark Boylan
03bf2a553f Allocate more memory to ElasticSearch indexing.
* modules/logstash/files/elasticsearch.yml: By default ElasticSearch
allocates 10% of its memory space to indexing. Bump this up to 40% to
help ensure indexing does not block on needing more memory.

Change-Id: Ic2b862235c12467213ab9916e7f85d92559efc37
Reviewed-on: https://review.openstack.org/30796
Reviewed-by: Khai Do <zaro0508@gmail.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-29 00:33:33 +00:00
Clark Boylan
5dd697f3c4 Fix elasticsearch connectivity on logstash.o.o.
The new elasticsearch server needs to be able to connect back to
logstash over the elasticsearch ports. Also make kibana talk to the new
elasticsearch server.

Change-Id: I88af40c7885b263fcef5ff16210f073cd0a696a5
Reviewed-on: https://review.openstack.org/30354
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-23 23:46:30 +00:00
Clark Boylan
e43e208a99 Switch to dedicated elasticsearch node.
Switch to a large dedicated elasticsearch node as sharing resources
between logstash, kibana, jenkins-log-pusher, and elasticsearch results
in a constrained environment.

Change-Id: I39e6210f2c577429be2cb38aca09111a0f56f9be
Reviewed-on: https://review.openstack.org/30344
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-23 21:37:01 +00:00
Clark Boylan
14f51e4185 Upgrade logstash and elasticsearch. Retune ES.
Upgrade logstash to 1.1.12. Upgrade elasticsearch to 0.20.5 (the version
corresponding to logstash 1.1.12). Go back to default elasticsearch
shard and replica counts.

Change-Id: I8e5497442e918f78677335ec29d03542c93734e5
Reviewed-on: https://review.openstack.org/29447
Reviewed-by: Elizabeth Krumbach Joseph <lyz@princessleia.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-17 19:15:53 +00:00
Clark Boylan
868ad1ff0f Further elasticsearch config tweaks.
* modules/logstash/files/elasticsearch.yml: Set the number of
elasticsearch shards to 1 and the number of replicas to 0 as we are
not running a cluster, just a single node.

* modules/openstack_project/manifests/logstash.pp: Update elasticsearch
indice optimization cron job's max_num_segments value from 1 to 2.

These options are suggested at
https://github.com/logstash/logstash/wiki/Elasticsearch-Storage-Optimization

Change-Id: I747ab8952f0102ba582e213407909aaeebcf2778
Reviewed-on: https://review.openstack.org/28616
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-10 16:45:30 +00:00
Clark Boylan
2beb4b2ef2 Fix elasticsearch mapping path and content.
* modules/logstash/manifests/elasticsearch.pp: The file name for the
default elasticsearch mapping is default-mapping.json not
default_mapping.json. Fix this path.

* modules/logstash/files/elasticsearch.mapping.json: Add a missing
comma.

Change-Id: I36bbe6e125157b97f3333f383a8949b2a6b3ef5c
Reviewed-on: https://review.openstack.org/28484
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
2013-05-08 18:39:34 +00:00
K Jonathan Harker
407534435d Add a default_mapping to elasticsearch.
* Don't store or index `_all`.
* Compress `_source` (which is stored but not indexed by default).
* Create a dynamic template for the string type.
* Only analyze the index of `@message` strings.

This is a very rudimentary config, and should probably be expanded
on by someone who has a deeper understanding of elasticsearch.

Change-Id: I14052c48c7bb6f05e259615a5203dc1ad87a08af
Reviewed-on: https://review.openstack.org/28216
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jesse Keating <jesse.keating@rackspace.com>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
2013-05-04 12:36:12 +00:00
Clark Boylan
4cb22b5643 Give elasticsearch more heap memory.
* modules/logstash/files/elasticseach.default: Elasticsearch does not
handle running out of memory very gracefully. Recommended allocation is
half of the hosts memory. Give elasticsearch 4GB of the 8GB host.

* modules/logstash/manifests/elasticseach.pp: Copy new elasticsearch
defaults file into place on the host.

Change-Id: I9ed14148c901cd3fd4ec38333a722821adb77ff5
Reviewed-on: https://review.openstack.org/27870
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-05-01 00:28:02 +00:00
K Jonathan Harker
901e706aea Use kibana as the web frontend to logstash.
Add a kibana module that installs kibana and flat configuration files,
and also disables the previous boring bland web frontend.

Change-Id: I772c66f07d2bc3a88128e101074be9d4162e3f8a
Reviewed-on: https://review.openstack.org/27089
Reviewed-by: Jesse Keating <jesse.keating@rackspace.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-04-25 20:45:19 +00:00
Clark Boylan
0a70f48641 Compress elasticsearch stored fields.
Configure elasticsearch to compress stored fields and the term vector.

Change-Id: Ic7f40e18fc950056abd84e083b5d9abb69e38dbe
Reviewed-on: https://review.openstack.org/21138
Reviewed-by: Paul Belanger <paul.belanger@polybeacon.com>
Reviewed-by: Khai Do <zaro0508@gmail.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-02-14 21:43:15 +00:00
Clark Boylan
457a9d8764 Add skeleton logstash module.
This new logstash module adds classes to install logstash agents and
indexers as well as redis and elasticsearch. The configuration for each
of these services is rudimentary but it shouldn't be difficult to expand
the configs and make them useful.

Also, add a logstash.openstack.org node that will have an agent,
indexer, web frontend, redis, and elasticsearch installed on it.

Change-Id: I25b635f088f99d45cfaa70ed122c6433d3784937
Reviewed-on: https://review.openstack.org/19871
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
2013-01-29 20:44:09 +00:00