To allow for reuse of elasticsearch I'm splitting it away from
logstash. Also, I'm doing a hash check of the elasticsearch wget
for better security.
Change-Id: Iff42d538cd941abd50b000879ea4a237ea48d40e
* modules/logstash/templates/elasticsearch.yml.erb:
Give 33% of memory to indexing instead of 40%.
* modules/logstash/files/es-logstash-template.json:
Flush the translog every 50k operations instead of the default 5k
(we can do 5k ops every second or two so the flushing was happening too
often). Limit the total number of shards per index per node to 3. We
have 5 nodes and 10 shards per index, 5*3 > 10 so we should be fine.
Set the field cache to soft type so that its entries can be garbage
collected in memory constricted situations.
* modules/logstash/manifests/elasticsearch.pp:
Apply es-logstash-template.json to the elasticsearch servers.
Change-Id: I2337fc41998fd00e090b0acfd29f007dfb6ec8df
* manifests/site.pp: List the two new elasticsearch nodes in the
appropriate lists.
* modules/logstash/manifests/elasticsearch.pp: Do not restart
elasticsearch when config files change. Service restarts are costly and
should be manually performed when necessary. Otherwise puppet should
simply update the config files.
* modules/logstash/templates/elasticsearch.yml.erb: Update elasticsearch
config with new cluster topology. Increase memory available for
indexing.
* modules/openstack_project/manifests/cacti.pp: Add new nodes to cacti
monitoring list.
Adding two more elasticsearch nodes to relieve memory pressure (more
nodes means fewer indexes per nodes which requires less memory to
manage). And two more nodes gives us more disk to retain older indexes
in. These new nodes should allow us to retain at least 3 weeks of
indexed logs.
Change-Id: I3a5a02311e939c8147e401110c7b96d085eb8274
Reviewed-on: https://review.openstack.org/36305
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Use mutliple discover nodes to determine elasticsearch cluster
membership. Put a timeout on recovery starting instead of the default
to recovery immediately. Describe cluster topology in elasticsearch yaml
config so that it can make smarter decisions. Round robin kibana
requests across each discover node.
Change-Id: I08ef9dd158ddf6a6ce01dfb2050626f543d45b10
Reviewed-on: https://review.openstack.org/34106
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
We need to expand our elasticsearch install base. Update puppet to make
this possible.
Change-Id: Id0dae839b12ebf47715cf40a363832e0f661a94f
Reviewed-on: https://review.openstack.org/33910
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Upgrade logstash to 1.1.12. Upgrade elasticsearch to 0.20.5 (the version
corresponding to logstash 1.1.12). Go back to default elasticsearch
shard and replica counts.
Change-Id: I8e5497442e918f78677335ec29d03542c93734e5
Reviewed-on: https://review.openstack.org/29447
Reviewed-by: Elizabeth Krumbach Joseph <lyz@princessleia.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
* modules/logstash/manifests/elasticsearch.pp: The file name for the
default elasticsearch mapping is default-mapping.json not
default_mapping.json. Fix this path.
* modules/logstash/files/elasticsearch.mapping.json: Add a missing
comma.
Change-Id: I36bbe6e125157b97f3333f383a8949b2a6b3ef5c
Reviewed-on: https://review.openstack.org/28484
Reviewed-by: James E. Blair <corvus@inaugust.com>
Approved: Jeremy Stanley <fungi@yuggoth.org>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Tested-by: Jenkins
* Don't store or index `_all`.
* Compress `_source` (which is stored but not indexed by default).
* Create a dynamic template for the string type.
* Only analyze the index of `@message` strings.
This is a very rudimentary config, and should probably be expanded
on by someone who has a deeper understanding of elasticsearch.
Change-Id: I14052c48c7bb6f05e259615a5203dc1ad87a08af
Reviewed-on: https://review.openstack.org/28216
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Jesse Keating <jesse.keating@rackspace.com>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
* modules/logstash/files/elasticseach.default: Elasticsearch does not
handle running out of memory very gracefully. Recommended allocation is
half of the hosts memory. Give elasticsearch 4GB of the 8GB host.
* modules/logstash/manifests/elasticseach.pp: Copy new elasticsearch
defaults file into place on the host.
Change-Id: I9ed14148c901cd3fd4ec38333a722821adb77ff5
Reviewed-on: https://review.openstack.org/27870
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
This new logstash module adds classes to install logstash agents and
indexers as well as redis and elasticsearch. The configuration for each
of these services is rudimentary but it shouldn't be difficult to expand
the configs and make them useful.
Also, add a logstash.openstack.org node that will have an agent,
indexer, web frontend, redis, and elasticsearch installed on it.
Change-Id: I25b635f088f99d45cfaa70ed122c6433d3784937
Reviewed-on: https://review.openstack.org/19871
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins