opendev/system-config
Code Issues Proposed changes

Merge "Update logstash doc for an elasticsearch cluster"

Browse Source
This commit is contained in:
Jenkins 2014-02-27 17:52:54 +00:00 committed by Gerrit Code Review
commit eb0fd1382f
1 changed files with 8 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
doc/source/logstash.rst
View File

@ -13,7 +13,7 @@ At a Glance
:Hosts: :Hosts:
* http://logstash.openstack.org * http://logstash.openstack.org
* logstash-worker\*.openstack.org * logstash-worker\*.openstack.org
* elasticsearch.openstack.org * elasticsearch\*.openstack.org
:Puppet: :Puppet:
* :file:`modules/logstash` * :file:`modules/logstash`
* :file:`modules/openstack_project/manifests/logstash.pp` * :file:`modules/openstack_project/manifests/logstash.pp`
@ -89,6 +89,7 @@ bottleneck very quickly. This looks something like:
\ | / \ | /
\ | / \ | /
elasticsearch elasticsearch
cluster
| |
| |
kibana kibana
@ -170,17 +171,12 @@ ElasticSearch
ElasticSearch is basically a REST API layer for Lucene. It provides ElasticSearch is basically a REST API layer for Lucene. It provides
the storage and search engine for Logstash. It scales horizontally and the storage and search engine for Logstash. It scales horizontally and
loves it when you give it more memory. Currently we run a single node loves it when you give it more memory. Currently we run a multi-node
cluster on a large VM to give ElasticSearch both memory and disk space. cluster on large VMs to give ElasticSearch both memory and disk space.
Per index (Logstash creates one index per day) we have one replica (on Per index (Logstash creates one index per day) we have N+1 replica
the same node, this does not provide HA, it speeds up searches) and redundancy to distribute disk utilization and provide high availability.
five shards (each shard is basically its own index, having multiple Each replica is broken into multiple shards providing inceased indexing
shards increases indexing throughput). and search throughput as each shard is essentially a valid mini index.
As this setup grows and handles more logs we may need to add more
ElasticSearch nodes and run a proper cluster. Haven't reached that point
yet, but will probably be necessary as disk and memory footprints
increase.
Kibana Kibana
------ ------