Adds parsing for Cisco NXOS/IOS syslog format, along with the
RFC5424 variant some devices can use. Messages which match these
patterns are fingerprinted based upon the message and host to
de-duplicate them when storing in elasticsearch.
Change-Id: I42fd441913d2095997c3493c37515362a5d732fc
Currently, we use only a single primary shard for each beat index,
as configured by the index templates. This is not usually an optimal
setup, as all index searching is handled on a single node. A more
efficient configuration for a small number of different indices is
to balance, with around one primary shard per node. This
commit exposes the config option to set the number of primary shards,
which was previously hardcoded to 1. This now defaults to one shard
per data node in the cluster.
Change-Id: Icacf76c2270c98f8676d57952c00773646ca7b51
It is currently assumed that the elasticsearch data nodes and the
logstash nodes are co-located within the same container during
deployment of the ELK stack. This commit is intended to remove that
assumption by requiring specification of separate elastic and
logstash groups. This is particularly important where instances of
elasticsearch and logstash are separated for resource reasons, a
common setup for an elastic cluster.
The default setup for the elastic nodes has been simplified, with
all non-coordinating nodes being both data and ingest nodes unless
otherwise specified. This was previously defined algorithmically,
with the elastic node setup tightly coupled to the placement of the
logstash instances.
Change-Id: Id06eb78b52705aefea9cfe1247f53bac58badd52
The elastic beats are currently deployed with a default logging level of
info, which can produce large amounts of data, particularly in the
case that beats are monitoring one another's logs, and then logging this fact.
This commit exposes the configuration, allowing it to be set by the
variable log_level. The default level is unchanged.
Change-Id: Idbf715fc99450003920a45323123fcba919f2a83
Processors for each of the beats are currently defined per template and
can be configured directly via template or once deployed. This change
allows processors for all or a subset of beats to be configured through
role data or via an overlay inventory.
Change-Id: I71fc96611082555b43fd0f505219c42c890464ef
When using Metricbeat in an environment where only the system module
is enabled, the configuration reports system metrics to logstash every
60s. However, the current connection timout is shorter than this,
leading to the connection resetting and re-opening repeatedly due to
i/o timeouts. This commit increases the timeout to 90s, such that the
connection stays open and errors aren't reported to the metricbeat
logs.
Change-Id: Ib30f983df71c0b0381bfa23d9d2c7f1220184a4b
The Elasticsearch endpoints in use by logstash are currently hardcoded
to localhost in both the logstash monitoring config and the default
logstash pipeline template. This commit allows this to be configured,
such that (multiple) remote endpoints can be specified for the case that
logstash is not co-located with elasticsearch. The default behaviour is
unchanged.
Change-Id: Id871f201168ff882a5b37677747c3484f908298f
Templates for the beat config files have been updated to allow more
configuration options to be passed through to the associated
Elasticsearch index templates. In particular, one can optionally
specify values of index_template_max_docvalue_search to be set at
the creation of the beat index template. This can prevent shard failure
errors when viewing output in Kibana relating to "Trying to retrieve
too many docvalue_fields". Any similar config options can in future
be passed into the template via the elastic_beat_settings object.
Change-Id: Ic9136c8e063bbd231ed280bb446661b251879407
Updated ELK config files to elk 7.x reference samples, bringing over
existing customisation from elk_metrics_6x.
Removed deprecated use of --pipeline in elastic_beat_setup/tasks/main.yml,
--pipeline is no longer a valid cli argument.
Updated logstash-pipelines and removed the dynamic insertion of the date into
index names. This function is now done with the new ILM feature in elasticsearch
rather than logstash.
Installation of each beat creates an ILM policy for that beat and this patch
does not change the default policy. It is possible that the default policy
will exhaust the available storage and future work needs to be done to address
this.
The non-beat elements of the logstash pipeline (syslog, collectd and others)
are not yet updated to be compatible with ILM.
Change-Id: I735b64c2b7b93e23562f35266134a176a00af1b7