Expose Journalbeat queue configuration

Currently the settings for the internal queue in journalbeat are left at their
default settings and unchangeable. This can result in a situation whereby logs
can remain in the queue for hours if there are not enough journal events
produced to trigger a flushing of the queue. One can observe this in kibana
through a large discrepancy between the output event.created and @timestamp
fields. This commit exposes the configuration options for the in-memory queue
which is used by default in journalbeat, such that the queue flushing can be
set with more appropriate timeouts etc. This commit leaves each value at its
current default.

Change-Id: I777c68a54226045966416d12d5bba56cb7eaeb68

elk_metrics_7x/roles/elastic_journalbeat/defaults/main.yml

@@ -32,3 +32,14 @@ processors: {}
 # Logging level
 # Available log levels are error, warning, info, debug
 journalbeat_log_level: "{{ elastic_beat_log_level | default('info') }}"
+
+## Journalbeat event queue configuration
+# Determines conditions under which the journalbeat event queue is flushed to logstash
+# Max # of events the queue can buffer
+journalbeat_queue_flush_max_events: 4096
+
+# Min # of events before queue is flushed
+journalbeat_queue_flush_min_events: 2048
+
+# Max time before queue is flushed (even if queue_min_events is not reached)
+journalbeat_queue_flush_timeout: 1s

elk_metrics_7x/roles/elastic_journalbeat/templates/journalbeat.yml.j2

@@ -77,25 +77,25 @@ tags:
 #fields_under_root: false
 
 # Internal queue configuration for buffering events to be published.
-#queue:
+queue:
   # Queue type by name (default 'mem')
   # The memory queue will present all available events (up to the outputs
   # bulk_max_size) to the output, the moment the output is ready to server
   # another batch of events.
-  #mem:
+  mem:
     # Max number of events the queue can buffer.
-    #events: 4096
+    events: {{ journalbeat_queue_flush_max_events }}
 
     # Hints the minimum number of events stored in the queue,
     # before providing a batch of events to the outputs.
     # The default value is set to 2048.
     # A value of 0 ensures events are immediately available
     # to be sent to the outputs.
-    #flush.min_events: 2048
+    flush.min_events: {{ journalbeat_queue_flush_min_events }}
 
     # Maximum duration after which events are available to the outputs,
     # if the number of events stored in the queue is < min_flush_events.
-    #flush.timeout: 1s
+    flush.timeout: {{ journalbeat_queue_flush_timeout }}
 
   # The spool queue will store events in a local spool file, before
   # forwarding the events to the outputs.