openstack/kolla-ansible
Code Issues Proposed changes

Increase the Fluentd request timeout for ES

Browse Source 
Fluentd has a default timeout of 5s for flushing data to ElasticSearch.
If there is a significant backlog of unsent log messages, this timeout
can be exceeded, resulting in Fluentd failing to make further progress.

Raise the default timeout to 60s.

This patch adopts the configuration parameters previously proposed by
Krzysztof Klimonda.

Closes-Bug: #1983031
Closes-Bug: #1896611
Change-Id: I1aaab654a5a0752fccef2cfb8cc0bde4a0ee2562
This commit is contained in:
Stig Telfer 2022-07-28 14:54:12 +01:00 committed by Michal Nasiadka
parent b0dc2423bc
commit ffb4767c05
4 changed files with 13 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
ansible/roles/common/defaults/main.yml
View File

@ -43,6 +43,7 @@ fluentd_elasticsearch_password: ""
fluentd_elasticsearch_ssl_version: "TLSv1_2"
fluentd_elasticsearch_ssl_verify: "true"
fluentd_elasticsearch_cacert: "{{ openstack_cacert }}"
fluentd_elasticsearch_request_timeout: "60s"
####################
# Docker

1
ansible/roles/common/templates/conf/output/00-local.conf.j2
View File

@ -37,6 +37,7 @@
logstash_format true
logstash_prefix {{ kibana_log_prefix }}
reconnect_on_error true
request_timeout {{ fluentd_elasticsearch_request_timeout }}
suppress_type_name true
<buffer>
@type file

1
ansible/roles/common/templates/conf/output/01-es.conf.j2
View File

@ -22,6 +22,7 @@
logstash_format true
logstash_prefix {{ kibana_log_prefix }}
reconnect_on_error true
request_timeout {{ fluentd_elasticsearch_request_timeout }}
suppress_type_name true
<buffer>
@type file

10
releasenotes/notes/fluentd-elasticsearch-request-timeout-f110883dd4412f65.yaml Normal file
View File

@ -0,0 +1,10 @@
---
fixes:
- |
Under circumstances of extended disruption to the Fluentd-ElasticSearch
central logging pipeline, it is possible to generate a sufficient buffer
of unsent log data that takes longer than the default Fluentd request
timeout (default 5 seconds) to transfer the buffer. The default request
timeout value is raised to ``60s``, and made configurable using new
parameter ``fluentd_elasticsearch_request_timeout``. `LP#1983031
<https://bugs.launchpad.net/kolla-ansible/+bug/1983031>`__