[queens-only] Fluentd - Fix multiline format

Fluentd was removed from TripleO recently.

Currently, there is a way to add multiline format for reading logs
when it is configured by services. However It is not possible with
the common parameter LoggingExtraSources.

This patch allows this "feature" deleting the filter of the
configuration parameters.

The following parameter is added configure the fluentd service:
  LoggingExtraSources:
  - format: multiline
    format_firstline: /(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+) (?<pid>\d+) (?<priority>\S+) (?<component>\S+) (?<message> Traceback .*)$/
    format1: /(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+) (?<pid>\d+) (?<priority>\S+) (?<component>\S+) (?<message>  .*)$/
    format2: /(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+) (?<pid>\d+) (?<priority>\S+) (?<component>\S+) (?<traceback> \S.*)$/
    path: /var/log/nova/nova-conductor.log
    tag: openstack.nova.conductor.traceback

The outcome withouth the patch is:
This can be seen in [2]
<source>
  format multiline
  path /var/log/containers/nova/nova-conductor.log
  pos_file /var/cache/fluentd/openstack.nova.conductor.traceback.pos
  tag openstack.nova.conductor.traceback
  @type tail
</source>

The outcome with the patch is that can be seen at /var/lib/config-data/puppet-generated/fluentd/etc/fluentd/config.d/100-openstack-sources.conf:
<source>
  format multiline
  format1 /(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+) (?<pid>\d+) (?<priority>\S+) (?<component>\S+) (?<message>  .*)$/
  format2 /(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+) (?<pid>\d+) (?<priority>\S+) (?<component>\S+) (?<traceback> \S.*)$/
  format_firstline /(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+) (?<pid>\d+) (?<priority>\S+) (?<component>\S+) (?<message> Traceback .*)$/
  path /var/log/containers/nova/nova-conductor.log
  tag openstack.nova.conductor.traceback
</source>

Change-Id: I16ca6882e82fa2abf0db61b76adadc13941c623d
[1] https://review.opendev.org/#/c/734892/
[2] https://6d82362f2cdc504b27f1-9f757b11a1d2b00e739d31e1ecad199a.ssl.cf5.rackcdn.com/734892/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/b321009/logs/subnode-1/var/lib/config-data/puppet-generated/fluentd/etc/fluentd/config.d/100-openstack-sources.conf
This commit is contained in:
Juan Badia Payno 2020-05-23 12:14:18 +02:00
parent cbfbeb3430
commit c66921b4fb
2 changed files with 8 additions and 7 deletions

View File

@ -167,6 +167,13 @@ parameter_defaults:
LoggingServers:
- host: 127.0.0.1
port: 24224
LoggingExtraSources:
- format: multiline
format_firstline: /(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+) (?<pid>\d+) (?<priority>\S+) (?<component>\S+) (?<message> Traceback .*)$/
format1: /(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+) (?<pid>\d+) (?<priority>\S+) (?<component>\S+) (?<message> .*)$/
format2: /(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+) (?<pid>\d+) (?<priority>\S+) (?<component>\S+) (?<traceback> \S.*)$/
path: /var/log/nova/nova-conductor.log
tag: openstack.nova.conductor.traceback
MonitoringRabbitHost: 127.0.0.1
MonitoringRabbitPort: 5676
MonitoringRabbitPassword: sensu

View File

@ -123,13 +123,7 @@ resources:
pos_file_path => coalesce($.data.pos_file_path, ''),
sources => coalesce($.data.sources, {}).flatten()
) ->
$sources.where($ != null).select({
'type' => 'tail',
'tag' => $.tag,
'path' => $.path,
'format' => $.get('format', $default_format),
'pos_file' => $.get('pos_file', $pos_file_path + '/' + $.tag + '.pos')
})
$sources.where($ != null)
data:
sources:
- {get_attr: [LoggingConfiguration, LoggingDefaultSources]}