fix error multiline folding for indexing

The previous folding rules for TRACE log level had the effect that all
TRACES were folded into the ERROR that was emitted before them. This
was fine and appropriate. When we dropped TRACE tag for these and used
ERROR, this was extended to ERROR.

This is incorrect.

While ERROR tags are used in stack traces in a multiline way, they
also just emit some times. We don't want to *always* fold them into
the previous line. Doing so means that logstash treats all our ERROR
log messages as what came before. Typically INFO.

One additional indication of a stacktrace is the log message always
has a process id in it. We can minimally modify the grok rule to
require %{NUMBER}, which should do the right thing here.

This should bring back ERROR log lines in logstash being listed under
loglevel:ERROR.

Change-Id: I20372686212d080d4bc5c5578c418546005260fd

modules/openstack_project/templates/logstash/indexer.conf.erb

@@ -39,7 +39,12 @@ filter {
       negate => false
       # NOTE(mriedem): oslo.log 1.2.0 changed the logging_exception_prefix
       # config option from using TRACE to ERROR so we have to handle both.
-      pattern => "^%{TIMESTAMP_ISO8601}%{SPACE}%{NUMBER}?%{SPACE}?(TRACE|ERROR)"
+      #
+      # NOTE(sdague): stack traces always include process id, so
+      # NUMBER being required element here is important, otherwise
+      # ERROR messages just fold into the previous messages, which are
+      # typically INFO.
+      pattern => "^%{TIMESTAMP_ISO8601}%{SPACE}%{NUMBER}%{SPACE}(TRACE|ERROR)"
       what => "previous"
       stream_identity => "%{host}.%{filename}"
     }