Commit Graph

9 Commits

Author SHA1 Message Date
Steve Wilkerson
4c29bafcbc Gates: Update fluent-logging/elasticsearch configurations
This updates the fluentd buffer output configurations to account
for the restraints of the jobs deploying fluentd. This also
renames the fluentd configuration key from td_agent to fluentd to
reflect the fact we're no longer deploying td-agent

This also updates the Elasticsearch default replicas and overrides
the replica counts in each Elasticsearch deployment to account for
resource constraints

Change-Id: I55dee410eced99c3e1645f7452e4306ad646e601
2018-10-19 17:30:08 +00:00
Steve Wilkerson
9a311475ba Charts: Use secrets for configs in chart
This updates the osh-infra charts to use a secret for their
configuration files instead of a configmap, allowing for the
storage of sensitive information

Change-Id: Ia32587162288df0b297c45fd43b55cef381cb064
2018-08-24 15:56:53 -05:00
Steve Wilkerson
dd986ed764 Fluentd: Change default image
This updates Fluentd to use the stable v1.2 debian fluentd
image instead of the kolla image. This images comes bundled
with the elasticsearch plugin, and provides more
flexibility in configuring the buffer behavior of the output
plugins

Change-Id: Id446ef1e050f5d9c005c94dae661cf9ae88fffea
2018-08-16 12:10:49 -05:00
Pete Birley
3470b17fc8 Fluent-Logging: update functions to live in correct locations
This PS simply moves functions within the chart to their correct location.

Change-Id: Ia5ac02a25a76ff759160cc352404b71b4208b216
Signed-off-by: Pete Birley <pete@port.direct>
2018-06-11 22:10:40 -05:00
Steve Wilkerson
4b8f46abee Fluent-logging: Support creation of arbitrary number of templates
This updates the fluent-logging chart to support the creation of
an arbitrary number of templates for elasticsearch. This allows
for the definition of multiple index mappings driven via the
chart's values. This provides flexibility in determining specific
structures for indexes that may differ between log types.

This also moves to define these mappings via json instead of XML.
As gotpl can convert yaml directly to json, and elasticsearch can
ingest json directly for index creation, we no longer need an XML
helper function to generate the required configuration. This helps
reduce the number of helper functions we need to maintain

Change-Id: I3c85fb9a1e700eb1592d96f83e632172d0eb2681
2018-06-04 12:46:34 -05:00
Chris Wedgwood
3a8c00764c yaml cleanup: trim multiline strings
Change-Id: I7e8f423be2efb84f3116258beca805265ca388f7
2018-03-08 20:18:53 +00:00
sungil
66919d28ef Add template for Fluent logging index
Fluent-logging stores logs on a elasticsearch by default. Elasticsearch
stores all fields as tokens by default, but some fields shoud be stored
for purposes such as retrieval, without splitting. Mapping in elasticsearch
is used to define a property of fields and Template can defines the mapping
for an index. fluent-logging use it to define the index structure. Specific
index type can be defined on the value file.

Change-Id: Id597111f478fcddf709b36d2db9ac5a5d6d8206f
2018-02-18 12:22:23 +09:00
Steve Wilkerson
09939a04de Move fluentbit and fluentd configs to values.yaml
Defines configuration files for fluentbit and fluentd via the
values.yaml file for fluent-logging. This provides flexibility in
defining parsers and routes for log gathering and routing

This functionality is added via helm-toolkit helper functions for
both fluentd and fluentbit to make the values configuration
cleaner

Change-Id: I8a43f36e487e651561bec8abf7752c8fac68aefc
2017-12-28 10:37:00 -06:00
sungil
2862f038e2 Fluent-logging helm chart
This introduces an initial helm chart for fluent logging.
It provides a functional fluent-bit and fluentd deployment to
use in conjunction with elasticsearch and kibana to consume
and aggregate logs from all resource types in a cluster.
It can deliver logs to kafka for external tools to consume.

This PS moves fluent-logging chart from osh-addons, osh to
osh-infra repo.
previous ps(addon): https://review.openstack.org/#/c/507023/
previous ps(osh): https://review.openstack.org/#/c/514622/

Specification: https://review.openstack.org/#/c/505491/
Partially implements: blueprint osh-logging-framework

Change-Id: I72e580aa3a197550060fc07af8396a7c8368d40b
2017-12-15 10:52:16 -06:00