The change that modified the output configurations for fluentd
accidentally removed the type_name from the default elasticsearch
output, which prevents the output from using the fluent template
that's defined in the chart. This replaces the type_name for that
output
Change-Id: I2098ca8c243d55f0446ea623a80b5b40e3acff8c
This updates the fluentbit configuration for tail inputs to remove
the values for utilizing mysqlite databases to track its location
in each file it's configured to tail. This is intended to reduce
the pressure fluentbit exerts on the host through writing to
/var/log/foo.db. To help mitigate large amounts of traffic
sent from fluentbit to fluentd upon a pod restart, this also
adds a throttle filter to fluentbit.
As a result, Fluentbit no longer needs a writable mount to its
hostPath on /var/log on the host. Thus, this change includes
updating the Fluentbit daemonset's mount on /var/log to be
readOnly
Change-Id: If4381f4ff47e887f3ea10beded4f6172edaf08ba
This removes an unused section of configuration for fluentd, as
well as cleans up the values for filtering fluentd logs
Change-Id: I0c58d3ac236af7723c64c3b9fcba877736b1f606
This adds a liveness probe to the fluentd chart. This probe will
simply perform a tcpSocket check on the same port the readiness
probe executes the check on.
Change-Id: I768b23d36d50d6f6938f5588bea71e97aeb624b9
This updates the fluentd configuration to use 8 threads for the
Elasticsearch output configuration by default. This uses the
correct buffer output settings for the fluent-elasticsearch
plugin
This also updates the buffer output settings to the defaults used
for fluentd
Change-Id: I976cddaa973e850dabe4de495cd3bf1a4acdd4e7
This adds the security context snippet to the fluentd and
fluentd exporter templates. This changes the users for these two
pods from root to the nobody user instead
This also adds the container security context to explicitly set
allowPrivilegeEscalation to false
Change-Id: Ibf1da152f4aa78d425bbd00f514c2787d8ad9c5f
This adds an input to Fluentbit for capturing all qemu instance
logs in /var/log/libvirt/qemu/, and adds an Elasticsearch output
for those entries
Change-Id: I0802023f9861a5944e7989fd5469133c325349e7
This modifies the libvirt chart to write logs directly to the
host by default. This also modifies the fluentbit and fluentd
charts to capture libvirt logs from the host and index them into
Elasticsearch
Change-Id: I0bbc49d2c0d4cf4895f797e48f309f308ffd021f
This PS implements the helm toolkit function to generate the
Egress in kubernetes network policy manifest based on overrideable values.
It also enbale the K8s network policy at Osh-infra gate.
Change-Id: Icbe2a18c98dba795d15398dcdcac64228f6a7b4c
This adds the Decode_Field_As configuration key to the docker
parser for fluentbit. This is required to escape utf-8 encoded
characters appropriately in the log field
Change-Id: Ie2600cfe22045e3ab651fddf61ed2f676ab8a1d5
Adding auth tags for the logs to support special filter
for openstack and application security logs
Change-Id: Ifbd2395e4268d8d8fc4a2a3ac4d351db3d3e0845
This removes the tolerations key from the labels entries. As the
boolean check is on the pod.tolerations.enabled key instead, the
labels.foo.tolerations key is no longer used and should be removed
Change-Id: I00536dabadf9bd354219058d8efd054c60952bbd
This updates the fluentd buffer output configurations to account
for the restraints of the jobs deploying fluentd. This also
renames the fluentd configuration key from td_agent to fluentd to
reflect the fact we're no longer deploying td-agent
This also updates the Elasticsearch default replicas and overrides
the replica counts in each Elasticsearch deployment to account for
resource constraints
Change-Id: I55dee410eced99c3e1645f7452e4306ad646e601
This removes the fluentbit sidecars from the ceph-mon and ceph-osd
charts. Instead, we mount /var/log/ceph as a hostpath, and use the
fluentbit daemonset to target the mounted log files instead
This also updates the fluentd configuration to better handle the
correct configuration type for flush_interval (time vs int), as
well as updates the fluentd elasticsearch output values to help
address the gate failures resulting from the Elasticsearch bulk
endpoints failing
Change-Id: If3f2ff6371f267ed72379de25ff463079ba4cddc
This patch set implements the helm toolkit function to generate a
kubernetes network policy manifest based on overrideable values.
This also adds a chart that shuts down all the ingress and egress
traffics in the namespace. This can be used to ensure the
whitelisted network policy works as intended.
Additionally, implementation is done for some infrastructure charts.
Change-Id: I78e87ef3276e948ae4dd2eb462b4b8012251c8c8
Co-Authored-By: Mike Pham <tp6510@att.com>
Signed-off-by: Tin Lam <tin@irrational.io>
This changes the image used for various jobs and helm tests in the
osh-infra charts. This replaces the kolla heat image with the loci
based heat image used for jobs and helm tests in openstack-helm in
order to drive consistency
Change-Id: Ie9deedadb7507282fe62723ec4641dd508040364
This updates the helm tests for the fluent-logging chart to make
them more robust in being able to check for indexes defined in the
chart. This is done by calculating the combined flush interval
for both fluentbit and fluentd, and sleeping for at least one
flush cycle to ensure all functional indexes have received logged
events.
Then, the test determines what indexes should exist by checking
all Elasticsearch output configuration entries, determining
whether to use the default logstash-* index or the logstash_prefix
configuration value if it exists. For each of these indexes, the
test checks whether the indexes have successful hits (ie: there
have been successful entries into these indexes)
Change-Id: I36ed7b707491e92da6ac4b422936a1d65c92e0ac
This updates the logging interval values for the Elasticsearch
outputs to integers (20) vs the previous string value (20s)
Change-Id: I681bdaf807ba0136fef3b6dc1c7ddaa689ae77a3
This updates the helm test pod templates in the charts with helm
tests defined. This change includes the addition of:
- Generate test pod cluster roles and role bindings
- Generate service accounts for test pods
- Add node selectors to the test pods
- Add service accounts to the test pods
- Addition of entrypoint container to the test pods
- Indentation fix for rabbitmq test pod template
Change-Id: I9a0dd8a1a87bfe5eaf1362e92b37bc004f9c2cdb
This adds inputs for kernel logs on the host, as well as dockerd
and kubelet logs via the systemd plugin. This also adds a filter
for adding the hostname to the kernel log events, for renaming the
fields for systemd logs as kibana can not visualize fields that
begin with an underscore, and adds elasticsearch indexes for both
kernel and systemd logs
Change-Id: I026470dd45a971047f1e5bd1cd49bd0889589d12
This updates the configuration for fluentd, providing a mechanism
for basic determination of the log level of a logged event via
entries from /var/log/containers. This log level is prepended to
the tag for that event, and also added as a new `level` key in
the resulting event. These two improvements allow for querying
specific log level events via the tag.
This also adds similar functionality to any events captured via
the oslo log fluentd handler/formatter. This allows for
elasticsearch queries akin to `error.openstack.keystone`, which
can be used by nagios or another alerting mechanism to raise
alerts when a particular level event has been captured.
Change-Id: I016ddcfcf7408de7b6511ddf7009e1e6a5f3a1d9
This updates the kubernetes plugin test for fluent-logging to
search across all indices instead of the default logstash-* index
to account for custom indexes created for the events tagged with
the kubernetes plugin.
This also makes the search pattern for the tag more flexible to
account for any arbitrary number of prefixes and/or suffixes
added to the 'kube' tag as a result of any processing done in
fluentd.
Change-Id: Ib1a431cc8b2ca2cc143a8c8337b87f54f56d1029
This adds a helm-toolkit template for injecting pod tolerations
via values, similar to how container resources are handled. This
allows for custom definition of tolerations instead of defining
tolerations for pods directly into the pod templates
Change-Id: Ice520fcece425b14ae890ca5980fec9d7428a34d
This updates fluentbit to version v0.14.2, which includes
the Modify plugin (required for trimming underscores from
systemd log fields, necessary for kibana visualization). This also
updates the fluentbit configuration util to allow for renaming
multiple entries in an event. This is required because the values
definition for a configuration section is defined as a map, and
does not supported multiple Rename directives
Change-Id: I05172e8236282a438587887f4a806cf35c4b6c68
This PS adds the ability to attach a release uuid to pods and rc
objects as desired. A follow up ps will add the ability to add arbitary
annotations to the same objects.
Change-Id: Iceedba457a03387f6fc44eb763a00fd57f9d84a5
Signed-off-by: Pete Birley <pete@port.direct>
This updates the osh-infra charts to use a secret for their
configuration files instead of a configmap, allowing for the
storage of sensitive information
Change-Id: Ia32587162288df0b297c45fd43b55cef381cb064
This updates Fluentd to use the stable v1.2 debian fluentd
image instead of the kolla image. This images comes bundled
with the elasticsearch plugin, and provides more
flexibility in configuring the buffer behavior of the output
plugins
Change-Id: Id446ef1e050f5d9c005c94dae661cf9ae88fffea
This filters out fluentd's logs for collection, as this can result
in infinite loops as fluentd will try to process the events in its
own logs repeatedly
Change-Id: I85cce909b6917901b964cb5cc479403143c4d211
Using a node selector can not run the fluent-bit or node-exporter
on the master node. So, This PS changes the scheduling to use
either taint/toleration or the node selector.
Change-Id: I0ca80a6e645b7047469288697387f0f5bf111345
In most cases, the ingress controller's nodeSelector key and value
are "node-role.kubernetes.io/ingress" and "true".
Using quote to treat the nodeSelector value as a string.
Change-Id: Ie1745629b90795e4d888d85f35565e6d6350e09b
This fixes the resource trees for the fluent-logging and
openstack-exporter charts to match the other charts. This
also fixes the elasticsearch master template to use the
correct indentation level for the resource template
Change-Id: Ic6ec270a880216daff10d1f22128c6377ebf9933
As of 0.12.14, fluentbit exposes a flag for setting the db_sync
behavior for writing the location of the tail input to its sqlite
database. The default setting is Full, which introduces additional
synchronizations before and after a transaction. This has the
potential to negatively affect disk performance with the extra
synchronizations. This moves the setting in the chart to Normal,
which performs fewer synchronizations and still maintains a high
level of safety with status writes
Change-Id: I3b437edd6bd7501ef37ce06f0a561bd1747eb290
This updates the helm tests and the elasticsearch template job.
This changes the tests to conditionally check whether the
template job is enabled and the templates key is not empty, and
uses the result to determine whether to test for the existence
of those templates (to account for situations where the job is
disabled).
This updates the job to also check whether there are templates
defined in additional to checking whether the job itself is
enabled.
Change-Id: I14cedeb8d8a4444a73ea974426c3b0f136d1b698
This adds missing readiness probes to the following charts in
openstack-helm-infra: elasticsearch, fluent-logging, kibana,
nagios, prometheus-kube-state-metrics, prometheus-node-exporter,
and prometheus-openstack-exporter
Change-Id: I6a2635b08667c31eadb1b05ba848c658935a17e5