Commit Graph

90 Commits

Author SHA1 Message Date
Steve Wilkerson
015665516e Fluentd: Add type_name to default elasticsearch output
The change that modified the output configurations for fluentd
accidentally removed the type_name from the default elasticsearch
output, which prevents the output from using the fluent template
that's defined in the chart. This replaces the type_name for that
output

Change-Id: I2098ca8c243d55f0446ea623a80b5b40e3acff8c
2019-02-13 12:49:57 -06:00
Steve Wilkerson
3614d025dc Fluentbit: Remove database used in tail inputs
This updates the fluentbit configuration for tail inputs to remove
the values for utilizing mysqlite databases to track its location
in each file it's configured to tail.  This is intended to reduce
the pressure fluentbit exerts on the host through writing to
/var/log/foo.db. To help mitigate large amounts of traffic
sent from fluentbit to fluentd upon a pod restart, this also
adds a throttle filter to fluentbit.

As a result, Fluentbit no longer needs a writable mount to its
hostPath on /var/log on the host.  Thus, this change includes
updating the Fluentbit daemonset's mount on /var/log to be
readOnly

Change-Id: If4381f4ff47e887f3ea10beded4f6172edaf08ba
2019-02-01 16:56:31 +00:00
Steve Wilkerson
f01e9d2391 Fluentd: remove unused configuration section
This removes an unused section of configuration for fluentd, as
well as cleans up the values for filtering fluentd logs

Change-Id: I0c58d3ac236af7723c64c3b9fcba877736b1f606
2019-01-30 16:03:59 -06:00
Steve Wilkerson
39410b16bc Fluentd: Remove unused liveness port
This removes an unused port for a previous implementation of the
fluentd liveness probe

Change-Id: I80367bcf6fedc75b3ee7054eba9c382fbb4bc79d
2019-01-29 14:31:50 -06:00
Steve Wilkerson
9f5b1a77bc Add liveness probe to fluentd
This adds a liveness probe to the fluentd chart. This probe will
simply perform a tcpSocket check on the same port the readiness
probe executes the check on.

Change-Id: I768b23d36d50d6f6938f5588bea71e97aeb624b9
2019-01-23 11:47:34 -06:00
Steve Wilkerson
181d7ebb34 Fluentd: Update buffer output settings for Elasticsearch
This updates the fluentd configuration to use 8 threads for the
Elasticsearch output configuration by default. This uses the
correct buffer output settings for the fluent-elasticsearch
plugin

This also updates the buffer output settings to the defaults used
for fluentd

Change-Id: I976cddaa973e850dabe4de495cd3bf1a4acdd4e7
2019-01-10 14:51:41 -06:00
Zuul
9de0d96739 Merge "Fluentd: Add security context for pods/containers" 2019-01-07 22:15:19 +00:00
Chris Wedgwood
0c4e37391f 'NOP' cleanup for more consistent white-space use in charts
Where we have the style '{{ ...' we should use the style '... }}'.

Change-Id: Ic3e779e4681370d396f95d3804ca27db5b9d3642
2019-01-03 22:45:49 +00:00
Steve Wilkerson
e7232313ea Fluentd: Add security context for pods/containers
This adds the security context snippet to the fluentd and
fluentd exporter templates. This changes the users for these two
pods from root to the nobody user instead

This also adds the container security context to explicitly set
allowPrivilegeEscalation to false

Change-Id: Ibf1da152f4aa78d425bbd00f514c2787d8ad9c5f
2019-01-03 16:10:23 -06:00
Steve Wilkerson
5c4e77d816 Fluent-logging: Add input and output for qemu logs
This adds an input to Fluentbit for capturing all qemu instance
logs in /var/log/libvirt/qemu/, and adds an Elasticsearch output
for those entries

Change-Id: I0802023f9861a5944e7989fd5469133c325349e7
2019-01-02 15:29:03 -06:00
Steve Wilkerson
281b0799f0 Write libvirt logs to host
This modifies the libvirt chart to write logs directly to the
host by default. This also modifies the fluentbit and fluentd
charts to capture libvirt logs from the host and index them into
Elasticsearch

Change-Id: I0bbc49d2c0d4cf4895f797e48f309f308ffd021f
2018-12-28 17:43:12 +00:00
Zuul
6d354f0f7b Merge "Revert "Add Egress Helm-toolkit function & enforce the nework policy at OSH-INFRA"" 2018-12-16 08:57:09 +00:00
Pete Birley
0bf3674539 Revert "Add Egress Helm-toolkit function & enforce the nework policy at OSH-INFRA"
This reverts commit 8d33a2911c.

Change-Id: Ic861b9bf9b337449b47a3558da8355e7a5bcacee
2018-12-16 04:21:46 +00:00
Zuul
b90bf10b89 Merge "Add Egress Helm-toolkit function & enforce the nework policy at OSH-INFRA" 2018-12-15 09:32:21 +00:00
Mike Pham
8d33a2911c Add Egress Helm-toolkit function & enforce the nework policy at OSH-INFRA
This PS implements the helm toolkit function to generate the
Egress in kubernetes network policy manifest based on overrideable values.
It also enbale the K8s network policy at Osh-infra gate.

Change-Id: Icbe2a18c98dba795d15398dcdcac64228f6a7b4c
2018-12-14 16:32:40 -05:00
Steve Wilkerson
f4e10f8839 Fluentbit: Add Decode_Field_As config to docker parser
This adds the Decode_Field_As configuration key to the docker
parser for fluentbit. This is required to escape utf-8 encoded
characters appropriately in the log field

Change-Id: Ie2600cfe22045e3ab651fddf61ed2f676ab8a1d5
2018-12-12 22:24:09 +00:00
Mike Pham
d09254c6de Modify Fluentbit to add appropriate tags
Adding auth tags for the logs to support special filter
for openstack and application security logs

Change-Id: Ifbd2395e4268d8d8fc4a2a3ac4d351db3d3e0845
2018-12-05 15:16:40 +00:00
Steve Wilkerson
26c3773983 Fluentbit/Node Exporter: Remove unused tolerations key
This removes the tolerations key from the labels entries. As the
boolean check is on the pod.tolerations.enabled key instead, the
labels.foo.tolerations key is no longer used and should be removed

Change-Id: I00536dabadf9bd354219058d8efd054c60952bbd
2018-11-27 12:38:16 -06:00
Steve Wilkerson
4c29bafcbc Gates: Update fluent-logging/elasticsearch configurations
This updates the fluentd buffer output configurations to account
for the restraints of the jobs deploying fluentd. This also
renames the fluentd configuration key from td_agent to fluentd to
reflect the fact we're no longer deploying td-agent

This also updates the Elasticsearch default replicas and overrides
the replica counts in each Elasticsearch deployment to account for
resource constraints

Change-Id: I55dee410eced99c3e1645f7452e4306ad646e601
2018-10-19 17:30:08 +00:00
Steve Wilkerson
92717bdc72 Ceph: Remove fluentbit sidecars, mount hostpath for logs
This removes the fluentbit sidecars from the ceph-mon and ceph-osd
charts. Instead, we mount /var/log/ceph as a hostpath, and use the
fluentbit daemonset to target the mounted log files instead

This also updates the fluentd configuration to better handle the
correct configuration type for flush_interval (time vs int), as
well as updates the fluentd elasticsearch output values to help
address the gate failures resulting from the Elasticsearch bulk
endpoints failing

Change-Id: If3f2ff6371f267ed72379de25ff463079ba4cddc
2018-10-17 11:05:03 -05:00
Tin Lam
92e68d33ea Add network policy toolkit function
This patch set implements the helm toolkit function to generate a
kubernetes network policy manifest based on overrideable values.
This also adds a chart that shuts down all the ingress and egress
traffics in the namespace. This can be used to ensure the
whitelisted network policy works as intended.

Additionally, implementation is done for some infrastructure charts.

Change-Id: I78e87ef3276e948ae4dd2eb462b4b8012251c8c8
Co-Authored-By: Mike Pham <tp6510@att.com>
Signed-off-by: Tin Lam <tin@irrational.io>
2018-10-15 13:50:50 +00:00
Zuul
75ea67e591 Merge "Fluent-logging: Update helm tests for checking index entries" 2018-10-13 03:11:39 +00:00
Zuul
c39b29e351 Merge "Fluentd: Update logging interval values" 2018-10-13 03:02:04 +00:00
Steve Wilkerson
c7cbb9f4dd Charts: Update heat image used for jobs and helm tests
This changes the image used for various jobs and helm tests in the
osh-infra charts. This replaces the kolla heat image with the loci
based heat image used for jobs and helm tests in openstack-helm in
order to drive consistency

Change-Id: Ie9deedadb7507282fe62723ec4641dd508040364
2018-10-11 14:47:58 -05:00
Steve Wilkerson
78283495f0 Fluent-logging: Update helm tests for checking index entries
This updates the helm tests for the fluent-logging chart to make
them more robust in being able to check for indexes defined in the
chart.  This is done by calculating the combined flush interval
for both fluentbit and fluentd, and sleeping for at least one
flush cycle to ensure all functional indexes have received logged
events.

Then, the test determines what indexes should exist by checking
all Elasticsearch output configuration entries, determining
whether to use the default logstash-* index or the logstash_prefix
configuration value if it exists.  For each of these indexes, the
test checks whether the indexes have successful hits (ie: there
have been successful entries into these indexes)

Change-Id: I36ed7b707491e92da6ac4b422936a1d65c92e0ac
2018-10-11 13:28:30 -05:00
Steve Wilkerson
9b5d4d9f17 Fluentd: Update logging interval values
This updates the logging interval values for the Elasticsearch
outputs to integers (20) vs the previous string value (20s)

Change-Id: I681bdaf807ba0136fef3b6dc1c7ddaa689ae77a3
2018-10-11 09:05:00 -05:00
Steve Wilkerson
bfa237d347 Charts: Update helm test pod templates
This updates the helm test pod templates in the charts with helm
tests defined. This change includes the addition of:

- Generate test pod cluster roles and role bindings
- Generate service accounts for test pods
- Add node selectors to the test pods
- Add service accounts to the test pods
- Addition of entrypoint container to the test pods
- Indentation fix for rabbitmq test pod template

Change-Id: I9a0dd8a1a87bfe5eaf1362e92b37bc004f9c2cdb
2018-10-09 21:00:00 +00:00
Steve Wilkerson
fa09705867 Fluentbit: Add kernel, kubelet, and dockerd logs
This adds inputs for kernel logs on the host, as well as dockerd
and kubelet logs via the systemd plugin. This also adds a filter
for adding the hostname to the kernel log events, for renaming the
fields for systemd logs as kibana can not visualize fields that
begin with an underscore, and adds elasticsearch indexes for both
kernel and systemd logs

Change-Id: I026470dd45a971047f1e5bd1cd49bd0889589d12
2018-10-01 11:56:58 +00:00
Zuul
bc1afb87d7 Merge "Helm-Toolkit: Add snippet for kubernetes tolerations" 2018-09-23 01:13:57 +00:00
Steve Wilkerson
ba736d9840 Fluent-logging: Update fluentd configuration
This updates the configuration for fluentd, providing a mechanism
for basic determination of the log level of a logged event via
entries from /var/log/containers. This log level is prepended to
the tag for that event, and also added as a new `level` key in
the resulting event. These two improvements allow for querying
specific log level events via the tag.

This also adds similar functionality to any events captured via
the oslo log fluentd handler/formatter. This allows for
elasticsearch queries akin to `error.openstack.keystone`, which
can be used by nagios or another alerting mechanism to raise
alerts when a particular level event has been captured.

Change-Id: I016ddcfcf7408de7b6511ddf7009e1e6a5f3a1d9
2018-09-19 14:22:27 -05:00
Zuul
e649ad529f Merge "Fluent-logging: Update kubernetes plugin test" 2018-09-19 19:20:33 +00:00
Steve Wilkerson
3f952be4c1 Fluent-logging: Update kubernetes plugin test
This updates the kubernetes plugin test for fluent-logging to
search across all indices instead of the default logstash-* index
to account for custom indexes created for the events tagged with
the kubernetes plugin.

This also makes the search pattern for the tag more flexible to
account for any arbitrary number of prefixes and/or suffixes
added to the 'kube' tag as a result of any processing done in
fluentd.

Change-Id: Ib1a431cc8b2ca2cc143a8c8337b87f54f56d1029
2018-09-19 08:20:18 -05:00
Steve Wilkerson
70afe83c16 Helm-Toolkit: Add snippet for kubernetes tolerations
This adds a helm-toolkit template for injecting pod tolerations
via values, similar to how container resources are handled. This
allows for custom definition of tolerations instead of defining
tolerations for pods directly into the pod templates

Change-Id: Ice520fcece425b14ae890ca5980fec9d7428a34d
2018-09-18 13:10:54 +00:00
Steve Wilkerson
8e2d3a5b4c Fluentbit: Update version, config util template
This updates fluentbit to version v0.14.2, which includes
the Modify plugin (required for trimming underscores from
systemd log fields, necessary for kibana visualization). This also
updates the fluentbit configuration util to allow for renaming
multiple entries in an event. This is required because the values
definition for a configuration section is defined as a map, and
does not supported multiple Rename directives

Change-Id: I05172e8236282a438587887f4a806cf35c4b6c68
2018-09-17 07:45:45 -05:00
Pete Birley
bb3ff98d53 Add release uuid to pods and rc objects
This PS adds the ability to attach a release uuid to pods and rc
objects as desired. A follow up ps will add the ability to add arbitary
annotations to the same objects.

Change-Id: Iceedba457a03387f6fc44eb763a00fd57f9d84a5
Signed-off-by: Pete Birley <pete@port.direct>
2018-09-13 05:35:35 +00:00
Steve Wilkerson
9a311475ba Charts: Use secrets for configs in chart
This updates the osh-infra charts to use a secret for their
configuration files instead of a configmap, allowing for the
storage of sensitive information

Change-Id: Ia32587162288df0b297c45fd43b55cef381cb064
2018-08-24 15:56:53 -05:00
Zuul
2bbf188cbd Merge "Fluentd: Change default image" 2018-08-22 16:47:30 +00:00
Zuul
dbd3841c6e Merge "Fluentd: Filter out fluentd's logs" 2018-08-22 16:47:29 +00:00
Steve Wilkerson
dd986ed764 Fluentd: Change default image
This updates Fluentd to use the stable v1.2 debian fluentd
image instead of the kolla image. This images comes bundled
with the elasticsearch plugin, and provides more
flexibility in configuring the buffer behavior of the output
plugins

Change-Id: Id446ef1e050f5d9c005c94dae661cf9ae88fffea
2018-08-16 12:10:49 -05:00
Steve Wilkerson
a7af54e0c8 Fluentd: Filter out fluentd's logs
This filters out fluentd's logs for collection, as this can result
in infinite loops as fluentd will try to process the events in its
own logs repeatedly

Change-Id: I85cce909b6917901b964cb5cc479403143c4d211
2018-08-16 10:14:18 -05:00
Seungkyu Ahn
6b6f277e7d Running agents on all nodes.
Using a node selector can not run the fluent-bit or node-exporter
on the master node. So, This PS changes the scheduling to use
either taint/toleration or the node selector.

Change-Id: I0ca80a6e645b7047469288697387f0f5bf111345
2018-08-10 08:40:52 +00:00
Seungkyu Ahn
a430533e6a Quoting node_select_value in Ingress Controller
In most cases, the ingress controller's nodeSelector key and value
are "node-role.kubernetes.io/ingress" and "true".
Using quote to treat the nodeSelector value as a string.

Change-Id: Ie1745629b90795e4d888d85f35565e6d6350e09b
2018-08-01 02:39:05 +00:00
Steve Wilkerson
397eebf995 Resources: Fix erroneous resource definitions
This fixes the resource trees for the fluent-logging and
openstack-exporter charts to match the other charts. This
also fixes the elasticsearch master template to use the
correct indentation level for the resource template

Change-Id: Ic6ec270a880216daff10d1f22128c6377ebf9933
2018-07-27 16:35:37 -05:00
Zuul
5b152643bb Merge "Fluent-logging: Update default fluentbit configuration" 2018-07-25 14:56:19 +00:00
Steve Wilkerson
7ea9623662 Fluent-logging: Update default fluentbit configuration
As of 0.12.14, fluentbit exposes a flag for setting the db_sync
behavior for writing the location of the tail input to its sqlite
database. The default setting is Full, which introduces additional
synchronizations before and after a transaction. This has the
potential to negatively affect disk performance with the extra
synchronizations. This moves the setting in the chart to Normal,
which performs fewer synchronizations and still maintains a high
level of safety with status writes

Change-Id: I3b437edd6bd7501ef37ce06f0a561bd1747eb290
2018-07-13 09:36:11 -05:00
Steve Wilkerson
5271d246fe Fluent-logging: Update tests and template job
This updates the helm tests and the elasticsearch template job.
This changes the tests to conditionally check whether the
template job is enabled and the templates key is not empty, and
uses the result to determine whether to test for the existence
of those templates (to account for situations where the job is
disabled).

This updates the job to also check whether there are templates
defined in additional to checking whether the job itself is
enabled.

Change-Id: I14cedeb8d8a4444a73ea974426c3b0f136d1b698
2018-07-13 09:31:46 -05:00
Steve Wilkerson
dc16a897d7 Add missing labels to helm test pods
This adds missing labels to the helm test pods in osh-infra

Change-Id: I618d9089bfde2d847411f5f876f0ff6afd9cce7f
2018-07-10 08:55:40 -05:00
Steve Wilkerson
cb7bf2c0b3 Add missing readiness probes to openstack-helm-infra charts
This adds missing readiness probes to the following charts in
openstack-helm-infra: elasticsearch, fluent-logging, kibana,
nagios, prometheus-kube-state-metrics, prometheus-node-exporter,
and prometheus-openstack-exporter

Change-Id: I6a2635b08667c31eadb1b05ba848c658935a17e5
2018-06-26 12:25:36 +00:00
zhulingjie
de8cc7f637 Remove the duplicated word
Change-Id: I4aff89407a59762eb6abef9287932f71daf3e51f
2018-06-10 19:04:54 -04:00
Zuul
1051065c2c Merge "Daemonsets: Use current kubernetes daemonset api version" 2018-06-14 16:24:33 +00:00