50 Commits

Author SHA1 Message Date
Steve Wilkerson
4c29bafcbc Gates: Update fluent-logging/elasticsearch configurations
This updates the fluentd buffer output configurations to account
for the restraints of the jobs deploying fluentd. This also
renames the fluentd configuration key from td_agent to fluentd to
reflect the fact we're no longer deploying td-agent

This also updates the Elasticsearch default replicas and overrides
the replica counts in each Elasticsearch deployment to account for
resource constraints

Change-Id: I55dee410eced99c3e1645f7452e4306ad646e601
2018-10-19 17:30:08 +00:00
Steve Wilkerson
92717bdc72 Ceph: Remove fluentbit sidecars, mount hostpath for logs
This removes the fluentbit sidecars from the ceph-mon and ceph-osd
charts. Instead, we mount /var/log/ceph as a hostpath, and use the
fluentbit daemonset to target the mounted log files instead

This also updates the fluentd configuration to better handle the
correct configuration type for flush_interval (time vs int), as
well as updates the fluentd elasticsearch output values to help
address the gate failures resulting from the Elasticsearch bulk
endpoints failing

Change-Id: If3f2ff6371f267ed72379de25ff463079ba4cddc
2018-10-17 11:05:03 -05:00
Tin Lam
92e68d33ea Add network policy toolkit function
This patch set implements the helm toolkit function to generate a
kubernetes network policy manifest based on overrideable values.
This also adds a chart that shuts down all the ingress and egress
traffics in the namespace. This can be used to ensure the
whitelisted network policy works as intended.

Additionally, implementation is done for some infrastructure charts.

Change-Id: I78e87ef3276e948ae4dd2eb462b4b8012251c8c8
Co-Authored-By: Mike Pham <tp6510@att.com>
Signed-off-by: Tin Lam <tin@irrational.io>
2018-10-15 13:50:50 +00:00
Steve Wilkerson
78283495f0 Fluent-logging: Update helm tests for checking index entries
This updates the helm tests for the fluent-logging chart to make
them more robust in being able to check for indexes defined in the
chart.  This is done by calculating the combined flush interval
for both fluentbit and fluentd, and sleeping for at least one
flush cycle to ensure all functional indexes have received logged
events.

Then, the test determines what indexes should exist by checking
all Elasticsearch output configuration entries, determining
whether to use the default logstash-* index or the logstash_prefix
configuration value if it exists.  For each of these indexes, the
test checks whether the indexes have successful hits (ie: there
have been successful entries into these indexes)

Change-Id: I36ed7b707491e92da6ac4b422936a1d65c92e0ac
2018-10-11 13:28:30 -05:00
Steve Wilkerson
bfa237d347 Charts: Update helm test pod templates
This updates the helm test pod templates in the charts with helm
tests defined. This change includes the addition of:

- Generate test pod cluster roles and role bindings
- Generate service accounts for test pods
- Add node selectors to the test pods
- Add service accounts to the test pods
- Addition of entrypoint container to the test pods
- Indentation fix for rabbitmq test pod template

Change-Id: I9a0dd8a1a87bfe5eaf1362e92b37bc004f9c2cdb
2018-10-09 21:00:00 +00:00
Steve Wilkerson
fa09705867 Fluentbit: Add kernel, kubelet, and dockerd logs
This adds inputs for kernel logs on the host, as well as dockerd
and kubelet logs via the systemd plugin. This also adds a filter
for adding the hostname to the kernel log events, for renaming the
fields for systemd logs as kibana can not visualize fields that
begin with an underscore, and adds elasticsearch indexes for both
kernel and systemd logs

Change-Id: I026470dd45a971047f1e5bd1cd49bd0889589d12
2018-10-01 11:56:58 +00:00
Zuul
bc1afb87d7 Merge "Helm-Toolkit: Add snippet for kubernetes tolerations" 2018-09-23 01:13:57 +00:00
Zuul
e649ad529f Merge "Fluent-logging: Update kubernetes plugin test" 2018-09-19 19:20:33 +00:00
Steve Wilkerson
3f952be4c1 Fluent-logging: Update kubernetes plugin test
This updates the kubernetes plugin test for fluent-logging to
search across all indices instead of the default logstash-* index
to account for custom indexes created for the events tagged with
the kubernetes plugin.

This also makes the search pattern for the tag more flexible to
account for any arbitrary number of prefixes and/or suffixes
added to the 'kube' tag as a result of any processing done in
fluentd.

Change-Id: Ib1a431cc8b2ca2cc143a8c8337b87f54f56d1029
2018-09-19 08:20:18 -05:00
Steve Wilkerson
70afe83c16 Helm-Toolkit: Add snippet for kubernetes tolerations
This adds a helm-toolkit template for injecting pod tolerations
via values, similar to how container resources are handled. This
allows for custom definition of tolerations instead of defining
tolerations for pods directly into the pod templates

Change-Id: Ice520fcece425b14ae890ca5980fec9d7428a34d
2018-09-18 13:10:54 +00:00
Steve Wilkerson
8e2d3a5b4c Fluentbit: Update version, config util template
This updates fluentbit to version v0.14.2, which includes
the Modify plugin (required for trimming underscores from
systemd log fields, necessary for kibana visualization). This also
updates the fluentbit configuration util to allow for renaming
multiple entries in an event. This is required because the values
definition for a configuration section is defined as a map, and
does not supported multiple Rename directives

Change-Id: I05172e8236282a438587887f4a806cf35c4b6c68
2018-09-17 07:45:45 -05:00
Pete Birley
bb3ff98d53 Add release uuid to pods and rc objects
This PS adds the ability to attach a release uuid to pods and rc
objects as desired. A follow up ps will add the ability to add arbitary
annotations to the same objects.

Change-Id: Iceedba457a03387f6fc44eb763a00fd57f9d84a5
Signed-off-by: Pete Birley <pete@port.direct>
2018-09-13 05:35:35 +00:00
Steve Wilkerson
9a311475ba Charts: Use secrets for configs in chart
This updates the osh-infra charts to use a secret for their
configuration files instead of a configmap, allowing for the
storage of sensitive information

Change-Id: Ia32587162288df0b297c45fd43b55cef381cb064
2018-08-24 15:56:53 -05:00
Zuul
2bbf188cbd Merge "Fluentd: Change default image" 2018-08-22 16:47:30 +00:00
Steve Wilkerson
dd986ed764 Fluentd: Change default image
This updates Fluentd to use the stable v1.2 debian fluentd
image instead of the kolla image. This images comes bundled
with the elasticsearch plugin, and provides more
flexibility in configuring the buffer behavior of the output
plugins

Change-Id: Id446ef1e050f5d9c005c94dae661cf9ae88fffea
2018-08-16 12:10:49 -05:00
Seungkyu Ahn
6b6f277e7d Running agents on all nodes.
Using a node selector can not run the fluent-bit or node-exporter
on the master node. So, This PS changes the scheduling to use
either taint/toleration or the node selector.

Change-Id: I0ca80a6e645b7047469288697387f0f5bf111345
2018-08-10 08:40:52 +00:00
Seungkyu Ahn
a430533e6a Quoting node_select_value in Ingress Controller
In most cases, the ingress controller's nodeSelector key and value
are "node-role.kubernetes.io/ingress" and "true".
Using quote to treat the nodeSelector value as a string.

Change-Id: Ie1745629b90795e4d888d85f35565e6d6350e09b
2018-08-01 02:39:05 +00:00
Steve Wilkerson
5271d246fe Fluent-logging: Update tests and template job
This updates the helm tests and the elasticsearch template job.
This changes the tests to conditionally check whether the
template job is enabled and the templates key is not empty, and
uses the result to determine whether to test for the existence
of those templates (to account for situations where the job is
disabled).

This updates the job to also check whether there are templates
defined in additional to checking whether the job itself is
enabled.

Change-Id: I14cedeb8d8a4444a73ea974426c3b0f136d1b698
2018-07-13 09:31:46 -05:00
Steve Wilkerson
dc16a897d7 Add missing labels to helm test pods
This adds missing labels to the helm test pods in osh-infra

Change-Id: I618d9089bfde2d847411f5f876f0ff6afd9cce7f
2018-07-10 08:55:40 -05:00
Steve Wilkerson
cb7bf2c0b3 Add missing readiness probes to openstack-helm-infra charts
This adds missing readiness probes to the following charts in
openstack-helm-infra: elasticsearch, fluent-logging, kibana,
nagios, prometheus-kube-state-metrics, prometheus-node-exporter,
and prometheus-openstack-exporter

Change-Id: I6a2635b08667c31eadb1b05ba848c658935a17e5
2018-06-26 12:25:36 +00:00
Zuul
1051065c2c Merge "Daemonsets: Use current kubernetes daemonset api version" 2018-06-14 16:24:33 +00:00
Pete Birley
fa629cdbbd Daemonsets: Use current kubernetes daemonset api version
This PS moves to use the current ga version for kubernetes daemonsets,
additionally any remaining deployments that were using the
`extensions/v1beta1` have been updated to `apps/v1`.

Story: 2002205
Task: 21735

Change-Id: If9703162dc472af1e6096bf2b9062802fd5ce8ab
Signed-off-by: Pete Birley <pete@port.direct>
2018-06-13 21:53:18 +00:00
Pete Birley
3470b17fc8 Fluent-Logging: update functions to live in correct locations
This PS simply moves functions within the chart to their correct location.

Change-Id: Ia5ac02a25a76ff759160cc352404b71b4208b216
Signed-off-by: Pete Birley <pete@port.direct>
2018-06-11 22:10:40 -05:00
Steve Wilkerson
4b8f46abee Fluent-logging: Support creation of arbitrary number of templates
This updates the fluent-logging chart to support the creation of
an arbitrary number of templates for elasticsearch. This allows
for the definition of multiple index mappings driven via the
chart's values. This provides flexibility in determining specific
structures for indexes that may differ between log types.

This also moves to define these mappings via json instead of XML.
As gotpl can convert yaml directly to json, and elasticsearch can
ingest json directly for index creation, we no longer need an XML
helper function to generate the required configuration. This helps
reduce the number of helper functions we need to maintain

Change-Id: I3c85fb9a1e700eb1592d96f83e632172d0eb2681
2018-06-04 12:46:34 -05:00
Steve Wilkerson
de9c46bcfa Charts: Tidy up openstack-helm-infra charts
This moves the charts in openstack-helm-infra closer towards a
standard structure. It addresses multiple deviations, including:
missing resources for init containers, incorrect indents for
disabled resources in some charts, incorrect indents for volumes
and volumemounts added via values, missing resources for some
helm test templates, missing helm-toolkit image functions, and
moving the resource template declarations to be under the image
template declarations

Change-Id: I4834a5d476ef7fc69c5583caacc0229050f20a76
2018-05-21 12:58:22 -07:00
Steve Wilkerson
e166432a98 Add manifest for image_repo_sync job
This ps proposes adding a common template for the image_repo_sync
jobs for consumption by the charts

Change-Id: I48476d1e4fd94bd1b08b13b46983e3d999f8d8ca
2018-04-19 14:10:08 +00:00
Steve Wilkerson
aaffc4caf0 OSH-Infra: Update labels for chart components
This ps adds more granular node selectors for the charts in osh
infra to match what is currently done in osh

Change-Id: I8957a95053b9fb3ea329fd37ff049cd223a7695d
2018-04-13 08:44:33 -05:00
Pete Birley
b9336ca613 Helm-Toolkit: Kubernetes Entrypoint, simplify image dependencies
This PS simplify the logic for dyanmicly merging the image management
depenencies into pod deps when active.

Change-Id: I0cf6c93173bc5fbce697ac15be8697d3b1326d0a
2018-04-13 08:42:37 -05:00
Steve Wilkerson
f28ef6ded2 Fluentd: Change elasticsearch endpoint port and creds reference
Updates the fluent-logging chart to reference the elasticsearch
endpoint via lookups on the `http` port to match the elasticsearch
chart's handling of the client port.  This also updates the helm
test pod to reference the elasticsearch credentials via the
secret used elsewhere in the fluent-logging chart

Change-Id: I352d912db5e231e14dc58cdf897ae642f3256373
2018-03-12 20:38:45 -05:00
Zuul
33cb0e8433 Merge "Revert Elasticsearch/Kibana image change" 2018-03-12 19:13:34 +00:00
Steve Wilkerson
8e4da9da55 Revert Elasticsearch/Kibana image change
This reverts the changes made to Elasticsearch, Kibana and fluent
logging charts in https://review.openstack.org/#/c/550229/7.

Specifically, this moves the images back to previous used versions
and makes the required changes to the fluent-logging elasticsearch
template job to include the correct mapping directives for the
elasticsearch template.

This change was made to give more time for evaluating a more
robust solution for switching to the official upstream images that
will not cause intermittent gate failures as seen since 550229 was
merged

Change-Id: I9f70b3412a8edc5cb1d80937b158aa2fe7b1ec82
2018-03-12 10:27:35 -05:00
Zuul
eb3cbf0f95 Merge "yaml cleanup: trim multiline strings" 2018-03-10 07:01:35 +00:00
Chris Wedgwood
3a8c00764c yaml cleanup: trim multiline strings
Change-Id: I7e8f423be2efb84f3116258beca805265ca388f7
2018-03-08 20:18:53 +00:00
Steve Wilkerson
417ce3f37b Fluent-logging: use endpoints section and lookups to set port
This PS moves fluent-logging to use the endpoints section and
lookups to set the port it serves on.

Change-Id: I7cbbd8d6287942eb36f70ae74872405038e523e8
2018-03-08 20:00:47 +00:00
Steve Wilkerson
d681396412 Address errors with Elasticsearch and Kibana
This moves Elasticsearch and Kibana to use the latest version
(6.2.2), as the images we were using are no longer supported with
the 6.x release.  There was a change in the doc reference in the
log entries that prevented the previous ES version from indexing
those entries, resulting in a busted gate.  Moving Kibana to 6.2.2
was required to match major/minor versions with Elasticsearch

The Elasticsearch version change also required changing config file
locations, changing the entrypoint used for launching the service,
changing the running user for the elasticsearch service, and
updated the ES tests as some of the API responses changed between
versions

This also required updating the elasticsearch template job as the
mapping definition entries changed between versions

Change-Id: Ia4cd9a66851754a1bb8f225c7e24513c43568e93
2018-03-08 10:27:06 -06:00
Pete Birley
3c101a6324 dependencies: move dynamic common deps under a 'dynamic.common' key
This PS moves existing dynamic common dependencies under a
'dynamic.common' key to simplify the yaml tree.

Change-Id: I4332bcfdf11197488e7bd5d8cf4c25565ea1c7b6
2018-02-24 17:42:10 -05:00
Pete Birley
e0c688d7ee dependencies: move static dependencies under a 'static' key
This PS moves static dependencies unser a 'static' key to allow
expansion to cover dynamic dependencies.

Change-Id: Ia0e853564955e0fbbe5a9e91a8b8924c703b1b02
2018-02-24 17:39:55 -05:00
Zuul
40b1c07f53 Merge "Add template for Fluent logging index" 2018-02-21 15:42:52 +00:00
portdirect
515494ca98 RBAC: Include release name in cluster roles to prevent collision
This PS includes the release name in the cluster role to prevent
colision if the chart is deployed multiple times in the same
cluster.

Change-Id: I7166e5ee25b3d4c89879393c5f84c869585a2681
2018-02-19 13:13:56 -06:00
sungil
66919d28ef Add template for Fluent logging index
Fluent-logging stores logs on a elasticsearch by default. Elasticsearch
stores all fields as tokens by default, but some fields shoud be stored
for purposes such as retrieval, without splitting. Mapping in elasticsearch
is used to define a property of fields and Template can defines the mapping
for an index. fluent-logging use it to define the index structure. Specific
index type can be defined on the value file.

Change-Id: Id597111f478fcddf709b36d2db9ac5a5d6d8206f
2018-02-18 12:22:23 +09:00
Zuul
1bfa439a6b Merge "Add fluentd prometheus exporter for logging metrics" 2018-02-14 15:49:01 +00:00
Steve Wilkerson
5d95b0e2cb Add fluentd prometheus exporter for logging metrics
This adds templates for a prometheus exporter for fluentd to
adequately capture log metrics and fluentd service metrics for
consumption by prometheus

Change-Id: I6d6a8c2be07af819dc6d99b8ce5f1d4b635a69f0
2018-02-13 19:09:35 +00:00
Sean Eagan
641c79c902 Add deep merge utility to helm-toolkit
Adds "helm-toolkit.utils.merge" which is a replacement for the
upstream sprig "merge" function which didn't quite do what we
wanted, specifically it didn't merge slices, it just overrode
one with the other.  This PS also updates existing callsites
of the sprig merge with "helm-toolkit.utils.merge".

Change-Id: I456349558d4cf941d1bcb07fc76d0688b0a10782
2018-02-13 10:08:50 -06:00
Steve Wilkerson
d197c4f9a2 Run elasticsearch behind apache
Run elasticsearch behind apache as a reverse proxy to supply basic
auth for elasticsearch, as xpack requires a suscription to support
security for elasticsearch

Change-Id: I72d06ed9cd2179ead86ddc67db33c68a1e40c437
2018-01-16 08:14:47 -06:00
sungil
85011f9c48 Add Permission for k8s plugin in fluent-logging
This PS adds permissions for k8s plugin in fluent-logging.
The k8s plugin in fluentbit gets information per pod and adds it to the
message(log) before output. But the plugin cannot get the pod in current chart.
This PS fix this issue.

Change-Id: Icdce8a0a5ed0975c4d6e72ba50df8ef9a3b76ca6
2018-01-10 11:02:02 +09:00
portdirect
abd7e78c65 Fluentd: tidy rbac roles and bindings to live with appropriate rc
This PS brings Fluentd (&bit) inline with other charts by placing the
RBAC roles and bindings in the same template as the pod rc they are
assocated with.

Change-Id: I622a2adfc0dc9f5044202cd6318e3ed803088c5f
2018-01-08 02:17:38 +00:00
Steve Wilkerson
09939a04de Move fluentbit and fluentd configs to values.yaml
Defines configuration files for fluentbit and fluentd via the
values.yaml file for fluent-logging. This provides flexibility in
defining parsers and routes for log gathering and routing

This functionality is added via helm-toolkit helper functions for
both fluentd and fluentbit to make the values configuration
cleaner

Change-Id: I8a43f36e487e651561bec8abf7752c8fac68aefc
2017-12-28 10:37:00 -06:00
Tin Lam
628fd3007d RBAC: Consolidate serviceaccounts and restrict rbac
Currently, services have two serviceaccounts: one specified in the
chart that cannot read anything, and one injected via helm-toolkit
that can read everything. This patch set refactors the logic to:

- cleanup the roles and their binding automatically when the helm
  chart is deleted;
- remove the need to separately mount a serviceaccount  with secret;
- better handling of namespaces resource restriction.

Co-Authored-By: portdirect <pete@port.direct>

Change-Id: I47d41e0cad9b5b002f59fc9652bad2cc025538dc
2017-12-19 20:22:57 -05:00
portdirect
611a78fb34 Fluent-Logging: Update fluent-bit to use common OSH entrypoint pattern
This PS updates the fluent-logging chart to use the same entrypoint
pattern as other OSH components.

Change-Id: I3bf9baf9824e1b7f7e46c4fcae292240566d9153
2017-12-17 17:33:24 +00:00
sungil
2862f038e2 Fluent-logging helm chart
This introduces an initial helm chart for fluent logging.
It provides a functional fluent-bit and fluentd deployment to
use in conjunction with elasticsearch and kibana to consume
and aggregate logs from all resource types in a cluster.
It can deliver logs to kafka for external tools to consume.

This PS moves fluent-logging chart from osh-addons, osh to
osh-infra repo.
previous ps(addon): https://review.openstack.org/#/c/507023/
previous ps(osh): https://review.openstack.org/#/c/514622/

Specification: https://review.openstack.org/#/c/505491/
Partially implements: blueprint osh-logging-framework

Change-Id: I72e580aa3a197550060fc07af8396a7c8368d40b
2017-12-15 10:52:16 -06:00