This patch uses the newly added td-agent repo on Debian.
It also makes code more robust regarding different distro handling.
We now have td-agent everywhere on x86_64.
Change-Id: I4d2449d7d100c68084f937ef9981596fc1625375
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
Fluentd packages in CentOS Opstools SIG repos is very old,
because of old-ish ruby (2.0) packaged with CentOS 7.
On the other hand - TreasureData packages for fluentd are packaged
with embedded ruby as dependencies - so this change moves to those.
Thanks to that we are able to move from fluentd 0.12 to 1.4.2.
Still we need ruby-devel for non-td-agent distros,
because we download monasca output plugin as source and build it
during installation.
* Bump up td.repo to version 3
* Merge sudoers into one file
* Change fluentd plugin install macro to reflect changes
Depends-On: https://review.opendev.org/679209/
Depends-On: https://review.opendev.org/679885/
Change-Id: I40a81103317ac044d9311b2505593056cf313bd2
In order to orchestrate a smooth migration of fluentd to newer
td-agent on CentOS x86_64, a label that points to fluentd binary
and version are needed.
Depends-On: https://review.opendev.org/677144/
Change-Id: Ie2257870d87c5b3a9b0572f8fd35085a3f95fedb
fluentd - we are not deploying kubernetes, no need for plugins
TripleO - CI currently broken (to reverse, FIXME in place)
Change-Id: I203f70176af7f2b53398ddeb5c21e6234a45e268
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
This resolves an issue where the previous version of the plugin
would fail to re-send buffered logs if a post to the Monasca Log
API failed. With this new version of the plugin, logs are correctly
buffered and resent.
Change-Id: Ic1d93a8ea5f2571edfacf50809d51a8b77959a73
If monasca is enabled on ubuntu, we see the following error in fluentd:
[error]: config error file="/etc/td-agent/td-agent.conf"
error_class=Fluent::ConfigError error="Unknown output plugin 'monasca'.
Run 'gem search -rd fluent-plugin' to find plugins"
This is because we use 'fluent-gem install' to install the gem, which
works on CentOS but on Ubuntu it is necessary to use 'td-agent-gem
install'.
Change-Id: I515f9764ca93a41a871e520696193a9298856d08
Closes-Bug: #1830147
The package used to install fluentd on debian/ubuntu is not
available for architectures other than x86_64. For those cases
install fluentd as a gem until the package 'td-agent' is available
for more architectures.
Change-Id: Ic1b25b37d06b429a112f1418a79d385d505bb1e9
This change updates the docker files to use base_package_type instead
of doing specific distro checks for the rhel/deb generic cases. The
base_distro is still available and is used when a specific distro needs
a customization but if the differences are purely rpm vs deb, then the
base_package_type can be used.
Change-Id: I8d720bb185df65a0178061ccf20b1ab2265da2c5
This is a simple output plugin for Fluentd which can forward logs
to the Monasca Log API.
Partially-Implements: blueprint monasca-containers
Change-Id: I599554052c3cb99ebcbcbc4d7ea1aeb57c573d2e
Currently, fluentd plugins are installed via gem install,
this may causes some issues when gems are updated upstream.
This patch uses the plugins packaged on CentOS opstools
repository.
The debian/ubuntu image has not been modified
Change-Id: Ib32ceaae0d9fa56c6ec25a47ae6f44259d3b6e6d
fluentd_plugins_install should contain only plugin installation operations,
so that the override does not have to handle irrelevant operations
Change-Id: I8f810d194a1e638683df1f445c187b8d98629730
For ubuntu, we actually use fluentd 0.14.22 and ruby 2.1.10 now.
Thus, we should use the specific gem version for the plugins:
- fluent-plugin-rewrite-tag-filter:2.0.0
According to the README.md of this plugin[1]
It shows:
----------------------------------------------------------
| fluent-plugin-rewrite-tag-filter | Fluentd | Ruby |
|----------------------------------|------------|--------|
| >= 2.0.0 | >= v0.14.2 | >= 2.1 |
| < 2.0.0 | >= v0.12.0 | >= 1.9 |
----------------------------------------------------------
- fluent-plugin-grok-parser:2.1.4
When using the fluent-plugin-rewrite-tag-filter 2.0.0 to fix this bug, another
bug[1] which is the same as this one[2] was triggered.
According to https://github.com/fluent/fluent-plugin-grok-parser, it shows:
---------------------------------------------------
| fluent-plugin-grok-parser | fluentd | Ruby |
|---------------------------|------------|--------|
| >= 1.0.0 | >= v0.14.0 | >= 2.1 |
| < 1.0.0 | >= v0.12.0 | >= 1.9 |
---------------------------------------------------
We actually use fluentd 0.14.22 now, but the version of fluent-plugin-grok-parser
is still pinned in 0.3.1. So we need to upgrade the fluent-plugin-grok-parser to
2.1.4.
[1] https://github.com/fluent/fluent-plugin-rewrite-tag-filter
[2] http://paste.openstack.org/show/626145/
[3] https://github.com/fluent/fluent-plugin-grok-parser/issues/32
backport: pike
Change-Id: I47985113fe732569d640a262ca832c6edc8c2bb6
Partial-Bug: #1730664
Use specific gem version for the plugins:
- fluent-plugin-rewrite-tag-filter
- fluent-plugin-elasticsearch
Change-Id: Icc5b8972e7eca01767cdc9321ee82d1fcde187d8
Closes-Bug: 1730346
This reverts commit 83baad304d.
The plugin is not used in any deployment tool. Was introduced to make multiline but finally standard multiline was used.
This change removes an unused plugin.
Change-Id: Ie38a876d56846ebdec02fa05707e9d22a02788e6
On the fluent Docker added the fluent-plugin-secure-forward,
so fluentd will be able to forward logs over SSL with
authentication
Closes-Bug: #1723408
Change-Id: I231ebc1052dac93fb9420dcfb10595a3647c35ce
the newest public_suffix gem required ruby>2.1.0, but the ruby on
centos' release is 2.0.0, we should install public_suffix==2.0.5
manual to avoid the ruby version not matched error.
Closes-Bug: #1713416
Change-Id: I725294da206a73b759027d3123d140bddafcfdfb
Actual Fluentd config doesn't manage multiline log pattern like:
2017-08-16 21:41:03.097 7 WARNING oslo_config.cfg [req-34ea3f23-f780-4059-b911-2788ea4af7b2 - - - - -] Option "firewall_driver" from group "DEFAULT" is deprecated for removal (
nova-network is deprecated, as are any related configuration options.
). Its value may be silently ignored in the future.
This log will generate 1 Fluentd records for each line (3 records),
this is really a wrong behaviour, and made Kibana interface unusable.
This patch add fluent-plugin-concat in order to concatenate multiline
log separated in multiple events.
Change-Id: I83e583605b49b7c421109d662de1291f38bd5c07
Closes-Bug: #1711399
-Use the fluentd package on the CentOS OpsTools repository
created by the CentOS OpsTools SIG.
-The user changed to fluentd for CentOS/RHEL/Oraclelinux
-Plugins are installed as gems on CentOS/RHEL/Oraclelinux
Change-Id: I4802618373cfa53d67d1dfe65be7abe1950b3d04
centos based images have wrong label info,
these changes fix own image's name and build-date.
Change-Id: I1d13f8f386c8db12b5fbe5f8ecbbf9e3fbb4ba1c
Closes-Bug: #1680341
Use LABEL instruction instead of MAINTAINER (deprecated) instruc-
tion as suggested by Docker's official dockerfile guide.
docs.docker.com/engine/reference/builder/#maintainer-deprecated
Closes-Bug: #1683652
Change-Id: Ie87a1ddf31aefcd0b623fd2837d78de420e76898
Debian support is not maintained in Kolla so it got a bit behind Ubuntu
one. This changeset enables Debian for all images. Jessie (even with
backports) may be too old for some images though.
Also unify distro check to ['debian', 'ubuntu'] to keep alphabetical order
like it is done for RPM distributions.
Partially-Implements: blueprint multiarch-and-arm64-containers
Change-Id: I056233fbfa277e0e2360c07c3f80d9558c554357
Some images have packages sorted alphabetically and some not.
Unify common style between all images.
Change-Id: I906ed89c10b12886665618752f525ba71d83d991
mariadb.log mode is 660, which can not be read by fluentd. Add td-agent
user into mysql group to allow fluent to read the log.
Closes-Bug: #1664867
Change-Id: If624002909d5fcd0bb82a5bc9937945518ec84b9
Additional repos for centos such as elasticsearch, ceph etc. are all set
up in the base image, fluentd should follow this both for consistency
and to make it easier for operators to remove/override.
Change-Id: If2ee20232d72ccf6b1c4be06c2fb3d1fa219e9cc
include_header and include_footer parameter is already removed, remove
them in all Dockerfiles.
Add missing footer block.
Change-Id: I90da03eb9f95a3827361d5f5ede65fde7d6be2b3
This centralizes all user and group creation into a single source. This
will fix any current and furture uid/gid mismatches (such as with
nova-libvirt).
In the process, we also unify users between the distros in a standard
way. The users in the following containers change from thier defaults:
Ubuntu: _chrony user is now chrony
Ubuntu: memcache user is now memcached
All: qemu user is used for ownership and socket permissions
All uid and gid numbers are customizable via kolla-build.conf
Co-Authored-By: Kris Lindgren <klindgren@godaddy.com>
Change-Id: I120f26ab0683dc87d69727c3df8d4707e52a4543
Partially-Implements: blueprint static-uid-gid
This PS adds Fluentd docker image to Kolla. Kolla
currently does not use Fluentd for log collection, but this image
is required for Kubernetes centralized logging solution.
Change-Id: I719661d75f5044239a1fd52080d322c77490ecae
Partially_Implements: blueprint kubernetes-images
Viktor Michalek