Unpin td-agent and cap elasticsearch gem

With [0] the version of td-agent being installed was pinned, which isn't
sustainable in the long run, so we drop the pin.

Latest version of the elasticsearch gem no longer works with older
(OSS) versions of Elasticsearch. This is fixed by downgrading the version
of the elasticsearch gems.

[0] Iefcdd3100b7e3c5320bc5f1286a18251bdeab885
Related-Bug: 1930867
Closes-Bug: 1954759
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Depends-On: https://review.opendev.org/c/openstack/kolla-ansible/+/823155
Change-Id: I3045786e863b098d7339d1066aef6c857aa5f97f

docker/base/apt_preferences.debian

@@ -7,11 +7,6 @@ Package: erlang*
 Pin: version 1:23.*
 Pin-Priority: 1000
 
-# FIXME(mgoddard): Pinning to 4.0.* to avoid bug 1930867.
-Package: td-agent*
-Pin: version 4.0.*
-Pin-Priority: 1000
-
 # NOTE(mgoddard): logstash 7.9.x is the last version that supports
 # Elasticsearch OSS.
 Package: logstash-oss

docker/base/apt_preferences.ubuntu

@@ -6,11 +6,6 @@ Package: erlang*
 Pin: version 1:23.*
 Pin-Priority: 1000
 
-# FIXME(mgoddard): Pinning to 4.0.* to avoid bug 1930867.
-Package: td-agent*
-Pin: version 4.0.*
-Pin-Priority: 1000
-
 # NOTE(mgoddard): logstash 7.9.x is the last version that supports
 # Elasticsearch OSS.
 Package: logstash-oss

docker/fluentd/Dockerfile.j2

@@ -13,11 +13,10 @@ LABEL maintainer="{{ maintainer }}" name="{{ image_name }}" build-date="{{ build
 
 {% if base_package_type == 'rpm' %}
 
-    # FIXME(mgoddard): Pinning to 4.0.* to avoid bug 1930867.
     {% set fluentd_packages = [
         'gcc-c++',
         'make',
-        'td-agent-4.0.*',
+        'td-agent',
     ] %}
 
 {% elif base_package_type == 'deb' %}
@@ -31,7 +30,13 @@ LABEL maintainer="{{ maintainer }}" name="{{ image_name }}" build-date="{{ build
 {% endif %}
 
 {{ macros.configure_user(name=fluentd_user, groups='mysql') }}
-{{ macros.install_packages(fluentd_packages | customizable("packages")) }}
+
+# NOTE(frickler): Downgrading elasticsearch gems for compatibility with OSS versions of ES
+{{ macros.install_packages(fluentd_packages | customizable("packages")) }} \
+    && td-agent-gem install elasticsearch:7.13.0 \
+    && td-agent-gem uninstall elasticsearch:7.15.0 \
+    && td-agent-gem uninstall elasticsearch-api:7.15.0 \
+    && td-agent-gem uninstall elasticsearch-transport:7.15.0
 
 COPY fluentd_sudoers /etc/sudoers.d/kolla_fluentd_sudoers
 

releasenotes/notes/cap-fluentd-elasticsearch-18c0ca8e90c1234c.yaml

@@ -0,0 +1,7 @@
+---
+fixes:
+  - |
+    Latest version of the elasticsearch gem no longer works with older
+    (OSS) versions of Elasticsearch. This is fixed by capping the version
+    of the elasticsearch gem installed into the fluentd container.
+    `LP#1954759 <https://launchpad.net/bugs/1954759>`__