By default podman 3.0.x sets the [engine]/events_logger to "file".
This causes every exec in podman to create a line of text in
/run/libpod/events/events.log like the following:
{"ID":"412b6770c0b418e6d49a4801e71a198ddb81bbbefdaf1c9aad4d7948f77910ee","Image":"quay.io/centos/centos:latest","Name":"leak-test-7","Status":"exec","Time":"2021-06-03T08:36:05.237964012Z","Type":"container","Attributes":{"org.label-schema.build-date":"20201204","org.label-schema.license":"GPLv2","org.label-schema.name":"CentOS Base Image","org.label-schema.schema-version":"1.0","org.label-schema.vendor":"CentOS"}}
Since by default /run is mounted on tmpfs, this has the side-effect of
increasing kernel slab objects over time indefinitely eventually causing
an OOM of the box.
We initially wanted to switch to the 'none' backend, but the podman
folks recommended using the journald backend because events logs are
used by podman in case of a rare race when running "podman run --rm".
Given that we call run with --rm from in a multithreaded fashion this
seems to be the safest approach. The drawback of using journald is
that events won't be logged for rootless containers unless the user
is part of the 'wheel' group. We believe we're not using those
containers in tripleo anyways, so this should be safe.
Tested by applying a backport of this patch to Train + podman 3.0.x and
got the following:
[root@controller-0 containers]# ls -la /run/libpod/events/
total 0
drwx------. 2 root root 40 Jun 3 11:55 .
drwxr-x--x. 5 root root 140 Jun 3 11:55 ..
[root@controller-0 containers]# more /etc/containers/containers.conf
[containers]
pids_limit = 4096
[engine]
events_logger = "journald"
Also tested the override via the corresponding THT change in
Ieffe2852111c3ec8347343a042dd78bbf691d79a.
Closes-Bug: #1923607
Change-Id: I780103e17f1bb42a0546c30bd6c001c642ad88b3
f31bab878b