Fix containerized logrotate configuration

Use copytruncate and 'hourly' log rotation by default.  Increase the
default max number of rotated files to 336, which corresponds to 14
days, so that default period retained as is.

With the copytruncate option enabled, logs should be hourly rotated to
decrease disk IO load when copying log files around. The default
maxsize of 10M is better maintained for often rotations done within a
day as well, so log files will not happen to become unexpectedly huge
at the end of it.

W/o copytruncate, the containerized logrotate sends no signals to
processes, as files are only renamed and not unlinked. That makes the
files deletion based filter failing, until the default period of 14
days expires. To fix that non-copytruncate case, post-rotate always
sends HUP (USR1 for httpd) signals to all processes holding open files
in the /var/log/containers host path. That also makes all services
reloaded hourly (there is still a random splay applied by cron though)
as a side effect.

With copytruncate ON, each rotation ensures the old log files will also be
deleted, so only affected services will be reloaded.

Additionally, send USR1 instead of HUP to reload httpd in containers
gracefully.

Closes-Bug: #1785659

Change-Id: I15fa0eab1625ac63fd57b6a6d5cd22a6ac85f221
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>

manifests/profile/base/logging/logrotate.pp

@@ -50,6 +50,10 @@
 #  [*user*]
 #    (optional) Defaults to 'root'. Configures cron job for logrotate.
 #
+#  [*copytruncate*]
+#    (optional) Defaults to True.
+#    Configures the logrotate copytruncate parameter.
+#
 #  [*delaycompress*]
 #    (optional) Defaults to True.
 #    Configures the logrotate delaycompress parameter.
@@ -63,11 +67,11 @@
 #    Configures the logrotate maxsize parameter.
 #
 #  [*rotation*]
-#    (optional) Defaults to 'daily'.
+#    (optional) Defaults to 'hourly'.
 #    Configures the logrotate rotation interval.
 #
 #  [*rotate*]
-#    (optional) Defaults to 14.
+#    (optional) Defaults to 336 (corresponds to 14 days).
 #    Configures the logrotate rotate parameter.
 #
 #  [*purge_after_days*]
@@ -91,11 +95,12 @@ class tripleo::profile::base::logging::logrotate (
   $weekday          = '*',
   Integer $maxdelay = 90,
   $user             = 'root',
+  $copytruncate     = true,
   $delaycompress    = true,
   $compress         = true,
-  $rotation         = 'daily',
+  $rotation         = 'hourly',
   $maxsize          = '10M',
-  $rotate           = 14,
+  $rotate           = 336,
   $purge_after_days = 14,
   # DEPRECATED PARAMETERS
   $size             = undef,

releasenotes/notes/logrotate-copytruncate-hourly-f0851bec551f5f5f.yaml

@@ -0,0 +1,13 @@
+---
+upgrade:
+  - |
+    Logrotate's copytruncate is used by default for containerized services logs
+    rotation.  The default period to keep old logs remains unchanged (14 days),
+    but rotation will happen on hourly basis instead of daily. That means that
+    old log files will be maintained for 336 versions of it, which represents
+    14 days multiplied by 24 hours per a day.
+fixes:
+  - |
+    Containerized logrotate configuration ensures httpd processes running in
+    containers will be reloaded gracefully via SIGUSR1. Containerized servicies
+    will reliably reopen its logfiles when logrotate happens.

templates/logrotate/containers_logrotate.conf.erb

@@ -10,6 +10,9 @@
   maxsize <%= @maxsize %>
   missingok
   notifempty
+<%- if @copytruncate %>
+  copytruncate
+<%- end %>
 <%- if @delaycompress %>
   delaycompress
 <%- end %>
@@ -22,8 +25,11 @@
     \( -mtime +<%= @purge_after_days %> -or \
     -atime +<%= @purge_after_days %> -or \
     -ctime +<%= @purge_after_days %> \) -exec rm -f {} \;;
-    /sbin/lsof -nPs +L1 +D /var/log/containers 2>&1|\
-    awk '/\S+\s+[0-9]+\s.*\/var\/log\/containers\/.*\(deleted\)/ {print $2}' |\
-    sort -u | /bin/xargs -n1 -r -t kill -HUP
+    /sbin/lsof -nPs <%- if @copytruncate -%>+L1<%- end -%> +D /var/log/containers 2>&1|\
+    awk '/\S+\s+[0-9]+\s.*\/var\/log\/containers\/<%- if @copytruncate -%>.*\(deleted\)<%- end -%>/ {print $2}' |\
+    sort -u | grep -v httpd | /bin/xargs -n1 -r -t kill -HUP;
+    /sbin/lsof -nPs <%- if @copytruncate -%>+L1<%- end -%> +D /var/log/containers/httpd 2>&1|\
+    awk '/\S+\s+[0-9]+\s.*\/var\/log\/containers\/httpd\/<%- if @copytruncate -%>.*\(deleted\)<%- end -%>/ {print $2}' |\
+    sort -u | /bin/xargs -n1 -r -t kill -USR1
   endscript
 }