docker: install rsyslog to capture container output

This started with me wondering why gerritbot was putting all its
output into /var/log/syslog -- it turns out Xenial docker is
configured to use journalctl (which forwards to syslog) and Bionic
onwards uses json-file.

Both are sub-optimial; but particularly the json-file because we lose
the logs when the container dies.  This proposes moving to a more
standard model of having the containers log to syslog and redirecting
that to files on disk.

Install a rsyslog configuration to capture "docker-*" program names
and put them into logfiles in /var/log/containers.  Also install
rotation for these files.

In an initial group of docker-compose files, setup logging to syslog
which should then be captured into these files.  Add some basic
testing.

If this works OK, I think we can standardise our docker-compose files
like this to caputure the logs the same everywhere.

Change-Id: I940a5b05057e832e2efad79d9a2ed5325020ed0c

playbooks/roles/gerritbot/files/docker-compose.yaml

@@ -7,6 +7,10 @@ services:
     image: docker.io/opendevorg/gerritbot:latest
     network_mode: host
     restart: always
+    logging:
+      driver: syslog
+      options:
+        tag: "docker-gerritbot"
     volumes:
       # This contains the main config, channel config, and ssh key
       - /etc/gerritbot:/etc/gerritbot

playbooks/roles/gitea/templates/docker-compose.yaml.j2

@@ -14,6 +14,10 @@ services:
       MYSQL_PASSWORD: "{{ gitea_db_password }}"
     volumes:
       - /var/gitea/db:/var/lib/mysql
+    logging:
+      driver: syslog
+      options:
+        tag: "docker-mariadb"
   gitea-web:
     depends_on:
       - mariadb
@@ -32,6 +36,10 @@ services:
       - /var/gitea/conf:/custom/conf
       - /var/gitea/logs:/logs
       - /var/gitea/certs:/certs
+    logging:
+      driver: syslog
+      options:
+        tag: "docker-gitea"
   gitea-ssh:
     depends_on:
       - mariadb
@@ -45,3 +53,7 @@ services:
       - /var/gitea/data:/data
       - /var/gitea/conf:/custom/conf
       - /var/gitea/logs:/logs
+    logging:
+      driver: syslog
+      options:
+        tag: "docker-gitea-ssh"

playbooks/roles/install-docker/README.rst

@@ -1,5 +1,16 @@
 An ansible role to install docker in the OpenStack infra production environment
 
+This also installs a log redirector for syslog ```docker-`` tags.  For
+most containers, they can be setup in the compose file with a section
+such as:
+
+.. code-block:: yaml
+
+   logging:
+     driver: syslog
+     options:
+       tag: docker-<appname>
+
 **Role Variables**
 
 .. zuul:rolevar:: use_upstream_docker

playbooks/roles/install-docker/files/98-docker.conf

@@ -0,0 +1,4 @@
+# Create a template for the target log file
+$template CUSTOM_LOGS,"/var/log/containers/%programname%.log"
+
+if $programname startswith  'docker-' then ?CUSTOM_LOGS

playbooks/roles/install-docker/handlers/main.yaml

@@ -0,0 +1,4 @@
+- name: Restart rsyslog
+  service:
+    name: rsyslog
+    state: restarted

playbooks/roles/install-docker/tasks/main.yaml

@@ -40,3 +40,30 @@
     name: docker-compose
     state: present
     executable: pip3
+
+- name: Install rsyslog redirector for container tags
+  copy:
+    src: '98-docker.conf'
+    dest: /etc/rsyslog.d/
+    owner: root
+    group: root
+    mode: 0644
+  notify:
+    - Restart rsyslog
+
+- name: Ensure rsyslog restarted now
+  meta: flush_handlers
+
+- name: Create container log directories
+  file:
+    state: directory
+    path: /var/log/containers/
+    owner: syslog
+    group: adm
+    mode: 0775
+
+- name: Install log rotation for docker files
+  include_role:
+    name: logrotate
+  vars:
+    logrotate_file_name: '/var/log/containers/*.log'

testinfra/test_eavesdrop.py

@@ -25,10 +25,11 @@ def test_eavesdrop(host):
     assert web in rules
 
 def test_gerritbot_logs(host):
+
+    log_file = host.file('/var/log/containers/docker-gerritbot.log')
     # A simple check that docker-compose and our container did something
-    cmd = host.run("docker logs gerritbot-docker_gerritbot_1")
     # We expect auth to fail so check that it did
-    assert "Authentication (publickey) failed" in cmd.stdout
+    assert log_file.contains("Authentication (publickey) failed")
 
 def test_gerritbot_running(host):
     # Check that the container hasn't stopped

testinfra/test_gitea.py

@@ -32,10 +32,6 @@ def test_ulimit(host):
                 "16777216 9223372036854775807 bytes")
     assert expected in cmd.stdout.split('\n')
 
-def test_sshd_logs(host):
-    cmd = host.run("docker logs gitea-docker_gitea-ssh_1")
-    assert cmd.stdout != '' or cmd.stderr != ''
-
 def test_robots(host):
     cmd = host.run('curl --insecure '
                    '--resolve gitea99.opendev.org:3000:127.0.0.1 '
@@ -54,3 +50,14 @@ def test_proxy_ua_blacklist(host):
                    '--resolve gitea99.opendev.org:3081:127.0.0.1 '
                    'https://gitea99.opendev.org:3081/')
     assert '403 Forbidden' in cmd.stdout
+
+def test_ondisk_logs(host):
+    mariadb_log = host.file('/var/log/containers/docker-mariadb.log')
+    assert mariadb_log.exists
+
+    gitea_log = host.file('/var/log/containers/docker-gitea.log')
+    assert gitea_log.exists
+
+    gitea_ssh_log = host.file('/var/log/containers/docker-gitea-ssh.log')
+    assert gitea_ssh_log.exists
+    assert gitea_ssh_log.contains("Server listening on :: port 222.")

zuul.d/system-config-run.yaml

@@ -16,6 +16,7 @@
         '/var/log/syslog': logs_txt
         '/var/log/messages': logs_txt
         '/var/log/docker': logs
+        '/var/log/containers': logs
         '/etc/iptables/rules.v4': logs_txt
         '/etc/iptables/rules.v6': logs_txt
     host-vars: