Add playbook to ship journals from hosts

The journal within systemd is able to be shipped from a physical hosts
to a centralized location. This change introduces
`systemd-journal-remote` which will ship all journals on the physical
host to the log host and store the journals under
"/var/log/journal/remote". This change gives deployers greater
visability into the cloud using the systemd built-ins.

> NOTE: This change is all accomplished in a playbook using our common
        roles. While this could be moved into a role by itself, it would
        be a waist of effort given how small this change is.

Given all services are inherently logging to the journal, this change
may allow us to one day deprecate or minimize the usage of our
rsyslog roles. If we were to remove the requirement for rsyslog to run
everywhere we could reduce overall internal cluster IO (CPU, network and
block) and remove the requirement for all services to ship log files from
all containers and hosts. This change is NOT modifying the integrated
logging architecture. At this time we're simply ensuring that the
journals on the physical host are co-located on the logging machines.

At this time there's no suitable package available for
systemd-journal-remote on suse so the playbook to install and setup
remote journalling is being omitted when the suse is detected. When a
suitable package is found the playbook omission should be removed.

Change-Id: I254d52df6303b7cc4d4071b4beaf347922b2616e
Related-Change: https://review.openstack.org/553707
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>

inventory/group_vars/all/all.yml

@@ -37,6 +37,15 @@ default_bind_mount_logs: true
 # in order to create a more sensible repo name for the distro.
 os_distro_version: "{{ (ansible_distribution | lower) | replace(' ', '_') }}-{{ ansible_distribution_version.split('.')[:2] | join('.') }}-{{ ansible_architecture | lower }}"
 
+# Set the systemd prefix based on the base OS.
+systemd_utils_distro_prefix:
+  apt: "/lib/systemd"
+  yum: "/lib/systemd"
+  dnf: "/lib/systemd"
+  zypper: "/usr/lib/systemd"
+
+systemd_utils_prefix: "{{ systemd_utils_distro_prefix[ansible_pkg_mgr] }}"
+
 # Ensure that the package state matches the global setting
 rsyslog_client_package_state: "{{ package_state }}"
 

playbooks/infra-journal-remote.yml

@@ -0,0 +1,104 @@
+---
+# Copyright 2018, Rackspace US, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: Install Journal-Remote
+  hosts: hosts
+  gather_facts: "{{ osa_gather_facts | default(True) }}"
+  become: true
+  pre_tasks:
+    # At this time there's no suitable package available for systemd-journal-remote/gateway
+    # When installing on SUSE 42.x. For now this playbook will omit suse when the package
+    # manager is "zypper". When a suitable package is available on SUSE this should be removed.
+    - name: Omit suse from this playbook
+      meta: end_play
+      when:
+        - ansible_pkg_mgr == 'zypper'
+
+    - name: Install systemd-journal-remote
+      package:
+        name: "{{ systemd_journal_remote_distro_package[ansible_pkg_mgr] }}"
+        state: "{{ package_state }}"
+
+    - name: Create journal directory
+      file:
+        path: "/var/log/journal"
+        state: "directory"
+        owner: "root"
+        group: "systemd-journal"
+
+    - name: Create journal remote directory
+      file:
+        path: "/var/log/journal/remote"
+        state: "directory"
+        owner: "systemd-journal-remote"
+        group: "systemd-journal"
+
+  roles:
+    - role: "systemd_service"
+      systemd_tempd_prefix: "openstack"
+      systemd_CPUAccounting: true
+      systemd_BlockIOAccounting: true
+      systemd_MemoryAccounting: true
+      systemd_TasksAccounting: true
+      systemd_services:
+        - service_name: "systemd-journal-remote"
+          enabled: "{{ (ansible_host != systemd_journal_remote_target) | ternary('no', 'yes') }}"
+          state: "{{ (ansible_host != systemd_journal_remote_target) | ternary('stopped', 'started') }}"
+          execstarts: >-
+            {{ systemd_utils_prefix }}/systemd-journal-remote
+            --listen-http=-3
+            --split-mode=host
+            --compress
+            --seal
+            --output=/var/log/journal/remote/
+          config_overrides:
+            Unit:
+              Description: "Journal Remote Sink Service"
+              Documentation: "man:systemd-journal-remote(8) man:journal-remote.conf(5)"
+              Requires: "systemd-journal-remote.socket"
+            Service:
+              WatchdogSec: "3min"
+              LimitNOFILE: 16384
+              User: "systemd-journal-remote"
+              Group: "systemd-journal-remote"
+
+        - service_name: "systemd-journal-upload"
+          enabled: "{{ (ansible_host == systemd_journal_remote_target) | ternary('no', 'yes') }}"
+          state: "{{ (ansible_host == systemd_journal_remote_target) | ternary('stopped', 'started') }}"
+          execstarts: >-
+            {{ systemd_utils_prefix }}/systemd-journal-upload
+            --save-state
+            --merge
+            --url=http://{{ systemd_journal_remote_target }}:19532
+          config_overrides:
+            Unit:
+              Description: "Journal Remote Upload Service"
+              Documentation: "man:systemd-journal-upload(8)"
+              After: "network.target"
+            Service:
+              WatchdogSec: "3min"
+              LimitNOFILE: 16384
+              User: "systemd-journal-upload"
+              Group: "systemd-journal"
+
+  vars:
+    systemd_journal_remote_target: "{{ hostvars[groups['log_hosts'][0]]['ansible_host'] }}"
+    systemd_journal_remote_distro_package:
+      apt: "systemd-journal-remote"
+      yum: "systemd-journal-gateway"
+      dnf: "systemd-journal-gateway"
+
+  tags:
+    - journal-remote

playbooks/setup-infrastructure.yml

@@ -27,3 +27,4 @@
 - include: etcd-install.yml
 - include: ceph-install.yml
 - include: rsyslog-install.yml
+- include: infra-journal-remote.yml

releasenotes/notes/systemd-journal-remote-25248628390b46d9.yaml

@@ -0,0 +1,9 @@
+---
+features:
+  - A new playbook ``infra-journal-remote.yml`` to ship journals has
+    been added. Physical hosts will now ship the all available systemd
+    journals to the logging infrastructure. The received journals will
+    be split up by host and stored in the `/var/log/journal/remote`
+    directory. This feature will give deployers greater access/insight
+    into how the cloud is functioning requiring nothing more that the
+    systemd built-ins.