Merge "Ansible roles for backup"

.zuul.yaml

@@ -687,6 +687,30 @@
       - testinfra/test_adns.py
       - testinfra/test_ns.py
 
+- job:
+    name: system-config-run-backup
+    parent: system-config-run
+    description: |
+      Run the playbook for backup configuration
+    nodeset:
+      nodes:
+        - name: bridge.openstack.org
+          label: ubuntu-bionic
+        - name: backup01.region.provider.opendev.org
+          label: ubuntu-bionic
+        - name: backup-test01.opendev.org
+          label: ubuntu-bionic
+        - name: backup-test02.opendev.org
+          label: ubuntu-xenial
+    vars:
+      run_playbooks:
+        - playbooks/service-backup.yaml
+    files:
+      - .zuul.yaml
+      - playbooks/roles/backup.*
+      - playbooks/zuul/templates/host_vars/backup.*
+      - testinfra/test_backups.py
+
 - job:
     name: system-config-run-mirror
     parent: system-config-run
@@ -870,6 +894,7 @@
         - system-config-run-base
         - system-config-run-base-ansible-devel:
             voting: false
+        - system-config-run-backup
         - system-config-run-dns
         - system-config-run-eavesdrop
         - system-config-run-lists

doc/source/sysadmin.rst

@@ -215,53 +215,21 @@ OpenStack CI infrastructure for another project.
 Backups
 =======
 
-Off-site backups are made to two servers:
+Infra uses the `bup <https://bup.github.io>`__ tool for backups.
 
-* backup01.ord.rax.ci.openstack.org
+Hosts in the ``backup`` Ansible inventory group will be backed up to
-* TBD
+servers in the ``backup-server`` group with ``bup``.  The
+``playbooks/roles/backup`` and ``playbooks/roles/backup-server`` roles
+implement the required setup.
 
-Puppet is used to perform the initial configuration of those machines,
+The backup server has a unique Unix user for each host to be backed
-but to protect them from unauthorized access in case access to the
+up.  The roles will setup required users, their home directories in
-puppet git repo is compromised, it is not run in agent or in cron mode
+the backup volume and relevant ``authorized_keys``.
-on them.  Instead, it should be manually run when changes are made
-that should be applied to the backup servers.
 
-To start backing up a server, some commands need to be run manually on
+Host backup happens via a daily cron job (managed by Ansible) on each
-both the backup server, and the server to be backed up.  On the server
+individual host to be backed up.  The host to be backed up initiates
-to be backed up::
+the backup process to the remote backup server(s) using a separate ssh
-
+key setup just for backup communication (see ``/root/.ssh/config``).
-  sudo su -
-  ssh-keygen -t rsa -f /root/.ssh/id_rsa -N ""
-  bup init
-
-And then ``cat /root/.ssh/id_rsa.pub`` for use later.
-
-On the backup servers::
-
-  # add bup user
-  BUPUSER=bup-<short-servername>  # eg, bup-jenkins-dev
-  sudo useradd -r $BUPUSER -s /bin/bash -d /opt/backups/$BUPUSER -m
-  sudo su - $BUPUSER
-
-  # initalise bup
-  bup init
-
-  # should be in home directory /opt/backups/$BUPUSER
-  mkdir .ssh
-  cat >.ssh/authorized_keys
-
-write this into the authorized_keys file and end with ^D on a blank line::
-
-  command="BUP_DEBUG=0 BUP_FORCE_TTY=3 bup server",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty <ssh key from earlier>
-
-Switching back to the server to be backed up, run::
-
-  ssh $BUPUSER@backup01.ord.rax.ci.openstack.org
-
-And verify the host key.  Note this will start the bup server on the
-remote end, you will not be given a pty. Use ^D to close the connection
-cleanly.  Add the "backup" class in puppet to the server
-to be backed up.
 
 Restore from Backup
 -------------------
@@ -276,9 +244,14 @@ how we restore content from backups::
   mkdir /root/backup-restore-$DATE
   cd /root/backup-restore-$DATE
 
+Root uses a separate ssh key and remote user to communicate with the
+backup server(s); the username and key to use for backup should be
+automatically configured in ``/root/.ssh/config``.  The backup server
+hostname can be taken from there.
+
 At this point we can join the tar that was split by the backup cron::
 
-  bup join -r bup-<short-servername>@backup01.ord.rax.ci.openstack.org: root > backup.tar
+  bup join -r backup.x.y.opendev.org: root > backup.tar
 
 At this point you may need to wait a while. These backups are stored on
 servers geographically distant from our normal servers resulting in less

playbooks/roles/backup-server/README.rst

@@ -0,0 +1,15 @@
+Setup backup server
+
+This role configures backup server(s) in the ``backup-server`` group
+to accept backups from remote hosts.
+
+Note that the ``backup`` role must have run on each host in the
+``backup`` group before this role.  That role will create a
+``bup_user`` tuple in the hostvars for for each host consisting of the
+required username and public key.
+
+Each required user gets a separate home directory in ``/opt/backups``.
+Their ``authorized_keys`` file is configured with the public key to
+allow the remote host to log in and only run ``bup``.
+
+**Role Variables**

playbooks/roles/backup-server/defaults/main.yaml

@@ -0,0 +1 @@
+bup_users: []

playbooks/roles/backup-server/tasks/main.yaml

@@ -0,0 +1,21 @@
+- name: Create backup directory
+  file:
+    state: directory
+    path: /opt/backups
+
+- name: Install bup
+  package:
+    name:
+      - bup
+    state: present
+
+- name: Build all bup users from backup hosts
+  set_fact:
+    bup_users: '{{ bup_users }} + [ {{ hostvars[item]["bup_user"] }} ]'
+  with_inventory_hostnames: backup
+
+- name: Create bup users
+  include_tasks: user.yaml
+  loop: '{{ bup_users }}'
+  loop_control:
+    loop_var: bup_user

playbooks/roles/backup-server/tasks/user.yaml

@@ -0,0 +1,32 @@
+# note bup_user is the parent loop variable name; this works on each
+# element from the bup_users global.
+- name: Set variables
+  set_fact:
+    user_name: '{{ bup_user[0] }}'
+    user_key: '{{ bup_user[1] }}'
+
+- name: Create bup user
+  user:
+    name: '{{ user_name }}'
+    comment: 'Backup user'
+    shell: /bin/bash
+    home: '/opt/backups/{{ user_name }}'
+    create_home: yes
+  register: homedir
+
+- name: Create bup user authorized key
+  authorized_key:
+    user: '{{ user_name }}'
+    state: present
+    key: '{{ user_key }}'
+    key_options: 'command="BUP_DEBUG=0 BUP_FORCE_TTY=3 bup server",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty'
+
+# ansible-lint wants this in a handler, it should be done here and
+# now; this isn't like a service restart where multiple things might
+# call it.
+- name: Initalise bup  # noqa 503
+  shell: |
+    BUP_DIR=/opt/backups/{{ user_name }}/.bup bup init
+  become: yes
+  become_user: '{{ user_name }}'
+  when: homedir.changed

playbooks/roles/backup/README.rst

@@ -0,0 +1,23 @@
+Configure a host to be backed up
+
+This role setups a host to use ``bup`` for backup to any hosts in the
+``backup-server`` group.
+
+A separate ssh key will be generated for root to connect to the backup
+server(s) and the host key for the backup servers will be accepted to
+the host.
+
+The ``bup`` tool is installed and a cron job is setup to run the
+backup periodically.
+
+Note the ``backup-server`` role must run after this to create the user
+correctly on the backup server.  This role sets a tuple ``bup_user``
+with the username and public key; the ``backup-server`` role uses this
+variable for each host in the ``backup`` group to initalise users.
+
+**Role Variables**
+
+.. zuul:rolevar:: bup_username
+
+   The username to connect to the backup server.  If this is left
+   undefined, it will be automatically set to ``bup-$(hostname)``

playbooks/roles/backup/files/bup-excludes

@@ -0,0 +1,22 @@
+/proc/*
+/sys/*
+/dev/*
+/tmp/*
+/floppy/*
+/cdrom/*
+/var/spool/squid/*
+/var/spool/exim/*
+/media/*
+/mnt/*
+/var/agentx/*
+/run/*
+/root/backup-restore-*
+/root/.bup
+/etc/puppet/modules/*
+/etc/puppet/hieradata/*
+/var/cache/*
+/var/lib/puppet/reports/*
+/var/lib/postgresql/*
+/var/lib/lxcfs/*
+/opt/system-config/*
+/afs/*

playbooks/roles/backup/tasks/main.yaml

@@ -0,0 +1,61 @@
+- name: Generate bup username for this host
+  set_fact:
+    bup_username: 'bup-{{ inventory_hostname.split(".", 1)[0] }}'
+  when: bup_username is not defined
+
+- debug:
+    var: bup_username
+
+- name: Install bup
+  package:
+    name:
+      - bup
+    state: present
+
+- name: Generate keypair for backups
+  openssh_keypair:
+    path: /root/.ssh/id_backup_ed25519
+    type: ed25519
+  register: bup_keypair
+
+- name: Initalise bup # noqa 503
+  command: bup init
+  when: bup_keypair.changed
+
+- name: Configure ssh for backup server
+  blockinfile:
+    path: /root/.ssh/ssh_config
+    create: true
+    block: |
+      Host {{ item }}
+      HostName {{ item }}
+      IdentityFile /root/.ssh/id_backup_ed25519
+      User {{ bup_username }}
+    mode: 0600
+  with_inventory_hostnames: backup-server
+
+- name: Generate bup_user info tuple
+  set_fact:
+    bup_user: '{{ [ bup_username, bup_keypair["public_key"] ] }}'
+
+- name: Accept hostkey of backup server
+  known_hosts:
+    state: present
+    key: '{{ item }} ecdsa-sha2-nistp256 {{ hostvars[item]["ansible_ssh_host_key_ed25519_public"] }}'
+    name: '{{ item }}'
+  with_inventory_hostnames: backup-server
+
+- name: Write /etc/bup-excludes
+  copy:
+    src: bup-excludes
+    dest: /etc/bup-excludes
+    mode: 0444
+
+- name: Install backup cron job
+  cron:
+    name: "Run bup backup"
+    job: "tar -X /etc/bup-excludes -cPF - / | bup split -r {{ bup_username }}@{{ item }}: -n root -q"
+    user: root
+    hour: '5'
+    minute: '{{ 59|random(seed=item) }}'
+  with_inventory_hostnames: backup-server

playbooks/service-backup.yaml

@@ -0,0 +1,10 @@
+# This needs to happen in order.  Backup hosts export their username/key
+# combos which are installed onto the backup server
+- hosts: "backup:!disabled"
+  name: "Base: Generate backup users and keys"
+  roles:
+    - backup
+- hosts: "backup-server:!disabled"
+  name: "Generate bup configuration"
+  roles:
+    - backup-server

playbooks/zuul/run-base.yaml

@@ -87,6 +87,8 @@
         - host_vars/letsencrypt02.opendev.org.yaml
         - host_vars/mirror01.openafs.provider.opendev.org.yaml
         - host_vars/mirror-update01.opendev.org.yaml
+        - host_vars/backup-test01.opendev.org.yaml
+        - host_vars/backup-test02.opendev.org.yaml
     - name: Display group membership
       command: ansible localhost -m debug -a 'var=groups'
     - name: Run base.yaml

playbooks/zuul/templates/gate-groups.yaml.j2

@@ -9,3 +9,10 @@ groups:
     - letsencrypt01.opendev.org
     - letsencrypt02.opendev.org
     - mirror01.openafs.provider.opendev.org
+
+  backup-server:
+    - backup01.region.provider.opendev.org
+
+  backup:
+    - backup-test01.opendev.org
+    - backup-test02.opendev.org

playbooks/zuul/templates/host_vars/backup-test01.opendev.org.yaml.j2

@@ -0,0 +1 @@
+bup_username: bup-backup01

playbooks/zuul/templates/host_vars/backup-test02.opendev.org.yaml.j2

@@ -0,0 +1,2 @@
+# Intentionally left blank to test autogeneration of name
+#bup_username: bup-backup-test02

testinfra/test_backups.py

@@ -0,0 +1,61 @@
+# Copyright 2019 Red Hat, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+import os.path
+import pytest
+
+testinfra_hosts = ['backup01.region.provider.opendev.org',
+                   'backup-test01.opendev.org',
+                   'backup-test02.opendev.org']
+
+
+def test_bup_installed(host):
+    package = host.package("bup")
+    assert package.is_installed
+
+def test_server_users(host):
+    hostname = host.backend.get_hostname()
+    if hostname.startswith('backup-test'):
+        pytest.skip()
+
+    for username in 'bup-backup01', 'bup-backup-test02':
+        homedir = os.path.join('/opt/backups/', username)
+        bup_config = os.path.join(homedir, '.bup', 'config')
+        authorized_keys = os.path.join(homedir, '.ssh', 'authorized_keys')
+
+        user = host.user(username)
+        assert user.exists
+        assert user.home == homedir
+
+        f = host.file(authorized_keys)
+        assert f.exists
+        assert f.contains("ssh-ed25519")
+
+        f = host.file(bup_config)
+        assert f.exists
+
+def test_backup_host_config(host):
+    hostname = host.backend.get_hostname()
+    if hostname == 'backup01.region.provider.opendev.org':
+        pytest.skip()
+
+    f = host.file('/root/.ssh/id_backup_ed25519')
+    assert f.exists
+
+    f = host.file('/root/.ssh/ssh_config')
+    assert f.exists
+    assert f.contains('Host backup01.region.provider.opendev.org')
+
+    f = host.file('/root/.bup/config')
+    assert f.exists