Browse Source

Ansible roles for backup

This introduces two new roles for managing the backup-server and hosts
that we wish to back up.

Firstly the "backup" role runs on hosts we wish to backup.  This
generates and configures a separate ssh key for running bup and
installs the appropriate cron job to run the backup daily.

The "backup-server" job runs on the backup server (or, indeed
servers).  It creates users for each backup host, accepts the remote
keys mentioned above and initalises bup.  It is then ready to receive
backups from the remote hosts.

This eliminates a fairly long-standing requirement for manual setup of
the backup server users and keys; this section is removed from the
documentation.

testinfra coverage is added.

Change-Id: I9bf74df351e056791ed817180436617048224d2c
changes/57/662657/27
Ian Wienand 3 months ago
parent
commit
814e4be128

+ 25
- 0
.zuul.yaml View File

@@ -687,6 +687,30 @@
687 687
       - testinfra/test_adns.py
688 688
       - testinfra/test_ns.py
689 689
 
690
+- job:
691
+    name: system-config-run-backup
692
+    parent: system-config-run
693
+    description: |
694
+      Run the playbook for backup configuration
695
+    nodeset:
696
+      nodes:
697
+        - name: bridge.openstack.org
698
+          label: ubuntu-bionic
699
+        - name: backup01.region.provider.opendev.org
700
+          label: ubuntu-bionic
701
+        - name: backup-test01.opendev.org
702
+          label: ubuntu-bionic
703
+        - name: backup-test02.opendev.org
704
+          label: ubuntu-xenial
705
+    vars:
706
+      run_playbooks:
707
+        - playbooks/service-backup.yaml
708
+    files:
709
+      - .zuul.yaml
710
+      - playbooks/roles/backup.*
711
+      - playbooks/zuul/templates/host_vars/backup.*
712
+      - testinfra/test_backups.py
713
+
690 714
 - job:
691 715
     name: system-config-run-mirror
692 716
     parent: system-config-run
@@ -870,6 +894,7 @@
870 894
         - system-config-run-base
871 895
         - system-config-run-base-ansible-devel:
872 896
             voting: false
897
+        - system-config-run-backup
873 898
         - system-config-run-dns
874 899
         - system-config-run-eavesdrop
875 900
         - system-config-run-lists

+ 18
- 45
doc/source/sysadmin.rst View File

@@ -215,53 +215,21 @@ OpenStack CI infrastructure for another project.
215 215
 Backups
216 216
 =======
217 217
 
218
-Off-site backups are made to two servers:
218
+Infra uses the `bup <https://bup.github.io>`__ tool for backups.
219 219
 
220
-* backup01.ord.rax.ci.openstack.org
221
-* TBD
220
+Hosts in the ``backup`` Ansible inventory group will be backed up to
221
+servers in the ``backup-server`` group with ``bup``.  The
222
+``playbooks/roles/backup`` and ``playbooks/roles/backup-server`` roles
223
+implement the required setup.
222 224
 
223
-Puppet is used to perform the initial configuration of those machines,
224
-but to protect them from unauthorized access in case access to the
225
-puppet git repo is compromised, it is not run in agent or in cron mode
226
-on them.  Instead, it should be manually run when changes are made
227
-that should be applied to the backup servers.
225
+The backup server has a unique Unix user for each host to be backed
226
+up.  The roles will setup required users, their home directories in
227
+the backup volume and relevant ``authorized_keys``.
228 228
 
229
-To start backing up a server, some commands need to be run manually on
230
-both the backup server, and the server to be backed up.  On the server
231
-to be backed up::
232
-
233
-  sudo su -
234
-  ssh-keygen -t rsa -f /root/.ssh/id_rsa -N ""
235
-  bup init
236
-
237
-And then ``cat /root/.ssh/id_rsa.pub`` for use later.
238
-
239
-On the backup servers::
240
-
241
-  # add bup user
242
-  BUPUSER=bup-<short-servername>  # eg, bup-jenkins-dev
243
-  sudo useradd -r $BUPUSER -s /bin/bash -d /opt/backups/$BUPUSER -m
244
-  sudo su - $BUPUSER
245
-
246
-  # initalise bup
247
-  bup init
248
-
249
-  # should be in home directory /opt/backups/$BUPUSER
250
-  mkdir .ssh
251
-  cat >.ssh/authorized_keys
252
-
253
-write this into the authorized_keys file and end with ^D on a blank line::
254
-
255
-  command="BUP_DEBUG=0 BUP_FORCE_TTY=3 bup server",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty <ssh key from earlier>
256
-
257
-Switching back to the server to be backed up, run::
258
-
259
-  ssh $BUPUSER@backup01.ord.rax.ci.openstack.org
260
-
261
-And verify the host key.  Note this will start the bup server on the
262
-remote end, you will not be given a pty. Use ^D to close the connection
263
-cleanly.  Add the "backup" class in puppet to the server
264
-to be backed up.
229
+Host backup happens via a daily cron job (managed by Ansible) on each
230
+individual host to be backed up.  The host to be backed up initiates
231
+the backup process to the remote backup server(s) using a separate ssh
232
+key setup just for backup communication (see ``/root/.ssh/config``).
265 233
 
266 234
 Restore from Backup
267 235
 -------------------
@@ -276,9 +244,14 @@ how we restore content from backups::
276 244
   mkdir /root/backup-restore-$DATE
277 245
   cd /root/backup-restore-$DATE
278 246
 
247
+Root uses a separate ssh key and remote user to communicate with the
248
+backup server(s); the username and key to use for backup should be
249
+automatically configured in ``/root/.ssh/config``.  The backup server
250
+hostname can be taken from there.
251
+
279 252
 At this point we can join the tar that was split by the backup cron::
280 253
 
281
-  bup join -r bup-<short-servername>@backup01.ord.rax.ci.openstack.org: root > backup.tar
254
+  bup join -r backup.x.y.opendev.org: root > backup.tar
282 255
 
283 256
 At this point you may need to wait a while. These backups are stored on
284 257
 servers geographically distant from our normal servers resulting in less

+ 15
- 0
playbooks/roles/backup-server/README.rst View File

@@ -0,0 +1,15 @@
1
+Setup backup server
2
+
3
+This role configures backup server(s) in the ``backup-server`` group
4
+to accept backups from remote hosts.
5
+
6
+Note that the ``backup`` role must have run on each host in the
7
+``backup`` group before this role.  That role will create a
8
+``bup_user`` tuple in the hostvars for for each host consisting of the
9
+required username and public key.
10
+
11
+Each required user gets a separate home directory in ``/opt/backups``.
12
+Their ``authorized_keys`` file is configured with the public key to
13
+allow the remote host to log in and only run ``bup``.
14
+
15
+**Role Variables**

+ 1
- 0
playbooks/roles/backup-server/defaults/main.yaml View File

@@ -0,0 +1 @@
1
+bup_users: []

+ 21
- 0
playbooks/roles/backup-server/tasks/main.yaml View File

@@ -0,0 +1,21 @@
1
+- name: Create backup directory
2
+  file:
3
+    state: directory
4
+    path: /opt/backups
5
+
6
+- name: Install bup
7
+  package:
8
+    name:
9
+      - bup
10
+    state: present
11
+
12
+- name: Build all bup users from backup hosts
13
+  set_fact:
14
+    bup_users: '{{ bup_users }} + [ {{ hostvars[item]["bup_user"] }} ]'
15
+  with_inventory_hostnames: backup
16
+
17
+- name: Create bup users
18
+  include_tasks: user.yaml
19
+  loop: '{{ bup_users }}'
20
+  loop_control:
21
+    loop_var: bup_user

+ 32
- 0
playbooks/roles/backup-server/tasks/user.yaml View File

@@ -0,0 +1,32 @@
1
+# note bup_user is the parent loop variable name; this works on each
2
+# element from the bup_users global.
3
+- name: Set variables
4
+  set_fact:
5
+    user_name: '{{ bup_user[0] }}'
6
+    user_key: '{{ bup_user[1] }}'
7
+
8
+- name: Create bup user
9
+  user:
10
+    name: '{{ user_name }}'
11
+    comment: 'Backup user'
12
+    shell: /bin/bash
13
+    home: '/opt/backups/{{ user_name }}'
14
+    create_home: yes
15
+  register: homedir
16
+
17
+- name: Create bup user authorized key
18
+  authorized_key:
19
+    user: '{{ user_name }}'
20
+    state: present
21
+    key: '{{ user_key }}'
22
+    key_options: 'command="BUP_DEBUG=0 BUP_FORCE_TTY=3 bup server",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty'
23
+
24
+# ansible-lint wants this in a handler, it should be done here and
25
+# now; this isn't like a service restart where multiple things might
26
+# call it.
27
+- name: Initalise bup  # noqa 503
28
+  shell: |
29
+    BUP_DIR=/opt/backups/{{ user_name }}/.bup bup init
30
+  become: yes
31
+  become_user: '{{ user_name }}'
32
+  when: homedir.changed

+ 23
- 0
playbooks/roles/backup/README.rst View File

@@ -0,0 +1,23 @@
1
+Configure a host to be backed up
2
+
3
+This role setups a host to use ``bup`` for backup to any hosts in the
4
+``backup-server`` group.
5
+
6
+A separate ssh key will be generated for root to connect to the backup
7
+server(s) and the host key for the backup servers will be accepted to
8
+the host.
9
+
10
+The ``bup`` tool is installed and a cron job is setup to run the
11
+backup periodically.
12
+
13
+Note the ``backup-server`` role must run after this to create the user
14
+correctly on the backup server.  This role sets a tuple ``bup_user``
15
+with the username and public key; the ``backup-server`` role uses this
16
+variable for each host in the ``backup`` group to initalise users.
17
+
18
+**Role Variables**
19
+
20
+.. zuul:rolevar:: bup_username
21
+
22
+   The username to connect to the backup server.  If this is left
23
+   undefined, it will be automatically set to ``bup-$(hostname)``

+ 22
- 0
playbooks/roles/backup/files/bup-excludes View File

@@ -0,0 +1,22 @@
1
+/proc/*
2
+/sys/*
3
+/dev/*
4
+/tmp/*
5
+/floppy/*
6
+/cdrom/*
7
+/var/spool/squid/*
8
+/var/spool/exim/*
9
+/media/*
10
+/mnt/*
11
+/var/agentx/*
12
+/run/*
13
+/root/backup-restore-*
14
+/root/.bup
15
+/etc/puppet/modules/*
16
+/etc/puppet/hieradata/*
17
+/var/cache/*
18
+/var/lib/puppet/reports/*
19
+/var/lib/postgresql/*
20
+/var/lib/lxcfs/*
21
+/opt/system-config/*
22
+/afs/*

+ 61
- 0
playbooks/roles/backup/tasks/main.yaml View File

@@ -0,0 +1,61 @@
1
+- name: Generate bup username for this host
2
+  set_fact:
3
+    bup_username: 'bup-{{ inventory_hostname.split(".", 1)[0] }}'
4
+  when: bup_username is not defined
5
+
6
+- debug:
7
+    var: bup_username
8
+
9
+- name: Install bup
10
+  package:
11
+    name:
12
+      - bup
13
+    state: present
14
+
15
+- name: Generate keypair for backups
16
+  openssh_keypair:
17
+    path: /root/.ssh/id_backup_ed25519
18
+    type: ed25519
19
+  register: bup_keypair
20
+
21
+- name: Initalise bup # noqa 503
22
+  command: bup init
23
+  when: bup_keypair.changed
24
+
25
+- name: Configure ssh for backup server
26
+  blockinfile:
27
+    path: /root/.ssh/ssh_config
28
+    create: true
29
+    block: |
30
+      Host {{ item }}
31
+      HostName {{ item }}
32
+      IdentityFile /root/.ssh/id_backup_ed25519
33
+      User {{ bup_username }}
34
+    mode: 0600
35
+  with_inventory_hostnames: backup-server
36
+
37
+- name: Generate bup_user info tuple
38
+  set_fact:
39
+    bup_user: '{{ [ bup_username, bup_keypair["public_key"] ] }}'
40
+
41
+- name: Accept hostkey of backup server
42
+  known_hosts:
43
+    state: present
44
+    key: '{{ item }} ecdsa-sha2-nistp256 {{ hostvars[item]["ansible_ssh_host_key_ed25519_public"] }}'
45
+    name: '{{ item }}'
46
+  with_inventory_hostnames: backup-server
47
+
48
+- name: Write /etc/bup-excludes
49
+  copy:
50
+    src: bup-excludes
51
+    dest: /etc/bup-excludes
52
+    mode: 0444
53
+
54
+- name: Install backup cron job
55
+  cron:
56
+    name: "Run bup backup"
57
+    job: "tar -X /etc/bup-excludes -cPF - / | bup split -r {{ bup_username }}@{{ item }}: -n root -q"
58
+    user: root
59
+    hour: '5'
60
+    minute: '{{ 59|random(seed=item) }}'
61
+  with_inventory_hostnames: backup-server

+ 10
- 0
playbooks/service-backup.yaml View File

@@ -0,0 +1,10 @@
1
+# This needs to happen in order.  Backup hosts export their username/key
2
+# combos which are installed onto the backup server
3
+- hosts: "backup:!disabled"
4
+  name: "Base: Generate backup users and keys"
5
+  roles:
6
+    - backup
7
+- hosts: "backup-server:!disabled"
8
+  name: "Generate bup configuration"
9
+  roles:
10
+    - backup-server

+ 2
- 0
playbooks/zuul/run-base.yaml View File

@@ -87,6 +87,8 @@
87 87
         - host_vars/letsencrypt02.opendev.org.yaml
88 88
         - host_vars/mirror01.openafs.provider.opendev.org.yaml
89 89
         - host_vars/mirror-update01.opendev.org.yaml
90
+        - host_vars/backup-test01.opendev.org.yaml
91
+        - host_vars/backup-test02.opendev.org.yaml
90 92
     - name: Display group membership
91 93
       command: ansible localhost -m debug -a 'var=groups'
92 94
     - name: Run base.yaml

+ 7
- 0
playbooks/zuul/templates/gate-groups.yaml.j2 View File

@@ -9,3 +9,10 @@ groups:
9 9
     - letsencrypt01.opendev.org
10 10
     - letsencrypt02.opendev.org
11 11
     - mirror01.openafs.provider.opendev.org
12
+
13
+  backup-server:
14
+    - backup01.region.provider.opendev.org
15
+
16
+  backup:
17
+    - backup-test01.opendev.org
18
+    - backup-test02.opendev.org

+ 1
- 0
playbooks/zuul/templates/host_vars/backup-test01.opendev.org.yaml.j2 View File

@@ -0,0 +1 @@
1
+bup_username: bup-backup01

+ 2
- 0
playbooks/zuul/templates/host_vars/backup-test02.opendev.org.yaml.j2 View File

@@ -0,0 +1,2 @@
1
+# Intentionally left blank to test autogeneration of name
2
+#bup_username: bup-backup-test02

+ 61
- 0
testinfra/test_backups.py View File

@@ -0,0 +1,61 @@
1
+# Copyright 2019 Red Hat, Inc.
2
+#
3
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
4
+# not use this file except in compliance with the License. You may obtain
5
+# a copy of the License at
6
+#
7
+#      http://www.apache.org/licenses/LICENSE-2.0
8
+#
9
+# Unless required by applicable law or agreed to in writing, software
10
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
11
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
12
+# License for the specific language governing permissions and limitations
13
+# under the License.
14
+
15
+import os.path
16
+import pytest
17
+
18
+testinfra_hosts = ['backup01.region.provider.opendev.org',
19
+                   'backup-test01.opendev.org',
20
+                   'backup-test02.opendev.org']
21
+
22
+
23
+def test_bup_installed(host):
24
+    package = host.package("bup")
25
+    assert package.is_installed
26
+
27
+def test_server_users(host):
28
+    hostname = host.backend.get_hostname()
29
+    if hostname.startswith('backup-test'):
30
+        pytest.skip()
31
+
32
+    for username in 'bup-backup01', 'bup-backup-test02':
33
+        homedir = os.path.join('/opt/backups/', username)
34
+        bup_config = os.path.join(homedir, '.bup', 'config')
35
+        authorized_keys = os.path.join(homedir, '.ssh', 'authorized_keys')
36
+
37
+        user = host.user(username)
38
+        assert user.exists
39
+        assert user.home == homedir
40
+
41
+        f = host.file(authorized_keys)
42
+        assert f.exists
43
+        assert f.contains("ssh-ed25519")
44
+
45
+        f = host.file(bup_config)
46
+        assert f.exists
47
+
48
+def test_backup_host_config(host):
49
+    hostname = host.backend.get_hostname()
50
+    if hostname == 'backup01.region.provider.opendev.org':
51
+        pytest.skip()
52
+
53
+    f = host.file('/root/.ssh/id_backup_ed25519')
54
+    assert f.exists
55
+
56
+    f = host.file('/root/.ssh/ssh_config')
57
+    assert f.exists
58
+    assert f.contains('Host backup01.region.provider.opendev.org')
59
+
60
+    f = host.file('/root/.bup/config')
61
+    assert f.exists

Loading…
Cancel
Save