borg-backup: send explicit email on backup failure

This sets a global BORG_UNDER_CRON=1 environment variable for
production hosts and makes the borg-backup script send an email if any
part of the backup job appears to fail (this avoids spamming ourselves
if we're testing backups, etc).

We should ideally never get this email, but if we do it's something we
want to investigate quickly.  There's nothing worse than thinking
backups are working when they aren't.

Change-Id: Ibb63f19817782c25a5929781b0f6342fe4c82cf0

playbooks/roles/borg-backup/tasks/main.yaml

@@ -53,6 +53,12 @@
     name: '{{ item }}'
   with_inventory_hostnames: borg-backup-server
 
+- name: Set cron flag to enable error reports
+  cron:
+    name: BORG_UNDER_CRON
+    env: yes
+    job: '1'
+
 - name: Install backup cron job
   cron:
     name: "Run borg backup to {{ item }}"

playbooks/roles/borg-backup/templates/borg-backup.j2

@@ -60,7 +60,12 @@ if [ ${backup_exit} -eq 0 ]; then
     info "Backup finished successfully"
 else
     info "Backup finished with errors"
+    if [ ${BORG_UNDER_CRON:-0} -eq 1 ]; then
+        echo "Backups failed on host $(hostname) at $(date)." | \
+            mail -s "ACTION REQUIRED: Backup failed on $(hostname)" infra-root@openstack.org
+    fi
 fi
 
+
 exit ${backup_exit}