kolla/specs/mariadb-backup-recovery.rst

MariaDB Backup and Recovery

Existing BP: https://blueprints.launchpad.net/kolla/+spec/database-backup-recovery

This blueprint attempts to outline the introduction of backup and recovery features in Kolla, for data hosted in MariaDB. It aims to do so by introducing tooling and options that are proven in deployments elsewhere, and with a degree of flexibility which facilitates integration with existing solutions.

Problem description

Kolla currently lacks an easy way for an operator to be able to take a backup of some or all of their MariaDB databases. Unrecoverable loss of data hosted in MariaDB can have disastrous consequences, so a feature which eases the introduction of a sensible backup and restore routine into an OpenStack operator's life is a worthwhile endeavour.

As backups are no use unless you can restore them, this solution should also include a feature - or at the very least, a documented set of steps - for an operator to be able to easily perform a restore and test the validity of their data.

As stated in the BP, general backup strategy should be considered out-of-scope as it's likely that each individual or organisation deploying OpenStack will have their own opinion on what should be done and the frequency with which these things should be performed. However, Kolla should at least offer a way to expose the necessary mechanisms to facilitate existing strategies.

Use cases

• As an operator, I wish to make an ad-hoc (on demand) backup of some or all MariaDB databases, prior to making any manual changes;
• As an operator, I'd like to include my MariaDB database(s) in the scope of my regularly scheduled backups, and would like to be able to do so via Kolla;
• As an operator, I want to be able to restore my database(s) to a particular point in time following a failed upgrade or a stray manual query.

For the first two use-cases, full and incremental backup options should be available.

Proposed change

There are several considerations as part of this proposed change. There's the tooling necessary to perform a backup, the ability to schedule backups, and the requirement to transfer the data elsewhere.

Backup Tooling

The linked Blueprint linked mentions the fact that there are several tools available which facilitate MariaDB backup (and restore). The most common is mysqldump, as this is included as standard with every installation and can be used to take a consistent backup of some or all databases. However, taking a backup with this tool has some limitations, chief amongst which is that it can have a significant performance impact when taking backups in a way that doesn't lock the database for the duration.

Instead, this proposed change will make use of Percona's XtraBackup tool, which has been designed specifically for 'hot-backups' avoiding locking and heavy performance impact. Because of the way XtraBackup functions, it also facilitates a simpler test / restore procedure as these are physical copies of the underlying database files, meaning a new instance of MariaDB can be spun up against these in order to test.

Percona provides pre-packaged binaries for this tool via their own mirrors in all of the major distributions supported by Kolla.

To implement this, this change will introduce a new Kolla container image hosting the XtraBackup binary plus dependencies necessary to be able to connect to MariaDB and retrieve data from some or all of the databases.

A Kolla-Ansible role will be created which will define tasks to:

• Establish the necessary backup-specific credentials;
• Start a container from this image with an associated volume and perform a full backup if no previous data exists, or an incremental backup if there is existing data. See below for a suggested default schedule.

The backup data will reside in a dedicated Docker volume. This can then be used to facilitate transfer of the data elsewhere (i.e mounted by another container with the tooling necessary to encrypt and upload) or be exported to another host for testing.

Backups will be performed by default locally, that is on the node currently running MariaDB, or on the designated master in a Galera cluster. However, it should be possible to nominate any node which has access to either the internal API address or the database node directly. It's up to the operator to choose which mode is best for them, as there are a number of different considerations and trade-offs to make. A new configuration option should be introduced to facilitate selection from a member of the MariaDB group.

Scheduling

Automatic scheduling of backups should be disabled by default, but Kolla could provide a mechanism to facilitate this if it's an operational requirement.

The approach described above doesn't introduce a new, persist container. Instead, it would be one which runs on demand and produces the target backup files.

Scheduling could be handled by changing this approach so that the container runs in perpetuity, with the localised backup scripts being triggered by cron according to a suggested (but configurable) default schedule. A proposed schedule would be:

• A full backup every 24 hours;
• An incremental backup every hour;
• Full backups are retained for two weeks;
• Incremental backups are retained for 24 hours.

Alternatively, backups could be triggered by another container or service running on the host.

Archival

Another tool is required to manage the backup lifecycle. This is currently considered out of scope.

Backup Restore

By targeting a discrete Docker volume for the data that's been backed up, facilitating a restore is relatively straightforward. Automating this is currently out of scope, but this piece of work should include an example procedure for how to handle this volume and access the data that's been backed up.

Security impact

Implementation of this BP will require the introduction of a dedicated backup role within MariaDB in order to give the tooling the necessary access. This will be read-only in nature and restricted to these specific privileges:

SELECT,RELOAD,LOCK TABLES,SHOW VIEW,REPLICATION CLIENT

Performance Impact

It's possible that there might be some performance degradation whilst taking a backup of a database node which has a significant amount of data, especially if the backup target device is the same as the source.

Aside from degradation incurred by way of I/O contention, the selection of XtraBackup is an attempt at mitigating any kind of performance impact.

Implementation

Assignee(s)

Primary assignee:

Nick Jones (yankcrime)

Work Items

1. Introduce a new Kolla image containing XtraBackup package plus dependencies such as scripts to handle triggering the backup;
2. Introduce a new Kolla-Ansible command and corresponding role to take a backup using a container launched from this image, saving data to a dedicated volume;
3. Documentation for new options and also restore process, along with examples.

Testing

Tests should be added to validate that a backup has been taken successfully with the default settings in place. This would take the form of starting another MariaDB container with the backup volume mounted as /var/lib/mysql and then performing some example queries to ensure expected data is returned.

Documentation Impact

Kolla and Kolla-Ansible documentation will need updating to introduce the new backup features and the various options that are available.

A dedicated and comprehensive section should be provide for restores, along with example scenarios.

References

[1] https://blueprints.launchpad.net/kolla/+spec/database-backup-recovery [2] https://etherpad.openstack.org/p/kolla-rocky-ptg-db-backup-restore [3] https://www.percona.com/doc/percona-xtrabackup/LATEST/index.html