Both the filesevers and db servers have common key material deployed
by the openafs-server-config role. Put both types of server in a new
group "afs-server-common" so we can define this key material in just
one group file on bridge.
Then separate out the two into afs-<file|db>-server groups for
consistent naming.
Rename afs-admin for consistent naming.
The service file is updated to reflect the new groups.
Change-Id: Ifa5f251fdfb8de737ad2ed96491d45294ce23a0c
We have setup rsyslogd/logrotate to handle anything with docker- tags
to be persisted to disk in /var/log/containers. Set this up here so
we keep track of the mariadb and refstack logs.
Change-Id: I760cfeb7226f79986fbf9d7dbc5f899fc87a0cd1
The mariadb container currently doesn't persist it's database
anywhere. Map /var/lib/refstack/db to /var/lib/mysql in the
container.
We have /var/refstack and /var/lib/refstack with various things.
While we're here move everythign under /var/lib/refstack.
Also use 127.0.0.1 to ensure mysql doesn't try to connect over a
socket, but tcp (I think pymsql does anyway, but it's a little
clearer).
Change-Id: I5605eac2848a6b2222698bf20c707baa4442fcd5
This slipped in with I4e80ad8ffe1d4992e405ea516b8762109758d7eb; it
should be openafs, not openstack.
Change-Id: Iefc41f9085d86e9fdaa13c6e5b90f1c99b7a2d83
It is buggy (throwing exceptions for undefinied variables which are
actualyl defined via set_fact), and we frequently run into problems
using it in this repo. It was designed to lint roles for Galaxy,
not the way we write ansible. As of the 5.0.0 release it's
generating >4.5K lines of complaints about files in this repository.
Change-Id: If9d8c19b5e663bdd6b6f35ffed88db3cff3d79f8
It's not obvious, but the if statements can change the PIPESTATUS
meaning we're not matching what we think we're matching. Save the
pipestatus of the backup commands so we exit the backup script with
the right code.
Change-Id: I83c7db45d3622067eb05107e26fbdc7a8aeecf63
Update the backup instructions for some recent changes. Make a note
of the streaming backup method, discuss some caveats with append-only
mode and discuss the pruning scripts and when to run
(c.f. I9559bb8aeeef06b95fb9e172a2c5bfb5be5b480e,
I250d84c4a9f707e63fef6f70cfdcc1fb7807d3a7).
Change-Id: Idb04ebfa5666cd3c20bc0132683d187e705da3f1
Due to backups running in append-only mode, we do not have a way to
safely automatically prune backups. To reduce the likelyhood we
forget about backups and end up with failing jobs, add a cron job to
send a email to infra-root if the backup partition goes over 90%
usage. At this point a manual prune should be run
(I9559bb8aeeef06b95fb9e172a2c5bfb5be5b480e).
Change-Id: I250d84c4a9f707e63fef6f70cfdcc1fb7807d3a7
Due to [1] --all-databases is no longer working with our version of
database. Move to explicitly backing up the only database we care
about now, which is accountPatchReviewDb; everything else is in
notedb.
[1] https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1914695
Change-Id: Iab2a8ab612cc0a0f10c90123f2936c0abda9e76f
openstack-forum, openstack-ptg, openstack-summit and openinfra-summit
will now be openinfra-events.
openstack-diversity will now be openinfra-diversity.
openstack-foundation will now be openinfra.
openstack-board will come later after we can get op rights and
register the channel.
Depends-On: https://review.opendev.org/c/openstack/project-config/+/774550
Change-Id: I6f3458d61800aad22f6e723a2d8b300460a0a03e
This adds a dockerfile to build an opendevorg/refstack image as well as
the jobs to build and publish it.
Change-Id: Icade6c713fa9bf6ab508fd4d8d65debada2ddb30
As noted inline, a recent mysql client update has broken the
"--all-databases" flag, at least for the client version and very old
server version we use.
Emperically, dumping individual databases still works with this
client. Switch this to stream the db directly into borg.
Ignore the old backups and remove the bup backup while we are here,
since this is all borg now.
Change-Id: I5fe762a003ce2c2ba4830367be87598f67f7e763
Despite be deprecated, the ask server is our 3rd biggest backup. Even
though the site is R/O we're still backing up the fresh rotations of
the gzipped backups every day.
To reduce the incremental space requirements, move to our plain-text
streaming for the db backup. This just needs a file dropped in /etc;
see the backup-borg role README documentation. We do this in puppet
to avoid complexity adding this deprecated service to ansible. This
then excludes the on-disk db backup dir.
Drop the bup backups while we are here.
Change-Id: Icfd81aca58b9a0dc3a3b74de04c1b00f03160327
These were gleaned from looking at what files are taking up space in
the deltas of backups. Nothing major, but mlocate in partiuclar is
taking up to a couple of hundred mb on some servers.
Change-Id: I4b08c4e2491fa7138045aabcb23017ff8cef7600
The floating IP of this host was changed during a network issue;
matches I898dbf7417fb01f608eded85faaae5a417ad2e98
Change-Id: Icf1daa4a761403a3927bcadab08656cd1f42f1aa
Because the bulk of this traffic originates with our load balancer we
need to use port info to differentiate between actual source clients in
the load balancer logs. That info is currently missing so add it in.
Change-Id: I737e6373c09669f0321b656ecd4b137b94be38a4
Add facility to borg-backup role to run a command and save the output
of it to a separate archive file during the backup process.
This is mostly useful for database backups. Compressed on-disk logs
are terrible for differential backups because revisions have
essentially no common data. By saving the uncompressed stream
directly from mysqldump, we allow borg the chance to de-duplicate,
saving considerable space on the backup servers.
This is implemented for our ansible-managed servers currently doing
dumps. We also add it to the testinfra.
This also separates the archive names for the filesystem and stream
backup with unique prefixes so they can be pruned separately.
Otherwise we end up keeping only one of the stream or filesystem
backups which isn't the intention. However, due to issues with
--append-only mode we are not issuing prune commands at this time.
Note the updated dump commands are updated slightly, particularly with
"--skip-extended-insert" which was suggested by mordred and
significantly improves incremental diff-ability by being slightly more
verbose but keeping much more of the output stable across dumps.
Change-Id: I500062c1c52c74a567621df9aaa716de804ffae7
Previously the test was checking that stderr reported "Cloning into
$PATH" which also happens in failure cases. We add an explicit check for
a successful command return code to ensure that we aren't failing with
that output.
Change-Id: Iec51217f2cc97e6a56ff9d8b7a260650010f229f
Ian Wienand