263 Commits

Author SHA1 Message Date
Jeremy Stanley
fd98a1750d Clean up OpenEdge configuration
The OpenEdge cloud has been offline for five months, initially
disabled in I4e46c782a63279d9c18ff4ba2944c15b3027114b, so go ahead
and clean up lingering references. If it is restored later, this can
be reverted fairly easily.

Depends-On: https://review.opendev.org/783989
Depends-On: https://review.opendev.org/783990
Change-Id: I544895003344bc8202363993b52f978e1c07d061
2021-03-31 01:42:36 +00:00
Zuul
27d58d3b57 Merge "Add review02.opendev.org" 2021-03-30 00:48:54 +00:00
Zuul
3180086559 Merge "Rename refstack group variables" 2021-03-29 21:33:02 +00:00
Ian Wienand
525d5d1c19 Add review02.opendev.org
review02.opendev.org is a much larger replacement server for review01
provided by Vexxhost.  It is up and running, with gerrit2 volume
attached and DNS entries.

This adds it to the staging group with no replication and a local h2
database configured for initial bringup.  There's quite a bit to
consider for full migration, but this will let us start experimenting.

Change-Id: I3638a5c0c7028dcc800ada42431b75395cff0c42
2021-03-26 14:53:31 +11:00
Ian Wienand
163d5b6133 Create review-staging group
Create a review-staging group so we can bring up a new server but
avoid running the project-management steps on it.

Change-Id: I93d2a36edcd58a48a36031f0692be3273a36f07c
2021-03-24 11:40:33 +11:00
Ian Wienand
9f11fc5c75 Remove references to review-dev
With our increased ability to test in the gate, there's not much use
for review-dev any more.  Remove references.

Change-Id: I97e9865e0b655cd157acf9ffa7d067b150e6fc72
2021-03-24 11:40:31 +11:00
Ian Wienand
aa94f2d831 Rename refstack group variables
When we cleaned up the puppet in
I6b6dfd0f8ef89a5362f64cfbc8016ba5b1a346b3 we renamed the group
s/refstack-docker/refstack/ but didn't move the variables and some
other references too.

Change-Id: Ib07d1e9ede628c43b4d5d94b64ec35c101e11be8
2021-03-19 16:01:46 +11:00
Zuul
b8874e4f51 Merge "kerberos-kdc: add database backups" 2021-03-19 00:06:59 +00:00
Zuul
f917044497 Merge "kerberos-kdc: add realm value" 2021-03-18 05:45:01 +00:00
Ian Wienand
ef62e1df31 kerberos-kdc: add realm value
I missed this in the production variables as it is set differently for
testing.

Change-Id: Ie9508cbcb11f8b342f05c98e8e85bc158e5ee4c1
2021-03-18 16:04:51 +11:00
Zuul
99a05bdf75 Merge "Add kerberos-client group" 2021-03-18 02:43:59 +00:00
Ian Wienand
dc827de23d Add kerberos-client group
We duplicate the KDC settings over all our kerberos clients.  Add
clients to a "kerberos-client" group and set the variables in a group
file.

Change-Id: I25ed5f8c68065060205dfbb634c6558488003a38
2021-03-18 11:59:30 +11:00
James E. Blair
96bac7b486 Add zookeeper-statsd
This adds a program, zookeeper-statsd, which monitors zookeeper
metrics and reports them to statsd.  It also adds a container to
run that program.  And it runs the container on each of the
ZooKeeper quorum members.  And it updates the graphite host to
allow statsd traffic from quorum members.  And it updates the
4-letter-word whitelist to allow the mntr command (which is used
to gather metrics) to be issued.

Change-Id: I298f0b13a05cc615d8496edd4622438507fc5423
2021-03-17 14:52:31 -07:00
Zuul
df9a85e45c Merge "kerberos: switch servers to Ansible control" 2021-03-17 04:03:03 +00:00
Zuul
4524a92caf Merge "kerberos-kdc: role to manage Kerberos KDC servers" 2021-03-16 22:28:46 +00:00
Zuul
b133afedfd Merge "refstack: cleanup old puppet" 2021-03-16 22:21:03 +00:00
Ian Wienand
3052ff4935 kerberos-kdc: add database backups
Add a script to save a db dump to borg backups.  Add the primary KDC
to our backup list.

Change-Id: I32f4ebc1bb4c1952034aba43c75e4d2f85a1b6d3
2021-03-17 08:31:52 +11:00
Ian Wienand
2254b6e43d kerberos: switch servers to Ansible control
This is a follow-on to I60b40897486b29beafc76025790c501b5055313d to
switch the KDC servers to Ansible control and remove any related
puppet configuration.

Change-Id: Ib8f6ec657ca10a3ba648bd154a035fc3d8da4be5
2021-03-17 08:30:52 +11:00
Ian Wienand
c1aff2ed38 kerberos-kdc: role to manage Kerberos KDC servers
This adds a role and related testing to manage our Kerberos KDC
servers, intended to replace the puppet modules currently performing
this task.

This role automates realm creation, initial setup, key material
distribution and replica host configuration.  None of this is intended
to run on the production servers which are already setup with an
active database, and the role should be effectively idempotent in
production.

Note that this does not yet switch the production servers into the new
groups; this can be done in a separate step under controlled
conditions and with related upgrades of the host OS to Focal.

Change-Id: I60b40897486b29beafc76025790c501b5055313d
2021-03-17 08:30:52 +11:00
Ian Wienand
018a14e34f refstack: cleanup old puppet
Remove old puppet configuration for the restack service, which is now
managed by Ansible.

Change-Id: I6b6dfd0f8ef89a5362f64cfbc8016ba5b1a346b3
2021-03-17 07:06:53 +11:00
Ian Wienand
ea48ffc596 refstack: fix backup script typo
This got copied from another command that also had this typo.

Also, don't bother backing up the on-disk backups, as we backup
directly via the stream dumps.

Change-Id: Ie200a29eec2b1a0725a8872ab548bcb0f26980e6
2021-03-16 15:12:41 +11:00
Martin Kopec
a5a0e5faba refstack: Fix openid endpoint
openid_endpoint was set to just base site url which is not
correct, it should be https://openstackid.org/accounts/openid2

Change-Id: I6624150f1ab78560347c8f82a13394b164860cad
2021-03-12 15:02:03 +00:00
Ian Wienand
753f9520e6 refstack: add backup
We should be backing up the user-generated refstack data

Change-Id: I1bd5f0de283a4436967dcae6da9c5d9cd055697c
2021-03-12 15:18:04 +11:00
Ian Wienand
d33ce951c0 refstack: use CNAME for production server
The production server is trying to send itself to
refstack01.openstack.org, causing cross-site scripting issues.  In
production, use the CNAME, but use the FQDN for testing.

Fix up job file matchers while here.

Change-Id: I18a5067ee25c59c5eaa17b7c2d9bd5a942a9173d
2021-03-12 10:24:06 +11:00
Zuul
8d67151838 Merge "Remove ze01.openstack.org" 2021-03-03 21:54:05 +00:00
Clark Boylan
a42c0b704a Remove ze01.openstack.org
This server has been replaced by ze01.opendev.org running Focal. Lets
remove the old ze01.openstack.org from inventory so that we can delete
the server. We will follow this up with a rotation of new focal servers
being put in place.

This also renames the xenial executor in testing to ze12.openstack.org
as that will be the last one to be rotated out in production. We will
remove it from testing at that point as well.

We also remove a completely unused zuul-executor-opendev.yaml group_vars
file to avoid confusion.

Change-Id: Ida9c9a5a11578d32a6de2434a41b5d3c54fb7e0c
2021-03-02 10:21:59 -08:00
Ian Wienand
fdd41cb850 Remove afs-admin group
This group no longer does anything.  This used to deploy a bunch of
keytabs for mirror-update, but that has all moved into
"mirror_update_keytab_*".

Change-Id: I3e2110a621d6946bc4838bfa2f743f0e9db391f3
2021-03-02 11:54:51 +11:00
Ian Wienand
c27915c3a7 translate: fix backup extras match
This should be called "_extra" ... currently it overrides the default
exclude list.  This means /var/lxcfs gets incorrectly included in the
backup and makes it error out as it has sockets and weird stuff that
can't be backed up; this is why we are getting failure mail.

Change-Id: Idea70c32b2d42f77fee2b35487d88a8ee982c856
2021-02-23 02:00:34 +00:00
Ian Wienand
39ffc685d6 backups: remove all bup
All hosts are now running thier backups via borg to servers in
vexxhost and rax.ord.

For reference, the servers being backed up at this time are:

 borg-ask01
 borg-ethercalc02
 borg-etherpad01
 borg-gitea01
 borg-lists
 borg-review-dev01
 borg-review01
 borg-storyboard01
 borg-translate01
 borg-wiki-update-test
 borg-zuul01

This removes the old bup backup hosts, the no-longer used ansible
roles for the bup backup server and client roles, and any remaining
bup related configuration.

For simplicity, we will remove any remaining bup cron jobs on the
above servers manually after this merges.

Change-Id: I32554ca857a81ae8a250ce082421a7ede460ea3c
2021-02-16 16:00:28 +11:00
Zuul
60b5f789ad Merge "Clean up ethercalc server replacement transition" 2021-02-15 22:20:10 +00:00
Jeremy Stanley
6d0c4b0b3b Update AFS group vars filenames
Ifa5f251fdfb8de737ad2ed96491d45294ce23a0c renamed the afs and afsdb
groups to afs-file-server and afs-db-server, but didn't update the
group files.

Previously the firewall rules were duplicated in the afs/afsdb group;
but now all afs servers are in the afs-server-common group.  Rename
afs.yaml->afs-server-common.yaml and remove the now unnecessary
afsdb.yaml.

Remove one of the old group vars files and rename the other to
afs-server-common so we can restore the udp ports they open in our
firewall rules.

Change-Id: I17dd0596660addf061ade31b4450bf040c01ffe8
2021-02-12 18:23:45 +11:00
Zuul
036ac31060 Merge "Refactor AFS groups" 2021-02-11 22:46:00 +00:00
Ian Wienand
312b9bec24 Refactor AFS groups
Both the filesevers and db servers have common key material deployed
by the openafs-server-config role.  Put both types of server in a new
group "afs-server-common" so we can define this key material in just
one group file on bridge.

Then separate out the two into afs-<file|db>-server groups for
consistent naming.

Rename afs-admin for consistent naming.

The service file is updated to reflect the new groups.

Change-Id: Ifa5f251fdfb8de737ad2ed96491d45294ce23a0c
2021-02-11 13:35:16 +11:00
Ian Wienand
32b48c81a2 refstack: use external https for API
Currently this variable is setting several URL's used in the config to
internal http links (port 8000).  This bubbles through to the UI which
then can't talk to the API.  Emperically, changing these values in the
container config and restarting it makes things work.  Update this
variable to make it talk to external https.

Change-Id: If61ec1e0383b98d34d092c55ca0095588487902a
2021-02-11 11:44:39 +11:00
Ian Wienand
5a7511f6a6 refstack: move non-private variables to public
These two variables can be deployed via system-config

Change-Id: If696945d7b01ee42eb822d2391405277eb6c23d3
2021-02-10 07:10:39 +11:00
Zuul
f526060e39 Merge "Deploy refstack with ansible docker" 2021-02-09 03:58:22 +00:00
Clark Boylan
a4604ae0b3 Deploy refstack with ansible docker
This adds a dockerfile to build an opendevorg/refstack image as well as
the jobs to build and publish it.

Change-Id: Icade6c713fa9bf6ab508fd4d8d65debada2ddb30
2021-02-05 19:23:34 +00:00
Ian Wienand
56277bf70a ask: fix backup typo and ignore live postgresql
This was overriding the main list of ignores; also ignore the live db.

Change-Id: Idf5ae8e88805829ee44e7f4ba003ac086f5f1206
2021-02-05 17:40:02 +11:00
Ian Wienand
01990670c9 translate: backup zanata db directly to borg
As noted inline, a recent mysql client update has broken the
"--all-databases" flag, at least for the client version and very old
server version we use.

Emperically, dumping individual databases still works with this
client.  Switch this to stream the db directly into borg.

Ignore the old backups and remove the bup backup while we are here,
since this is all borg now.

Change-Id: I5fe762a003ce2c2ba4830367be87598f67f7e763
2021-02-05 14:05:24 +11:00
Ian Wienand
f9184ce323 ask: stream db backup
Despite be deprecated, the ask server is our 3rd biggest backup.  Even
though the site is R/O we're still backing up the fresh rotations of
the gzipped backups every day.

To reduce the incremental space requirements, move to our plain-text
streaming for the db backup.  This just needs a file dropped in /etc;
see the backup-borg role README documentation.  We do this in puppet
to avoid complexity adding this deprecated service to ansible.  This
then excludes the on-disk db backup dir.

Drop the bup backups while we are here.

Change-Id: Icfd81aca58b9a0dc3a3b74de04c1b00f03160327
2021-02-05 13:24:57 +11:00
Zuul
89cd6972f2 Merge "borg-backup: implement saving a stream, use for database backups" 2021-02-03 03:11:11 +00:00
Zuul
70bd9166f7 Merge "Manage afsdb servers with Ansible" 2021-02-03 02:03:28 +00:00
Ian Wienand
51733e5623 borg-backup: implement saving a stream, use for database backups
Add facility to borg-backup role to run a command and save the output
of it to a separate archive file during the backup process.

This is mostly useful for database backups.  Compressed on-disk logs
are terrible for differential backups because revisions have
essentially no common data.  By saving the uncompressed stream
directly from mysqldump, we allow borg the chance to de-duplicate,
saving considerable space on the backup servers.

This is implemented for our ansible-managed servers currently doing
dumps.  We also add it to the testinfra.

This also separates the archive names for the filesystem and stream
backup with unique prefixes so they can be pruned separately.
Otherwise we end up keeping only one of the stream or filesystem
backups which isn't the intention.  However, due to issues with
--append-only mode we are not issuing prune commands at this time.

Note the updated dump commands are updated slightly, particularly with
"--skip-extended-insert" which was suggested by mordred and
significantly improves incremental diff-ability by being slightly more
verbose but keeping much more of the output stable across dumps.

Change-Id: I500062c1c52c74a567621df9aaa716de804ffae7
2021-02-03 11:43:12 +11:00
Zuul
e762fd3677 Merge "gitea backup: prune some large directories" 2021-01-21 00:22:06 +00:00
Ian Wienand
c98505c8f2 Manage afsdb servers with Ansible
Move common setup steps into a openafs-server-config role, and create
openafs-file-server and openafs-db-server roles to manage fileserver
and db servers respectively.

Modify the playbook to run these roles against the AFS servers.

Change-Id: I4e80ad8ffe1d4992e405ea516b8762109758d7eb
2021-01-21 07:08:37 +11:00
Ian Wienand
92250eca82 Remove afs-1.8 group
With all AFS file-servers upgraded to 1.8, we can move afs01.dfw back
and rename the group to just "afs".

Change-Id: Ib31bde124e01cd07d6ff7eb31679c55728b95222
2021-01-21 07:08:29 +11:00
Ian Wienand
99a36d790e gitea backup: prune some large directories
It's not necessary to capture the live db or git trees, so prune these
from the backups.

Change-Id: I7a27c49035eb0590d0157766eb3392a0f6331aea
2021-01-20 16:01:16 +11:00
Ian Wienand
60a7bfc5f6 Move afs02.dfw.openstack.org to afs-1.8 group
This host is now running OpenAFS 1.8 and should be Ansible managed
now.

Change-Id: Ia0cf0672f3e924a3b6d8e337d3355f6216796e92
2021-01-19 09:34:26 +11:00
Ian Wienand
7683fa11b3 openafs-server : add ansible roles for OpenAFS servers
This starts at migrating OpenAFS server setup to Ansible.

Firstly we split up the groups and explicitly name hosts, as we will
me migrating each one step-by-step.  We split out 1.8 hosts into a new
afs-1.8 group; the first host is afs01.ord.openstack.org which already
has openafs 1.8 installed manually.

An openafs-server role is introduced that does the same setup as the
extant puppet.

The AFS job is renamed to infra-prod-afs as the puppet component will
eventually disappear.  Otherwise it runs in the same way, but also
runs the openafs-server role for the 1.8 servers.

Once this is merged, we can run it against afs01.ord.openstack.org to
ensure it works and is idempotent.  We can then take on upgrading the
other file servers, and work further on the database servers.

Change-Id: I7998af43961999412f58a78214f4b5387713d30e
2021-01-19 08:08:33 +11:00
Jeremy Stanley
7d48d972b5 Clean up ethercalc server replacement transition
The old ethercalc01 server has been deleted as have its DNS entries.
Belatedly update cacti to query the new server, and remove an old
unused reference which was at one time disabling the former server.

Change-Id: Ide70c7d03bfff5bd695272c696913dfb3decc525
2021-01-05 16:27:09 +00:00