79 Commits

Author SHA1 Message Date
Jeremy Stanley
89c4fd9b3d Remove configuration management for wiki servers
We never finished puppeting the OpenStack wiki, and if we do manage
to get it under configuration management in the future it will
likely not use Puppet anyway. The dev server is already gone, and
deployment has been explicitly disabled for the other, so let's go
ahead and remove the references here and then we should be able to
retire the separate Puppet module we've been hosting.

Change-Id: I3f9ada3eb3d6f16545270135fab994ac460be94b
2022-02-14 22:32:18 +00:00
James E. Blair
2a9553ef25 Add Zuul load balancer
This adds a load balancer for zuul-web and fingergw.

Change-Id: Id5aa01151f64f3c85e1532ad66999ef9471c5896
2022-02-10 13:24:42 -08:00
Jeremy Stanley
f469f1189e Drop wiki-dev03 from inventory
The wiki-dev03.openstack.org server was a test deployment working
through completing the puppetry for our Mediawiki environment. Since
it's on a now-EoL Ubuntu version, and that configuration management
work has stalled, delete this test server from our inventory rather
than needlessly consuming resources and an ESM entitlement.

Also clean up an old disabled entry for wiki-dev01.openstack.org
which no longer exists (it was a predecessor of this server). Leave
the templating for wiki-dev* in place for now in case we decide to
launch a replacement.

Change-Id: I5beed4dde8e4e84d92f510f8726f8443daf774c1
2022-01-27 16:40:26 +00:00
Jeremy Stanley
fa0c1b495c Generate HTTPS certs for Mailman sites
We're going to want Mailman 3 served over HTTPS for security
reasons, so start by generating certificates for each of the sites
we have in v2. Also collect the acme.sh logs for verification.

Change-Id: I261ae55c6bc0a414beb473abcb30f9a86c63db85
2021-12-17 22:25:22 +00:00
James E. Blair
e79dbbe6bb Add a keycloak server
This adds a keycloak server so we can start experimenting with it.

It's based on the docker-compose file Matthieu made for Zuul
(see https://review.opendev.org/819745 )

We should be able to configure a realm and federate with openstackid
and other providers as described in the opendev auth spec.  However,
I am unable to test federation with openstackid due its inability to
configure an oauth app at "localhost".  Therefore, we will need an
actual deployed system to test it.  This should allow us to do so.

It will also allow use to connect realms to the newly available
Zuul admin api on opendev.

It should be possible to configure the realm the way we want, then
export its configuration into a JSON file and then have our playbooks
or the docker-compose file import it.  That would allow us to drive
change to the configuration of the system through code review.  Because
of the above limitation with openstackid, I think we should regard the
current implementation as experimental.  Once we have a realm
configuration that we like (which we will create using the GUI), we
can chose to either continue to maintain the config with the GUI and
appropriate file backups, or switch to a gitops model based on an
export.

My understanding is that all the data (realms configuration and session)
are kept in an H2 database.  This is probably sufficient for now and even
production use with Zuul, but we should probably switch to mariadb before
any heavy (eg gerrit, etc) production use.

This is a partial implementation of https://docs.opendev.org/opendev/infra-specs/latest/specs/central-auth.html

We can re-deploy with a new domain when it exists.

Change-Id: I2e069b1b220dbd3e0a5754ac094c2b296c141753
Co-Authored-By: Matthieu Huin <mhuin@redhat.com>
2021-12-03 14:17:23 -08:00
Clark Boylan
cf91bc0971 Remove the gerrit group in favor of the review group
Having two groups here was confusing. We seem to use the review group
for most ansible stuff so we prefer that one. We move contents of the
gerrit group_vars into the review group_vars and then clean up the use
of the old group vars file.

Change-Id: I7fa7467f703f5cec075e8e60472868c60ac031f7
2021-10-12 09:48:53 -07:00
Clark Boylan
63f5674e6f Switch test gerrit hostname to review99.opendev.org
Previously we had set up the test gerrit instance to use the same
hostname as production: review02.opendev.org. This causes some confusion
as we have to override settings specifically for testing like a reduced
heap size, but then also copy settings from the prod host vars as we
override the host vars entirely. Using a new hostname allows us to use a
different set of host vars with unique values reducing confusion.

Change-Id: I4b95bbe1bde29228164a66f2d3b648062423e294
2021-10-12 09:48:53 -07:00
Jeremy Stanley
2fbf6d9e7a Stop managing OpenStackID servers
The Open Infrastructure Foundation's developers who maintain the
OpenStackID software are taking over management of the site itself,
and have deployed it on new servers. DNS records have already been
updated to the new IP address, so it's time to clean up our end in
preparation for deleting the old servers we've been running.

OpenStackID is still used by some services we run, like RefStack and
Zanata, and we're still hosting the OpenStackID Git repository and
documentation, so this does not get rid of all references to it.

Change-Id: I1d625d5204f1e9e3a85ba9605465f6ebb9433021
2021-08-31 19:53:13 +00:00
James E. Blair
8d76a7cd99 Test port 9001 on eavesdrop
We merged change I9459e47ecfd19b27b7adcaee9ce91f80d51c124d which
should have opened this port but did not.  Add testing for it.

Remove eavesdrop from webservers group

This was overridding the custom iptables ports that were being set
in the eavesdrop group vars file.  There appears to be no other use
for the webservers group.

Change-Id: I7109f1472176ff39482f9bdfc8462e5f525f791c
2021-08-11 14:20:41 -07:00
Ian Wienand
c1278d18bb Remove review-test
With our system-config-run gerrit/review jobs we have much less need
for a dedicated server to stage changes on.  Remove in prepartion of
server cleanup.

Change-Id: I9430f7a2432324a184e3a4f7e41f9e5150c0200c
2021-07-21 13:12:43 +10:00
Ian Wienand
0142bc10eb backups: add review02.opendev.org
Start backing up the new review server.  Stop backing up the old
server.  Fix the group matching test for the new server.

Change-Id: I8d84b80099d5c4ff7630aca9df312eb388665b86
2021-07-19 15:29:42 +10:00
Zuul
f1b559bb7a Merge "review02: move out of staging group" 2021-07-19 04:49:37 +00:00
Ian Wienand
8607ff7d81 review02: move out of staging group
This moves review02 out of the review-staging group and into the main
review group.  At this point, review01.openstack.org is inactive so we
can remove all references to openstack.org from the groups.  We update
the system-config job to run against a focal production server, and
remove the unneeded rsync setup used to move data.

This additionally enables replication; this should be a no-op when
applied as part of the transition process is to manually apply this,
so that DNS setup can pull zone changes from opendev.org.

It also switches to the mysql connector, as noted inline we found some
issues with mariadb.

Note backups follow in a separate step to avoid doing too much at
once, hence dropping the backup group from the testing list.

Change-Id: I7ee3e3051ea8f3237fd5f6bf1dcc3e5996c16d10
2021-07-18 19:45:35 -07:00
Zuul
b895af4d35 Merge "Remove paste01.openstack.org" 2021-07-16 03:03:50 +00:00
Ian Wienand
5e52befdfa Remove paste01.openstack.org
This has been replaced by paste01.opendev.org and Ansible deployment.

Change-Id: I0f8f5374a3f5d269b317bde4ae2b37435e0871d5
2021-07-15 23:25:10 +00:00
Ian Wienand
d4c613a07a Add paste01.opendev.org to backup
Change-Id: Iec6b916bd27a5333d28d1fdc931d4f41165bf50c
2021-07-15 15:02:52 +10:00
Ian Wienand
916c1d3dc8 Add paste service
The paste service needs an upgrade; since others have created a
lodgeit container it seems worth us keeping the service going if only
to maintain the historical corpus of pastes.

This adds the ansible to deploy lodgeit and a sibling mariadb
container.  I have imported a dump of the old data as a test.  The
dump is ~4gb and imported it takes up about double that; certainly
nothing we need to be too concerned over.  The server will be more
than capable of running the db container alongside the lodgeit
instance.

This should have no effect on production until we decide to switch
DNS.

Change-Id: I284864217aa49d664ddc3ebdc800383b2d7e00e3
2021-07-07 15:12:04 +10:00
Ian Wienand
0e9b950086 Add eavesdrop01.opendev.org to backup group
This saves a copy of our channel/meeting logs.

Change-Id: I376d1426573416ff0c2e633fa40e4d93adc89483
2021-06-23 10:48:38 +10:00
Zuul
084879c1fa Merge "limnoria/meetbot setup on eavesdrop01.opendev.org" 2021-06-10 02:04:53 +00:00
Ian Wienand
403773d55a limnoria/meetbot setup on eavesdrop01.opendev.org
This installs our Limnoira/meetbot container and configures it on
eavesdrop01.opendev.org.  I have ported the configuration from the old
puppet as best I can (it is very verbose); my procedure was to use the
Limnoira wizard to start a new config file then backport everything
from the old file.  I felt this was best to not miss any new options.

This does channel logging (via built-in ChannelLogger plugin, along
with a cron job for logs2html) and runs our fork of meetbot.

It exports the channel logs via HTTP to /irclogs and meetings logs to
/meetings.  meetings.opendev.org will proxy to these two locations
when the server is active.

Note this has not ported the channel list; so the bot will not be
listening in our channels.

Change-Id: I9f9a466c271e1a706f9f98f816de0e84047519f1
2021-06-10 09:02:16 +10:00
Zuul
632b2f9df7 Merge "Cleanup ask.openstack.org" 2021-06-09 05:42:26 +00:00
Ian Wienand
f66efc0d9c Restore eavesdrop01.openstack.org to webservers group
This host is no longer under puppet control, but should still be a
webserver to export the logs it is still collecting until we finish
moving that to the new server.  Restore the match to open*

See I809f9af3e78f566362142790f6c79654ef5b8959

Change-Id: I524c0a7c5cc93313c180eca68b67a0f0582474df
2021-06-08 16:07:55 +10:00
Ian Wienand
7de885b5ee Cleanup ask.openstack.org
This was retired with I8a31f8fcf9b3064c0ae58e463a6014dc14b518a7

Change-Id: Ieafac856b0feb91f41f05084aa669e2ccb92569d
2021-06-08 14:35:28 +10:00
Ian Wienand
fec8018581 Move gerritbot/accessbot to new eavesdrop server
This moves these services to eavesdrop01.opendev.org, a new
Focal-based server to host IRC services.

We have stopped running puppet on eavesdrop01.openstack.org so there
is nothing left for it to do (note the server is still running
meetbot/ptgbot).  Remove the commented out puppet run, and remove the
server from puppet groups.  Update the host in the Zuul jobs to the
new node.

Change-Id: I809f9af3e78f566362142790f6c79654ef5b8959
2021-06-08 08:16:56 +10:00
Ian Wienand
fb94b79e82 Add eavesdrop01.opendev.org server
This adds a new server to take over from eavesdrop01.openstack.org.

We limit the puppet installs, etc. to the openstack.org server.  The
new server is in the group eavesdrop_opendev as we cut over services.
A stub for basic installation is added to the service playbook.

Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/795004
Change-Id: I88c3059532e4d6ab267fdec5b390daefa5b0c4a1
2021-06-07 12:59:02 +10:00
Clark Boylan
399ade787b More puppetry and inventory cleanups
This cleans up ask-staging which hasn't been a thing in a log time.
We remove some puppet stubs for nodepool builders (they are all ansible
now).

We also cleanup the inventory file to remove corvustest, lists-dev,
pbx, mirror-update*.openstack.org (is opendev.org now), and sort the
LE list.

Change-Id: I8da025640e16bf6e8aca1eb6ec7799d26bd03f12
2021-05-27 14:49:39 -07:00
Clark Boylan
7a0ab6c94e Provision LE certs for openstackid.org
This will provision LE certs for openstackid.org. If we are happy with
the results then the child change can be merged to to swap apache over
to using the new cert.

Change-Id: Icc9fdd8a39630323916d1f33d9867f93fc6f2b85
2021-05-26 13:28:27 -07:00
Clark Boylan
06d021e6e6 Provision LE cert for translate.openstack.org
This provisions the cert then when we are happy with the results we can
land the child change to swap the cert over in apache.

Change-Id: Id8e66102cf26a3b9819d4638b7589f44f6400634
2021-05-24 12:45:15 -07:00
Clark Boylan
ff99f21404 Provision LE cert for storyboard.openstack.org
This provisions the cert but doesn't switch apache to it. When we are
happy with the new cert we can land the child change which will flip
apache over to the new cert.

Change-Id: I9cffd26a51317ea569b078b89cc30dc34c7e7747
2021-05-24 12:35:09 -07:00
Clark Boylan
46edf8aeb0 Provision ethercalc LE cert
This runs the LE ansible alongside the ethercalc puppetry to get an LE
cert provision for this service. Once we are happy with the new cert we
can land the followup change to switch to the LE cert.

Note we don't add an altname for the host because that will require
extra DNS records in rax DNS.

Change-Id: I04c062eb994f672283aa30ffcc0c4d45fc8c50f6
2021-05-24 08:25:39 -07:00
Zuul
9fbd1ccf2c Merge "Ansible mailman configs" 2021-05-19 15:55:09 +00:00
Clark Boylan
c743b7e484 Clean up zuul01 from inventory
This cleans up zuul01 as it should no longer be used at this point. We
also make the inventory groups a bit more clear that all zuul servers
are under the opendev.org domain now.

Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/790483
Change-Id: I7885fe60028fbd87688f3ae920a24bce4d1a3acd
2021-05-13 06:58:36 -07:00
Clark Boylan
4c4e27cb3a Ansible mailman configs
This converts our existing puppeted mailman configuration into a set of
ansible roles and a new playbook. We don't try to do anything new and
instead do our best to map from puppet to ansible as closely as
possible. This helps reduce churn and will help us find problems more
quickly if they happen.

Followups will further cleanup the puppetry.

Change-Id: If8cdb1164c9000438d1977d8965a92ca8eebe4df
2021-05-11 08:40:01 -07:00
Jeremy Stanley
1df1001cb4 Deprovision Limesurvey config management and docs
The Limesurvey service hosted at survey.openstack.org was a beta
which saw limited use. The platform it runs on, Xenial, is now EOL
from Ubuntu/Canonical and in order to upgrade to a newer
distribution release we would need to rewrite all the configuration
management (the version of Puppet supported by newer Ubuntu is not
backward-compatible with what we've been running).

If a similar service becomes interesting to users of our
collaboratory in the future, it will need to be reintroduced with
freshly written configuration management anyway. The old configs and
documentation remain in our Git history should anyone wish to use
them as inspiration.

Change-Id: I59b419cf112d32f20084ab93eb6f2417a7f93fdb
2021-05-01 15:12:00 +00:00
Zuul
cb5898ae0a Merge "Remove firehose.openstack.org" 2021-04-14 18:50:16 +00:00
Clark Boylan
2eebb858af Remove firehose.openstack.org
Once we are satisfied that we have disabled the inputs to firehose we
can land this change to stop managing it in config management. Once that
is complete the server can be removed.

Change-Id: I7ebd54f566f8d6f940a921b38139b54a9c4569d8
2021-04-13 13:51:48 -07:00
Ian Wienand
db76061c71 Stop managing planet01.openstack.org
This server has been retired.
If141aca5efbdbe60c91ceefaa4e05c98cd0ba5bb has redirected this.

Change-Id: I8d3c089e6e845d98a46ae39c0b32b1c845436add
2021-04-13 16:17:14 +10:00
Ian Wienand
525d5d1c19 Add review02.opendev.org
review02.opendev.org is a much larger replacement server for review01
provided by Vexxhost.  It is up and running, with gerrit2 volume
attached and DNS entries.

This adds it to the staging group with no replication and a local h2
database configured for initial bringup.  There's quite a bit to
consider for full migration, but this will let us start experimenting.

Change-Id: I3638a5c0c7028dcc800ada42431b75395cff0c42
2021-03-26 14:53:31 +11:00
Ian Wienand
163d5b6133 Create review-staging group
Create a review-staging group so we can bring up a new server but
avoid running the project-management steps on it.

Change-Id: I93d2a36edcd58a48a36031f0692be3273a36f07c
2021-03-24 11:40:33 +11:00
Ian Wienand
9f11fc5c75 Remove references to review-dev
With our increased ability to test in the gate, there's not much use
for review-dev any more.  Remove references.

Change-Id: I97e9865e0b655cd157acf9ffa7d067b150e6fc72
2021-03-24 11:40:31 +11:00
Zuul
b8874e4f51 Merge "kerberos-kdc: add database backups" 2021-03-19 00:06:59 +00:00
Ian Wienand
dc827de23d Add kerberos-client group
We duplicate the KDC settings over all our kerberos clients.  Add
clients to a "kerberos-client" group and set the variables in a group
file.

Change-Id: I25ed5f8c68065060205dfbb634c6558488003a38
2021-03-18 11:59:30 +11:00
Zuul
df9a85e45c Merge "kerberos: switch servers to Ansible control" 2021-03-17 04:03:03 +00:00
Zuul
b133afedfd Merge "refstack: cleanup old puppet" 2021-03-16 22:21:03 +00:00
Ian Wienand
3052ff4935 kerberos-kdc: add database backups
Add a script to save a db dump to borg backups.  Add the primary KDC
to our backup list.

Change-Id: I32f4ebc1bb4c1952034aba43c75e4d2f85a1b6d3
2021-03-17 08:31:52 +11:00
Ian Wienand
2254b6e43d kerberos: switch servers to Ansible control
This is a follow-on to I60b40897486b29beafc76025790c501b5055313d to
switch the KDC servers to Ansible control and remove any related
puppet configuration.

Change-Id: Ib8f6ec657ca10a3ba648bd154a035fc3d8da4be5
2021-03-17 08:30:52 +11:00
Ian Wienand
018a14e34f refstack: cleanup old puppet
Remove old puppet configuration for the restack service, which is now
managed by Ansible.

Change-Id: I6b6dfd0f8ef89a5362f64cfbc8016ba5b1a346b3
2021-03-17 07:06:53 +11:00
Ian Wienand
753f9520e6 refstack: add backup
We should be backing up the user-generated refstack data

Change-Id: I1bd5f0de283a4436967dcae6da9c5d9cd055697c
2021-03-12 15:18:04 +11:00
Ian Wienand
fdd41cb850 Remove afs-admin group
This group no longer does anything.  This used to deploy a bunch of
keytabs for mirror-update, but that has all moved into
"mirror_update_keytab_*".

Change-Id: I3e2110a621d6946bc4838bfa2f743f0e9db391f3
2021-03-02 11:54:51 +11:00
Ian Wienand
39ffc685d6 backups: remove all bup
All hosts are now running thier backups via borg to servers in
vexxhost and rax.ord.

For reference, the servers being backed up at this time are:

 borg-ask01
 borg-ethercalc02
 borg-etherpad01
 borg-gitea01
 borg-lists
 borg-review-dev01
 borg-review01
 borg-storyboard01
 borg-translate01
 borg-wiki-update-test
 borg-zuul01

This removes the old bup backup hosts, the no-longer used ansible
roles for the bup backup server and client roles, and any remaining
bup related configuration.

For simplicity, we will remove any remaining bup cron jobs on the
above servers manually after this merges.

Change-Id: I32554ca857a81ae8a250ce082421a7ede460ea3c
2021-02-16 16:00:28 +11:00