263 Commits

Author SHA1 Message Date
Ian Wienand
2791684d39 review02 : bump heap limit to 96gb
This host has 128gb RAM.  96gb still leaves a considerable amount for
cache.

Change-Id: I1245c03ae6fbfa77743296e28b52a6a62395fc36
2021-06-18 13:20:37 +10:00
Zuul
2a1505dd5b Merge "review02 : switch reviewdb to mariadb_container type" 2021-06-17 22:57:51 +00:00
Zuul
9181d5198d Merge "gerrit: add mariadb_container option" 2021-06-16 23:14:48 +00:00
Ian Wienand
d1924491d6 review02 : switch reviewdb to mariadb_container type
This switches review02 to use a mariadb container for the change
review database.

Change-Id: Idc6183d63e22e7484a4127a3b71b29cb53c23c51
2021-06-16 13:57:19 +10:00
Ian Wienand
570ca85cd8 gerrit: add mariadb_container option
This adds a local mariadb container to the gerrit host to hold the
accountPatchReviewDb database.  This is inspired by a few things

 - since migration to NoteDB, there is only one table left where
   Gerrit records what files have been reviewed for a change.  This
   logically scales with the number of reviews users are doing.
   Pulling the stats on this, we can see since the NoteDB upgrade this
   went from a very busy database (~300 queries/70 commits per second)
   to barely registering one hit per second :
   https://imgur.com/a/QGJV7Fw

   Thus separating the db to an external host for performance reasons
   is not a large concern any more.

 - emperically we've done a bad job in keeping the existing hosted db
   up-to-date; it's still running mysql 5.1 and we have been hit by
   bugs such as the one referenced in-line which silently drops
   backups.

 - The other gerrit option is to use an on-disk H2 database.  This is
   certainly an option, however you need special tools to interact
   with it for migration, etc. and it's not safe to backup from files
   on disk (as opposed to mysqldump).  Upstream advice is unclear, and
   varies between H2 being a performance bottleneck to this being
   ephemeral data that users don't care about.  We know how to admin
   mariadb/mysql and this allows us to migrate and backup data, so
   seems like the best choice.

 - we have a pressing need to update the server to a new operating
   system.  Running the db alongside the gerrit instance minimises
   fiddling we have to do manging connections to and migrating the
   hosted db systems.

 - related to that, we are tending towards more provider independence
   for control-plane servers.  A hosted database product is not always
   provided, so this gives us more flexibility in moving things
   around.

 - the main concern here is memory usage.  "docker stats" reports a
   quiescent container, freshly started on a 8GB host:

    gerrit-compose_mariadb_1  67.32MiB

   After loading a copy of the production table, and then dumping it
   back to a file the same container reports:

    gerrit-compose_mariadb_1  462.6MiB

The existing remote mysql configuration path remains mostly the same.
We move the gerrit startup into a script rather than a CMD so we can
call it after a "wait for db" script in the mariadb_container case
(this is the reccommeded way to enforce ordering [1]).

Backups of the local container need different dump commands; backups
are relocated to a new file and updated.

Testing is converted to use this rather than a local H2 database.

[1] https://docs.docker.com/compose/startup-order/

Change-Id: Iec981ef3c2e38889f91e9759e66295dbfb499c2e
2021-06-16 13:57:13 +10:00
Ian Wienand
868a42a85a Move statusbot channels out of hiera
This makes I246b2723372594e65bcd1ba90215d6831d4c0c72 active

Change-Id: I5a9efa2edc2fe6fb70e21d4b58fd4283d2d5972d
2021-06-11 18:15:48 +10:00
Zuul
f80ab86043 Merge "Move meetbot config to eavesdrop01.opendev.org" 2021-06-11 00:10:56 +00:00
Zuul
084879c1fa Merge "limnoria/meetbot setup on eavesdrop01.opendev.org" 2021-06-10 02:04:53 +00:00
Ian Wienand
ccda6d08a1 Move meetbot config to eavesdrop01.opendev.org
This enables the new eavesdrop01.opendev.org server in all current
channels.  Puppet has been disabled on the old server and we will
manually stop supybot/meetbot and mirgrate logs before this applies.

Change-Id: I4a422bb9589c8a8761191313a656f8377e93422f
2021-06-10 09:02:23 +10:00
Ian Wienand
403773d55a limnoria/meetbot setup on eavesdrop01.opendev.org
This installs our Limnoira/meetbot container and configures it on
eavesdrop01.opendev.org.  I have ported the configuration from the old
puppet as best I can (it is very verbose); my procedure was to use the
Limnoira wizard to start a new config file then backport everything
from the old file.  I felt this was best to not miss any new options.

This does channel logging (via built-in ChannelLogger plugin, along
with a cron job for logs2html) and runs our fork of meetbot.

It exports the channel logs via HTTP to /irclogs and meetings logs to
/meetings.  meetings.opendev.org will proxy to these two locations
when the server is active.

Note this has not ported the channel list; so the bot will not be
listening in our channels.

Change-Id: I9f9a466c271e1a706f9f98f816de0e84047519f1
2021-06-10 09:02:16 +10:00
Zuul
632b2f9df7 Merge "Cleanup ask.openstack.org" 2021-06-09 05:42:26 +00:00
Zuul
39731fd614 Merge "Forward openstack-security ML to openstack-discuss" 2021-06-08 17:50:23 +00:00
Ian Wienand
f66efc0d9c Restore eavesdrop01.openstack.org to webservers group
This host is no longer under puppet control, but should still be a
webserver to export the logs it is still collecting until we finish
moving that to the new server.  Restore the match to open*

See I809f9af3e78f566362142790f6c79654ef5b8959

Change-Id: I524c0a7c5cc93313c180eca68b67a0f0582474df
2021-06-08 16:07:55 +10:00
Ian Wienand
7de885b5ee Cleanup ask.openstack.org
This was retired with I8a31f8fcf9b3064c0ae58e463a6014dc14b518a7

Change-Id: Ieafac856b0feb91f41f05084aa669e2ccb92569d
2021-06-08 14:35:28 +10:00
Ian Wienand
fec8018581 Move gerritbot/accessbot to new eavesdrop server
This moves these services to eavesdrop01.opendev.org, a new
Focal-based server to host IRC services.

We have stopped running puppet on eavesdrop01.openstack.org so there
is nothing left for it to do (note the server is still running
meetbot/ptgbot).  Remove the commented out puppet run, and remove the
server from puppet groups.  Update the host in the Zuul jobs to the
new node.

Change-Id: I809f9af3e78f566362142790f6c79654ef5b8959
2021-06-08 08:16:56 +10:00
Ian Wienand
fb94b79e82 Add eavesdrop01.opendev.org server
This adds a new server to take over from eavesdrop01.openstack.org.

We limit the puppet installs, etc. to the openstack.org server.  The
new server is in the group eavesdrop_opendev as we cut over services.
A stub for basic installation is added to the service playbook.

Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/795004
Change-Id: I88c3059532e4d6ab267fdec5b390daefa5b0c4a1
2021-06-07 12:59:02 +10:00
Jeremy Stanley
84c63ff1bf Forward openstack-security ML to openstack-discuss
The openstack-security mailing list is officially closing, and wants
future attempts at posting to end up on openstack-discuss instead:

http://lists.openstack.org/pipermail/openstack-security/2021-June/006077.html

This was also the only remaining user of the notify-impact Gerrit
hook, so we can stop installing/running it.

Change-Id: Id60b781beb072366673b32326e32fd79637c1219
2021-06-03 17:57:54 +00:00
Ian Wienand
0cfedd2318 Add static eavesdrop.openstack.org site
We are trying to replace eavesdrop01.openstack.org

The main landing page serves meeting information which has been moved
to a static site served from AFS at meeting.opendev.org.  Redirect
everything to there.

The IRC logs are currently still hosted on eavesdrop01, so while we
work on migrating these, proxy meeting.opendev.org/<irclogs|meetings>
to this server.

Note this will be a no-op until we move the DNS, but we should make
the eavesdrop acme records before merging.

Change-Id: I5c9c23e619dbe930a77f657b5cd6fdd862034301
2021-06-03 14:34:20 +10:00
Ian Wienand
270daa1b1a Serve meetings.opendev.org
This site replaces eavesdrop.openstack.org.  I think this name makes
more sense.

That is/was being published by jobs directly pushing this onto the
eavesdrop server.  Instead, the publishing jobs for irc-meetings now
publish to /afs/openstack.org/project/meetings.opendev.org.  This
makes the site available via the static server.

This is actually a production no-op; nothing has changed for the
current publishing.  It is still todo to figure out the correct
redirects to keep things working from the existing
eavesdrop.openstack.org and stop the old publishing method.

Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/794085
Change-Id: Ia582c4cee1f074e78cee32626be86fd5eb1d81bd
2021-06-02 13:56:19 +10:00
Clark Boylan
399ade787b More puppetry and inventory cleanups
This cleans up ask-staging which hasn't been a thing in a log time.
We remove some puppet stubs for nodepool builders (they are all ansible
now).

We also cleanup the inventory file to remove corvustest, lists-dev,
pbx, mirror-update*.openstack.org (is opendev.org now), and sort the
LE list.

Change-Id: I8da025640e16bf6e8aca1eb6ec7799d26bd03f12
2021-05-27 14:49:39 -07:00
Clark Boylan
7a0ab6c94e Provision LE certs for openstackid.org
This will provision LE certs for openstackid.org. If we are happy with
the results then the child change can be merged to to swap apache over
to using the new cert.

Change-Id: Icc9fdd8a39630323916d1f33d9867f93fc6f2b85
2021-05-26 13:28:27 -07:00
Zuul
715dda2c8d Merge "ask.openstack.org static site" 2021-05-26 01:05:14 +00:00
Ian Wienand
1fbd156697 ask.openstack.org static site
We have decided to decommision the ask.openstack.org server as it is
running EOL Xenial, and its manually purchased certiface is about to
expire.  Although it has been deprecated for some time, we feel like
it has been around long-enough as a resource that it is best if we
replace it with a place-holder.  The links included here are the same
as the currently shown header explaining the site is read-only.

There's nowhere particularly relevant to redirect the site, so we add
a static file here, and some minimal Ansible to put it in the right
place in a generic way in-case we want to do the same for another
service.

Change-Id: I8a31f8fcf9b3064c0ae58e463a6014dc14b518a7
2021-05-25 16:09:52 +10:00
Clark Boylan
06d021e6e6 Provision LE cert for translate.openstack.org
This provisions the cert then when we are happy with the results we can
land the child change to swap the cert over in apache.

Change-Id: Id8e66102cf26a3b9819d4638b7589f44f6400634
2021-05-24 12:45:15 -07:00
Clark Boylan
ff99f21404 Provision LE cert for storyboard.openstack.org
This provisions the cert but doesn't switch apache to it. When we are
happy with the new cert we can land the child change which will flip
apache over to the new cert.

Change-Id: I9cffd26a51317ea569b078b89cc30dc34c7e7747
2021-05-24 12:35:09 -07:00
Clark Boylan
46edf8aeb0 Provision ethercalc LE cert
This runs the LE ansible alongside the ethercalc puppetry to get an LE
cert provision for this service. Once we are happy with the new cert we
can land the followup change to switch to the LE cert.

Note we don't add an altname for the host because that will require
extra DNS records in rax DNS.

Change-Id: I04c062eb994f672283aa30ffcc0c4d45fc8c50f6
2021-05-24 08:25:39 -07:00
Zuul
9fbd1ccf2c Merge "Ansible mailman configs" 2021-05-19 15:55:09 +00:00
Clark Boylan
c743b7e484 Clean up zuul01 from inventory
This cleans up zuul01 as it should no longer be used at this point. We
also make the inventory groups a bit more clear that all zuul servers
are under the opendev.org domain now.

Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/790483
Change-Id: I7885fe60028fbd87688f3ae920a24bce4d1a3acd
2021-05-13 06:58:36 -07:00
Clark Boylan
533594d959 Add zuul02 to inventory
This zuul02 instance will replace zuul01. There are a few items to
coordinate when doing an actual switch so we haven't removed zuul01 from
inventory here. In particular we need to update gearman server config
values in the zuul cluster and we need to save queues, shutdown zuul01,
then start zuul02's scheduler and restore queues there.

I believe landing this change is safe as we don't appear to start zuul
on new instances by default. Reviewers should double check this.

Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/791039
Change-Id: I524b456e494124d8293fbe8e1468de40f3800772
2021-05-13 06:58:30 -07:00
Clark Boylan
4c4e27cb3a Ansible mailman configs
This converts our existing puppeted mailman configuration into a set of
ansible roles and a new playbook. We don't try to do anything new and
instead do our best to map from puppet to ansible as closely as
possible. This helps reduce churn and will help us find problems more
quickly if they happen.

Followups will further cleanup the puppetry.

Change-Id: If8cdb1164c9000438d1977d8965a92ca8eebe4df
2021-05-11 08:40:01 -07:00
Zuul
f778e7cd9d Merge "host_vars : add .yaml extension" 2021-05-07 02:42:51 +00:00
Jeremy Stanley
1df1001cb4 Deprovision Limesurvey config management and docs
The Limesurvey service hosted at survey.openstack.org was a beta
which saw limited use. The platform it runs on, Xenial, is now EOL
from Ubuntu/Canonical and in order to upgrade to a newer
distribution release we would need to rewrite all the configuration
management (the version of Puppet supported by newer Ubuntu is not
backward-compatible with what we've been running).

If a similar service becomes interesting to users of our
collaboratory in the future, it will need to be reintroduced with
freshly written configuration management anyway. The old configs and
documentation remain in our Git history should anyone wish to use
them as inspiration.

Change-Id: I59b419cf112d32f20084ab93eb6f2417a7f93fdb
2021-05-01 15:12:00 +00:00
Clark Boylan
82a5445ae0 Cleanup mirror01.iad3.inmotion
This server has been replaced with a mirror02 host with a slightly
different network config. Clean this server up.

Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/787630
Change-Id: I6eaa51db47d8b4d4596928f6a7ef105bebe0e8f1
2021-04-22 12:20:33 -07:00
Clark Boylan
ac2b661cf2 Add mirror02 to inmotion
We are doing this so that we can cleanup the private network + floating
IP setup that the existing mirror does. Once this new mirror is up and
happy we can cname to it and then clean up the old mirror and its
networking config. We do this in order to save an IP that the current
private network router is consuming.

Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/787628
Change-Id: I50c311087c6c28726e36913c7e081f3b3d0ee049
2021-04-22 12:15:28 -07:00
Clark Boylan
6b190a4751 Use External inmotion cloud network for zuul nodes
We have limited ipv4 address space in this cloud. Currently we can do
about 6 IP addresses for test nodes after we account for network
infrastructure and the mirror. By switching these instances to using the
external network directly we can clean up some of the neutron network
infrastructure which we think may free up 2 more IP addresses. That
should get us our originally intended max-servers of 8.

Change-Id: I705ff082ff06ae1c97f4c229a22893e6d87d206d
2021-04-22 09:53:19 -07:00
Clark Boylan
dbc316ba49 Add iad3.inmotion mirror node
This updates out inventory to add the new inmotion mirror. This is a
necessary step in bootstrapping this cloud for nodepool usage.

Change-Id: Ie66cdb010c0772310f1cfa8187ca0a2d7f1de1b8
2021-04-21 16:37:19 -07:00
Clark Boylan
ada70387e9 Set the correct cloud for opendevzuul-inmotion enrollment
Due to bad copy pasta the new inmotion cloud enrollmant for opendevzuul
was run against osuosl. Fix this to get the right cloud sorted. I will
manually cleanup osuosl.

Change-Id: I56a71734b3e0fd648443ef7d1894ee79cd23077a
2021-04-21 15:23:02 -07:00
Clark Boylan
f1df36145d Add inmotion cloud to cloud launcher
This adds the new inmotion cloud to clouds.yaml files and the cloud
launcher config. This cloud is running on an openstack as a service
platform so we have quite a bit of freedom to make changes here within
the resource limitations if necessary.

Change-Id: I2aed6dffde4a1d6e3044c4bd8df4ca60065ae1ea
2021-04-21 11:18:40 -07:00
Ian Wienand
28fed0bcd5 nodepool-builder: configure upload workers, reduce nb03
Add a variable to configure upload-workers for nodepool-builder
daemons.

Reduce our defaults for nb03 to see if we can get more reliable
uploads.

Change-Id: I819bdd262c7118cbde4e6ffdc12aa3ac64569a96
2021-04-15 09:10:37 +10:00
Zuul
cb5898ae0a Merge "Remove firehose.openstack.org" 2021-04-14 18:50:16 +00:00
Zuul
410ee03d82 Merge "Stop managing planet01.openstack.org" 2021-04-14 04:13:46 +00:00
Zuul
bb2188d298 Merge "Add planet.openstack.org redirect to static" 2021-04-14 02:01:02 +00:00
Clark Boylan
2eebb858af Remove firehose.openstack.org
Once we are satisfied that we have disabled the inputs to firehose we
can land this change to stop managing it in config management. Once that
is complete the server can be removed.

Change-Id: I7ebd54f566f8d6f940a921b38139b54a9c4569d8
2021-04-13 13:51:48 -07:00
Ian Wienand
77a197bd91 OSU OSL: fix typo in certificate name
Same typo as I9839dfb167e853c167d94da2adcf297e074678d3.  Too many
OS's!

Change-Id: I894bdf30ac1626535a2033a03ee6bee9ba7c8435
2021-04-13 20:17:54 +10:00
Ian Wienand
b112f1f756 host_vars : add .yaml extension
We have an mix of files with .yaml extension and without.  Standardise
with the .yaml.

Change-Id: I8b657e29d4c112cdb2c17e7b89a3efbcd824b846
2021-04-13 17:25:57 +10:00
Ian Wienand
6c3c101cb3 Fix OSU OSL mirror host variable file typo
Change-Id: I9839dfb167e853c167d94da2adcf297e074678d3
2021-04-13 17:22:10 +10:00
Ian Wienand
db76061c71 Stop managing planet01.openstack.org
This server has been retired.
If141aca5efbdbe60c91ceefaa4e05c98cd0ba5bb has redirected this.

Change-Id: I8d3c089e6e845d98a46ae39c0b32b1c845436add
2021-04-13 16:17:14 +10:00
Ian Wienand
609986634f Add planet.openstack.org redirect to static
This handles planet.openstack.org and redirects it to the
opendev.org/openstack/planet-openstack repo, where we will put a
README and the OPML file of the last state as we deprecate this
service.

Change-Id: If141aca5efbdbe60c91ceefaa4e05c98cd0ba5bb
2021-04-13 16:17:09 +10:00
Ian Wienand
8e9d250293 Add OSUOSL mirror
Change-Id: Ia065fb30cfd69c5ab3fe96541d168b5722ff59ce
Depends-On: https://review.opendev.org/c/opendev/zone-opendev.org/+/786003
2021-04-13 14:19:10 +10:00
Ian Wienand
28ffbfb12c Add OSUOSL cloud
The Oregon State University Open Source Lab (OSUOSL;
https://osuosl.org/) has kindly donated some ARM64 resources.  Add
initial cloud config.

Change-Id: I43ed7f0cb0b193db52d9908e39c04e351b3887e3
2021-04-12 09:31:51 +10:00