263 Commits

Author SHA1 Message Date
Jeremy Stanley
f5268d956c Add check to remainder of balance_git_https
Now that we can confirm this hasn't broken for gitea01, set check on
all the remaining server lines as well.

Change-Id: I11f1f15210dafed66e1209329ddf7f3838592881
2022-03-07 18:14:19 +00:00
Jeremy Stanley
4061acd3e7 Add check keyword to balance_zuul_https servers
Apparently the check-ssl option only modifies check behavior, but
does not actually turn it on. The check option also needs to be set
in order to activate checks of the server. See §5.2 of the haproxy
docs for details:
https://git.haproxy.org/?p=haproxy-2.5.git;a=blob;f=doc/configuration.txt;h=e3949d1eebe171920c451b4cad1d5fcd07d0bfb5;hb=HEAD#l14396

Turn it on for all of our balance_zuul_https server entries.

Also set this on the gitea01 server entry in balance_git_https, so
we can make sure it's still seen as "up" once this change takes
effect. A follow-up change will turn it on for the other
balance_git_https servers out of an abundance of caution around that
service.

Change-Id: I4018507f6e0ee1b5c30139de301e09b3ec6fc494
2022-03-07 18:11:46 +00:00
Zuul
53fbf72fdd Merge "Allow zuul-lb to send stats to graphite" 2022-03-07 05:23:29 +00:00
James E. Blair
e97efd5fd5 Allow zuul-lb to send stats to graphite
Change-Id: Ib6bcd21555d34f80e1ace58cbd1cc7f479f92f7a
2022-03-04 15:26:21 -08:00
Clark Boylan
f24bbf97a7 Do more robust checks against zuul-web with haproxy
Switch the port 80 and 443 endpoints over to doing http checks instead
of tcp checks. This ensures that both apache and the zuul-web backend
are functional before balancing to them.

The fingergw remains a tcp check.

Change-Id: Iabe2d7822c9ef7e4514b9a0eb627f15b93ad48e2
2022-03-04 14:17:51 -08:00
James E. Blair
3f8acefbe1 Run zuul-web on zuul01 and add to load balancer
Change-Id: Ia8b10338fa3a1876993404276e0759f4b10d6b54
2022-03-04 13:11:09 -08:00
Jack Morgan
ded27cbb5d Adds support for running zuul-registry as a non-root user
Signed-off-by: Jack Morgan <jack@jento.io>
Change-Id: I89594affb04639b49b409a569036d6afac997251
2022-03-03 09:06:51 -08:00
Clark Boylan
b7ccc12a6b Remove airship-citycloud resources
We've been told these resources are going away. Trying to remove them
gracefully from nodepool. Once that is done we can remove our configs
here.

Depends-On: https://review.opendev.org/c/openstack/project-config/+/831398
Change-Id: I396ca49ab33c09622dd398012528fe7172c39fe8
2022-03-01 11:39:53 -08:00
Jeremy Stanley
9a25740961 Clean up two retired mailing lists
The enterprise-wg and product-wg lists were deleted from the
openstack site per the announcement[*] on 2022-02-01, but I
neglected to push a change to remove them from our configuration
management, so Ansible helpfully recreated them for me. Clean this
up so I can re-remove the lists once and for all.

[*] http://lists.openinfra.dev/pipermail/foundation/2022-February/003048.html

Change-Id: Iddcb5cbac68d426e0ad13dd41541ad1371366bb1
2022-02-24 19:34:19 +00:00
Clark Boylan
5b5be7cd02 Remove mirror ports 4444 and 8081 from the firewall
The docker v1 protocol proxy listened on these ports and was removed
by 9b6398394d5d5d9e9e9aff244ccac2f98a4317d1 as everything uses v2 now.
The firewall holes were left open though. Clean that up.

Change-Id: Ie00acd5bfb657153b9bc49222ae5d9778ad36e70
2022-02-17 08:31:58 -08:00
Zuul
7dfa0f5fa8 Merge "Haproxy http checks for Gitea" 2022-02-16 22:08:26 +00:00
Zuul
d0a4710eb0 Merge "Remove configuration management for wiki servers" 2022-02-16 17:58:05 +00:00
Clark Boylan
df335525ab Haproxy http checks for Gitea
Previously we were only checking that Apache can open TCP connections to
determine if Gitea is up or down on a backend. This is insufficient
because Gitea itself may be down while Apache is up. In this situation
TCP connection to Apache will function, but if we make an HTTP request
we should get back an error.

To check if both Apache and Gitea are working properly we switch to
using http checks instead. Then if Gitea is down Apache can return a 500
and the Gitea backend will be removed from the pool. Similarly if Apache
is non functional the check will fail to connect via TCP.

Note we don't verify ssl certs for simplicity as checking these in
testing is not straightforward. We didn't have verification with the old
tcp checks so this isn't a regression, but does represent something we
could try and improve in the future.

Change-Id: Id47a1f9028c7575e8fbbd10fabfc9730095cb541
2022-02-15 09:59:52 -08:00
Zuul
9db437f2ba Merge "Switch refstack's IDP to OpenInfraID" 2022-02-15 04:52:24 +00:00
Jeremy Stanley
89c4fd9b3d Remove configuration management for wiki servers
We never finished puppeting the OpenStack wiki, and if we do manage
to get it under configuration management in the future it will
likely not use Puppet anyway. The dev server is already gone, and
deployment has been explicitly disabled for the other, so let's go
ahead and remove the references here and then we should be able to
retire the separate Puppet module we've been hosting.

Change-Id: I3f9ada3eb3d6f16545270135fab994ac460be94b
2022-02-14 22:32:18 +00:00
James E. Blair
2a9553ef25 Add Zuul load balancer
This adds a load balancer for zuul-web and fingergw.

Change-Id: Id5aa01151f64f3c85e1532ad66999ef9471c5896
2022-02-10 13:24:42 -08:00
Zuul
8dafc621d7 Merge "Remove gearman from Zuul" 2022-02-01 23:11:30 +00:00
James E. Blair
14f4a20628 Remove gearman from Zuul
Zuul no longer uses gearman, so we can remove the infrastructure
around it.

Change-Id: I3613d812971add4733d3fe509ee22835e5814ec6
2022-02-01 13:52:47 -08:00
Zuul
6d25c4a5c3 Merge "Add openstack-skyline channel in statusbot/meetbot/logging" 2022-01-31 00:57:14 +00:00
Zuul
24b91e5726 Merge "Drop wiki-dev03 from inventory" 2022-01-28 17:32:34 +00:00
James E. Blair
535b7162a1 Move Zuul SQL connection to "database"
The sql connection is no longer supported, we need to use "database"
instead.  The corresponding hostvars change has already been made
on bridge.

Change-Id: Ibcac56568f263bd50b2be43baa26c8c514c5272b
2022-01-27 16:46:32 -08:00
Jeremy Stanley
f469f1189e Drop wiki-dev03 from inventory
The wiki-dev03.openstack.org server was a test deployment working
through completing the puppetry for our Mediawiki environment. Since
it's on a now-EoL Ubuntu version, and that configuration management
work has stalled, delete this test server from our inventory rather
than needlessly consuming resources and an ESM entitlement.

Also clean up an old disabled entry for wiki-dev01.openstack.org
which no longer exists (it was a predecessor of this server). Leave
the templating for wiki-dev* in place for now in case we decide to
launch a replacement.

Change-Id: I5beed4dde8e4e84d92f510f8726f8443daf774c1
2022-01-27 16:40:26 +00:00
Ghanshyam Mann
b57f954456 Add openstack-skyline channel in statusbot/meetbot/logging
openstack/skyline project is newly added
- https://review.opendev.org/c/openstack/governance/+/814037
and channel is being added in project-config accessbot
by depends on patch.

Depends-On: https://review.opendev.org/c/openstack/project-config/+/825881
Change-Id: I5df1704b4dade9bf3c5b0ee717a72f6d04fac43a
2022-01-21 15:21:20 -06:00
Jeremy Stanley
2d450e29bc Switch refstack's IDP to OpenInfraID
The OpenStackID project has been rebranded, and the old
openstackid.org deployment is being retained temporarily in order to
ease transition, but id.openinfra.dev is in place now and intended
as its successor.

Note that when this merges, a manual database edit will be required
to associate every user's new ID with their existing accounts, so
this should only be merged when we're ready to do that part just
prior to deploying and then check it again after to make sure we
didn't race any user additions.

Change-Id: I2716e469bc61e53645c23d362b8637bab0a32bb1
2022-01-10 21:21:28 +00:00
Jeremy Stanley
c51521fffe Add a summit-programming-committee mailing list
The old openstack-track-chairs mailing list was retired recently.
This new summit-programming-committee list will take its place.

Change-Id: I4e0c3fc65e0da8f17dec518a867e4b0a59ead94b
2021-12-21 19:48:24 +00:00
Jeremy Stanley
215fd6da98 Retire defunct or unused foundation mailing lists
The following OpenInfra Foundation mailing lists are no longer
required (some were never used at all, some haven't had a post in
many years):

    * admin-cert-wg
    * ambassadors
    * analyst-relations
    * app-catalog-admin
    * defcore-committee
    * foundation-testing-standards
    * nov-2013-track-chairs
    * openstack-track-chairs
    * openstack-travel-committee
    * superuser
    * tax-affairs

Remove them from our configuration, they'll be manually retired with
the rmlist command once this merges (leaving any public archives
behind for historical reference).

Change-Id: I30dcdd52aa16bed8af866f629d85fde3b9502fa1
2021-12-21 19:39:31 +00:00
Jeremy Stanley
3858a0bc23 Move edge-computing ML to opendev Mailman site
The edge-computing discussion list is not OpenStack-specific. It was
originally included on the lists.openstack.org site when we didn't
yet have a more neutral list hosting location. While we're in the
process of moving other non-OpenStack mailing lists off the
lists.openstack.org site, rehome this one to lists.opendev.org by
setting up address forwarding and Web redirects, and moving the
existing mailman list entry for it in our configuration.

Note that this should be a no-op when it merges, as the list move
will be handled manually while deployment is temporarily disabled
for the server.

Change-Id: If5207f0237bee1571924855b769a22d653964af7
2021-12-21 19:28:22 +00:00
Jeremy Stanley
eadb01c5d8 Move summitsponsors ML to openinfra Mailman site
In keeping with its name change to the Open Infrastructure
Foundation, the summit sponsors mailing list is moving from
lists.openstack.org to lists.openinfra.dev. Set up address
forwarding and Web redirects to reflect this, and move the existing
mailman list entry for it in our configuration.

Note that this should be a no-op when it merges, as the list move
will be handled manually while deployment is temporarily disabled
for the server.

Change-Id: I29e1e94885fd16b0edd7001662f367caec591439
2021-12-21 19:20:02 +00:00
Jeremy Stanley
56e8aaa870 Move marketing ML to openinfra Mailman site
In keeping with its name change to the Open Infrastructure
Foundation, the foundation marketing mailing list is moving from
lists.openstack.org to lists.openinfra.dev. Set up address
forwarding and Web redirects to reflect this, and add a mailman list
entry for it (there's no old one to remove as it wasn't previously
included in our configuration).

Note that this should be a no-op when it merges, as the list move
will be handled manually while deployment is temporarily disabled
for the server.

Change-Id: Ibadc4bfc430656286774e25b4dce6d8e29b5acf7
2021-12-21 19:15:09 +00:00
Jeremy Stanley
71d566bf20 Move goldmembers ML to openinfra Mailman site
In keeping with its name change to the Open Infrastructure
Foundation, the foundation gold member mailing list is moving from
lists.openstack.org to lists.openinfra.dev. Set up address
forwarding and Web redirects to reflect this, and add a mailman list
entry for it (there's no old one to remove as it wasn't previously
included in our configuration).

Note that this should be a no-op when it merges, as the list move
will be handled manually while deployment is temporarily disabled
for the server.

Change-Id: I6cd92e052b26705bd16a4b38b3725248cb5691fd
2021-12-21 18:58:16 +00:00
Jeremy Stanley
5d6cee89f4 Move foundation-board-confidential ML to openinfra
In keeping with its name change to the Open Infrastructure
Foundation, the confidential board mailing list is moving from
lists.openstack.org to lists.openinfra.dev. Set up address
forwarding and Web redirects to reflect this, and add a mailman list
entry for it (there's no old one to remove as it wasn't previously
included in our configuration).

Note that this should be a no-op when it merges, as the list move
will be handled manually while deployment is temporarily disabled
for the server.

Change-Id: I191676bcb7f878afab17ec3c1735219d91b4de4d
2021-12-21 18:54:10 +00:00
Jeremy Stanley
558528c66d Move foundation-board ML to openinfra Mailman site
In keeping with its name change to the Open Infrastructure
Foundation, the foundation board mailing list is moving from
lists.openstack.org to lists.openinfra.dev. Set up address
forwarding and Web redirects to reflect this, and add a mailman list
entry for it (there's no old one to remove as it wasn't previously
included in our configuration).

Note that this should be a no-op when it merges, as the list move
will be handled manually while deployment is temporarily disabled
for the server.

Change-Id: Idcac72c067fab66b6322f08c027e9c451a488ca3
2021-12-21 18:50:43 +00:00
Jeremy Stanley
1a6d341a7d Move community ML to openinfra Mailman site
In keeping with its name change to the Open Infrastructure
Foundation, the foundation community mailing list is moving from
lists.openstack.org to lists.openinfra.dev. Set up address
forwarding and Web redirects to reflect this, and add a mailman list
entry for it (there's no old one to remove as it wasn't previously
included in our configuration).

Note that this should be a no-op when it merges, as the list move
will be handled manually while deployment is temporarily disabled
for the server.

Change-Id: I9fff3b920a7fd0f75a3cc7a704003eeb3aab4d8a
2021-12-21 18:46:34 +00:00
Jeremy Stanley
b5583429b2 Move foundation ML to openinfra Mailman site
In keeping with its name change to the Open Infrastructure
Foundation, the general foundation mailing list is moving from
lists.openstack.org to lists.openinfra.dev. Set up address
forwarding and Web redirects to reflect this, and add a mailman list
entry for it (there's no old one to remove as it wasn't previously
included in our configuration).

Note that this should be a no-op when it merges, as the list move
will be handled manually while deployment is temporarily disabled
for the server.

Change-Id: I367dd2a3d9a1c70c14915efa729d643419375060
2021-12-21 18:15:12 +00:00
Jeremy Stanley
81f8cdfb7b Add HTTPS vhosts to mailman servers
Add secondary vhosts for HTTPS to each mailman site, but don't
remove the plain HTTP ones for now. Before switching to Mailman 3
we'll replace the current HTTP vhosts with blanket redirects to
HTTPS.

Add tests to make sure this is working, and also add a command-line
test for the lists.openinfra.dev site now that it's got a first
non-default list of its own. Also collect Apache logs from the test
nodes so we can see for sure what might break.

Change-Id: I4d93d643381f17c9a968595587909f0ba3dd6f92
2021-12-20 20:35:14 +00:00
Zuul
05044aad41 Merge "Generate HTTPS certs for Mailman sites" 2021-12-20 00:38:55 +00:00
Zuul
857bc42e7a Merge "Forward messages for OpenInfra Foundation staff ML" 2021-12-19 21:01:31 +00:00
Jeremy Stanley
fa0c1b495c Generate HTTPS certs for Mailman sites
We're going to want Mailman 3 served over HTTPS for security
reasons, so start by generating certificates for each of the sites
we have in v2. Also collect the acme.sh logs for verification.

Change-Id: I261ae55c6bc0a414beb473abcb30f9a86c63db85
2021-12-17 22:25:22 +00:00
Zuul
8133805f29 Merge "Create an OpenInfra Foundation staff ML" 2021-12-16 23:18:07 +00:00
Zuul
ef24d3e9ce Merge "Add a domain aliases mechanism to lists.o.o" 2021-12-16 23:14:15 +00:00
Jeremy Stanley
75c8739bf9 Forward messages for OpenInfra Foundation staff ML
Once the staff mailing list has been migrated to its new Mailman
site, merge this in order to forward posts destined for its old
address to the new one.

Add a test to make sure domain aliases are bein written as expected.

Change-Id: I5fea8e9ee6460417283c0ed7339d0dd447b2ff63
2021-12-16 19:22:16 +00:00
Jeremy Stanley
f906b06555 Create an OpenInfra Foundation staff ML
This is a new mailing list into which the current staff ML from the
lists.openstack.org site will be manually migrated. The existing one
is not included in our current configuration anyway, but a followup
change will set up an appropriate forward for its old address once
migration is complete.

Change-Id: I15f47d210e38a8f04925ffba27e44b2ad5e97dd5
2021-12-16 19:22:16 +00:00
Jeremy Stanley
1addce7dbc Add a domain aliases mechanism to lists.o.o
In order to be able to redirect list addresses which have moved from
one domain to another, we need a solution to alias the old addresses
to the new ones. We have simple aliases but they only match on the
local part. Add a new /etc/aliases.domain which matches full
local_part@domain addresses instead. Also collect this file in the
Mailman deployment test for ease of inspection.

Change-Id: I16f871e96792545e1a8cc8eb3834fa4eb82e31c8
2021-12-16 19:22:11 +00:00
Ghanshyam Mann
9dde035e8a Add openstack-venus channel in statusbot
openstack/venus project is newly added
- https://review.opendev.org/c/openstack/project-config/+/808149
and channel is being added in project-config
accessbot by depends on patch.

Depends-On: https://review.opendev.org/c/openstack/project-config/+/821875
Change-Id: Ibf98e54850f65968710a5161d77d3d0880642f38
2021-12-15 15:29:44 -06:00
Jeremy Stanley
196b081159 Add "mailman" meta-list to lists.katacontainers.io
Mailman uses a (usually hidden) mailing list named "mailman" to
handle things like password reminders and certain other sorts of
notifications. We have one in the configuration for all the sites on
lists.openstack.org but not on lists.katacontainers.io, even though
the production server has one. Not creating this list will cause
the services to fail to start, and since we want to test restarting
them in an upcoming change, add the missing entry (it will be a
no-op in production anyway).

Change-Id: If06d9d060e40055f95c1df337eb6f32c6064a89f
2021-12-14 21:04:41 +00:00
Zuul
94bc7c1455 Merge "Add a keycloak server" 2021-12-04 16:50:26 +00:00
James E. Blair
e79dbbe6bb Add a keycloak server
This adds a keycloak server so we can start experimenting with it.

It's based on the docker-compose file Matthieu made for Zuul
(see https://review.opendev.org/819745 )

We should be able to configure a realm and federate with openstackid
and other providers as described in the opendev auth spec.  However,
I am unable to test federation with openstackid due its inability to
configure an oauth app at "localhost".  Therefore, we will need an
actual deployed system to test it.  This should allow us to do so.

It will also allow use to connect realms to the newly available
Zuul admin api on opendev.

It should be possible to configure the realm the way we want, then
export its configuration into a JSON file and then have our playbooks
or the docker-compose file import it.  That would allow us to drive
change to the configuration of the system through code review.  Because
of the above limitation with openstackid, I think we should regard the
current implementation as experimental.  Once we have a realm
configuration that we like (which we will create using the GUI), we
can chose to either continue to maintain the config with the GUI and
appropriate file backups, or switch to a gitops model based on an
export.

My understanding is that all the data (realms configuration and session)
are kept in an H2 database.  This is probably sufficient for now and even
production use with Zuul, but we should probably switch to mariadb before
any heavy (eg gerrit, etc) production use.

This is a partial implementation of https://docs.opendev.org/opendev/infra-specs/latest/specs/central-auth.html

We can re-deploy with a new domain when it exists.

Change-Id: I2e069b1b220dbd3e0a5754ac094c2b296c141753
Co-Authored-By: Matthieu Huin <mhuin@redhat.com>
2021-12-03 14:17:23 -08:00
Zuul
fc257bdcaa Merge "Create a new lists.openinfra.dev mailing list site" 2021-12-02 17:57:48 +00:00
Ian Wienand
f29aa2da16 Make haproxy role more generic
This makes the haproxy role more generic so we can run another (or
potentially even more) haproxy instance(s) to manage other services.

The config file is moved to a variable for the haproxy role.  The
gitea specific config is then installed for the gitea-lb service by a
new gitea-lb role.

statsd reporting is made optional with an argument.  This
enables/disables the service in the docker compose.

Role documenation is updated.

Needed-By: https://review.opendev.org/678159
Change-Id: I3506ebbed9dda17d910001e71b17a865eba4225d
2021-12-01 09:55:45 +11:00
Jeremy Stanley
33fc2a4d4e Create a new lists.openinfra.dev mailing list site
The Open Infrastructure Foundation has a number of mailing lists
located in the lists.openstack.org site due to historical reasons
(from when they were the OpenStack Foundation). In order to better
disambiguate their mailing lists, a new Mailman site is being
created into which they'll be moved, leaving the old site
exclusively for OpenStack project-specific lists.

As a first step, create the new lists.openinfra.dev site with the
default "mailman" meta-list (which will be hidden once created).
Subsequent changes will create new lists, and remove/redirect the
old ones once configuration is manually replicated.

Change-Id: I64770fbc33184374f1d24f4a2c234f849ab47bce
2021-11-22 19:46:48 +00:00