17628 Commits

Author SHA1 Message Date
Zuul
82edab1d39 Merge "Add comments to manage-projects about project-config syncing" 2021-12-07 00:39:57 +00:00
Zuul
8d192c1c4a Merge "Update the gerritbot-matrix image to support arbitrary uid with docker" 2021-12-06 23:26:12 +00:00
Zuul
5fd1766e65 Merge "Update letsencrypt role docs to suggest a specific order" 2021-12-06 22:23:18 +00:00
Zuul
5ad4066547 Merge "Update gitea to 1.15.7" 2021-12-06 22:21:34 +00:00
Tristan Cacqueray
990b9c1905 Update the gerritbot-matrix image to support arbitrary uid with docker
This change pulls in https://softwarefactory-project.io/r/c/software-factory/gerritbot-matrix/+/23348 to achieve abritary uid with docker.

Change-Id: I43198af4feebd7f175ffdede376e2e4d19928d62
2021-12-06 20:36:11 +00:00
Clark Boylan
a57f3bab08 Update gitea to 1.15.7
Changelog here:
  https://github.com/go-gitea/gitea/blob/v1.15.7/CHANGELOG.md#1157---2021-12-01

Seems to mostly be a number of bugfixes which we like to get.

Note there is one tiny mostly inconsequently template update that we
make to a commented out portion of the head navbar file. But we do this
to avoid unnecessary future delta that needs to be resolved.

Change-Id: I9bf19c4f63d1713701ec889a705f3c8855d943d7
2021-12-06 11:02:13 -08:00
Clark Boylan
d4f98bb426 Unset the matrix gerritbot uid:gid settings
We thought the latest image would be happy based on updates that
tristanC made. Unfortunately ssh complains with:

  2021-12-06 18:33:14.616 [ThreadId 19]: [ERROR] ssh process exited: 255
  2021-12-06 18:33:15.618 [ThreadId 19]: Connecting to review.opendev.org:29418
  No user exists for uid 11000

Undo only the uid:gid settings as the new image seems to work otherwise.

Change-Id: I9c0963ea5c78cecb99e0070b06f9ebd8876a3157
2021-12-06 10:37:12 -08:00
Zuul
3026c0a77b Merge "Update the gerritbot-matrix image to support arbitrary uid" 2021-12-06 18:18:53 +00:00
James E. Blair
f131ae98a8 Correct keycloak proxy config
Some extra steps are needed to use keycloak with a reverse proxy.
This adjusts the apache config to send the required headers and
the keycloak server config to use them.

Since the openid configuration json page is constructed entirely
from these headers (and not from static configuration), this is
a good test that the entire system is working.

Change-Id: I662dc85836d640cb732f12f39e9a61607767fcf3
2021-12-04 10:49:11 -08:00
Zuul
94bc7c1455 Merge "Add a keycloak server" 2021-12-04 16:50:26 +00:00
James E. Blair
da492287ca Add keycloak01
Change-Id: I68c33a7fc8a5fb8ca1133eb6b03961931db2fb01
2021-12-03 16:37:19 -08:00
James E. Blair
7f96224ef9 Update letsencrypt role docs to suggest a specific order
In reviews on https://review.opendev.org/819923 we discovered we
are inconsistent in how we create certs.  Suggest a specific course
of action and record the reasoning.

Change-Id: I974a1717a74e759ca8805dcb707efc7fe29ba53f
2021-12-03 14:24:13 -08:00
James E. Blair
e79dbbe6bb Add a keycloak server
This adds a keycloak server so we can start experimenting with it.

It's based on the docker-compose file Matthieu made for Zuul
(see https://review.opendev.org/819745 )

We should be able to configure a realm and federate with openstackid
and other providers as described in the opendev auth spec.  However,
I am unable to test federation with openstackid due its inability to
configure an oauth app at "localhost".  Therefore, we will need an
actual deployed system to test it.  This should allow us to do so.

It will also allow use to connect realms to the newly available
Zuul admin api on opendev.

It should be possible to configure the realm the way we want, then
export its configuration into a JSON file and then have our playbooks
or the docker-compose file import it.  That would allow us to drive
change to the configuration of the system through code review.  Because
of the above limitation with openstackid, I think we should regard the
current implementation as experimental.  Once we have a realm
configuration that we like (which we will create using the GUI), we
can chose to either continue to maintain the config with the GUI and
appropriate file backups, or switch to a gitops model based on an
export.

My understanding is that all the data (realms configuration and session)
are kept in an H2 database.  This is probably sufficient for now and even
production use with Zuul, but we should probably switch to mariadb before
any heavy (eg gerrit, etc) production use.

This is a partial implementation of https://docs.opendev.org/opendev/infra-specs/latest/specs/central-auth.html

We can re-deploy with a new domain when it exists.

Change-Id: I2e069b1b220dbd3e0a5754ac094c2b296c141753
Co-Authored-By: Matthieu Huin <mhuin@redhat.com>
2021-12-03 14:17:23 -08:00
Jeremy Stanley
1987f86a9a Revert "infra-prod: clone source once"
This reverts commit 42df57b545d6f8dd314678174c281c249171c1d0.
This reverts commit 9cccb02bb09671fc98e42b335e649589610b33cf.

Change-Id: I56be9bcf54b634b7403e71af8b4d08d234cbb91a
Depends-On: https://review.opendev.org/820251
2021-12-02 19:18:43 +00:00
Zuul
fc257bdcaa Merge "Create a new lists.openinfra.dev mailing list site" 2021-12-02 17:57:48 +00:00
Zuul
89597880ec Merge "Upgrade to gerrit 3.3.8" 2021-12-02 03:14:10 +00:00
Zuul
45045ae521 Merge "haproxy: map in config as ro" 2021-12-01 18:45:58 +00:00
Dr. Jens Harbott
26805b2bb5
Fix name for haproxy-statsd dependency
Mixed up with gitea-lb naming.
Fixes I19db98fcec5715c33b62c9c9ba5234fd55700fd8

Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I91d077102904a2144d12bc60eb7341f1065473b4
2021-12-01 17:32:31 +01:00
Ian Wienand
42df57b545 infra-prod: fix name of clone source job
This was introduced with I19db98fcec5715c33b62c9c9ba5234fd55700fd8

opendev-infra-prod-setup-src is the abstract parent job, we should be
using infra-prod-setup-src.

Change-Id: I7fdefe7ce60ab248f9a90b6be363eefc826f8e1f
2021-12-01 13:48:44 +11:00
Zuul
c663d9b4ea Merge "Make haproxy role more generic" 2021-12-01 00:50:45 +00:00
Zuul
8b4c6487c8 Merge "infra-prod: clone source once" 2021-11-30 23:47:34 +00:00
Ian Wienand
3c993c317b haproxy: map in config as ro
The container shouldn't need to write to its config

Change-Id: Iadbe7b663615623fbd6201aa1f2557b0eb267179
2021-12-01 09:57:05 +11:00
Ian Wienand
f29aa2da16 Make haproxy role more generic
This makes the haproxy role more generic so we can run another (or
potentially even more) haproxy instance(s) to manage other services.

The config file is moved to a variable for the haproxy role.  The
gitea specific config is then installed for the gitea-lb service by a
new gitea-lb role.

statsd reporting is made optional with an argument.  This
enables/disables the service in the docker compose.

Role documenation is updated.

Needed-By: https://review.opendev.org/678159
Change-Id: I3506ebbed9dda17d910001e71b17a865eba4225d
2021-12-01 09:55:45 +11:00
Zuul
6bcf28b5a2 Merge "Cache Ansible Galaxy on CI mirror servers" 2021-11-30 21:03:56 +00:00
Clark Boylan
4285b40928 Upgrade to gerrit 3.3.8
There are new gerrit releases. This change updates our production 3.3
image to 3.3.8. We also update Our 3.4 image to 3.4.2 to keep up there.

Release notes for both:

  https://www.gerritcodereview.com/3.3.html#338
  https://www.gerritcodereview.com/3.4.html#342

Seems to largely be bugfixes and reindexing improvements.

Change-Id: Iae8aa403b4001937320767d4166a6af2bc89a2ea
2021-11-29 16:18:15 -08:00
Zuul
81155d6e8f Merge "Update etherpad to 1.18.16" 2021-11-29 22:01:48 +00:00
Zuul
b2c1a9c545 Merge "Refactor infra-prod jobs for parallel running" 2021-11-29 19:58:44 +00:00
Clark Boylan
ce67fad1f5 Update etherpad to 1.18.16
This updates our etherpad image to version 1.18.16. To do this we've had
to adopt the upstream Dockerfile locally and introduce our own edits as
upstream hasn't build images for this release or the previous one.

Minor updates are made to the upstream Dockerfile to update the
maintainer directive and convert from using a local copy of the source
code to a git clone.

Change-Id: I561680072085caff751e08b6f2fd79dee1d4efe8
2021-11-29 08:44:55 -08:00
Jeremy Stanley
a96bbca946 Add zuul01 to cacti
It would be nice to get some idea of how its resource utilization
compares to 02, especially as it runs on a smaller flavor.

Change-Id: If00a949a575949cb3b1a2d8268ae29e4c4965a0b
2021-11-23 22:43:54 +00:00
Jeremy Stanley
33fc2a4d4e Create a new lists.openinfra.dev mailing list site
The Open Infrastructure Foundation has a number of mailing lists
located in the lists.openstack.org site due to historical reasons
(from when they were the OpenStack Foundation). In order to better
disambiguate their mailing lists, a new Mailman site is being
created into which they'll be moved, leaving the old site
exclusively for OpenStack project-specific lists.

As a first step, create the new lists.openinfra.dev site with the
default "mailman" meta-list (which will be hidden once created).
Subsequent changes will create new lists, and remove/redirect the
old ones once configuration is manually replicated.

Change-Id: I64770fbc33184374f1d24f4a2c234f849ab47bce
2021-11-22 19:46:48 +00:00
Jeremy Stanley
0613cfda31 Cache Ansible Galaxy on CI mirror servers
Ansible Galaxy indexes tarballs of Ansible roles and collections at
a central site, which in turn points to a dedicated Amazon S3
subdomain. The tools which consume it support overriding the default
Galaxy URL with any arbitrary one, so should be able to take
advantage of this in CI jobs.

Change-Id: Ib5664e5588f7237a19a2cdb6eec3109452e8a107
2021-11-22 18:49:58 +00:00
Tristan Cacqueray
91783d0380 Update the gerritbot-matrix image to support arbitrary uid
This change pulls in https://softwarefactory-project.io/r/c/software-factory/gerritbot-matrix/+/23196
to achieve abritary uid.

Change-Id: I973b1621aad3ed5ca9d65611ff87fa8124e2fd55
2021-11-19 21:46:45 +00:00
Clark Boylan
ee9cec707b Partial revert of matrix-gerritbot user change
It complains about not being able to get or create the default cache
directory (but doesn't tell us what that directory is). We'll have to
sort this out later.

Change-Id: I5ce7a875ede77c6203d1b5d06da97f8c52ee48e1
2021-11-19 10:10:22 -08:00
Zuul
76a605a1dc Merge "Run matrix-gerritbot with gerritbot user" 2021-11-19 16:49:23 +00:00
Zuul
7cea2a9f83 Merge "infra-prod: remove master override steps" 2021-11-19 02:43:22 +00:00
Zuul
4f2d4cef4c Merge "Run gerritbot with a user that will be shared with matrix-gerritbot" 2021-11-18 19:00:16 +00:00
Zuul
2dfb8db232 Merge "infra-prod: remove duplicated bridge key setup" 2021-11-18 04:39:08 +00:00
Ian Wienand
9cccb02bb0 infra-prod: clone source once
The current opendev-infra-prod-base job sets up the executor to log
into bridge AND copies in Zuul's checkout of system-config to
/home/zuul/src.

This presents an issue for parallel operation, as every production job
is cloning system-config ontop of each other.

Since they all operate in the same buildset, we only need to clone
system-config from Zuul once, and then all jobs can share that repo.

This adds a new job "infra-prod-setup-src" which does this.  It is a
dependency of the base job so should run first.

All other jobs now inhert from opendev-infra-prod-setup-keys, which
only sets up the executor for logging into bridge.

Change-Id: I19db98fcec5715c33b62c9c9ba5234fd55700fd8
Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/807807
2021-11-18 10:31:16 +11:00
Ian Wienand
d0467bfc98 Refactor infra-prod jobs for parallel running
Refactor the infra-prod jobs to specify dependencies so they can run
in parallel.

Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-11-18 10:31:11 +11:00
Ian Wienand
2bc34a1593 infra-prod: remove master override steps
The dependent change moves this into the common infra-prod-base job so
we don't have to do this in here.

Change-Id: I444d2844fe7c7560088c7ef9112893da1496ae62
Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/818189
2021-11-18 09:07:00 +11:00
Ian Wienand
150b3a70da infra-prod: remove duplicated bridge key setup
The known_host key is written out by the parent infra-prod-base job in
the run-production-playbook.yaml step [1].  We don't need to do this
here again.

[1] 2c194e5cbf/playbooks/zuul/run-production-playbook.yaml (L1)

Change-Id: I514132b2dbc20ac321a79ca2eb6d4c8b11c4296d
2021-11-18 09:05:55 +11:00
Zuul
0a279742fc Merge "gerrit: test reviewed flag add/delete/add cycle" 2021-11-17 20:33:42 +00:00
Zuul
2c194e5cbf Merge "mirror: Add centos-stream link" 2021-11-16 00:27:45 +00:00
Ian Wienand
ebc90eac5b mirror: Add centos-stream link
Missed this with I483c2982a6931e7d6fc97ab82f7750b72d2ef265; this
ensure the mirror webserver exports the directory.

Change-Id: I6e14cdace213a6af6df65b8ddb09bb3a167fbf9b
2021-11-16 09:25:56 +11:00
Zuul
1129e6855a Merge "Lower UID/GID range max to make way for containers" 2021-11-15 20:20:19 +00:00
Zuul
23e218a789 Merge "Cleanup users launch-node.py might have used" 2021-11-15 17:56:55 +00:00
Ian Wienand
855efc9010 Enable mirroring of 9-stream
This is a re-implementation of
I195ebee548071b0b89bd5bf64b251595271178ca that puts 9-stream in a
separate AFS volume

(Note the automated volume name "mirror.centos-stream" comes just
short of the limit)

Change-Id: I483c2982a6931e7d6fc97ab82f7750b72d2ef265
2021-11-15 17:54:54 +11:00
Ian Wienand
d97eff271c Revert "Enable mirroring of centos stream 9 contents"
This reverts commit 8591ce2b5c689b5e438fc0bfe9d410be7e344fb1.

It did not click that this is written to use
/afs/.openstack.org/mirror/centos-stream as the base directory.  The
mirror/ directory has volumes mounted in it -- i.e. centos-stream has
to be a new volume (and also has to be "vos released" separately, the
existing script won't do it).

The simplest way to do this is to treat this separately.  I'll propose
this in a follow-on.

Change-Id: If7b8239adf7635da4f0c317287d23daf5ab0f4bf
2021-11-15 17:05:33 +11:00
Zuul
3c4a93fa7c Merge "Enable mirroring of centos stream 9 contents" 2021-11-15 05:23:13 +00:00
Chandan Kumar (raukadah)
8591ce2b5c Enable mirroring of centos stream 9 contents
It picks the rackspace mirror from this list
https://admin.fedoraproject.org/mirrormanager/mirrors/CentOS/9-stream/x86_64
which is present in US.

It moves base directory to centos-stream to be consistent to centos
mirrors.

We will only synchronize x86_64 and aarch64 arches as those are the only
ones used in opendev CI. We also exculde source and debug directories to
optimize space usage as those are only required for debugging purposes.

Change-Id: I195ebee548071b0b89bd5bf64b251595271178ca
2021-11-12 09:41:26 +01:00