130 Commits

Author SHA1 Message Date
Zuul
a1885ef992 Merge "Update limboria ircbot to bullseye" 2021-12-15 22:27:33 +00:00
Zuul
29fbc1f078 Merge "Update matrix-eavesdrop image to bullseye" 2021-12-15 17:46:36 +00:00
Zuul
57d5e116a0 Merge "Update the accessbot image to bullseye" 2021-12-14 23:40:39 +00:00
Zuul
63fb188aa3 Merge "Update the hound image to bullseye" 2021-12-13 22:08:29 +00:00
Clark Boylan
22957c6549 Update limboria ircbot to bullseye
Spring cleaning updates of our docker images now that bullseye is out.

Change-Id: I5e4b84edd2c5a8e196659e4815c5b349c0226393
2021-12-13 09:22:17 -08:00
Clark Boylan
ed0526cd8b Update the accessbot image to bullseye
This is general spring cleaning that we are going to try and do for our
images now that bullseye is out.

Change-Id: Iad8f5b76896b88a6aafbfba0c38d0749b9d5c88f
2021-12-13 09:18:56 -08:00
Clark Boylan
b07d5eca37 Update matrix-eavesdrop image to bullseye
Just some spring cleaning now that bullseye is released.

Change-Id: I9641dae9ee7679fb45bef93e770f69d9673d75bf
2021-12-13 09:12:10 -08:00
Clark Boylan
8530ed39a1 Update the hound image to bullseye
Just some spring cleaning now that bullseye has released.

Change-Id: I1202400932860a04841d376b9f10beb89acc175c
2021-12-13 09:04:20 -08:00
Ian Wienand
5a215e0654 infra-prod: fix infra-prod-service-zookeeper soft dependency
This is a typo from the job shuffle in
I8f6150ec2f696933c93560c11fed0fd16b11bf65 -- this should be a soft
dependency.

It is currently causing periodic jobs to fail

Change-Id: Ia420e74a1d64b12b63b1697e61992c46119451dc
2021-12-13 11:01:45 +11:00
Ian Wienand
73a9acc7ad Rename install-ansible to bootstrap-bridge
This used to be called "bridge", but was then renamed with
Ia7c8dd0e32b2c4aaa674061037be5ab66d9a3581 to install-ansible to be
clearer.

It is true that this is installing Ansible, but as part of our
reworking for parallel jobs this is the also the synchronisation point
where we should be deploying the system-config code to run for the
buildset.

Thus naming this "boostrap-bridge" should hopefully be clearer again
about what's going on.

I've added a note to the job calling out it's difference to the
infra-prod-service-bridge job to hopefully also avoid some of the
inital confusion.

Change-Id: I4db1c883f237de5986edb4dc4c64860390cc8e22
2021-12-07 16:24:53 +11:00
Zuul
94bc7c1455 Merge "Add a keycloak server" 2021-12-04 16:50:26 +00:00
James E. Blair
e79dbbe6bb Add a keycloak server
This adds a keycloak server so we can start experimenting with it.

It's based on the docker-compose file Matthieu made for Zuul
(see https://review.opendev.org/819745 )

We should be able to configure a realm and federate with openstackid
and other providers as described in the opendev auth spec.  However,
I am unable to test federation with openstackid due its inability to
configure an oauth app at "localhost".  Therefore, we will need an
actual deployed system to test it.  This should allow us to do so.

It will also allow use to connect realms to the newly available
Zuul admin api on opendev.

It should be possible to configure the realm the way we want, then
export its configuration into a JSON file and then have our playbooks
or the docker-compose file import it.  That would allow us to drive
change to the configuration of the system through code review.  Because
of the above limitation with openstackid, I think we should regard the
current implementation as experimental.  Once we have a realm
configuration that we like (which we will create using the GUI), we
can chose to either continue to maintain the config with the GUI and
appropriate file backups, or switch to a gitops model based on an
export.

My understanding is that all the data (realms configuration and session)
are kept in an H2 database.  This is probably sufficient for now and even
production use with Zuul, but we should probably switch to mariadb before
any heavy (eg gerrit, etc) production use.

This is a partial implementation of https://docs.opendev.org/opendev/infra-specs/latest/specs/central-auth.html

We can re-deploy with a new domain when it exists.

Change-Id: I2e069b1b220dbd3e0a5754ac094c2b296c141753
Co-Authored-By: Matthieu Huin <mhuin@redhat.com>
2021-12-03 14:17:23 -08:00
Jeremy Stanley
1987f86a9a Revert "infra-prod: clone source once"
This reverts commit 42df57b545d6f8dd314678174c281c249171c1d0.
This reverts commit 9cccb02bb09671fc98e42b335e649589610b33cf.

Change-Id: I56be9bcf54b634b7403e71af8b4d08d234cbb91a
Depends-On: https://review.opendev.org/820251
2021-12-02 19:18:43 +00:00
Dr. Jens Harbott
26805b2bb5
Fix name for haproxy-statsd dependency
Mixed up with gitea-lb naming.
Fixes I19db98fcec5715c33b62c9c9ba5234fd55700fd8

Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I91d077102904a2144d12bc60eb7341f1065473b4
2021-12-01 17:32:31 +01:00
Ian Wienand
42df57b545 infra-prod: fix name of clone source job
This was introduced with I19db98fcec5715c33b62c9c9ba5234fd55700fd8

opendev-infra-prod-setup-src is the abstract parent job, we should be
using infra-prod-setup-src.

Change-Id: I7fdefe7ce60ab248f9a90b6be363eefc826f8e1f
2021-12-01 13:48:44 +11:00
Ian Wienand
9cccb02bb0 infra-prod: clone source once
The current opendev-infra-prod-base job sets up the executor to log
into bridge AND copies in Zuul's checkout of system-config to
/home/zuul/src.

This presents an issue for parallel operation, as every production job
is cloning system-config ontop of each other.

Since they all operate in the same buildset, we only need to clone
system-config from Zuul once, and then all jobs can share that repo.

This adds a new job "infra-prod-setup-src" which does this.  It is a
dependency of the base job so should run first.

All other jobs now inhert from opendev-infra-prod-setup-keys, which
only sets up the executor for logging into bridge.

Change-Id: I19db98fcec5715c33b62c9c9ba5234fd55700fd8
Depends-On: https://review.opendev.org/c/opendev/base-jobs/+/807807
2021-11-18 10:31:16 +11:00
Ian Wienand
d0467bfc98 Refactor infra-prod jobs for parallel running
Refactor the infra-prod jobs to specify dependencies so they can run
in parallel.

Change-Id: I8f6150ec2f696933c93560c11fed0fd16b11bf65
2021-11-18 10:31:11 +11:00
Zuul
9c29fd8324 Merge "Remove the gerrit group in favor of the review group" 2021-10-22 16:15:56 +00:00
Zuul
0017bdc468 Merge "Replace testing group vars with host vars for review02" 2021-10-13 17:16:31 +00:00
Clark Boylan
cf91bc0971 Remove the gerrit group in favor of the review group
Having two groups here was confusing. We seem to use the review group
for most ansible stuff so we prefer that one. We move contents of the
gerrit group_vars into the review group_vars and then clean up the use
of the old group vars file.

Change-Id: I7fa7467f703f5cec075e8e60472868c60ac031f7
2021-10-12 09:48:53 -07:00
Clark Boylan
76baae4e3f Replace testing group vars with host vars for review02
Previously we had a test specific group vars file for the review Ansible
group. This provided junk secrets to our test installations of Gerrit
then we relied on the review02.opendev.org production host vars file to
set values that are public.

Unfortunately, this meant we were using the production heapLimit value
which is far too large for our test instances leading to the occasionaly
failure:

  There is insufficient memory for the Java Runtime Environment to continue.
  Native memory allocation (mmap) failed to map 9596567552 bytes for committing reserved memory.

We cannot set the heapLimit in the group var file because the hostvar
file overrides those values. To fix this we need to replace the test
specific group var contents with a test specific host var file instead.
To avoid repeating ourselves we also create a new review.yaml group_vars
file to capture common settings between testing and prod. Note we should
look at combining this new file with the gerrit.yaml group_vars.

On the testing side of things we set the heapLimit to 6GB, we change the
serverid value to prevent any unexpected notedb confusion, and we remove
replication config.

Change-Id: Id8ec5cae967cc38acf79ecf18d3a0faac3a9c4b3
2021-10-12 09:48:45 -07:00
Clark Boylan
46faa6626b Remove Gerrit 3.2 images
This should be merged after we are on 3.3 and happy with the state of
things.

Depends-On: https://review.opendev.org/c/openstack/project-config/+/813081
Change-Id: I4173df5e4ae38af6423402be0299470323762da2
2021-10-07 20:07:38 +00:00
Clark Boylan
0f6c29c0ee Test upgrade from Gerrit 3.3 to 3.4
This shifts our Gerrit upgrade testing ahead to testing 3.3 to 3.4
upgrades as we have upgraded to 3.3 at this point.

Change-Id: Ibb45113dd50f294a2692c65f19f63f83c96a3c11
2021-10-07 11:57:04 -07:00
Monty Taylor
d49f399b17 Start building gerrit 3.4
Change-Id: I9cd8c9e1fb837dae91057da9bc80a3f15e566a59
2021-10-07 11:54:50 -07:00
Clark Boylan
e47dccdc34 Upgrade Gerrit to 3.3
This bumps the gerrit image up to our 3.3 image. Followup changes will
shift upgrade testing to test 3.3 to 3.4 upgrades, clean up no longer
needed 3.2 images, and start building 3.4 images.

Change-Id: Id0f544846946d4c50737a54ceb909a0a686a594e
2021-10-07 11:54:46 -07:00
Ian Wienand
e772abaf96 gitea: use assets bundle
This uses the opendev assets bundle image created with
I3166679bde6d771276289b9d32e7e4407957b2f8.

The mount options require using BuildKit, hence the Dockerfile update.

Otherwise conceptually it's fairly simple; copy in the files from the
opendevorg/assets image rather than the file-system.

Change-Id: I36bdc76471eec5380a676ebcdd885a88d3985976
2021-09-06 15:07:36 +10:00
Ian Wienand
25cdc97950 Add assets and a related docker image/bundle
Move some common assets into a top-level assets/ directory.  Services
can reference these assets via

 https://opendev.org/opendev/system-config/raw/branch/master/assets/<file>

in <img> tags, etc.

Some services want to embed these into their images, but we wish to
only keep one canonical copy.  For this, add a Dockerfile and jobs
that creates a simple bundle of assets in opendevorg/assets.  This can
be referenced in other builds; the new BuildKit bind-mount is
particularly useful for this
(c.f. I36bdc76471eec5380a676ebcdd885a88d3985976).

Change-Id: I3931566eb86a0618705d276445fa0a5f659692ea
2021-09-01 06:15:43 +10:00
Monty Taylor
8dbf0a3d82 Produce both buster and bullseye container images
This will allow us to roll out consumption forward in a
methodical manner.

This reverts commit 45caec4d43900bc66fb0b8c219c6dcc3180ca8aa.

Note the weird ARG definitions are there for a reason:
https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact

Change-Id: I81174ac035164695f1c27c9662f25335b78c2e64
2021-08-28 16:46:29 -07:00
Clark Boylan
aeddc1bf17 Test a gerrit 3.2 -> 3.3 upgrade
We create (a currently test only) playbook that upgrades zuul. This job
then runs through project creation and renaming and testinfra testing on
the upgraded gerrit version.

Future improvements should consider loading state on the old gerrit
install before we upgrade that can be asserted as well.

Change-Id: I364037232cf0e6f3fa150f4dbb736ef27d1be3f8
2021-08-19 13:19:05 -07:00
Clark Boylan
ce5d207dbb Run remote-puppet-else daily instead of hourly
Update the file matchers to actually match the current set of puppet
things. This ensure the deploy job runs when we want it and we can catch
up daily instead of hourly.

Previously a number of the matchers didn't actually match the puppet
things because the path prefix was wrong or works were in different
orders for the dir names.

Change-Id: I3510da81d942cf6fb7da998b8a73b0a566ea7411
2021-08-17 15:54:38 -07:00
Clark Boylan
ffe06527de Run infra-prod-service-zuul-preview daily instaed of hourly
This is being done beacuse we don't make many changes to the
zuul-preview service but it runs in the hourly buildset starving deploy
runs. Since this doesn't change much we can move it to the daily run
instead.

If we need to update it we can run the playbook manually or land a
change to trigger it.

Change-Id: I89d2c712fcfd18bd4f694b2c90067295253b8836
2021-08-17 15:45:17 -07:00
Clark Boylan
268fc98bd7 Remove extra service-codesearch job in deploy
This job was listed twice. Remove the extra one for clarity.

Change-Id: I7aa39e3757d6562af474ec7c9cfdda7d8024cd1c
2021-08-16 11:42:37 -07:00
Clark Boylan
711bf9e9f8 Run the cloud launcher daily instead of hourly
This is a job that takes quite a bit of time, but only rarely do we need
the updates encoded in this job. Move the job from our hourly deployment
to the daily deployment to make its impact less painful.

Change-Id: I724bcdd67f4c324f497a9d8239bcfd8d37528956
2021-08-16 11:41:37 -07:00
Zuul
af5fcdcb13 Merge "Run matrix-eavesdrop on eavesdrop" 2021-08-02 17:00:09 +00:00
Zuul
ab092e721f Merge "Add matrix-eavesdrop container image" 2021-08-02 16:35:25 +00:00
James E. Blair
82c966e6da Run matrix-eavesdrop on eavesdrop
Thin runs the new matrix-eavesdrop bot on the eavesdrop server.

It will write logs out to the limnoria logs directory, which is mounted
inside the container.

Change-Id: I867eec692f63099b295a37a028ee096c24109a2e
2021-07-28 18:34:58 -05:00
James E. Blair
b58b204a8e Add matrix-eavesdrop container image
This builds a container image with a simple eavesdrop bot for Matrix.

Change-Id: I5304b4ec974b84886ac969b59cfcec8dec2febf9
2021-07-23 14:28:22 -07:00
Ian Wienand
cec6372288 Add infra-service-deploy-paste to deploy pipeline
This was added in periodic, but it should also be in the deploy
pipline.

Change-Id: I392e955667ed56e38c0c1b2386562e04b8dd8dd1
2021-07-13 11:27:15 +10:00
Ian Wienand
916c1d3dc8 Add paste service
The paste service needs an upgrade; since others have created a
lodgeit container it seems worth us keeping the service going if only
to maintain the historical corpus of pastes.

This adds the ansible to deploy lodgeit and a sibling mariadb
container.  I have imported a dump of the old data as a test.  The
dump is ~4gb and imported it takes up about double that; certainly
nothing we need to be too concerned over.  The server will be more
than capable of running the db container alongside the lodgeit
instance.

This should have no effect on production until we decide to switch
DNS.

Change-Id: I284864217aa49d664ddc3ebdc800383b2d7e00e3
2021-07-07 15:12:04 +10:00
Zuul
bfaa4713eb Merge "Remove system-config-legacy-logstash-filters job" 2021-06-10 17:29:17 +00:00
Ian Wienand
403773d55a limnoria/meetbot setup on eavesdrop01.opendev.org
This installs our Limnoira/meetbot container and configures it on
eavesdrop01.opendev.org.  I have ported the configuration from the old
puppet as best I can (it is very verbose); my procedure was to use the
Limnoira wizard to start a new config file then backport everything
from the old file.  I felt this was best to not miss any new options.

This does channel logging (via built-in ChannelLogger plugin, along
with a cron job for logs2html) and runs our fork of meetbot.

It exports the channel logs via HTTP to /irclogs and meetings logs to
/meetings.  meetings.opendev.org will proxy to these two locations
when the server is active.

Note this has not ported the channel list; so the bot will not be
listening in our channels.

Change-Id: I9f9a466c271e1a706f9f98f816de0e84047519f1
2021-06-10 09:02:16 +10:00
Ian Wienand
0d00b28da8 Create ircbot container
This container installs Limnoria, the supybot replacement as the
generic ircbot container.  We install meetbot plugin as a sibling
project.

Previously we've conflated supybot with meetbot, which is a bit
confusing because meetbot is a plugin, but we also use other plugins
such as the channel logger.  We also hope to convert some of our other
bots to Limnoria (ptgbot?) to consolidate everything.  For this reason
I've called this the more generic "ircbot".  The image installs
meetbot as a sibling project, with the idea being any other plugins
would also be installed as siblings.

The siblings install expects the work directory to be a relative
directory.  I'm not sure we run this from other projects, but this
will work the same if we do.

Depends-On: https://review.opendev.org/c/opendev/meetbot/+/793876
Change-Id: Icee4c6bbb5ea235ba69c10f800a14bbf5beef3d5
2021-06-10 09:00:43 +10:00
Clark Boylan
6e04e500fd Remove system-config-legacy-logstash-filters job
We're trying to phase out the ELK systems. While we have agreed to not
immediately turn anything off we probably don't need to keep running the
system-config-legacy-logstash-filters job as ELK should remain fairly
fixed unless someone rewrites config management for it and modernizes
it. And if that happens they will want new modern testing too.

Depends-On: https://review.opendev.org/c/openstack/project-config/+/792710
Change-Id: I9ac6f12ec3245e3c1be0471d5ed17caec976334f
2021-05-21 17:03:32 -07:00
Zuul
be4f67f23e Merge "Add infra-prod-service-lists job" 2021-05-19 19:16:32 +00:00
Clark Boylan
caedb11d3d Add infra-prod-service-lists job
This job is not added in the parent so that we can manually run
playbooks after the parent lands. Once we are happy with the results
from the new service-lists.yaml playbook we can land this change and
have zuul automatically apply it when necessary.

Change-Id: I38de8b98af9fb08fa5b9b8849d65470cbd7b3fdc
2021-05-11 08:40:06 -07:00
Ian Wienand
629fdec768 Build Python 3.9 python-builder/base containers
Python 3.9 is released, so let's build containers.

This splits the docker-images/ files up as they are becoming a bit
crowded.

Change-Id: Id68080575a30e4a08c99df0af603fbb65a0983bd
2021-05-05 09:55:56 +10:00
Ian Wienand
9f11fc5c75 Remove references to review-dev
With our increased ability to test in the gate, there's not much use
for review-dev any more.  Remove references.

Change-Id: I97e9865e0b655cd157acf9ffa7d067b150e6fc72
2021-03-24 11:40:31 +11:00
James E. Blair
96bac7b486 Add zookeeper-statsd
This adds a program, zookeeper-statsd, which monitors zookeeper
metrics and reports them to statsd.  It also adds a container to
run that program.  And it runs the container on each of the
ZooKeeper quorum members.  And it updates the graphite host to
allow statsd traffic from quorum members.  And it updates the
4-letter-word whitelist to allow the mntr command (which is used
to gather metrics) to be issued.

Change-Id: I298f0b13a05cc615d8496edd4622438507fc5423
2021-03-17 14:52:31 -07:00
Zuul
77b1c14a9a Merge "Use upstream jitsi-meet web image" 2021-03-17 00:22:50 +00:00
Ian Wienand
c1aff2ed38 kerberos-kdc: role to manage Kerberos KDC servers
This adds a role and related testing to manage our Kerberos KDC
servers, intended to replace the puppet modules currently performing
this task.

This role automates realm creation, initial setup, key material
distribution and replica host configuration.  None of this is intended
to run on the production servers which are already setup with an
active database, and the role should be effectively idempotent in
production.

Note that this does not yet switch the production servers into the new
groups; this can be done in a separate step under controlled
conditions and with related upgrades of the host OS to Focal.

Change-Id: I60b40897486b29beafc76025790c501b5055313d
2021-03-17 08:30:52 +11:00