145 Commits

Author SHA1 Message Date
Zuul
91c6b2ae84 Merge "Build our gerrit images on Bullseye" 2021-10-07 00:12:30 +00:00
Clark Boylan
88a6a202e9 Build our gerrit images on Bullseye
This is a bit of spring cleaning. Previously we based on images on
Buster but Bullseye exists now so give it a go.

Change-Id: Icc3d79b361e41df2f2f063993fd206ab7d992f75
2021-09-15 13:45:31 -07:00
Clark Boylan
17ba858b19 Update our gitea images to bullseye
To do this we also update jinja-init to bullseye and gitea seems to be
the only user of this image. The impact of this should be fairly self
contained to gitea.

Note this update isn't urgent, but good hygiene. We should coordinate
this update with the 1.15.x gitea upgrade and do them in such a sequence
that we can identify problems easily if they pop up.

Change-Id: Ia0075416a1d8a067cfecd26c03f8db9641cbcb89
2021-09-15 10:49:42 -07:00
Ian Wienand
e772abaf96 gitea: use assets bundle
This uses the opendev assets bundle image created with
I3166679bde6d771276289b9d32e7e4407957b2f8.

The mount options require using BuildKit, hence the Dockerfile update.

Otherwise conceptually it's fairly simple; copy in the files from the
opendevorg/assets image rather than the file-system.

Change-Id: I36bdc76471eec5380a676ebcdd885a88d3985976
2021-09-06 15:07:36 +10:00
Ian Wienand
25cdc97950 Add assets and a related docker image/bundle
Move some common assets into a top-level assets/ directory.  Services
can reference these assets via

 https://opendev.org/opendev/system-config/raw/branch/master/assets/<file>

in <img> tags, etc.

Some services want to embed these into their images, but we wish to
only keep one canonical copy.  For this, add a Dockerfile and jobs
that creates a simple bundle of assets in opendevorg/assets.  This can
be referenced in other builds; the new BuildKit bind-mount is
particularly useful for this
(c.f. I36bdc76471eec5380a676ebcdd885a88d3985976).

Change-Id: I3931566eb86a0618705d276445fa0a5f659692ea
2021-09-01 06:15:43 +10:00
Monty Taylor
8dbf0a3d82 Produce both buster and bullseye container images
This will allow us to roll out consumption forward in a
methodical manner.

This reverts commit 45caec4d43900bc66fb0b8c219c6dcc3180ca8aa.

Note the weird ARG definitions are there for a reason:
https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact

Change-Id: I81174ac035164695f1c27c9662f25335b78c2e64
2021-08-28 16:46:29 -07:00
Clark Boylan
f4c1735dfb Update Gerrit images to most recent releases
There are new releases of Gerrit. Let's update our 3.2 and 3.3 images to
match.

Change-Id: Id06cbb5eb2c7e6af30d579327b4a4c89b92e6d8d
2021-08-20 15:55:53 -07:00
Clark Boylan
e5ea6ebc38 Rebuild gerrit images to include fixes
We upstreamed fixes for the mariadb jdbc connector and users being able
to orphan their accounts through accidental deletion of their openid
external ids. These fixes are now present in both the stable-3.2 and
stable-3.3 branches of gerrit. We should rebuild these images to ensure
our images include the fixes.

Note that stable-3.4 does not yet include these fixes but should in
time.

We will update our jdbc connection url in a followup change as we don't
auto update our images. This way we can ensure that the new image is
ready to go before updating that config.

Change-Id: Id23215ddfb3bd4424109e77ecd3480ed2375431d
2021-08-03 09:05:14 -07:00
James E. Blair
b58b204a8e Add matrix-eavesdrop container image
This builds a container image with a simple eavesdrop bot for Matrix.

Change-Id: I5304b4ec974b84886ac969b59cfcec8dec2febf9
2021-07-23 14:28:22 -07:00
Clark Boylan
928bb2facb Update gerrit image to v3.2.11
This includes a number of bugfixes that seem worthwhile. Release notes
at: https://www.gerritcodereview.com/3.2.html#3211

Change-Id: I4e568fbd5c7c9d21437ead50e698fa14c9b66b93
2021-07-02 08:59:53 -07:00
Zuul
9181d5198d Merge "gerrit: add mariadb_container option" 2021-06-16 23:14:48 +00:00
Ian Wienand
570ca85cd8 gerrit: add mariadb_container option
This adds a local mariadb container to the gerrit host to hold the
accountPatchReviewDb database.  This is inspired by a few things

 - since migration to NoteDB, there is only one table left where
   Gerrit records what files have been reviewed for a change.  This
   logically scales with the number of reviews users are doing.
   Pulling the stats on this, we can see since the NoteDB upgrade this
   went from a very busy database (~300 queries/70 commits per second)
   to barely registering one hit per second :
   https://imgur.com/a/QGJV7Fw

   Thus separating the db to an external host for performance reasons
   is not a large concern any more.

 - emperically we've done a bad job in keeping the existing hosted db
   up-to-date; it's still running mysql 5.1 and we have been hit by
   bugs such as the one referenced in-line which silently drops
   backups.

 - The other gerrit option is to use an on-disk H2 database.  This is
   certainly an option, however you need special tools to interact
   with it for migration, etc. and it's not safe to backup from files
   on disk (as opposed to mysqldump).  Upstream advice is unclear, and
   varies between H2 being a performance bottleneck to this being
   ephemeral data that users don't care about.  We know how to admin
   mariadb/mysql and this allows us to migrate and backup data, so
   seems like the best choice.

 - we have a pressing need to update the server to a new operating
   system.  Running the db alongside the gerrit instance minimises
   fiddling we have to do manging connections to and migrating the
   hosted db systems.

 - related to that, we are tending towards more provider independence
   for control-plane servers.  A hosted database product is not always
   provided, so this gives us more flexibility in moving things
   around.

 - the main concern here is memory usage.  "docker stats" reports a
   quiescent container, freshly started on a 8GB host:

    gerrit-compose_mariadb_1  67.32MiB

   After loading a copy of the production table, and then dumping it
   back to a file the same container reports:

    gerrit-compose_mariadb_1  462.6MiB

The existing remote mysql configuration path remains mostly the same.
We move the gerrit startup into a script rather than a CMD so we can
call it after a "wait for db" script in the mariadb_container case
(this is the reccommeded way to enforce ordering [1]).

Backups of the local container need different dump commands; backups
are relocated to a new file and updated.

Testing is converted to use this rather than a local H2 database.

[1] https://docs.docker.com/compose/startup-order/

Change-Id: Iec981ef3c2e38889f91e9759e66295dbfb499c2e
2021-06-16 13:57:13 +10:00
Zuul
084879c1fa Merge "limnoria/meetbot setup on eavesdrop01.opendev.org" 2021-06-10 02:04:53 +00:00
Zuul
c51b860620 Merge "Create ircbot container" 2021-06-10 01:04:32 +00:00
Ian Wienand
403773d55a limnoria/meetbot setup on eavesdrop01.opendev.org
This installs our Limnoira/meetbot container and configures it on
eavesdrop01.opendev.org.  I have ported the configuration from the old
puppet as best I can (it is very verbose); my procedure was to use the
Limnoira wizard to start a new config file then backport everything
from the old file.  I felt this was best to not miss any new options.

This does channel logging (via built-in ChannelLogger plugin, along
with a cron job for logs2html) and runs our fork of meetbot.

It exports the channel logs via HTTP to /irclogs and meetings logs to
/meetings.  meetings.opendev.org will proxy to these two locations
when the server is active.

Note this has not ported the channel list; so the bot will not be
listening in our channels.

Change-Id: I9f9a466c271e1a706f9f98f816de0e84047519f1
2021-06-10 09:02:16 +10:00
Ian Wienand
0d00b28da8 Create ircbot container
This container installs Limnoria, the supybot replacement as the
generic ircbot container.  We install meetbot plugin as a sibling
project.

Previously we've conflated supybot with meetbot, which is a bit
confusing because meetbot is a plugin, but we also use other plugins
such as the channel logger.  We also hope to convert some of our other
bots to Limnoria (ptgbot?) to consolidate everything.  For this reason
I've called this the more generic "ircbot".  The image installs
meetbot as a sibling project, with the idea being any other plugins
would also be installed as siblings.

The siblings install expects the work directory to be a relative
directory.  I'm not sure we run this from other projects, but this
will work the same if we do.

Depends-On: https://review.opendev.org/c/opendev/meetbot/+/793876
Change-Id: Icee4c6bbb5ea235ba69c10f800a14bbf5beef3d5
2021-06-10 09:00:43 +10:00
Clark Boylan
b53914f812 Update Gerrit plugin versions for Gerrit 3.2 and 3.3
This updates our plugin versions to keep up with Gerrit releases. We
build from the stable branch which means we need to periodically bump
plugin tagged versions to keep up when stable branches for the plugins
are not available.

Change-Id: I5bb50d8610ebdbdb9c6a70ad8dde732067cc368f
2021-06-08 13:06:00 -07:00
Ian Wienand
5357b33e57 gerrit docker: match some more files
These files are involved in creating gerrit docker images; make sure
we trigger jobs when they are modified.

Change-Id: I7c4436e066cfb0c2d0b2ca7adf54c99b09dac95f
2021-05-07 11:06:13 +10:00
Ian Wienand
629fdec768 Build Python 3.9 python-builder/base containers
Python 3.9 is released, so let's build containers.

This splits the docker-images/ files up as they are becoming a bit
crowded.

Change-Id: Id68080575a30e4a08c99df0af603fbb65a0983bd
2021-05-05 09:55:56 +10:00
Clark Boylan
c2d46f4247 Set up gitea image provides and requires for gating
This should ensure that if we have a parent job that updates the gitea
version and a do not merge child job that induces an artificial failure
for zuul hold purposes that we test the correct image in the child job's
changes.

Prior to this we were testing the existing published images, but
provides + requires will give the correct signaling to make the desired
"test new proposed image" behavior happen in the child change builds.

Change-Id: Ie6b827b650e0f32606dc5ec7f4aa0adfeebdeb5e
2021-03-19 10:33:09 -07:00
James E. Blair
96bac7b486 Add zookeeper-statsd
This adds a program, zookeeper-statsd, which monitors zookeeper
metrics and reports them to statsd.  It also adds a container to
run that program.  And it runs the container on each of the
ZooKeeper quorum members.  And it updates the graphite host to
allow statsd traffic from quorum members.  And it updates the
4-letter-word whitelist to allow the mntr command (which is used
to gather metrics) to be issued.

Change-Id: I298f0b13a05cc615d8496edd4622438507fc5423
2021-03-17 14:52:31 -07:00
James E. Blair
b768325480 Use upstream jitsi-meet web image
This has our change to open etherpad on join, so we should no longer need
to run a fork of the web server.  Switch to the upstream container image
and stop building our own.

Change-Id: I3e8da211c78b6486a3dcbd362ae7eb03cc9f5a48
2021-03-09 12:35:46 -08:00
Clark Boylan
7320c8e6ed Build Gerrit 3.3 images
Gerrit 3.3 has released. Lets start building images for it so that we
can do testing when ready to start that.

We also add testing files to the list of things that trigger the 3.3
builds. Strictly this isn't necessary since the test will continue to
use 3.2 images until we upgrade to 3.3, but this helps us avoid
forgetting to do this when we do upgrade. Little extra jobs run today to
ensure we continue to run the right jobs tomorrow.

Change-Id: Ib7e7d7313e0827a40009df840119444611d74ca2
2021-02-10 15:07:19 -08:00
Zuul
f526060e39 Merge "Deploy refstack with ansible docker" 2021-02-09 03:58:22 +00:00
Clark Boylan
a4604ae0b3 Deploy refstack with ansible docker
This adds a dockerfile to build an opendevorg/refstack image as well as
the jobs to build and publish it.

Change-Id: Icade6c713fa9bf6ab508fd4d8d65debada2ddb30
2021-02-05 19:23:34 +00:00
Zuul
137d518bf8 Merge "Update Gerrit 3.2 plugin versions on image builds" 2021-02-01 09:21:23 +00:00
Ian Wienand
9f4cbcfbc2 Expand gerrit testing to multiple changes
This reworks the gerrit testing slightly to give some broader
coverage.

It sets up ssh keys for the user; not really necessary but can be
helpful when interacting on a held host.

It sets up groups and verification labels just so Zuul can comment
with -2/+2; again this is not really necessary, but makes things a
little closer to production reality.

We make multiple changes, so we can better test navigating between
them.  The change comments are updated to have some randomness in them
so they don't all look the same.  We take screen shots of two change
pages to validate the navigation between them.

Change-Id: I60b869e4fdcf8849de836e33db643743128f8a70
2021-02-01 14:06:08 +11:00
Clark Boylan
2f06ba77cc Update Gerrit 3.2 plugin versions on image builds
Gerrit's 3.2 release is moving ahead and we should keep up with our
plugin version specifications.

Change-Id: I552760a98c00abb7c31c7c6864d74fe951491be7
2021-01-28 10:19:34 -08:00
Ian Wienand
738b4ba739 gerrit: Install zuul-summary-results plugin
This installs the zuul-summary-results plugin into our gerrit
container.  testinfra is updated to take a screenshot of the plugin in
action.

Change-Id: Ie0a165cc6ffc765c03457691901a1dd41ce99d5a
2021-01-18 07:58:23 -08:00
Ian Wienand
1858f15a8a gerrit: move plugins to common code
The "additional_plugins" variable is so different builds gerrit can
specify additional plugins specific to their version to install into
the base image.

Since we've moved to only building 3.2 and master images, a bunch of
plugins that used to be additional (because they weren't 2.XX era) are
now common.  Move them into the common plugin code in the playbook,
and leave the only one different for master, the "checks" plugin, as
separate.

Change-Id: I8966ed7b5436fbe012486dccc1028bc8cb1cf9e4
2021-01-18 07:58:23 -08:00
Clark Boylan
c882808578 Remove container image builds for old gerrit versions
Having upgraded to 3.2, we don't need these versions any more.

Change-Id: Ifc37a75aa62b2498e649a4c81b589a04c794184a
Depends-On: https://review.opendev.org/763617
2020-11-21 13:58:19 -08:00
Zuul
1b16dae681 Merge "Migrate codesearch site to container" 2020-11-19 22:26:12 +00:00
Ian Wienand
368466730c Migrate codesearch site to container
The hound project has undergone a small re-birth and moved to

 https://github.com/hound-search/hound

which has broken our deployment.  We've talked about leaving
codesearch up to gitea, but it's not quite there yet.  There seems to
be no point working on the puppet now.

This builds a container than runs houndd.  It's an opendev specific
container; the config is pulled from project-config directly.

There's some custom scripts that drive things.  Some points for
reviewers:

 - update-hound-config.sh uses "create-hound-config" (which is in
   jeepyb for historical reasons) to generate the config file.  It
   grabs the latest projects.yaml from project-config and exits with a
   return code to indicate if things changed.

 - when the container starts, it runs update-hound-config.sh to
   populate the initial config.  There is a testing environment flag
   and small config so it doesn't have to clone the entire opendev for
   functional testing.

 - it runs under supervisord so we can restart the daemon when
   projects are updated.  Unlike earlier versions that didn't start
   listening till indexing was done, this version now puts up a "Hound
   is not ready yet" message when while it is working; so we can drop
   all the magic we were doing to probe if hound is listening via
   netstat and making Apache redirect to a status page.

 - resync-hound.sh is run from an external cron job daily, and does
   this update and restart check.  Since it only reloads if changes
   are made, this should be relatively rare anyway.

 - There is a PR to monitor the config file
   (https://github.com/hound-search/hound/pull/357) which would mean
   the restart is unnecessary.  This would be good in the near and we
   could remove the cron job.

 - playbooks/roles/codesearch is unexciting and deploys the container,
   certificates and an apache proxy back to localhost:6080 where hound
   is listening.

I've combined removal of the old puppet bits here as the "-codesearch"
namespace was already being used.

Change-Id: I8c773b5ea6b87e8f7dfd8db2556626f7b2500473
2020-11-20 07:41:12 +11:00
Clark Boylan
33bbdbcc81 Build new gerrit images
This change modifies the Dockerfile to ensure the promote job runs.

It also updates the branches used to build against to pull recent
security updates which don't seem to hvae made it to the proper stable
branches yet.

Change-Id: I2cc9a67515ffdf256fccf5771cf4cf327ed9dbd5
2020-11-18 20:01:24 -08:00
Clark Boylan
44307d119a Update gerrit plugins on 2.16 and 3.0
The gerrit 2.16 and 3.0 releases have been updated. Bump the plugin
versions to match which will rebuild those images on top of the latest
releases.

Change-Id: Icabb786560c71ac664aaadc0c0518a426e7f3392
2020-11-05 09:15:27 -08:00
Clark Boylan
1008a1fc90 Build images for gerrit 3.1 and 3.2
This will allow us to test further gerrit upgrades while we sort out how
far into the gerrit releases we will be upgrading to on our next
upgrade.

Change-Id: Ic9d07b76e41ad4262cc0e2e1ff8a5d554f88239e
2020-10-06 15:37:06 -07:00
Zuul
870dd0ecd2 Merge "Use the new stable-2.16 branch of its-storyboard" 2020-10-02 15:39:17 +00:00
Zuul
b89f4d6245 Merge "Update deb-docker path" 2020-09-18 00:55:32 +00:00
Clark Boylan
64969913e0 Use the new stable-2.16 branch of its-storyboard
This is equivalent to master but if we make any 2.16 fixes they will
likely end up on this branch so build from it.

Change-Id: Ic0f53c31c88d9eae3f8c1f7404918f4b296ff2e1
2020-08-26 12:45:03 -07:00
Ian Wienand
32640cd026 Update deb-docker path
As a follow-on to I728d38daf9a953a64364689da0648c9339a27693, use the
per-release distro path for the mirror.

Change-Id: Ib75904c801a7de4a914db55ca70b192d586461cd
2020-08-14 15:01:01 +10:00
Clark Boylan
3279a7ec10 Gerrit image cleanups/fixes
We name the base image we build gerrit-base and we expose port 8081
not 8080 as opendev's gerrit listens on 8081.

Also explicitly build the javamelody plugin deps jar and copy it into
the review_site/lib dir on Gerrit 2 bazel builds. This is necessary
according to javamelody plugin build docs. In order to split Gerrit 2.x
and 3.x behavior in the Bazel builds we convert our Dockerfile into a
multi stage build.

All this ended up down a thread pull where the script in the Dockerfile
dir called build-gerrit.sh isn't actually used to build gerrit :/
clarify that. The script may be useful for local builds so we haven't
removed it yet.

Finally update gerrit plugin checkouts to tags or master as appropriate
where stable branches don't exist for the specified version.

Change-Id: I155a20685b3462e965c4216d134b3b36978fbcc7
2020-08-11 13:15:06 -07:00
Monty Taylor
2302879244 Build multi-arch python-base/python-builder
In order to build multi-arch python images, we need
multi-arch python base and builder images.

Change-Id: Ifc0d6f7c16876bf55db8e1ee459a3eaa07744547
2020-07-15 09:09:35 -07:00
Ian Wienand
b146181174 Grafana container deployment
This uses the Grafana container created with
Iddfafe852166fe95b3e433420e2e2a4a6380fc64 to run the
grafana.opendev.org service.

We retain the old model of an Apache reverse-proxy; it's well tested
and understood, it's much easier than trying to map all the SSL
termination/renewal/etc. into the Grafana container and we don't have
to convince ourselves the container is safe to be directly web-facing.

Otherwise this is a fairly straight forward deployment of the
container.  As before, it uses the graph configuration kept in
project-config which is loaded in with grafyaml, which is included in
the container.

Once nice advantage is that it makes it quite easy to develop graphs
locally, using the container which can talk to the public graphite
instance.  The documentation has been updated with a reference on how
to do this.

Change-Id: I0cc76d29b6911aecfebc71e5fdfe7cf4fcd071a4
2020-07-03 07:17:22 +10:00
Ian Wienand
330e297318 Add a grafana/grafyaml image
This is a docker image based on the latest upstream Grafana with
grafyaml also installed inside.  It includes a small script to run a
refresh of the dashboards.

Change-Id: Iddfafe852166fe95b3e433420e2e2a4a6380fc64
2020-06-24 08:21:26 +10:00
Clark Boylan
cfc83807b7 Organize zuul jobs in zuul.d/ dir
Our .zuul.yaml file has grown quite large. Try to make this more
manageable by splitting it into zuul.d/ directory with jobs organized by
function.

Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882
2020-05-07 17:30:48 -05:00