This is a bit of spring cleaning. Previously we based on images on
Buster but Bullseye exists now so give it a go.
Change-Id: Icc3d79b361e41df2f2f063993fd206ab7d992f75
To do this we also update jinja-init to bullseye and gitea seems to be
the only user of this image. The impact of this should be fairly self
contained to gitea.
Note this update isn't urgent, but good hygiene. We should coordinate
this update with the 1.15.x gitea upgrade and do them in such a sequence
that we can identify problems easily if they pop up.
Change-Id: Ia0075416a1d8a067cfecd26c03f8db9641cbcb89
This uses the opendev assets bundle image created with
I3166679bde6d771276289b9d32e7e4407957b2f8.
The mount options require using BuildKit, hence the Dockerfile update.
Otherwise conceptually it's fairly simple; copy in the files from the
opendevorg/assets image rather than the file-system.
Change-Id: I36bdc76471eec5380a676ebcdd885a88d3985976
Move some common assets into a top-level assets/ directory. Services
can reference these assets via
https://opendev.org/opendev/system-config/raw/branch/master/assets/<file>
in <img> tags, etc.
Some services want to embed these into their images, but we wish to
only keep one canonical copy. For this, add a Dockerfile and jobs
that creates a simple bundle of assets in opendevorg/assets. This can
be referenced in other builds; the new BuildKit bind-mount is
particularly useful for this
(c.f. I36bdc76471eec5380a676ebcdd885a88d3985976).
Change-Id: I3931566eb86a0618705d276445fa0a5f659692ea
We upstreamed fixes for the mariadb jdbc connector and users being able
to orphan their accounts through accidental deletion of their openid
external ids. These fixes are now present in both the stable-3.2 and
stable-3.3 branches of gerrit. We should rebuild these images to ensure
our images include the fixes.
Note that stable-3.4 does not yet include these fixes but should in
time.
We will update our jdbc connection url in a followup change as we don't
auto update our images. This way we can ensure that the new image is
ready to go before updating that config.
Change-Id: Id23215ddfb3bd4424109e77ecd3480ed2375431d
This adds a local mariadb container to the gerrit host to hold the
accountPatchReviewDb database. This is inspired by a few things
- since migration to NoteDB, there is only one table left where
Gerrit records what files have been reviewed for a change. This
logically scales with the number of reviews users are doing.
Pulling the stats on this, we can see since the NoteDB upgrade this
went from a very busy database (~300 queries/70 commits per second)
to barely registering one hit per second :
https://imgur.com/a/QGJV7Fw
Thus separating the db to an external host for performance reasons
is not a large concern any more.
- emperically we've done a bad job in keeping the existing hosted db
up-to-date; it's still running mysql 5.1 and we have been hit by
bugs such as the one referenced in-line which silently drops
backups.
- The other gerrit option is to use an on-disk H2 database. This is
certainly an option, however you need special tools to interact
with it for migration, etc. and it's not safe to backup from files
on disk (as opposed to mysqldump). Upstream advice is unclear, and
varies between H2 being a performance bottleneck to this being
ephemeral data that users don't care about. We know how to admin
mariadb/mysql and this allows us to migrate and backup data, so
seems like the best choice.
- we have a pressing need to update the server to a new operating
system. Running the db alongside the gerrit instance minimises
fiddling we have to do manging connections to and migrating the
hosted db systems.
- related to that, we are tending towards more provider independence
for control-plane servers. A hosted database product is not always
provided, so this gives us more flexibility in moving things
around.
- the main concern here is memory usage. "docker stats" reports a
quiescent container, freshly started on a 8GB host:
gerrit-compose_mariadb_1 67.32MiB
After loading a copy of the production table, and then dumping it
back to a file the same container reports:
gerrit-compose_mariadb_1 462.6MiB
The existing remote mysql configuration path remains mostly the same.
We move the gerrit startup into a script rather than a CMD so we can
call it after a "wait for db" script in the mariadb_container case
(this is the reccommeded way to enforce ordering [1]).
Backups of the local container need different dump commands; backups
are relocated to a new file and updated.
Testing is converted to use this rather than a local H2 database.
[1] https://docs.docker.com/compose/startup-order/
Change-Id: Iec981ef3c2e38889f91e9759e66295dbfb499c2e
This installs our Limnoira/meetbot container and configures it on
eavesdrop01.opendev.org. I have ported the configuration from the old
puppet as best I can (it is very verbose); my procedure was to use the
Limnoira wizard to start a new config file then backport everything
from the old file. I felt this was best to not miss any new options.
This does channel logging (via built-in ChannelLogger plugin, along
with a cron job for logs2html) and runs our fork of meetbot.
It exports the channel logs via HTTP to /irclogs and meetings logs to
/meetings. meetings.opendev.org will proxy to these two locations
when the server is active.
Note this has not ported the channel list; so the bot will not be
listening in our channels.
Change-Id: I9f9a466c271e1a706f9f98f816de0e84047519f1
This container installs Limnoria, the supybot replacement as the
generic ircbot container. We install meetbot plugin as a sibling
project.
Previously we've conflated supybot with meetbot, which is a bit
confusing because meetbot is a plugin, but we also use other plugins
such as the channel logger. We also hope to convert some of our other
bots to Limnoria (ptgbot?) to consolidate everything. For this reason
I've called this the more generic "ircbot". The image installs
meetbot as a sibling project, with the idea being any other plugins
would also be installed as siblings.
The siblings install expects the work directory to be a relative
directory. I'm not sure we run this from other projects, but this
will work the same if we do.
Depends-On: https://review.opendev.org/c/opendev/meetbot/+/793876
Change-Id: Icee4c6bbb5ea235ba69c10f800a14bbf5beef3d5
This updates our plugin versions to keep up with Gerrit releases. We
build from the stable branch which means we need to periodically bump
plugin tagged versions to keep up when stable branches for the plugins
are not available.
Change-Id: I5bb50d8610ebdbdb9c6a70ad8dde732067cc368f
These files are involved in creating gerrit docker images; make sure
we trigger jobs when they are modified.
Change-Id: I7c4436e066cfb0c2d0b2ca7adf54c99b09dac95f
Python 3.9 is released, so let's build containers.
This splits the docker-images/ files up as they are becoming a bit
crowded.
Change-Id: Id68080575a30e4a08c99df0af603fbb65a0983bd
This should ensure that if we have a parent job that updates the gitea
version and a do not merge child job that induces an artificial failure
for zuul hold purposes that we test the correct image in the child job's
changes.
Prior to this we were testing the existing published images, but
provides + requires will give the correct signaling to make the desired
"test new proposed image" behavior happen in the child change builds.
Change-Id: Ie6b827b650e0f32606dc5ec7f4aa0adfeebdeb5e
This adds a program, zookeeper-statsd, which monitors zookeeper
metrics and reports them to statsd. It also adds a container to
run that program. And it runs the container on each of the
ZooKeeper quorum members. And it updates the graphite host to
allow statsd traffic from quorum members. And it updates the
4-letter-word whitelist to allow the mntr command (which is used
to gather metrics) to be issued.
Change-Id: I298f0b13a05cc615d8496edd4622438507fc5423
This has our change to open etherpad on join, so we should no longer need
to run a fork of the web server. Switch to the upstream container image
and stop building our own.
Change-Id: I3e8da211c78b6486a3dcbd362ae7eb03cc9f5a48
Gerrit 3.3 has released. Lets start building images for it so that we
can do testing when ready to start that.
We also add testing files to the list of things that trigger the 3.3
builds. Strictly this isn't necessary since the test will continue to
use 3.2 images until we upgrade to 3.3, but this helps us avoid
forgetting to do this when we do upgrade. Little extra jobs run today to
ensure we continue to run the right jobs tomorrow.
Change-Id: Ib7e7d7313e0827a40009df840119444611d74ca2
This adds a dockerfile to build an opendevorg/refstack image as well as
the jobs to build and publish it.
Change-Id: Icade6c713fa9bf6ab508fd4d8d65debada2ddb30
This reworks the gerrit testing slightly to give some broader
coverage.
It sets up ssh keys for the user; not really necessary but can be
helpful when interacting on a held host.
It sets up groups and verification labels just so Zuul can comment
with -2/+2; again this is not really necessary, but makes things a
little closer to production reality.
We make multiple changes, so we can better test navigating between
them. The change comments are updated to have some randomness in them
so they don't all look the same. We take screen shots of two change
pages to validate the navigation between them.
Change-Id: I60b869e4fdcf8849de836e33db643743128f8a70
Gerrit's 3.2 release is moving ahead and we should keep up with our
plugin version specifications.
Change-Id: I552760a98c00abb7c31c7c6864d74fe951491be7
This installs the zuul-summary-results plugin into our gerrit
container. testinfra is updated to take a screenshot of the plugin in
action.
Change-Id: Ie0a165cc6ffc765c03457691901a1dd41ce99d5a
The "additional_plugins" variable is so different builds gerrit can
specify additional plugins specific to their version to install into
the base image.
Since we've moved to only building 3.2 and master images, a bunch of
plugins that used to be additional (because they weren't 2.XX era) are
now common. Move them into the common plugin code in the playbook,
and leave the only one different for master, the "checks" plugin, as
separate.
Change-Id: I8966ed7b5436fbe012486dccc1028bc8cb1cf9e4
Having upgraded to 3.2, we don't need these versions any more.
Change-Id: Ifc37a75aa62b2498e649a4c81b589a04c794184a
Depends-On: https://review.opendev.org/763617
The hound project has undergone a small re-birth and moved to
https://github.com/hound-search/hound
which has broken our deployment. We've talked about leaving
codesearch up to gitea, but it's not quite there yet. There seems to
be no point working on the puppet now.
This builds a container than runs houndd. It's an opendev specific
container; the config is pulled from project-config directly.
There's some custom scripts that drive things. Some points for
reviewers:
- update-hound-config.sh uses "create-hound-config" (which is in
jeepyb for historical reasons) to generate the config file. It
grabs the latest projects.yaml from project-config and exits with a
return code to indicate if things changed.
- when the container starts, it runs update-hound-config.sh to
populate the initial config. There is a testing environment flag
and small config so it doesn't have to clone the entire opendev for
functional testing.
- it runs under supervisord so we can restart the daemon when
projects are updated. Unlike earlier versions that didn't start
listening till indexing was done, this version now puts up a "Hound
is not ready yet" message when while it is working; so we can drop
all the magic we were doing to probe if hound is listening via
netstat and making Apache redirect to a status page.
- resync-hound.sh is run from an external cron job daily, and does
this update and restart check. Since it only reloads if changes
are made, this should be relatively rare anyway.
- There is a PR to monitor the config file
(https://github.com/hound-search/hound/pull/357) which would mean
the restart is unnecessary. This would be good in the near and we
could remove the cron job.
- playbooks/roles/codesearch is unexciting and deploys the container,
certificates and an apache proxy back to localhost:6080 where hound
is listening.
I've combined removal of the old puppet bits here as the "-codesearch"
namespace was already being used.
Change-Id: I8c773b5ea6b87e8f7dfd8db2556626f7b2500473
This change modifies the Dockerfile to ensure the promote job runs.
It also updates the branches used to build against to pull recent
security updates which don't seem to hvae made it to the proper stable
branches yet.
Change-Id: I2cc9a67515ffdf256fccf5771cf4cf327ed9dbd5
The gerrit 2.16 and 3.0 releases have been updated. Bump the plugin
versions to match which will rebuild those images on top of the latest
releases.
Change-Id: Icabb786560c71ac664aaadc0c0518a426e7f3392
This will allow us to test further gerrit upgrades while we sort out how
far into the gerrit releases we will be upgrading to on our next
upgrade.
Change-Id: Ic9d07b76e41ad4262cc0e2e1ff8a5d554f88239e
This is equivalent to master but if we make any 2.16 fixes they will
likely end up on this branch so build from it.
Change-Id: Ic0f53c31c88d9eae3f8c1f7404918f4b296ff2e1
As a follow-on to I728d38daf9a953a64364689da0648c9339a27693, use the
per-release distro path for the mirror.
Change-Id: Ib75904c801a7de4a914db55ca70b192d586461cd
We name the base image we build gerrit-base and we expose port 8081
not 8080 as opendev's gerrit listens on 8081.
Also explicitly build the javamelody plugin deps jar and copy it into
the review_site/lib dir on Gerrit 2 bazel builds. This is necessary
according to javamelody plugin build docs. In order to split Gerrit 2.x
and 3.x behavior in the Bazel builds we convert our Dockerfile into a
multi stage build.
All this ended up down a thread pull where the script in the Dockerfile
dir called build-gerrit.sh isn't actually used to build gerrit :/
clarify that. The script may be useful for local builds so we haven't
removed it yet.
Finally update gerrit plugin checkouts to tags or master as appropriate
where stable branches don't exist for the specified version.
Change-Id: I155a20685b3462e965c4216d134b3b36978fbcc7
This uses the Grafana container created with
Iddfafe852166fe95b3e433420e2e2a4a6380fc64 to run the
grafana.opendev.org service.
We retain the old model of an Apache reverse-proxy; it's well tested
and understood, it's much easier than trying to map all the SSL
termination/renewal/etc. into the Grafana container and we don't have
to convince ourselves the container is safe to be directly web-facing.
Otherwise this is a fairly straight forward deployment of the
container. As before, it uses the graph configuration kept in
project-config which is loaded in with grafyaml, which is included in
the container.
Once nice advantage is that it makes it quite easy to develop graphs
locally, using the container which can talk to the public graphite
instance. The documentation has been updated with a reference on how
to do this.
Change-Id: I0cc76d29b6911aecfebc71e5fdfe7cf4fcd071a4
This is a docker image based on the latest upstream Grafana with
grafyaml also installed inside. It includes a small script to run a
refresh of the dashboards.
Change-Id: Iddfafe852166fe95b3e433420e2e2a4a6380fc64
Our .zuul.yaml file has grown quite large. Try to make this more
manageable by splitting it into zuul.d/ directory with jobs organized by
function.
Change-Id: I0739eb1e2bc64dcacebf92e25503f67302f7c882