Commit Graph

18803 Commits (master)

Author SHA1 Message Date
Zuul 9f953277e3 Merge "Add codesearch to cacti" 2023-09-22 12:12:20 +00:00
Radosław Piliszek 3929acb7a4 Add codesearch to cacti
Change-Id: I8c55262b5ce783668d4780511223f620b90d3992
2023-09-22 11:12:36 +00:00
Zuul 6e53e7e198 Merge "Move OpenInfra and StarlingX lists to Mailman 3" 2023-09-21 14:09:07 +00:00
Clark Boylan 00c098bcad Bump refstack to Bookworm and python3.11
Refstack doesn't rely on much in the base operating system as far as I
can tell. That said refstack seems to test with python3.10 and not 3.11
so this may not work, but our testing should give us a good idea. Bump
these things up as we are updating all the services we can in order to
eventually cleanup image builds for old debian and old python.

Change-Id: Id39027691484e8f81bd097c174f0a4a1e81463af
2023-09-19 10:22:56 -07:00
Clark Boylan 2888408833 Convert haproxy-statsd to bookworm base image
We keep the same python 3.11 version we had on bullseye but switch the
rest of the userspace to bookworm. Since the python version doesn't
change this should have minimal impact.

Change-Id: I59ad8c2a92159f51d567dd0212e2ab8bec1b45b1
2023-09-14 13:36:18 -07:00
Jeremy Stanley 4a115ed54d Move OpenInfra and StarlingX lists to Mailman 3
This uncomments the list additions for the lists.openinfra.dev and
lists.starlingx.io sites on the new mailman server, removing the
configuration for them from the lists.openstack.org server and also
cleaning up some benign entries which were missed in the previous
migration change. With this, the old server should only be hosting
specifically lists.openstack.org mailing lists.

Change-Id: I1e2d332cd4addb8970a3759157bbeceddd77ea95
2023-09-14 18:25:12 +00:00
Jeremy Stanley a6ab3543fc Move Airship and Kata lists to Mailman 3
This uncomments the list additions for the lists.airshipit.org and
lists.katacontainers.io sites on the new mailman server, removing
the configuration for them from the lists.opendev.org server and, in
the case of the latter, removing all our configuration management
for the server as it was the only site hosted there.

Change-Id: Ic1c735469583e922313797f709182f960e691efc
2023-09-14 12:08:34 +00:00
Clark Boylan c06b31df3a Update to gitea 1.20.4
This is a small update from what we are currently running (1.20.4). The
full changelog can be found here:

  https://github.com/go-gitea/gitea/blob/v1.20.4/CHANGELOG.md

There is one small template update in 1.20.4, but it is to documentation
urls which we are already overriding with our own documentation links so
doesn't affect us.

Change-Id: I5ed374e2e6e0056397e05404e0bf42ffd3906469
2023-09-08 08:10:29 -07:00
Zuul 53391950e1 Merge "Run bootstrap-bridge with empty nodeset" 2023-09-04 12:53:13 +00:00
Zuul c91d217681 Merge "Update Gitea images to bookworm" 2023-09-01 21:29:18 +00:00
Zuul 90620b284b Merge "Rebuild gitea on bullseye" 2023-09-01 19:35:49 +00:00
Clark Boylan dea0930962 Update Gitea images to bookworm
There are two main components that I expect will be affected by this
bookworm update. The first is git. Git is updated from 2.30.2-1+deb11u2
to 2.39.2-1.1. In general git has been very good about maintaining
compatibility, but there is some risk of a behavior change impacting us.

Second is openssh-server. In particular we upgrade from 8.4p1-5+deb11u1
to 9.2p1-2 which crosses the 8.8 release threshold. 8.8 changed how RSA
keys are handled. Now by default only RSA + SHA2 is negotiated by
default and RSA + SHA1 is not allowed. Gerrit currently uses RSA keys
for replication. This should all be fine because MINA added support for
negotiating RSA + SHA2 as both a client and server in version prior to
that running in Gerrit 3.7, but there is still some risk this will
break.

We can test this with held nodes, or we can assume it will work given
the fixes in MINA. Then if it breaks we can switched to ed25519 keys or
update config to openssh to reenable SHA1 or we can revert to bullseye.

Note, we make a small update to the image to set `ENV USER git` in both
the web and ssh iamges as the ssh image uses this env var to dynamically
set sshd_config's AllowUsers value. We weren't setting this value
previously which older sshd seems to ignore. Bookworm sshd gets angry
about this directly being set without a value in its config.

Change-Id: I5a923798e90be4dcd9486a97014180ed1790fab1
2023-09-01 09:44:52 -07:00
Clark Boylan a63f8c6dd7 Rebuild gitea on bullseye
We want to rebuild gitea on bullseye before upgrading to bookworm. The
reason for this is we only prune images that are more than 72 hours old.
Deploying a new bullseye image then deploying the bookworm image ensures
we have 72 hours before that bullseye image is pruned allowing us to
easily revert if necessary.

Change-Id: I5cc8078e0c5f6e55215e9419ac3569a686060b05
2023-09-01 09:44:51 -07:00
Zuul a4011f3808 Merge "Explicitly disable offline reindexing during project renames" 2023-09-01 00:13:59 +00:00
Jeremy Stanley 9bbe19d4a3 Temporarily limit node image upload concurrency
One of our providers is struggling with Glance backend tasks after
upload, which causes increasingly lengthy delays for image readiness
the more images we're uploading. Globally reduce our upload
concurrency to the minimum possible of one per builder, for now, in
order to alleviate the delays and increase the chances images will
come ready in that provider.

This can be reverted once the situation has been resolved.

Change-Id: I29daa8f6d2d13055baf74215184ce0987fc20be0
2023-08-30 21:16:01 +00:00
Zuul af6c43bce3 Merge "Update jinja-init image to bookworm" 2023-08-30 16:13:58 +00:00
Zuul d148cb51e1 Merge "mailman3: re-sync custom web/settings.py" 2023-08-28 22:15:22 +00:00
Clark Boylan df66e2fd6f Exclude i686 rpms in our centos mirrors
For some reason the x86_64/ directories of centos mirrors contain i686
packages too. Since none of our systems rely on i686 lets go ahead and
exclude these packages entirely. This should save quite a bit of disk
space (almost halving the required disk space?).

Change-Id: I146765da021e8436365e39a6dca4db470a07fe81
2023-08-28 08:46:35 -07:00
Zuul 52b865a9eb Merge "Update zookeeper-statsd image to bookworm" 2023-08-25 21:21:16 +00:00
Jeremy Stanley e13ea50e68 mailman3: re-sync custom web/settings.py
The version in the container config repository has moved on, update
our copy with overrides for the allowed hosts and site ID. Adjust
how we generate the allowed hosts envvar list to replace Exim's
field separators with those expected by Mailman.

Change-Id: Ia6831ca10e1cd1ad057475e0f78eacc02857eef2
2023-08-25 21:14:50 +00:00
Jeremy Stanley c9c8febd84 Trigger mm3 deployment when containers change
Add the docker/mailman tree to the infra-prod-service-lists3 job so
that we deploy new versions whenever we make changes to the
container images.

Change-Id: Ife5e878b1f81c2879c2959fe6d4de22fe841583b
2023-08-25 16:35:46 +00:00
Zuul aa4afe593b Merge "Upgrade to latest Mailman 3 releases" 2023-08-25 16:27:25 +00:00
Zuul 8e00fda3c2 Merge "Restart matrix-eavesdrop when config changes" 2023-08-24 21:04:41 +00:00
Zuul 1c5bee394c Merge "Update hound image to bookworm" 2023-08-24 21:02:41 +00:00
James E. Blair 44b0e32063 Restart matrix-eavesdrop when config changes
Change-Id: I8b8846d5451159ddf17966a58526f70ab3a258dc
2023-08-24 12:59:13 -07:00
Zuul c2ea42a86e Merge "Update to Gitea 1.20" 2023-08-24 18:58:57 +00:00
James E. Blair 55abbf33bf Update hound image to bookworm
Change-Id: I51b8630b27149aa96c63d13d6fae240c5c81cd96
2023-08-24 11:31:46 -07:00
James E. Blair d39162a79a Update zookeeper-statsd image to bookworm
And upgrade to Python 3.11.

Change-Id: I863aabd21e0518879604b187509aa9b46b15bace
2023-08-24 11:19:23 -07:00
James E. Blair 1354a1e6da Update jinja-init image to bookworm
And upgrade to python 3.11.

Change-Id: I47a99263b93be027ffe251fce24d677d07f519d7
2023-08-24 11:18:11 -07:00
Zuul 1b64bb2e31 Merge "Add StarlingX Matrix channels to the logbot" 2023-08-23 15:22:25 +00:00
Zuul 9d7dc8e73a Merge "gerrit: bump index.maxTerms" 2023-08-23 15:22:23 +00:00
Jeremy Stanley 62e9af560c Add StarlingX Matrix channels to the logbot
StarlingX has decided to move from IRC to Matrix, and 11 channels
have been established for them in a dedicated space on our
homeserver. Add those channels to the matrix-eavesdrop bot for
logging.

Change-Id: I3cb50ebbe892e837e85cefc094bc99bbb4b0a759
2023-08-22 17:19:23 +00:00
Zuul cf4dec4391 Merge "Upgrade Etherpad to 1.9.2" 2023-08-21 18:12:16 +00:00
Clark Boylan ef450d1bce Update to Gitea 1.20
The 1.20 release is here. Upgrade to this version.

Things we change:
 * Nodejs is updated to v20 to match the alpine 3.18 package version
   that gitea switched to.
 * Templates are updated to match upstream 1.20 templates.
 * We drop the deprecated LFS_CONTENT_PATH from our server config and
   add an equivalent [lfs] config section.
 * Normalize app.ini content so that gitea won't write it back out to
   disk which fails due to permissions (and we don't want it overriding
   our configs anyway). For this we need to add WORK_PATH,
   oauth2.JWT_SECRET, and normliazing spacing and quoting for entries.
 * Set JWT_SIGNING_PRIVATE_KEY_FILE explicitly to be located at
   /data/gitea/jwt/private.pem otherwise gitea attempts to create the
   jwt/ directory somewhere it doesn't have permissions to (I think /)
   and won't be persisted across containers.
 * Replace log.ENABLE_ACCESS_LOG with log.logger.access.MODE = file as
   log.ENABLE_ACCESS_LOG is deprecated and doesn't appear to work
   anymore. This appears to be a documentation issue or they deprecated
   and removed things more quickly than originaly anticipated.
 * Add log.ACCESS_LOG_TEMPLATE to readd source port info to the access
   logs.
 * Add a templates/custom/header.tmpl file to set theme-color as the
   config item for this has been removed.

The 1.20.0 changelog [0] lists a number of breaking changes. I have
tried to capture there here as well as potential impacts to us:

 * Fix WORK_DIR for docker (root) image (#25738) (#25811)
   * We set APP_DATA_PATH to /data/gitea in our app.ini config which
     means we aren't relying on the inferred value from WORK_DIR. I
     think this isolates us from this chnage. But we can check for any
     content in /app/gitea on our running containers to be sure.
     Note we hardcode WORK_PATH to /data/gitea because gitea attempts to
     write this back to our config file otherwise as a result of this
     change.
 * Restrict [actions].DEFAULT_ACTIONS_URL to only github or self (#25581) (#25604)
   * We disable actions. This shouldn't affect us.
 * Refactor path & config system (#25330) (#25416)
   * This is related to the first breaking changes. Basically we need
     to check our use of WORK_PATH and determine if we need to hardcode
     it to something. Probably a good idea given how they keep changing
     this on us...
 * Fix all possible setting error related storages and added some tests (#23911) (#25244)
   * We don't use storage configs. This shouldn't affect us.
 * Use a separate admin page to show global stats, remove actions stat (#25062)
   * The breaking change only affects the use of Prometheus which we
     don't have yet.
 * Remove the service worker (#25010)
   * Is listed as a breaking change for UI cleanup that we don't need to
     cleanup. (ui.USE_SERVICE_WORKER can be removed).
 * Remove meta tags theme-color and default-theme (#24960)
   * https://github.com/go-gitea/gitea/pull/24960
   * Addressed by adding a custome templates/custom/header.tmpl file
     that sets this meta tag to the existing value. Note this only
     affects mobile clients so needs to be double checked via a mobile
     device.
 * Use [git.config] for reflog cleaning up (#24958)
   * Affects git.reflog config entries and we don' thave any.
 * Allow all URL schemes in Markdown links by default (#24805)
   * TODO determine if we need to limit link types and add that
     change if so. A point release was made to exclude bad types
     already. Not sure if there are others we need to add.
 * Redesign Scoped Access Tokens (#24767)
   * This breaks scoped tokens with scopes that don't exist anymore.
     I don't think we use scoped tokens.
 * Fix team members API endpoint pagination (#24754)
   * They 1 index the pagination of this endpoint now instead of 0
     indexing it.
 * Rewrite logger system (#24726)
   * They made changes to the loggers and encourage people to check
     their logs work as expected when upgrading. Using our test instance
     logs I don't see anything that is a problem.
 * Increase default LFS auth timeout from 20m to 24h (#24628)
   * We don't LFS but can change the timeout if necssary.
 * Rewrite queue (#24505)
   * Check for 'Removed queue option:' log entries and clean up
     corresponding entries in app.ini. We don't have any of these
     entries in our logs.
 * Remove unused setting time.FORMAT (#24430)
   * We didn't have this entry in app.ini.
 * Refactor setting.Other and remove unused SHOW_FOOTER_BRANDING (#24270)
   * This setting can be removed from app.ini, but we don't set it.
 * Correct the access log format (#24085)
   * We uncorrect it because they removed source port info in the
     correction step. They did this because some log parsers don't
     understand having the port info present, but if you are behind a
     reverse proxy this information is very important. We run gitea behind
     a reverse proxy.
 * Reserve ".png" suffix for user/org names (#23992)
   * .png is no longer a valid user/org name (it didn't work before
     anyway).
 * Prefer native parser for SSH public key parsing (#23798)
   * If you relied on the openssh ssh-keygen executable for public key
     parsing then you must explicitly set config to use it. I don't
     think we do as the golang native parser should handle the keytypes
     we use.
 * Editor preview support for external renderers (#23333)
   * This removed an app.ini settings we don't seem to set.
 * Add Gitea Profile Readmes (#23260)
   * Readmes in .profile repositories will always be shown now. We don't
     have .profiles repos so this doesn't affect us.
 * Refactor ctx in templates (#23105)
   * This affects custom templates as we may need to replace ctx with
     ctxData in our templates.
   * I've searched our templates for 'root', 'ctx', and 'ctxData' and
     have found no instances. Looking at the files modifying by the
     commits related to this change:
     bd7f218dce
     7c01260e1d
     we don't seem to override the affected files. I think we are fine
     as is.

The 1.20.1 changelog indicates there are no breaking changes, and git
diff shows no changes to the templates between 1.20.0 and 1.20.1.

The 1.20.2 changelog indicates there are no breaking changes, and git
diff shows no changes to the templates between 1.20.1 and 1.20.2.

The 1.20.3 changelog indicates there is a single breaking change:
 * Fix the wrong derive path (#26271) (#26318)
   * If I'm reading the code correctly, I think the problem was storage
     configuration inheriting the base storage config and particularly
     the related path. Then when archival storage looked for its config
     the path was the root gitea storage path and it would inadverdently
     delete all repos when deleting a single repo or something like
     that. We don't use these features and these are mirrors anyway so I
     don't think this really affects us.

[0] https://github.com/go-gitea/gitea/blob/v1.20.3/CHANGELOG.md

Change-Id: I265f0ad16c0e757a11c1d889996ffe2198625a1a
2023-08-21 08:49:46 -07:00
Dr. Jens Harbott 08610609e5 gerrit: bump index.maxTerms
The default value is 1024, which causes issues for users that have
starred more than that number of changes. Bump by 50% hoping that the
possible performance impact will be moderate.

[0] https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#index

Change-Id: I0c00110cfd6ba6d235821f6a5db7e1b91e2a8945
2023-08-19 22:59:57 +02:00
Zuul e8a274e8cc Merge "Mailman3: check docker-compose stderr not stdout" 2023-08-17 21:19:59 +00:00
Jeremy Stanley 188ea92675 Mailman3: check docker-compose stderr not stdout
For the conditional check as to whether docker-compose up was a
no-op, we look for a recognizable string in the output it generates.
This information is emitted on the stderr descriptor, not stdout, so
correct the match condition accordingly.

Change-Id: I82015e615071458c95342eaea2e6a17aeae44c07
2023-08-17 20:04:09 +00:00
Zuul 508ebe2dfc Merge "Restart Mailman 3 containers when configs change" 2023-08-17 19:55:51 +00:00
Clark Boylan 9f0b4b7df9 Upgrade Etherpad to 1.9.2
The day after we upgrade to 1.9.1 we've got a new release. That is just
how it goes sometimes. The changelog can be found here:

  https://github.com/ether/etherpad-lite/blob/v1.9.2/CHANGELOG.md#192

Should be fairly straightforward to update to. I did make a minor change
to set a production build flag for the installDeps.sh script (which
lives upstream) so that we'll get a more production quality build in the
end. The updates to installDeps.sh happened in the 1.9.2 release so
there is no need to do this outside of the 1.9.2 upgrade.

Change-Id: Ia22ebd3c01393af5a7e593ceb64fc7c63ad46ffa
2023-08-17 10:48:17 -07:00
Clark Boylan 9fdbed9c27 Run bootstrap-bridge with empty nodeset
We are currently using the default nodeset on the
infra-prod-bootstrap-bridge job which results in us waiting for a node
that we end up ignoring. As far as I can tell this job runs against
localhost and the add_host bridge entry. It ignores the default test
node from the nodeset.

Speed up job execution and reduce node waste by setting an empty nodeset
on the job.

Change-Id: I8c3ffda60b92a8655989579335a49423fbdd18a2
2023-08-17 09:59:09 -07:00
Jeremy Stanley eec60a2cb9 Restart Mailman 3 containers when configs change
Add a handler for restarting Mailman 3 containers if they're up, and
notify it from all of the copy tasks for configuration files. Check
for the uwsgi processes that Django runs under to determine whether
the containers are running.

Change-Id: I73be59be773fdde100999c7872520aab1d9e2066
2023-08-17 15:52:26 +00:00
Zuul 0b86e9e148 Merge "Remove old insecure-ci-registry01 from our inventory" 2023-08-17 04:38:58 +00:00
Zuul 5ab69e4a00 Merge "Setup #openstack-charms IRC channel." 2023-08-17 01:40:20 +00:00
Zuul b5e069dbbe Merge "Update etherpad to 1.9.1" 2023-08-16 16:29:21 +00:00
Zuul 920afd7087 Merge "Use magic domain guessing in Mailman 3" 2023-08-15 22:58:42 +00:00
Zuul 0c40d48422 Merge "Make mailman3 DB migration check PyVer-agnostic" 2023-08-15 22:58:40 +00:00
Zuul 1c04f23d99 Merge "Pin importlib_resources<6 in mailman images" 2023-08-15 22:36:00 +00:00
Jeremy Stanley 330a5e02e6 Correct static known_hosts entry for goaccess jobs
When the static.opendev.org service was replaced up a new server, we
neglected to update its SSH known_hosts entry in *-goaccess-report
jobs, so they began failing. Get them back on track.

Change-Id: I7a4902d6edfbc0bbfedbc6dbf9ea7e93acd04386
2023-08-07 21:27:40 +00:00
Felipe Reyes 12208a4b5a Setup #openstack-charms IRC channel.
This change configures meetbot to join the #openstack-charms where team
meetings are hosted.

Change-Id: I2e0697b24a06b78cf410807bbbd46fe40b5deb2e
2023-08-02 10:30:23 -04:00
Jeremy Stanley 18a91c272e Upgrade to latest Mailman 3 releases
New releases info:

https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/YDUYHVWBGBLASJHAYE2PQKY2F7PNEIKP/

Sync all our forked files from mailman-docker to the current
upstream state, except for our overridden hyperkitty->archives and
postorius->mailman3 URL routes in
docker/mailman/web/mailman-web/urls.py.

Change-Id: I26f45b0f019b9212c148f145a5c36ed777fb111c
2023-08-01 20:58:34 +00:00