Commit Graph

2779 Commits

Author SHA1 Message Date
Tony Breeds
f223a237a2 Add a jammy test node for regional mirrors
Change-Id: I922af92e523407b7324f020732fad52b98f027e1
2023-10-31 18:27:59 -05:00
Zuul
5b837c1799 Merge "Convert commentlinks to new no html system" 2023-10-30 20:18:38 +00:00
Zuul
31a430db07 Merge "Revert "Cap ruamel.yaml install for ARA"" 2023-10-26 22:10:14 +00:00
Zuul
b79818feae Merge "Add OpenInfra EU mailing lists" 2023-10-25 16:42:34 +00:00
Zuul
5d0f944c3e Merge "Update to Ansible 8 on bridge" 2023-10-25 16:25:10 +00:00
Clark Boylan
5aec9da11e Revert "Cap ruamel.yaml install for ARA"
This reverts commit a77eebe911.

Ruamel.yaml 0.18.2 converted the error assocaited with the use of this
deprecated method from a sys.exit(1) to a raised Exception. It is
believed that this will allow Ara to run in some capacity and we don't
need to pin this dependency anymore.

More details in the upstream bug here:

  https://github.com/ansible-community/ara/issues/524

Change-Id: I694b8a016755d828490f0bcf4c6ceb812edf43d9
2023-10-25 09:04:57 -07:00
Jeremy Stanley
704321653b Add OpenInfra EU mailing lists
The OpenInfra Foundation executive team is requesting creation of
new mailing lists on lists.openinfra.dev for the foundation's new EU
hub. One list will have an open subscription policy and publicly
available archives, while the other will be utilized by the advisory
board for any sensitive topics that must be kept private.

Change-Id: I138bcdddd8b8feeb94adb71f0ba5e03d8c809e20
2023-10-25 15:31:37 +00:00
Clark Boylan
a77eebe911 Cap ruamel.yaml install for ARA
ARA is not compatible with latest ruamel.yaml which leads to errors
running ansible. Fix this by capping the ruamel.yaml version we install.

Change-Id: Ia5db3ba8579e7e5c1fe375b156323b94f341ad3e
2023-10-24 09:44:16 -07:00
Clark Boylan
8f9b1f2c9c Convert commentlinks to new no html system
Gerrit 3.8 drops support for html in commentlinks entirely. Gerrit 3.7
supports both html and the new non html system. Update our 3.7
installation to the new system on 3.7 so that we are ready for the
Gerrit 3.8 upgrade later.

Most of our comment links did not use html entries so we drop the html
lines entirely. A single commentlink does use html and there we convert
it to the new prefix, link, text, suffix system. More details can be
found here:

  https://gerrit.googlesource.com/gerrit/+/refs/tags/v3.8.2/tools/migration/html_to_link_commentlink.md

This should be a 1:1 mapping for our config and not change any behavior.

Change-Id: I0b87aac7b90814d242338be8fd03cfc9a76200f7
2023-10-23 14:03:06 +00:00
Zuul
bd3fd30462 Merge "Remove the old mailing list server" 2023-10-20 23:04:26 +00:00
Jeremy Stanley
cab53d10ac Remove the old mailing list server
Clean up references to lists.openstack.org other than as a virtual
host on the new lists01.opendev.org Mailman v3 server. Update a few
stale references to the old openstack-infra mailing list (and
accompanying stale references to the OpenStack Foundation and
OpenStack Infra team). Update our mailing list service documentation
to reflect the new system rather than the old one. Once this change
merges, we can create an archival image of the old server and delete
it (as well as removing it from our emergency skip list for
Ansible).

Side note, the lists.openstack.org server will be 11.5 years old on
November 1, created 2012-05-01 21:14:53 UTC. Farewell, old friend!

Change-Id: I54eddbaaddc7c88bdea8a1dbc88f27108c223239
2023-10-20 18:10:08 +00:00
Clark Boylan
27e65ba84f Update to Ansible 8 on bridge
Zuul has already made the move; we should catch up. Part of this is
motivated by the weird failures we've seen when creating the LE
certcheck domains list in an Ansible loop though I've no real evidence
that upgrading would fix this. Python on bridge is 3.10 which should be
compatible with Ansible 8.

Full (and probably far too dense) changelogs can be found here:

  https://github.com/ansible-community/ansible-build-data/blob/main/8/CHANGELOG-v8.rst

A prior patchset temporarily updated zuul configs to run most of our
system-config-run-* jobs using ansible 8. They all passed implying that
our playbooks and roles will function under the newer version of
ansible.

Change-Id: Ie1b4e5363c56c0dcd61721fb0ea061d5198ecfed
2023-10-18 09:20:31 -07:00
Clark Boylan
4929d84f49 Fix the linaro cloud certcheck entry
The entry we had did not match the name used by nodepool. This caused us
to not detect that this cert would expire (and has now expired).

Change-Id: Ibd4885f2f3fbbc776f76fef92736b98b8f87c664
2023-10-18 08:43:33 -07:00
Zuul
0218fe84ed Merge "Bump zookeeper from 3.7 to 3.8" 2023-10-17 17:10:00 +00:00
Jeremy Stanley
82b5640ff4 Drop the mailman_copy Exim router
In Ic1156849957bc326e9216c2aca0ab9d180e158e6 we added a temporary
router named mailman_copy to dump raw messages for the
openstack-discuss mailing list to an mbox file at
/var/mail/openstack-discuss in order to be able to compare
pre-Mailman state of messages for DKIM signature debugging. Since
this file doesn't exist and Exim lacks permission to create it, the
resulting router errors are leading to message deferrals for the
openstack-discuss mailing list.

Rather than add Ansible to create the mbox file for this, just drop
the router and accompanying transport definitions from our Exim
config. We can always set it up more thoroughly in the future if we
ever want to re-add it.

Change-Id: If4f6c7b90b7b312b23a7736251f704dace668879
2023-10-15 01:04:47 +00:00
Clark Boylan
93f423b3be Bump zookeeper from 3.7 to 3.8
Note that we'll probably end up manually performing this upgrade to
ensure the leader is upgraded last as well as performing sanity checks
along the way. Please don't merge this change until we are certain we
are ready for it. In the meaintime it gives us early feedback for any
unexpected problems with the new zookeeper version.

Change-Id: I84c8f3d05edba03cd4ab526ab0105d7512e3984f
2023-10-11 08:56:18 -07:00
Zuul
cac37a7a3c Merge "Update gerrit image to bookworm" 2023-10-09 16:19:47 +00:00
Takashi Kajinami
9d89477326 reprepro: mirror Ubuntu UCA Bobcat for Ubuntu Jammy
Change-Id: Iaef326ad9218808e1b2a8e47c31a10f45d06f7b6
2023-10-08 13:46:35 +09:00
Zuul
16744d8336 Merge "Blackhole deliveries for Mailman v3 local user" 2023-10-06 16:56:39 +00:00
Clark Boylan
9beab7723b Update the apache ua filter set
There are always more UAs to add to the list. It does look like they are
now using all the chrome versions with android. To match these I've
added regexes to the very end of the list so that we don't have to list
hundreds of rules separately. We might be able to collapse of the
earlier rules too, but in theory having direct matches first is faster?

Change-Id: Ic0a84aabea4327d40fa89aef6eb9f9a2d42a658f
2023-10-05 16:01:05 -07:00
Zuul
40bdcd848c Merge "Add Element X Ignition support for Matrix" 2023-10-05 21:44:50 +00:00
Jeremy Stanley
fcef589bdc Add mailing list for Nordix environment
As part of the transition from the Nordix group to OpenInfra Europe,
some systems and services will remain under the Nordix name for now.
The people managing these resources need a mailing list to better
coordinate their activities.

Change-Id: I03b679b4d5f57b1953e1815555b79caf5b6452ff
2023-10-03 14:36:32 +00:00
Jeremy Stanley
222414b585 Blackhole deliveries for Mailman v3 local user
On Mailman v2 the "mailman" addresses were mapped to special mailing
lists used for monthly password notifications and some other tasks.
This does not exist on Mailman v3, but spammers still have the old
mailman list addresses and send junk to them, which the server
attempts to deliver because there's a local user account with the
same name.

Reject messages for the old "mailman" addresses at receipt, so they
never enter our message queue.

Change-Id: I9db93ae98f4b3952400c1e478612ab70a6241dd1
2023-10-02 21:33:11 +00:00
Jeremy Stanley
f8a528cf8a Add Element X Ignition support for Matrix
We received a notification from Element Matrix Services recommending
we make this change in order to support the new Element X mobile
client. More info:

https://element.io/blog/element-x-ignition/

Change-Id: Ieaa3bc7c5c6e397de4c1a63a2b67a150dff1f8e2
2023-09-27 13:15:06 +00:00
Jeremy Stanley
a6ab3543fc Move Airship and Kata lists to Mailman 3
This uncomments the list additions for the lists.airshipit.org and
lists.katacontainers.io sites on the new mailman server, removing
the configuration for them from the lists.opendev.org server and, in
the case of the latter, removing all our configuration management
for the server as it was the only site hosted there.

Change-Id: Ic1c735469583e922313797f709182f960e691efc
2023-09-14 12:08:34 +00:00
James E. Blair
fec1277185 Update gerrit image to bookworm
And upgrade to Python 3.11, and JDK to 17.

Change-Id: I7c9415e6706141db6cd9cab056a439da81469def
2023-09-08 08:28:22 -07:00
Zuul
a4011f3808 Merge "Explicitly disable offline reindexing during project renames" 2023-09-01 00:13:59 +00:00
Jeremy Stanley
9bbe19d4a3 Temporarily limit node image upload concurrency
One of our providers is struggling with Glance backend tasks after
upload, which causes increasingly lengthy delays for image readiness
the more images we're uploading. Globally reduce our upload
concurrency to the minimum possible of one per builder, for now, in
order to alleviate the delays and increase the chances images will
come ready in that provider.

This can be reverted once the situation has been resolved.

Change-Id: I29daa8f6d2d13055baf74215184ce0987fc20be0
2023-08-30 21:16:01 +00:00
Zuul
d148cb51e1 Merge "mailman3: re-sync custom web/settings.py" 2023-08-28 22:15:22 +00:00
Clark Boylan
df66e2fd6f Exclude i686 rpms in our centos mirrors
For some reason the x86_64/ directories of centos mirrors contain i686
packages too. Since none of our systems rely on i686 lets go ahead and
exclude these packages entirely. This should save quite a bit of disk
space (almost halving the required disk space?).

Change-Id: I146765da021e8436365e39a6dca4db470a07fe81
2023-08-28 08:46:35 -07:00
Jeremy Stanley
e13ea50e68 mailman3: re-sync custom web/settings.py
The version in the container config repository has moved on, update
our copy with overrides for the allowed hosts and site ID. Adjust
how we generate the allowed hosts envvar list to replace Exim's
field separators with those expected by Mailman.

Change-Id: Ia6831ca10e1cd1ad057475e0f78eacc02857eef2
2023-08-25 21:14:50 +00:00
James E. Blair
44b0e32063 Restart matrix-eavesdrop when config changes
Change-Id: I8b8846d5451159ddf17966a58526f70ab3a258dc
2023-08-24 12:59:13 -07:00
Zuul
c2ea42a86e Merge "Update to Gitea 1.20" 2023-08-24 18:58:57 +00:00
Zuul
1b64bb2e31 Merge "Add StarlingX Matrix channels to the logbot" 2023-08-23 15:22:25 +00:00
Zuul
9d7dc8e73a Merge "gerrit: bump index.maxTerms" 2023-08-23 15:22:23 +00:00
Jeremy Stanley
62e9af560c Add StarlingX Matrix channels to the logbot
StarlingX has decided to move from IRC to Matrix, and 11 channels
have been established for them in a dedicated space on our
homeserver. Add those channels to the matrix-eavesdrop bot for
logging.

Change-Id: I3cb50ebbe892e837e85cefc094bc99bbb4b0a759
2023-08-22 17:19:23 +00:00
Clark Boylan
ef450d1bce Update to Gitea 1.20
The 1.20 release is here. Upgrade to this version.

Things we change:
 * Nodejs is updated to v20 to match the alpine 3.18 package version
   that gitea switched to.
 * Templates are updated to match upstream 1.20 templates.
 * We drop the deprecated LFS_CONTENT_PATH from our server config and
   add an equivalent [lfs] config section.
 * Normalize app.ini content so that gitea won't write it back out to
   disk which fails due to permissions (and we don't want it overriding
   our configs anyway). For this we need to add WORK_PATH,
   oauth2.JWT_SECRET, and normliazing spacing and quoting for entries.
 * Set JWT_SIGNING_PRIVATE_KEY_FILE explicitly to be located at
   /data/gitea/jwt/private.pem otherwise gitea attempts to create the
   jwt/ directory somewhere it doesn't have permissions to (I think /)
   and won't be persisted across containers.
 * Replace log.ENABLE_ACCESS_LOG with log.logger.access.MODE = file as
   log.ENABLE_ACCESS_LOG is deprecated and doesn't appear to work
   anymore. This appears to be a documentation issue or they deprecated
   and removed things more quickly than originaly anticipated.
 * Add log.ACCESS_LOG_TEMPLATE to readd source port info to the access
   logs.
 * Add a templates/custom/header.tmpl file to set theme-color as the
   config item for this has been removed.

The 1.20.0 changelog [0] lists a number of breaking changes. I have
tried to capture there here as well as potential impacts to us:

 * Fix WORK_DIR for docker (root) image (#25738) (#25811)
   * We set APP_DATA_PATH to /data/gitea in our app.ini config which
     means we aren't relying on the inferred value from WORK_DIR. I
     think this isolates us from this chnage. But we can check for any
     content in /app/gitea on our running containers to be sure.
     Note we hardcode WORK_PATH to /data/gitea because gitea attempts to
     write this back to our config file otherwise as a result of this
     change.
 * Restrict [actions].DEFAULT_ACTIONS_URL to only github or self (#25581) (#25604)
   * We disable actions. This shouldn't affect us.
 * Refactor path & config system (#25330) (#25416)
   * This is related to the first breaking changes. Basically we need
     to check our use of WORK_PATH and determine if we need to hardcode
     it to something. Probably a good idea given how they keep changing
     this on us...
 * Fix all possible setting error related storages and added some tests (#23911) (#25244)
   * We don't use storage configs. This shouldn't affect us.
 * Use a separate admin page to show global stats, remove actions stat (#25062)
   * The breaking change only affects the use of Prometheus which we
     don't have yet.
 * Remove the service worker (#25010)
   * Is listed as a breaking change for UI cleanup that we don't need to
     cleanup. (ui.USE_SERVICE_WORKER can be removed).
 * Remove meta tags theme-color and default-theme (#24960)
   * https://github.com/go-gitea/gitea/pull/24960
   * Addressed by adding a custome templates/custom/header.tmpl file
     that sets this meta tag to the existing value. Note this only
     affects mobile clients so needs to be double checked via a mobile
     device.
 * Use [git.config] for reflog cleaning up (#24958)
   * Affects git.reflog config entries and we don' thave any.
 * Allow all URL schemes in Markdown links by default (#24805)
   * TODO determine if we need to limit link types and add that
     change if so. A point release was made to exclude bad types
     already. Not sure if there are others we need to add.
 * Redesign Scoped Access Tokens (#24767)
   * This breaks scoped tokens with scopes that don't exist anymore.
     I don't think we use scoped tokens.
 * Fix team members API endpoint pagination (#24754)
   * They 1 index the pagination of this endpoint now instead of 0
     indexing it.
 * Rewrite logger system (#24726)
   * They made changes to the loggers and encourage people to check
     their logs work as expected when upgrading. Using our test instance
     logs I don't see anything that is a problem.
 * Increase default LFS auth timeout from 20m to 24h (#24628)
   * We don't LFS but can change the timeout if necssary.
 * Rewrite queue (#24505)
   * Check for 'Removed queue option:' log entries and clean up
     corresponding entries in app.ini. We don't have any of these
     entries in our logs.
 * Remove unused setting time.FORMAT (#24430)
   * We didn't have this entry in app.ini.
 * Refactor setting.Other and remove unused SHOW_FOOTER_BRANDING (#24270)
   * This setting can be removed from app.ini, but we don't set it.
 * Correct the access log format (#24085)
   * We uncorrect it because they removed source port info in the
     correction step. They did this because some log parsers don't
     understand having the port info present, but if you are behind a
     reverse proxy this information is very important. We run gitea behind
     a reverse proxy.
 * Reserve ".png" suffix for user/org names (#23992)
   * .png is no longer a valid user/org name (it didn't work before
     anyway).
 * Prefer native parser for SSH public key parsing (#23798)
   * If you relied on the openssh ssh-keygen executable for public key
     parsing then you must explicitly set config to use it. I don't
     think we do as the golang native parser should handle the keytypes
     we use.
 * Editor preview support for external renderers (#23333)
   * This removed an app.ini settings we don't seem to set.
 * Add Gitea Profile Readmes (#23260)
   * Readmes in .profile repositories will always be shown now. We don't
     have .profiles repos so this doesn't affect us.
 * Refactor ctx in templates (#23105)
   * This affects custom templates as we may need to replace ctx with
     ctxData in our templates.
   * I've searched our templates for 'root', 'ctx', and 'ctxData' and
     have found no instances. Looking at the files modifying by the
     commits related to this change:
     bd7f218dce
     7c01260e1d
     we don't seem to override the affected files. I think we are fine
     as is.

The 1.20.1 changelog indicates there are no breaking changes, and git
diff shows no changes to the templates between 1.20.0 and 1.20.1.

The 1.20.2 changelog indicates there are no breaking changes, and git
diff shows no changes to the templates between 1.20.1 and 1.20.2.

The 1.20.3 changelog indicates there is a single breaking change:
 * Fix the wrong derive path (#26271) (#26318)
   * If I'm reading the code correctly, I think the problem was storage
     configuration inheriting the base storage config and particularly
     the related path. Then when archival storage looked for its config
     the path was the root gitea storage path and it would inadverdently
     delete all repos when deleting a single repo or something like
     that. We don't use these features and these are mirrors anyway so I
     don't think this really affects us.

[0] https://github.com/go-gitea/gitea/blob/v1.20.3/CHANGELOG.md

Change-Id: I265f0ad16c0e757a11c1d889996ffe2198625a1a
2023-08-21 08:49:46 -07:00
Dr. Jens Harbott
08610609e5 gerrit: bump index.maxTerms
The default value is 1024, which causes issues for users that have
starred more than that number of changes. Bump by 50% hoping that the
possible performance impact will be moderate.

[0] https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#index

Change-Id: I0c00110cfd6ba6d235821f6a5db7e1b91e2a8945
2023-08-19 22:59:57 +02:00
Zuul
e8a274e8cc Merge "Mailman3: check docker-compose stderr not stdout" 2023-08-17 21:19:59 +00:00
Jeremy Stanley
188ea92675 Mailman3: check docker-compose stderr not stdout
For the conditional check as to whether docker-compose up was a
no-op, we look for a recognizable string in the output it generates.
This information is emitted on the stderr descriptor, not stdout, so
correct the match condition accordingly.

Change-Id: I82015e615071458c95342eaea2e6a17aeae44c07
2023-08-17 20:04:09 +00:00
Zuul
508ebe2dfc Merge "Restart Mailman 3 containers when configs change" 2023-08-17 19:55:51 +00:00
Jeremy Stanley
eec60a2cb9 Restart Mailman 3 containers when configs change
Add a handler for restarting Mailman 3 containers if they're up, and
notify it from all of the copy tasks for configuration files. Check
for the uwsgi processes that Django runs under to determine whether
the containers are running.

Change-Id: I73be59be773fdde100999c7872520aab1d9e2066
2023-08-17 15:52:26 +00:00
Zuul
0b86e9e148 Merge "Remove old insecure-ci-registry01 from our inventory" 2023-08-17 04:38:58 +00:00
Zuul
b5e069dbbe Merge "Update etherpad to 1.9.1" 2023-08-16 16:29:21 +00:00
Zuul
920afd7087 Merge "Use magic domain guessing in Mailman 3" 2023-08-15 22:58:42 +00:00
Zuul
0c40d48422 Merge "Make mailman3 DB migration check PyVer-agnostic" 2023-08-15 22:58:40 +00:00
Jeremy Stanley
330a5e02e6 Correct static known_hosts entry for goaccess jobs
When the static.opendev.org service was replaced up a new server, we
neglected to update its SSH known_hosts entry in *-goaccess-report
jobs, so they began failing. Get them back on track.

Change-Id: I7a4902d6edfbc0bbfedbc6dbf9ea7e93acd04386
2023-08-07 21:27:40 +00:00
Jeremy Stanley
b3794ea1d5 Use magic domain guessing in Mailman 3
Setting SITE_ID=0 in settings.py is a signal to Postorius and
Hyperkitty that they should guess the domain and filter lists based
on the hostname supplied by the browser:

https://docs.mailman3.org/en/latest/faq.html#the-domain-name-displayed-in-hyperkitty-shows-example-com-or-something-else

Change-Id: I34df58fd95df3ff8075850ac370be9260a245021
2023-08-01 20:58:34 +00:00
Jeremy Stanley
0ba4d751a2 Make mailman3 DB migration check PyVer-agnostic
Embedding the Python version in the path to the Alembic config file
breaks down any time we change Python versions. Skirt this gotcha by
using a shell in the container to expand a file glob pattern.

Change-Id: I0e842524c53515f43af3f3284bc31476a9fe7ade
2023-08-01 20:58:34 +00:00
Ian Wienand
f61aa1cfde
nodepool osuosl : use public7
At the request of the upstream admins, use the public7 network which
should have more floating IP's we can use for tests nodes.

Change-Id: I1ed46a23832a1b9761875d61f2d8779914d061ae
2023-07-20 11:44:34 +10:00