Merge "Add a specification for Mailman 3"

Zuul 2021-11-05 21:08:18 +00:00 committed by Gerrit Code Review
commit 56c1288bc5
2 changed files with 259 additions and 0 deletions

View File

@ -47,6 +47,7 @@ permits.
Help Wanted

specs/mailman3.rst Normal file
View File

@ -0,0 +1,258 @@
Copyright 2021 Jeremy Stanley
This work is licensed under a Creative Commons Attribution 3.0
Unported License.
Mailman 3 Migration
This specification covers the implementation of a new Mailman 3
based listserv in OpenDev, and subsequent migration of the existing
Mailman 2 based mailing lists we provide to the new server,
concluding with taking down the old server once all lists have been
migrated. Care will be taken to preserve existing list
configurations as much as the existing migration tooling allows, as
well as import all historical list archives and ensure old archive
hyperlinks still take people to the same content.
Problem Description
For years now, the Mailman community has been producing two series
of mailing list software:
- Mailman 2, which we currently use and have since circa 2010,
offers a stable and battle-hardened mailing list service focused
on exchanging information via E-mail, with an optional public Web
archive called Pipermail.
- Mailman 3, which was experimental for a very long time but has
become more stable and essentially reached feature parity with its
predecessor some years ago now, is functionally the same
internally as far as exchanging E-mail messages is concerned, but
has expanded substantially on its Web interface with a new
Postorius list management interface and Hyperkitty interactive Web
In the decade since we started running Mailman, significant shifts
have occurred in the ways people interface with E-mail and hold
group discussions. The mass freemail providers have become
entrenched, ubiquitous, and emboldened to begin breaking
long-established protocol standards. Portable handheld devices are
becoming the primary way many people interact. Web forums have
morphed into social media platforms, and their popularity has
further altered expectations of the next generation of computer
users. At the same time, these shifts in behavior have driven
changes in how Web applications are developed and delivered.
Mailman 3's Hyperkitty archive interface is essentially a bolt-on
Web forum frontend, with more advanced authentication and some
(mis?)features borrowed from popular social media. Its Postorius
list management interface uses the same new authentication
subsystems, having distinct per-user accounts and role-based access
controls instead of the shared password model used by MM2. Both of
these are now Django based applications, so can take advantage of
the Django ecosystem of additional features for further
customization such as social authentication and integration with
external identity providers.
Among relevant improvements in Mailman 3 is proper virtual domain
support, which should allow us to greatly simplify our current
implementation by hosting all our various list sites in a single
deployment without fragile workarounds, and more easily add new
sites in the future.
Proposed Change
Since there are semi-official `Mailman 3 containers
<>`_ already available on
GHCR/Quay/DockerHub, this seems like the easiest path forward and
most compatible with our present model of configuration management
and orchestration. They publish three applications as individual
containers: mailman-core (the listserv), mailman-web (a.k.a.
Hyperkitty), and postorius. In addition, it's assumed that the
system(s) where these run will provide a Web server to act as a
proxying front-end, an SMTP server, and an RDBMS.
We'll write Ansible playbooks and Zuul jobs to deploy the three
Mailman service containers along with an accompanying MariaDB
container, and then configure system packages of Apache and Exim to
support these (similar to how we do with our existing services, e.g.
Gerrit). The Mailman container documentation provides examples for
configuring `Exim <>`_
and `Nginx
but the latter can be easily adapted to Apache for increased
consistency with our other Web-based services, and the Mailman
documentation also has some `tips for this
Following our usual test-driven development methodology, once basic
functionality is working and tested successfully, we'll hold one or
more test nodes and perform direct interactive testing, first with
sample mailing lists, but later with imported production
configuration and archives (taking sufficient measures to avoid
annoying people with unwanted tests or duplicate messages). The
Mailman developers provide a convenient `migration tool
<>`_ which we can use
for this purpose. We should also be sure to note any significant
changes in behavior for things like bounce processing or which
mailing-list-specific headers get included in messages, so that we
can provide users with better guidance around the migration.
Once we're confident the deployment and config/archive imports are
working as expected, we'll boot a new production mailing list
server, then schedule and announce a migration window when we'll be
moving all the existing sites and their lists to the new server. A
full copy of the old pre-migration Pipermail archives will also be
served from the new server in order to preserve the functionality of
any existing bookmarks or historical hyperlinks. The virtual domain will migrate first in order to test
the waters, and then the remaining virtual domains will be migrated
individually as is convenient for their communities.
After all migrations are completed and any remaining issues sorted,
the old mailing list servers will be imaged for posterity and then
We could consider using distro packaged Mailman 3 like we presently
do for Mailman 2, the packages for it in Ubuntu 20.04 LTS are quite
usable at this point, though slightly outdated compared to what we
get from the Mailman 3 container images. The down-side here is that
we only consume new features when upgrading the operating system
itself to a new major release, and this tight coupling tends to
complicate things as MM3 is still a bit of a moving target and two
years of deferred upgrades could mean many significant changes in
behavior to juggle and untangle at the same time. On the other hand,
relying on distribution packages in theory shields us from
unexpected changes outside operating system upgrades, and (if the
packages are well maintained) means we get to skip bugs impacting
upstream releases which may be ironed out in packaging before they
ever trickle down to us.
We could also of course consider switching to a different mailing
list software entirely, or sticking with Mailman 2 (it's still
maintained), or migrate everyone to a SaaS list provider, or decide
to stop running mailing lists entirely. Any of these would require a
much different spec however.
- fungi
- clarkb
Gerrit Topic
Use Gerrit topic "mailman3" for all patches related to this spec.
.. code-block:: bash
git-review -t mailman3
Work Items
1. Create deployment playbooks and related testing.
2. Test config migration and archive import on a held test node.
3. Create new production list server.
4. Schedule and announce virtual domain migrations.
5. Migrate all Mailman virtual domains to the new server.
6. Stop the old servers, image them for posterity, and then delete.
No new git repositories need to be created.
A new server will be created using our default
image (currently Ubuntu 20.04 LTS), and the old and server will be
DNS Entries
The,,,,, and DNS
records will need updating to the addresses of the new server during
migration, as will the DNS records for any other virtual domains
added to the old servers prior to migration. New
records will be added, as will ACME records to support HTTPS for all
the virtual domains.
Borg backups will be configured for the new
server prior to migration of any virtual domains, and backups of the
old server will be retained after migration for as long as is
The system-config `mailing lists
document will be updated to reflect the details of the new
deployment. Additionally, user-facing documentation in the
Infrastructure Manual or maintained by individual project
communities may need updating if it mentions details of list
subscription or includes screenshots.
This is overall a net improvement in security of the service, since
Mailman 3 uses stronger (effectively non-brute-forcible) password
confirmation tokens, a proper password reset option rather than
simple reminders, stores password hashes internally instead of
plaintext copies, provides role-based access control rather than
relying on shared passwords, can integrate with other authentication
and identity providers if we want, and will be deployed with HTTPS
in order to reinforce the stronger security model.
The only significant security risk is that we operate a handful of
private mailing lists, and so extra care should be taken during
migration to avoid exposing or otherwise leaking the archives for
these. We should also make sure to double-check lists with moderated
subscription settings to be sure unauthorized users can't subscribe
after the list has been moved to the new system.
Basic functional testing of deployment and Web interfaces (including
some screen capture artifacts) will be added as part of creating the
configuration management and orchestration for the service.
Additional on-demand config and archive migration testing will be
performed with held job nodes. In future, end-to-end testing of
message delivery and archiving can easily be added to help prevent
regressions when evaluating new versions of the container images.
There are no dependencies outside of the spec itself.