Remove the old mailing list server

Clean up references to lists.openstack.org other than as a virtual
host on the new lists01.opendev.org Mailman v3 server. Update a few
stale references to the old openstack-infra mailing list (and
accompanying stale references to the OpenStack Foundation and
OpenStack Infra team). Update our mailing list service documentation
to reflect the new system rather than the old one. Once this change
merges, we can create an archival image of the old server and delete
it (as well as removing it from our emergency skip list for
Ansible).

Side note, the lists.openstack.org server will be 11.5 years old on
November 1, created 2012-05-01 21:14:53 UTC. Farewell, old friend!

Change-Id: I54eddbaaddc7c88bdea8a1dbc88f27108c223239
This commit is contained in:
Jeremy Stanley 2023-10-11 18:54:47 +00:00
parent 8991d4b160
commit cab53d10ac
14 changed files with 23 additions and 416 deletions

View File

@ -5,33 +5,33 @@
Mailing Lists
#############
`Mailman <http://www.gnu.org/software/mailman/>`_ is installed on
lists.openstack.org to run OpenStack related mailing lists, as well as
host list archives.
`The Mailman Suite <https://www.list.org/>`_ (Mailman Core, Postorius,
Hyperkitty) is installed on lists.opendev.org to run mailing lists for the
OpenDev Collaboratory and projects it hosts, as well as archives of those
lists.
At a Glance
===========
:Hosts:
* http://lists.openstack.org
* https://lists.opendev.org (and numerous other aliases)
:Ansible:
* :git_file:`playbooks/service-lists.yaml`
* :git_file:`playbooks/roles/mailman`
* :git_file:`playbooks/roles/mailman-site`
* :git_file:`playbooks/roles/mailman-list`
* :git_file:`playbooks/service-lists3.yaml`
* :git_file:`playbooks/roles/mailman3`
:Projects:
* http://www.gnu.org/software/mailman/
* https://www.list.org/
* https://gitlab.com/mailman/
:Bugs:
* https://storyboard.openstack.org/#!/project/748
* https://bugs.launchpad.net/mailman
* https://gitlab.com/mailman/
:Resources:
* `Mailman Documentation <http://www.gnu.org/software/mailman/docs.html>`_
* `Mailman Documentation <https://docs.mailman3.org/>`_
Adding a List
=============
A list may be added by adding it to the ``openstack-infra/system-config``
repository in :git_file:`inventory/service/host_vars/lists.openstack.org.yaml`.
repository in :git_file:`inventory/service/host_vars/lists01.opendev.org.yaml`.
For example:
.. code-block:: yaml
@ -41,61 +41,14 @@ For example:
admin: 'admin@example.com'
password: "{{ mailman_list_password }}"
Scripted Changes to Lists
=========================
Scripted Interaction with Lists
===============================
This may only be performed with root access to the list server.
Mailman supports running a python code snippet in the context of
individual lists or every list on the system. The following example
adds an address to the list of banned addresses for every list. This
has proved useful in the case of attackers abusing the HTTP
subscription interface to subscribe a target's address to multiple
mailing lists.
Banning an Address from All Lists
---------------------------------
Create the file `/usr/lib/mailman/bin/ban.py` with the following
content:
.. code-block:: python
def ban(m, address):
try:
m.Lock()
if address not in m.ban_list:
m.ban_list.append(address)
m.Save()
finally:
m.Unlock()
And then run the withlist script as:
.. code-block:: bash
sudo -u list /usr/lib/mailman/bin/withlist -a -r ban "<address to ban>"
sudo docker-compose -f /etc/mailman-compose/docker-compose.yaml exec -T -u mailman mailman-core mailman help
Because the script itself handles locking, do not use the `-l`
argument to withlist. To run the same script on a single list, use:
.. code-block:: bash
sudo -u list /usr/lib/mailman/bin/withlist -r ban listname "<address to ban>"
Note that the ban list accepts regular expressions, so to ban an
address and all suffixes, use '^address.*@example.com' as the "address
to ban".
Lock Files
----------
If a list stops handling traffic for some time, it may be due to a
stale lock file. Mailman locks are in /srv/mailman/openstack/locks.
If a lock is held for a list, then ``listname.lock`` will exist. The
contents of the file will be the name of the lock sequence file which
was used to obtain the lock. That file is in the form
``listname.lock.hostname.pid.sequence``. If the process id in that
string no longer exists, it's safe to assume the process died without
cleaning up the lock. It should generally be safe to remove the
lockfile in that case.
Use context help for the CLI's many subcommands, or see the Mailman v3
documentation for more details.

View File

@ -39,7 +39,11 @@ Contributing
this list to get smaller over time.
We welcome contributions from new contributors. Reading this
documentation is the first step. You should also join our `mailing list <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra>`_.
documentation is the first step. You should also join our `announcements
<https://lists.opendev.org/mailman3/lists/service-announce.lists.opendev.org/>`_
and `discussion
<https://lists.opendev.org/mailman3/lists/service-discuss.lists.opendev.org/>`_
mailing lists.
We are most active on IRC, so please join the **#opendev**
channel on OFTC.

View File

@ -29,7 +29,6 @@ cacti_hosts:
- kdc03.openstack.org
- kdc04.openstack.org
- keycloak01.opendev.org
- lists.openstack.org
- lists01.opendev.org
- nb01.opendev.org
- nb02.opendev.org

View File

@ -308,17 +308,6 @@ all:
- 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKUtLplUhod5VnjVoTY5WHhjOHrRM6puFpFpcr9iJmOKkbnJ5V2SA8U0thFEne4XUoa/eZ3SiQ9Yt923+1MAcKQ='
- 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5qje1++4tUZ1U4sQ2Jsju/S4BdpCeiauSxZ2uMdQSegtjZ4GclxRjP4zJjL6P/iixTwjsu4dOEnvPt8B9JZGEaYERzKiqjIRT3I80mTjI0wsx+bN38Z2xg5Tm1O5xrOxT0rjA2zGJDRtMhk6IwmUg4DELlxUfalsWgpoZV0fYxUFneOgVuG8XY841b1igh2ScyOuSfu8RQFF3YTulzoT7o8QzgdKiliciLAWujy+4okN8wln5/atqiDuN7oi+9WYLt/HW2YZTUHd2/u+ZghgvbVVJ8xsB2gQ+BESS3P4YZsWMqM/7lz/7GVUQfolRnC5dyPOa9cwuoBW9ru6VGYH/'
- 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDSUpspKrIHEXRkP9xIa/hyKkauDvuPX0nVwWpUzQkIh'
lists.openstack.org:
ansible_host: 50.56.173.222
location:
cloud: openstackci-rax
region_name: DFW
public_v4: 50.56.173.222
public_v6: 2001:4800:780e:510:3bc3:d7f6:ff04:b736
host_keys:
- 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJXPszbNKhny/BiR9JAC/yvV5Knqh6ZgUlk21XzJlk7D3kCBcyuMCSlWskvQU/59pYM5pDSBf8+a//pvxoR3Cm0MBPrMEYGT3LcGY/lxsgHoHaHhMgRvdHKVNPsO852CuecCB31HnUZ5c0cZJQ3V5UzZzld5FRvp9tV6vxKdaLrkA8HEuktMj59Tgm3zUv7a6Ou6JV78NC8zFAw73ZUZYNgRZ118MSd3ZogBoHOLY0FEWF6vwYHOC3qp3655+SEHEzJOVxknntxeeaopVLqxuIOkfjDCAc8DIBhbFr/IevK34XWSlRJjXvQnYKpzA1/urwugsLvQKqFwjeVQZOn5Bb'
- 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH7wzqKuzV5f6N5lCoOMqdXE0j/LpGAU5/vfPadqzadQCY4xApfpbJgCnGWJ6dESAKhv8HRIZ7nVKuns8NfNQIU='
- 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL4fR5POWlUrnu8JGSKLCxPFY5VCOwveqsDgCDWimWCD'
lists01.opendev.org:
ansible_host: 162.209.78.70
location:

View File

@ -39,7 +39,6 @@ groups:
- lists99.opendev.org
# All these servers are "special-cased" in specifically
# as they are puppet and should be replaced "soon"
- lists.openstack.org
- storyboard01.opendev.org
- translate01.openstack.org
borg-backup-server:
@ -92,7 +91,6 @@ groups:
- graphite[0-9]*.opendev.org
- insecure-ci-registry[0-9]*.opendev.org
- keycloak[0-9]*.opendev.org
- lists.openstack.org
- lists[0-9]*.opendev.org
- meetpad[0-9]*.opendev.org
- mirror[0-9]*.opendev.org
@ -105,8 +103,6 @@ groups:
- tracing[0-9]*.opendev.org
- translate[0-9]*.open*.org
- zuul[0-9]*.opendev.org
mailman:
- lists.openstack.org
mailman3:
- lists[0-9]*.opendev.org
meetpad:

View File

@ -1,168 +0,0 @@
mm_domains: 'lists.openstack.org'
exim_local_domains: "@:{{ mm_domains }}"
exim_enable_spf: true
exim_aliases:
root: "{{ ','.join(listadmins|default([])) }}"
interop-wg: openstack-discuss
openstack: openstack-discuss
openstack-dev: openstack-discuss
openstack-infra: openstack-discuss
openstack-operators: openstack-discuss
openstack-security: openstack-discuss
openstack-sigs: openstack-discuss
openstack-tc: openstack-discuss
user-committee: openstack-discuss
airship-discuss-owner: spam
community-owner: spam
edge-computing-owner: spam
foundation-board-confidential-owner: spam
foundation-board-owner: spam
foundation-owner: spam
legal-discuss-owner: spam
mailman-owner: spam
marketing-owner: spam
openstack-announce-owner: spam
openstack-docs-owner: spam
openstack-fr-owner: spam
openstack-i18n-owner: spam
openstack-infra-owner: spam
openstack-ko-owner: spam
openstack-qa-owner: spam
product-wg-owner: spam
user-committee-owner: spam
spam: ':fail: delivery temporarily disabled due to ongoing spam flood'
exim_domain_aliases:
community@lists.openstack.org: community@lists.openinfra.dev
edge-computing@lists.openstack.org: edge-computing@lists.opendev.org
foundation@lists.openstack.org: foundation@lists.openinfra.dev
foundation-board@lists.openstack.org: foundation-board@lists.openinfra.dev
foundation-board-confidential@lists.openstack.org: foundation-board-confidential@lists.openinfra.dev
goldmembers@lists.openstack.org: goldmembers@lists.openinfra.dev
marketing@lists.openstack.org: marketing@lists.openinfra.dev
staff@lists.openstack.org: staff@lists.openinfra.dev
summit-programming-committee@lists.openinfra.dev: summit-track-chairs@lists.openinfra.dev
summitsponsors@lists.openstack.org: summitsponsors@lists.openinfra.dev
exim_routers:
- mailman_verp_router: |
{% raw -%}
driver = dnslookup
condition = ${if or{{eq{$sender_host_address}{127.0.0.1}}\
{eq{$sender_host_address}{::1}}}{yes}{no}}
{% endraw %}
domains = !+local_domains
ignore_target_hosts = <; 0.0.0.0; \
127.0.0.0/8; \
::1/128;fe80::/10;fe \
c0::/10;ff00::/8
senders = "*-bounces@*"
transport = mailman_verp_smtp
- dnslookup: '{{ exim_dnslookup_router }}'
- system_aliases: '{{ exim_system_aliases_router }}'
- domain_aliases: |
driver = redirect
allow_fail
allow_defer
data = ${lookup{$local_part@$domain}lsearch{/etc/aliases.domain}}
file_transport = address_file
pipe_transport = address_pipe
- localuser: '{{ exim_localuser_router }}'
- mailman_copy: |
driver = accept
domains = lists.openstack.org
local_parts = openstack-discuss
transport = local_copy
unseen
- mailman_router: |
driver = accept
domains = {{ mm_domains }}
local_part_suffix = -admin : \
-bounces : -bounces+* : \
-confirm : -confirm+* : \
-join : -leave : \
-owner : -request : \
-subscribe : -unsubscribe
local_part_suffix_optional
require_files = ${lookup{${lc::$domain}}lsearch{/etc/mailman/sites}}/lists/${lc::$local_part}/config.pck
transport = mailman_transport
exim_transports:
- local_copy: |
driver = appendfile
file = /var/mail/$local_part
group = mail
mode = 0660
- mailman_transport: |
driver = pipe
command = /var/lib/mailman/mail/mailman \
'${if def:local_part_suffix \
{${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} \
{post}}' \
$local_part
current_directory = /var/lib/mailman
environment = HOST=${lc:$domain}
group = list
home_directory = /var/lib/mailman
user = list
- mailman_verp_smtp: |
driver = smtp
headers_add = Errors-To: ${return_path}
headers_remove = Errors-To
max_rcpt = 1
return_path = ${local_part:$return_path}+$local_part=$domain@${domain:$return_path}
# We put lists.openstack.org first as it's the current servername
letsencrypt_certs:
lists-openstack-org-main:
- lists.openstack.org
mailman_multihost: true
mailman_sites:
- name: openstack
listdomain: lists.openstack.org
install_languages: ['de', 'fr', 'it', 'ko', 'ru', 'vi', 'zh_TW']
lists:
- name: mailman
description: 'The mailman site list'
admin: 'nobody@openstack.org'
password: "{{ mailman_list_password }}"
- name: openstack-es
description: 'Lista de correo acerca de OpenStack en español'
admin: 'flavio@redhat.com'
password: "{{ mailman_list_password }}"
- name: openstack-fr
description: 'List of the OpenStack french user group'
admin: 'erwan@erwan.com'
password: "{{ mailman_list_password }}"
- name: openstack-i18n
description: 'List of the OpenStack Internationalization team.'
admin: 'guoyingc@cn.ibm.com'
password: "{{ mailman_list_password }}"
- name: openstack-it
description: 'Discussioni su OpenStack in italiano'
admin: 'stefano@openstack.org'
password: "{{ mailman_list_password }}"
- name: openstack-ko
description: 'OpenStack Korea Community Discussions in Korean (오픈스택 한국 커뮤니티 메일링리스트)'
admin: 'ianyrchoi@gmail.com'
password: "{{ mailman_list_password }}"
- name: openstack-zh
description: 'OpenStack社区中文讨论群组'
admin: 'yeluaiesec@gmail.com'
password: "{{ mailman_list_password }}"
- name: release-job-failures
description: 'Notification messages for failures from release-related build jobs.'
admin: 'doug@doughellmann.com'
password: "{{ mailman_list_password }}"
- name: embargo-notice
description: 'Announcements to stakeholders for embargoed security vulnerabilities.'
admin: 'jeremy@openstack.org'
password: "{{ mailman_list_password }}"
- name: release-announce
description: 'Announcement of official OpenStack releases.'
admin: 'thierry@openstack.org'
password: "{{ mailman_list_password }}"
- name: openstack-mentoring
description: 'List to coordinate interactions between mentors and mentees of the OpenStack mentoring program. Also for questions about the mentoring program (i.e. how to get involved, how it works, etc.'
admin: 'amy@demarco.com'
password: "{{ mailman_list_password }}"
- name: openstack-discuss
description: 'Discussion of OpenStack usage and development.'
admin: 'fungi@yuggoth.org'
password: "{{ mailman_list_password }}"

View File

@ -18,7 +18,7 @@
<p>The following options are available for community-based support.</p>
<ul>
<li><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss">openstack-discuss</a> mailing list.</li>
<li><a href="https://lists.openstack.org/mailman3/lists/openstack-discuss.lists.openstack.org/">openstack-discuss</a> mailing list.</li>
<li><a href="https://stackoverflow.com/questions/tagged/openstack">stackoverflow.com</a> for code questions.</li>
<li><a href="https://serverfault.com/tags/openstack/info">serverfault.com</a> for operations.</li>
<li>The <a href="https://web.archive.org/web/*/ask.openstack.org">Internet
@ -27,4 +27,3 @@
</body>
</html>

View File

@ -1,26 +0,0 @@
- hosts: "lists.openstack.org"
tasks:
# Make sure Mailman services for each site are running so that they
# will attempt to deliver any pending list admin notifications and
# we can capture that activity in the Exim logs.
- name: Restart Mailman services
service:
name: "mailman-{{ zuul_mailman_site.name }}"
state: restarted
loop: "{{ mailman_sites }}"
loop_control:
loop_var: zuul_mailman_site
- hosts: "localhost"
tasks:
# Pause briefly so that mail deliveries are attempted, otherwise the
# job may finish too quickly and we won't have a complete picture in
# the MTA logs by the time we start collecting them.
#
# TODO(fungi): We could improve this to wait for specific log entries
# instead, I suppose.
- name: Wait for delivery attempts
wait_for:
timeout: 10

View File

@ -142,7 +142,6 @@
- host_vars/etherpad99.opendev.org.yaml
- host_vars/letsencrypt01.opendev.org.yaml
- host_vars/letsencrypt02.opendev.org.yaml
- host_vars/lists.openstack.org.yaml
- host_vars/gitea99.opendev.org.yaml
- host_vars/grafana01.opendev.org.yaml
- host_vars/mirror01.openafs.provider.opendev.org.yaml

View File

@ -1,11 +0,0 @@
- hosts: lists.openstack.org
tasks:
- name: Collect per-site mailman log directories
shell:
cmd: |
mkdir -p /var/log/mailman
for logs in /srv/mailman/*/logs ; do
cp -a $logs /var/log/mailman/$(basename $(dirname $logs))
done
become: yes
ignore_errors: true

View File

@ -1,62 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
testinfra_hosts = ['lists.openstack.org']
def test_mm_list_is_present(host):
cmd = host.run('HOST=lists.openstack.org list_lists --bare')
assert 'openstack-discuss' in cmd.stdout
def test_mm_list_site(host):
cmd = host.run('curl --insecure '
'--resolve lists.openstack.org:443:127.0.0.1 '
'https://lists.openstack.org/cgi-bin/mailman/listinfo')
assert '<TITLE>lists.openstack.org Mailing Lists</TITLE>' in cmd.stdout
def test_mm_list_site_redirect_http(host):
cmd = host.run('curl '
'--resolve lists.openstack.org:80:127.0.0.1 '
'http://lists.openstack.org/cgi-bin/mailman/listinfo')
assert ('The document has moved <a href="'
'https://lists.openstack.org/cgi-bin/mailman/listinfo'
'">here</a>') in cmd.stdout
def test_mm_list_site_redirect_listinfo(host):
cmd = host.run('curl --insecure '
'--resolve lists.openstack.org:443:127.0.0.1 '
'https://lists.openstack.org/cgi-bin/mailman/listinfo/staff')
assert ('The document has moved <a href="'
'https://lists.openinfra.dev/cgi-bin/mailman/listinfo/staff'
'">here</a>') in cmd.stdout
def test_mm_list_site_redirect_archives(host):
cmd = host.run('curl --insecure '
'--resolve lists.openstack.org:443:127.0.0.1 '
'https://lists.openstack.org/pipermail/staff/')
assert ('The document has moved <a href="'
'https://lists.openinfra.dev/pipermail/staff/'
'">here</a>') in cmd.stdout
def test_mm_list_site_static_files(host):
cmd = host.run('curl --insecure '
'--resolve lists.openstack.org:443:127.0.0.1 '
'https://lists.openstack.org/archives.yaml')
assert 'openstack-discuss' in cmd.stdout
cmd = host.run('curl --insecure '
'--resolve lists.openstack.org:443:127.0.0.1 '
'https://lists.openstack.org/robots.txt')
assert 'Disallow: /' in cmd.stdout
def test_domain_aliases(host):
domain_aliases = host.file('/etc/aliases.domain')
assert domain_aliases.exists
assert domain_aliases.contains('staff@lists.openstack.org: staff@lists.openinfra.dev')

View File

@ -561,20 +561,6 @@
- playbooks/roles/graphite/
- playbooks/roles/iptables/
- job:
name: infra-prod-service-lists
parent: infra-prod-service-base
description: Run service-lists.yaml playbook.
vars:
playbook_name: service-lists.yaml
files:
- inventory/base
- inventory/service/host_vars/lists.openstack.org.yaml
- playbooks/roles/iptables/
- playbooks/roles/base/exim
- playbooks/roles/mailman/
- playbooks/service-lists.yaml
- job:
name: infra-prod-service-lists3
parent: infra-prod-service-base

View File

@ -29,7 +29,6 @@
- name: system-config-build-image-hound
soft: true
- system-config-run-kerberos
- system-config-run-lists
- system-config-run-lists3:
dependencies:
- name: opendev-buildset-registry
@ -196,7 +195,6 @@
- name: system-config-upload-image-hound
soft: true
- system-config-run-kerberos
- system-config-run-lists
- system-config-run-lists3:
dependencies:
- name: opendev-buildset-registry
@ -509,12 +507,6 @@
dependencies:
- name: infra-prod-letsencrypt
soft: true
- infra-prod-service-lists: &infra-prod-service-lists
dependencies:
- name: infra-prod-service-borg-backup
soft: true
- name: infra-prod-letsencrypt
soft: true
- infra-prod-service-lists3: &infra-prod-service-lists3
dependencies:
- name: infra-prod-service-borg-backup
@ -665,7 +657,6 @@
- infra-prod-service-graphite: *infra-prod-service-graphite
- infra-prod-service-keycloak: *infra-prod-service-keycloak
- infra-prod-service-meetpad: *infra-prod-service-meetpad
- infra-prod-service-lists: *infra-prod-service-lists
- infra-prod-service-lists3: *infra-prod-service-lists3
- infra-prod-service-mirror: *infra-prod-service-mirror
- infra-prod-service-nodepool: *infra-prod-service-nodepool

View File

@ -277,48 +277,6 @@
- playbooks/roles/letsencrypt-install-txt-record
- playbooks/roles/letsencrypt-request-certs
- job:
name: system-config-run-lists
parent: system-config-run
description: |
Run the playbook for a list server.
post-run: playbooks/zuul/run-lists-post.yaml
nodeset:
nodes:
- <<: *bridge_node_x86
- name: lists.openstack.org
label: ubuntu-focal
groups:
- <<: *bastion_group
required-projects:
- opendev/system-config
files:
- playbooks/bootstrap-bridge.yaml
- inventory/service/host_vars/lists.openstack.org.yaml
- inventory/service/group_vars/mailman.yaml
- playbooks/roles/base/exim
- playbooks/roles/mailman/
- playbooks/service-lists.yaml
- playbooks/test-lists.yaml
- playbooks/zuul/templates/host_vars/lists.openstack.org.yaml.j2
- testinfra/test_lists_o_o.py
- playbooks/zuul/run-lists-post.yaml
vars:
run_playbooks:
- playbooks/letsencrypt.yaml
- playbooks/service-lists.yaml
# Run this twice to check idempotency
- playbooks/service-lists.yaml
run_test_playbook: playbooks/test-lists.yaml
host-vars:
lists.openstack.org:
host_copy_output:
'/etc/aliases.domain': logs_txt
'/var/log/acme.sh': logs
'/var/log/apache2': logs
'/var/log/mailman': logs
'/etc/apache2/sites-enabled': logs
- job:
name: system-config-run-lists3
# We don't use the system-config-run-containers base job because we