Re-expose our Mailman archives.yaml and robots.txt

In switching to all-HTTPS for Mailman sites, it was missed that only
the plain HTTP vhosts set a DocumentRoot of /var/www. This was only
used for publishing metadata so went unnoticed until now. Rather
than add a DocumentRoot to the new HTTPS vhosts, simply use Aliases
to map the specific files we want to expose, for improved clarity
and to make it less likely they'll be overlooked in configuration in
the future.

In order to make sure the archives.yaml file exists at server
creation, before its cronjob fires for the first time, add a direct
invocation of the script which builds it. Move all tasks related to
this after the tasks which create the mailing lists, so that the
generated file will include them. This also simplifies testing.

For the non-multihost configuration, only robots.txt is expected to
be present, so don't add an alias for archives.yaml there.

Also add regression tests to ensure we keep these working.

Change-Id: I6b54b0386f0ea9f888c1f23580ad8698314474b9

playbooks/roles/mailman-site/templates/mailman_multihost.vhost.j2

@@ -39,6 +39,8 @@
   ScriptAlias /cgi-bin/mailman/ /usr/lib/cgi-bin/mailman/
   Alias /pipermail/ /srv/mailman/{{ mailman_site.name }}/archives/public/
   Alias /images/mailman/ /usr/share/images/mailman/
+  Alias /archives.yaml /var/www/archives.yaml
+  Alias /robots.txt /var/www/robots.txt
 
   <Directory /usr/lib/cgi-bin/mailman/>
     AllowOverride None

playbooks/roles/mailman/tasks/main.yaml

@@ -62,25 +62,6 @@
         owner: root
         group: root
         mode: '0444'
-    - name: Install mailman mk-archives-index
-      copy:
-        src: mk-archives-index
-        dest: /usr/local/sbin/mk-archives-index
-        owner: root
-        group: root
-        mode: '0744'
-    - name: Set cron PATH for mk-archives-index
-      cron:
-        name: PATH
-        env: yes
-        job: /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
-    - name: Enable mk-archives-index cron
-      cron:
-        name: "mk-archives-index cron"
-        state: present
-        job: mk-archives-index > /var/www/archives.yaml
-        minute: "0"
-        hour: "0"
     - name: Create mm sites file
       template:
         src: sites.j2
@@ -101,6 +82,30 @@
         src: "domain_aliases.j2"
         dest: "/etc/aliases.domain"
         mode: 0444
+    - name: Install mailman mk-archives-index
+      copy:
+        src: mk-archives-index
+        dest: /usr/local/sbin/mk-archives-index
+        owner: root
+        group: root
+        mode: '0744'
+    - name: Set cron PATH for mk-archives-index
+      cron:
+        name: PATH
+        env: yes
+        job: /usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
+    - name: Pre-run mk-archives-index
+      shell:
+        cmd: /usr/local/sbin/mk-archives-index > /var/www/archives.yaml
+      args:
+        creates: /var/www/archives.yaml
+    - name: Enable mk-archives-index cron
+      cron:
+        name: "mk-archives-index cron"
+        state: present
+        job: mk-archives-index > /var/www/archives.yaml
+        minute: "0"
+        hour: "0"
   when: mailman_multihost
 
 - name: normal mailman configs

playbooks/roles/mailman/templates/mailman.vhost.j2

@@ -35,6 +35,7 @@
   ScriptAlias /cgi-bin/mailman/ /usr/lib/cgi-bin/mailman/
   Alias /pipermail/ /var/lib/mailman/archives/public/
   Alias /images/mailman/ /usr/share/images/mailman/
+  Alias /robots.txt /var/www/robots.txt
 
   <Directory /usr/lib/cgi-bin/mailman/>
     AllowOverride None

testinfra/test_lists_k_i.py

@@ -29,3 +29,9 @@ def test_mm_list_site_redirect_http(host):
     assert ('The document has moved <a href="'
             'https://lists.katacontainers.io/cgi-bin/mailman/listinfo'
             '">here</a>') in cmd.stdout
+
+def test_mm_list_site_static_files(host):
+    cmd = host.run('curl --insecure '
+                   '--resolve lists.katacontainers.io:443:127.0.0.1 '
+                   'https://lists.katacontainers.io/robots.txt')
+    assert 'Disallow: /' in cmd.stdout

testinfra/test_lists_o_o.py

@@ -121,6 +121,16 @@ def test_mm_list_site_redirect_archives(host):
                    'https://lists.openstack.org/pipermail/staff/')
     assert '<h1>The Staff Archives </h1>' in cmd.stdout
 
+def test_mm_list_site_static_files(host):
+    cmd = host.run('curl --insecure '
+                   '--resolve lists.opendev.org:443:127.0.0.1 '
+                   'https://lists.opendev.org/archives.yaml')
+    assert 'service-discuss' in cmd.stdout
+    cmd = host.run('curl --insecure '
+                   '--resolve lists.opendev.org:443:127.0.0.1 '
+                   'https://lists.opendev.org/robots.txt')
+    assert 'Disallow: /' in cmd.stdout
+
 def test_domain_aliases(host):
     domain_aliases = host.file('/etc/aliases.domain')
     assert domain_aliases.exists