letsencrypt : don't use staging in the gate

Currently we connect to the LE staging environment with acme.sh during
CI to get the DNS-01 tokens (but we never follow-through and actually
generate the certificate, as we have nowhere to publish the tokens).
We've known for a while that LE staging isn't really meant to be used
by CI like this, and recent instability has made the issue pronounced.

This modifies the driver script to generate fake tokens which work to
ensure all the DNS processing, etc. is happening correctly.

I have put this behind a flag so the letsencrypt job still does this
however.  I think it is worth this job actually calling acme.sh to
validate this path; this shouldn't be required too often.

Change-Id: I7c0b471a0661aa311aaa861fd2a0d47b07e45a72

playbooks/roles/letsencrypt-acme-sh-install/files/driver.sh

@@ -48,6 +48,27 @@ if [[ ${1} == "issue" ]]; then
                 #  - extract everything between ' '
                 #  - stick every two lines together, separated by a :
     done
+elif [[ ${1} == "issue-selfsign" ]]; then
+    shift;
+    for arg in "$@"; do
+        # looks like
+        #  "-d foo01.com -d foo.com " "-d bar01.com -d bar.com"
+        arr=(${arg})
+        len=${#arr[@]}
+        for (( i=0; i<$len; i++ )); do
+            if [[ $((i%2)) -eq 0 ]]; then
+                continue # this should be a "-d"
+            else
+                # The ACME protocol hashes "stuff" and the TXT record
+                # is ultimately a sha256 encoded into base64url
+                # (RFC8555); emulate that here.
+                base64url=$(echo -n ${arr[$i]}-${RANDOM} | \
+                                openssl dgst -binary -sha256 | \
+                                openssl base64 |  sed 's/+/-/g; s,/,_,g; s/=//g')
+                echo "${arr[$i]}:${base64url}"
+            fi
+        done
+    done
 elif [[ ${1} == "renew" ]]; then
     shift;
     for arg in "$@"; do
@@ -81,19 +102,22 @@ elif [[ ${1} == "selfsign" ]]; then
             mkdir -p ${CERT_HOME}/${domain}
             cd ${CERT_HOME}/${domain}
             echo "Creating certs in ${CERT_HOME}/${domain}"
+            # Create key for domain
+            openssl genrsa -out ${domain}.key 2048
+            # openssl makes this 0600; match the permissions acme.sh
+            # makes it with for general sanity
+            chmod 0640 ${domain}.key
             # Generate a fake CA key
             openssl genrsa -out ca.key 2048
             # Create fake CA root certificate
             openssl req -x509 -new -nodes -key ca.key -sha256 -days 1024 -subj "/C=US/ST=CA/O=opendev" -out ca.cer
-            # Create key for localhost
-            openssl genrsa -out ${domain}.key 2048
             # Create localhost certificate signing request
             openssl req -sha256 -new -key ${domain}.key -out ${domain}.csr -subj '/CN=localhost'
             # Create localhost certificate signed by fake CA
             openssl x509 -req -CA ca.cer -CAkey ca.key -CAcreateserial \
                     -sha256 -days 365 -in ${domain}.csr -out ${domain}.cer
             cp ${domain}.cer fullchain.cer
-        } | tee -a ${LOG_FILE}
+        } 2>&1 | tee -a ${LOG_FILE}
     done
 else
     echo "Unknown driver arg: $1"

playbooks/roles/letsencrypt-create-certs/README.rst

@@ -8,13 +8,26 @@ on the host.
 **Role Variables**
 
 .. zuul:rolevar:: letsencrypt_self_sign_only
+   :default: False
 
    If set to True, will locally generate self-signed certificates in
    the same locations the real script would, instead of contacting
    letsencrypt.  This is set during gate testing as the
    authentication tokens are not available.
 
+.. zuul:rolevar:: letsencrypt_self_generate_tokens
+   :default: False
+
+   When set to ``True``, self-generate fake DNS-01 TXT tokens rather
+   than acquiring them through the ACME process with letsencrypt.
+   This avoids leaving "half-open" challenges during gate testing,
+   where we have no way to publish the DNS TXT records letsencrypt
+   gives us to complete the certificate issue.  This should be
+   ``True`` if ``letsencrypt_self_sign_only`` is ``True`` (unless you
+   wish to specifically test the ``acme.sh`` operation).
+
 .. zuul:rolevar:: letsencrypt_use_staging
+   :default: False
 
    If set to True will use the letsencrypt staging environment, rather
    than make production requests.  Useful during initial provisioning

playbooks/roles/letsencrypt-request-certs/tasks/acme.yaml

@@ -7,7 +7,7 @@
 - name: Run acme.sh driver for certificate issue
   shell:
     cmd: |
-      /opt/acme.sh/driver.sh issue {{ acme_args }}
+      /opt/acme.sh/driver.sh {{ 'issue-selfsign' if letsencrypt_self_generate_tokens else 'issue' }} {{ acme_args }}
   args:
     chdir: /opt/acme.sh/
   register: acme_output

playbooks/zuul/templates/group_vars/letsencrypt.yaml.j2

@@ -3,6 +3,7 @@
 # issues.  As we don't have the authentication keys exposed in the
 # gate, only generate a place-holder self-signed cert for testing.
 letsencrypt_use_staging: True
+letsencrypt_self_generate_tokens: {{ letsencrypt_self_generate_tokens|default(True) }}
 letsencrypt_self_sign_only: True
 
-letsencrypt_account_email: le-test@opendev.org
+letsencrypt_account_email: le-test@opendev.org

zuul.d/system-config-run.yaml

@@ -187,6 +187,8 @@
       run_playbooks:
         - playbooks/service-nameserver.yaml
         - playbooks/letsencrypt.yaml
+      # Make sure this test runs acme.sh
+      letsencrypt_self_generate_tokens: False
     host-vars:
       bridge.openstack.org:
         host_copy_output: