Clark Boylan e2df5a8b47 Add docs for linaro cloud cert renewal process

These docs had lived on the server in question in a text file as they
were sort of cobbled together from emails. Since then multiple renewals
have been successfully performed so we may as well add the process to
our actual documentation.

Change-Id: I13267ad08c1e4ef6007e5cbea040c274ea2f27d5

2024-03-26 10:33:58 -07:00

6.9 KiB

Raw Blame History

title: letsencrypt

Let's Encrypt Certificates

We support provisioning certificates from https://letsencrypt.org for hosts in the opendev.org namespace.

At a Glance

Ansible

Resources

Chat

#opendev on OFTC

Overview

We support automatic provisioning of certificates from Let's Encrypt to hosts in the opendev.org domain.

This is implemented in OpenDev via the roles driven from :git_file:`playbooks/letsencrypt.yaml`. The overall actions implemented by the above roles are roughly:

Hosts that want a certificate use the amce.sh tool to request it from the Let's Encrypt CA.

Creation or renewal requests receive a TXT record authentication value that must be published to prove ownership of the domain. We implement this by making the challenge-request hostname _acme-challenge.hostname.opendev.org a CNAME record to a special "signing domain" acme.opendev.org.

Note if valid certificates are present and they are not within the renewal period (which is most of the time) no further action is taken.
The provided TXT record authentication values are installed and published to the acme.opendev.org domain via the OpenDev nameservers.
The host can now finalise certificate creation. Let's Encrypt checks _acme-chellenge.hostname.opendev.org, which is a CNAME to acme.opendev.org. Let's Encrypt then enumerates the TXT records there, and once finding the required key will return the signed keys to the host, which saves them to disk.

Configuring a host to get certificates

A basic configuration consists of the following steps:

Ensure the host is matched by the letsencrypt group in :git_file:`inventory/groups.yaml`.
DNS entries for _acme-chellenge.hostname as a CNAME to opendev.org must be added and live in the opendev.org zone.db file. Follow the other examples to ensure other fields such as CAA records are set too.

Take care to list all hostnames that you wish covered by the certificate (e.g. hostname01.opendev.org and hostname.opendev.org)
Configure the certificates to be issued to the host.

The roles look for certificate configuration in a letsencrypt_certs variable defined for each host. This is usually done via specific host variables in inventory/service/host_vars/<hostname>.opendev.org.yaml. For a simple host that wants a single certificate to cover its numeric hostname and regular CNAME this would look like :
```
letsencrypt_certs:
  hostname01-opendev-org:
   - hostname01.opendev.org
   - hostname.opendev.org
```
This will result in certificate material in /etc/letsencrypt-certs/hostname01.opendev.org/ on the host.

Note that the "certificate name" dictionary keys (just hostname01-opendev-org above) are essentially a free-form string, but are used in the next step. Follow the naming conventions for similar hosts.

For full details, including information on issuing multiple certificates for a single host, see :git_file:`playbooks/roles/letsencrypt-request-certs/README.rst`.
Define a handler for certificate creation and renewal actions.

When the certificate is created or renewed, the letsencrypt-create-certs role calls a predefined handler so action can be taken. This handler name is constructed by prepending letsencrypt updated to the certificate name above. Thus in this example it would be :
```
- name: letsencrypt updated hostname01-opendev-org
  ...
```
Usually these handlers are defined centrally in :git_file:`playbooks/roles/letsencrypt-create-certs/handlers/main.yaml` and common tasks such as restarting Apache have pre-defined tasks available for easy import.

You may choose to define the handler in another way, but it must exist (Ansible does not have a way to say "call this handler only if it exists", thus a missing handler will cause an Ansible error at runtime).

Debugging

The Ansible run logs on bridge.opendev.org should be consulted if the certificate material is not being created as expected.

Hosts will log their acme.sh output to /var/log/acme.sh/acme.sh.log

The G Suite Toolbox Dig tool can be useful for checking DNS entries from a remote location.

Refreshing keys

In normal operation there should be no need to manually refresh keys on hosts. However there have been situations (such as LetsEncrypt revoking certificates made during a certain period due to bugs) which may necessitate a manual renewal.

The best way to do this is to move the .conf files from /etc/letsencrypt-certs/<certname> on the affected host and allow the next Ansible pulse to renew.

# cd /etc/letsencrypt-certs/<name>
# rename 's/.conf/.conf.old/' *.conf
# tail -f /var/log/acme.sh/acme.sh.log
... watch and should be renewed on next pulse
# rm *.conf.old

Linaro ARM64 Cloud Cert Renewal

The Linaro ARM64 cloud relies on Let's Encrypt certs for API endpoints, but these certs are not automatically provisioned. The reason for this is that cloud is not completely enrolled into our Ansible automation (we share management of this install with Linaro and full integration has not be done). We can manually refresh the SSL certs in this cloud though.

To access the cloud backend ssh via bridge as root to openinfraci.linaro.cloud.

First we provision a new certificate using acme.sh on the cloud node:

/root/acme.sh/acme.sh --server letsencrypt --issue \
  --dns dns_aws -d openinfraci.linaro.cloud

Next backup the old cert:

cp /root/us.linaro.cloud/secret/openinfraci.linaro.cloud.pem \
  /root/us.linaro.cloud/secret/openinfraci.linaro.cloud.pem.$DATE

Copy the new cert into the kolla-ansible secrets:

cat /root/.acme.sh/openinfraci.linaro.cloud/openinfraci.linaro.cloud.key \
  /root/.acme.sh/openinfraci.linaro.cloud/fullchain.cer \
  > /root/us.linaro.cloud/secret/openinfraci.linaro.cloud.pem

Activate the kolla-ansible virtualenv to run ansible:

source /root/venv3/bin/activate

Run kolla-ansible to deploy the cert:

/root/venv3/bin/kolla-ansible -i ~/all-in-one deploy

6.9 KiB Raw Blame History