:title: letsencrypt

.. _letsencrypt:

Let's Encrypt Certificates
##########################

We support provisioning certificates from https://letsencrypt.org for
hosts in the ``opendev.org`` namespace.

At a Glance
===========

:Ansible:
  * :git_file:`playbooks/letsencrypt.yaml`
  * :git_file:`playbooks/roles/letsencrypt-acme-sh-install`
  * :git_file:`playbooks/roles/letsencrypt-request-certs`
  * :git_file:`playbooks/roles/letsencrypt-install-txt-record`
  * :git_file:`playbooks/roles/letsecnrypt-create-certs`
:Resources:
  * https://letsencrypt.org
  * https://github.com/Neilpang/acme.sh
:Chat:
  * #opendev on OFTC

Overview
========

We support automatic provisioning of certificates from Let's Encrypt
to hosts in the ``opendev.org`` domain.

This is implemented in OpenDev via the roles driven from
:git_file:`playbooks/letsencrypt.yaml`.  The overall
actions implemented by the above roles are roughly:

* Hosts that want a certificate use the ``amce.sh`` tool to request it
  from the Let's Encrypt CA.

  Creation or renewal requests receive a TXT record authentication
  value that must be published to prove ownership of the domain.  We
  implement this by making the challenge-request hostname
  ``_acme-challenge.hostname.opendev.org`` a ``CNAME`` record to a
  special "signing domain" ``acme.opendev.org``.

  Note if valid certificates are present and they are not within the
  renewal period (which is most of the time) no further action is
  taken.

* The provided TXT record authentication values are installed and
  published to the ``acme.opendev.org`` domain via the OpenDev
  nameservers.

* The host can now finalise certificate creation.  Let's Encrypt
  checks ``_acme-chellenge.hostname.opendev.org``, which is a
  ``CNAME`` to ``acme.opendev.org``.  Let's Encrypt then enumerates
  the TXT records there, and once finding the required key will return
  the signed keys to the host, which saves them to disk.


Configuring a host to get certificates
======================================

A basic configuration consists of the following steps:

1. Ensure the host is matched by the ``letsencrypt`` group in
   :git_file:`inventory/groups.yaml`.
#. DNS entries for ``_acme-chellenge.hostname`` as a ``CNAME`` to
   ``opendev.org`` must be added and live in the ``opendev.org``
   `zone.db
   <https://opendev.org/opendev/zone-opendev.org/src/branch/master/zones/opendev.org/zone.db>`__
   file.  Follow the other examples to ensure other fields such as
   ``CAA`` records are set too.

   Take care to list `all` hostnames that you wish covered by the
   certificate (e.g. ``hostname01.opendev.org`` and
   ``hostname.opendev.org``)
#. Configure the certificates to be issued to the host.

   The roles look for certificate configuration in a
   ``letsencrypt_certs`` variable defined for each host.  This is
   usually done via specific host variables in
   ``inventory/service/host_vars/<hostname>.opendev.org.yaml``.  For a simple
   host that wants a single certificate to cover its numeric hostname
   and regular ``CNAME`` this would look like ::

     letsencrypt_certs:
       hostname01-opendev-org:
        - hostname01.opendev.org
        - hostname.opendev.org

   This will result in certificate material in
   ``/etc/letsencrypt-certs/hostname01.opendev.org/`` on the host.

   Note that the "certificate name" dictionary keys (just
   ``hostname01-opendev-org`` above) are essentially a free-form
   string, but are used in the next step.  Follow the naming
   conventions for similar hosts.

   For full details, including information on issuing multiple
   certificates for a single host, see
   :git_file:`playbooks/roles/letsencrypt-request-certs/README.rst`.
#. Define a handler for certificate creation and renewal actions.

   When the certificate is created or renewed, the
   ``letsencrypt-create-certs`` role calls a predefined handler so
   action can be taken.  This handler name is constructed by
   prepending ``letsencrypt updated`` to the certificate name above.
   Thus in this example it would be ::

     - name: letsencrypt updated hostname01-opendev-org
       ...

   Usually these handlers are defined centrally in
   :git_file:`playbooks/roles/letsencrypt-create-certs/handlers/main.yaml`
   and common tasks such as restarting Apache have pre-defined tasks
   available for easy import.

   You may choose to define the handler in another way, but it *must*
   exist (Ansible does not have a way to say "call this handler only
   if it exists", thus a missing handler will cause an Ansible error
   at runtime).

Debugging
=========

The Ansible run logs on ``bridge.opendev.org`` should be consulted if
the certificate material is not being created as expected.

Hosts will log their ``acme.sh`` output to
``/var/log/acme.sh/acme.sh.log``

The `G Suite Toolbox Dig <https://toolbox.googleapps.com/apps/dig/>`__
tool can be useful for checking DNS entries from a remote location.

Refreshing keys
===============

In normal operation there should be no need to manually refresh keys
on hosts.  However there have been situations (such as LetsEncrypt
revoking certificates made during a certain period due to bugs) which
may necessitate a manual renewal.

The best way to do this is to move the ``.conf`` files from
``/etc/letsencrypt-certs/<certname>`` on the affected host and allow
the next Ansible pulse to renew.

.. code-block:: console

   # cd /etc/letsencrypt-certs/<name>
   # rename 's/.conf/.conf.old/' *.conf
   # tail -f /var/log/acme.sh/acme.sh.log
   ... watch and should be renewed on next pulse
   # rm *.conf.old