opendev/system-config
Code Issues Proposed changes
4a631dd201
system-config/doc/source/signing.rst
Jeremy Stanley 738f42760a Update artifact signing key management process 
Now that the SKS keyserver network is no more, and there's no
convenient way to share third-party key signatures, we need to
adjust our key management and rollover process accordingly.

Change-Id: I7008706aae06b6e4a16db2dd85a8c7f91530cd50
2021-10-26 19:41:28 +00:00

494 lines
19 KiB
ReStructuredText
Raw Blame History

:title: Signing System
.. _signing:
Signing System
##############
Our standard signing automation model leverages an OpenPGP signing
subkey, encrypted as a Zuul secret, to create detached signatures
for release artifacts (tarballs, wheels, et cetera) and to sign and
push Git tags as part of managed release automation. For OpenStack's
releases, the master key corresponding to this subkey is replaced
near the start of each new development cycle and set to expire soon
after the cycle is scheduled to conclude (with enough overlap to
allow for graceful replacement).
At a Glance
===========
:Secrets:
* `gpg_key
<https://opendev.org/openstack/project-config/src/branch/master/zuul.d/secrets.yaml>`_
:Roles:
* `add-gpgkey
<https://docs.openstack.org/infra/zuul-jobs/general-roles.html#role-add-gpgkey>`_
* `sign-artifacts
<https://docs.openstack.org/infra/zuul-jobs/general-roles.html#role-sign-artifacts>`_
Key Management Overview
=======================
The signing system is implemented as a set of Zuul jobs; these
utilize the signing subkey via an encrypted Zuul secret imported
into a job's ``~/.gnupg/secring.gpg`` at runtime. It's used by jobs
to create detached signatures of release artifacts and to sign Git
tags in release management automation.
Storage
-------
While the signing subkey is installed unencrypted on some job nodes,
so that it can be used unattended by job automation, for OpenStack
the corresponding master key is kept symmetrically encrypted in the
root home directory of the Infra systems management bastion. At the
time of key creation a revocation certificate is also generated, for
which sysadmins are encouraged to retrieve and keep local copies in
case control over or access to the original master key is lost.
Rotation
--------
For OpenStack, the master key is rotated at the start of each
development cycle (usually shortly after cycle-trailing deliverables
are released), signed by the previous key before being put into
service, and has an expiration date set for at least a month after
the end of the targeted development cycle (or best guess, often
longer for safety). New key fingerprints are also submitted to the
openstack/releases repository, for publication on the
releases.openstack.org Web site.
Revocation
----------
Under normal circumstances, keys should be allowed to expire
gracefully. If the key is compromised but still accessible, a
revocation certificate can be generated and published to the key
network at that time. If access to the private key is lost
completely, the revocation certificate generated at key creation
time should be used as a last resort.
Key Management Process
======================
Configuration
-------------
This is the content of the ``/root/signing.gnupg/gpg.conf`` file on
our management bastion host::
# A basic gpg.conf using secure keyserver transport and some more
# verbose display options. This configuration assumes you have
# installed both the gnupg and gnupg-curl packages. Set your umask
# to 077, create a /root/signing.gnupg directory and place this
# configuration file in it.
# Receive, send and search for keys in the SKS keyservers pool using
# HKPS (OpenPGP HTTP Keyserver Protocol via TLS/SSL).
keyserver hkps://keys.openpgp.org
# Display key IDs in a more accurate 16-digit hexidecimal format
# and add 0x at the beginning for clarity.
keyid-format 0xlong
# Display the calculated validity of user IDs when listing keys or
# showing signatures.
list-options show-uid-validity
verify-options show-uid-validity
Generation
----------
Make sure we start with a restrictive umask so that files and
directories we write from this point forward are only accessible by
the root user:
.. code-block:: shell-session
root@bridge:~# umask 077
Now create a master key for the coming development cycle, taking
mostly the GnuPG recommended default values. Set a validity period
sufficient to last through the release process at the conclusion of
the cycle. Use a sufficiently long, randomly-generated passphrase
string (it's fine to reuse the one stored in our passwords list for
earlier keys unless we know it to have been compromised):
.. code-block:: shell-session
root@bridge:~# gpg --homedir signing.gnupg --full-generate-key --expert
gpg (GnuPG) 2.2.4; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
(7) DSA (set your own capabilities)
(8) RSA (set your own capabilities)
(9) ECC and ECC
(10) ECC (sign only)
(11) ECC (set your own capabilities)
(13) Existing key
Your selection? 9
Please select which elliptic curve you want:
(1) Curve 25519
(3) NIST P-256
(4) NIST P-384
(5) NIST P-521
(6) Brainpool P-256
(7) Brainpool P-384
(8) Brainpool P-512
(9) secp256k1
Your selection? 1
Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0) 9m
Key expires at Thu 02 Feb 2017 08:41:39 PM UTC
Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
"Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"
Real name: OpenStack Infra
Email address: infra-root@openstack.org
Comment: Some Cycle
You selected this USER-ID:
"OpenStack Infra (Some Cycle) <infra-root@openstack.org>"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key.
Enter passphrase: ********************************
Repeat passphrase: ********************************
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.+++++
......+++++
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.+++++
+++++
gpg: key 0x120D3C23C6D5584D marked as ultimately trusted
gpg: revocation certificate stored as '/root/signing.gnupg/openpgp-revocs.d/7222E5A05730B7670F93035A120D3C23C6D5584D.rev'
public and secret key created and signed.
pub ed25519/0x120D3C23C6D5584D 2016-07-07 [expires: 2017-02-02]
7222E5A05730B7670F93035A120D3C23C6D5584D
uid OpenStack Infra (Some Cycle) <infra-root@openstack.org>
sub cv25519/0x1F215B56867C5D9A 2016-07-07 [E] [expires: 2017-02-02]
Save the revocation certificate for the master key, for use in the
case extreme case that this master key itself becomes inaccessible,
for example because the decryption passphrase is lost (under any
other circumstances, a revocation certificate with a more detailed
description can be generated using the master key on an as-needed
basis). Replace ``some`` in the output filename with the lower-cased
cycle name:
.. code-block:: shell-session
root@bridge:~# mv \
> signing.gnupg/openpgp-revocs.d/7222E5A05730B7670F93035A120D3C23C6D5584D.rev
> signing.gnupg/some.revoke.asc
Use the interactive key editor to add a subkey constrained to
signing purposes only. It does not need an expiration since it will
be valid only for as long as its associated master key is valid:
.. code-block:: shell-session
root@bridge:~# gpg --homedir signing.gnupg --expert --edit-key 0x120D3C23C6D5584D
gpg (GnuPG) 2.2.4; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Secret key is available.
sec ed25519/0x120D3C23C6D5584D
created: 2016-07-07 expires: 2017-02-02 usage: SC
trust: ultimate validity: ultimate
ssb cv25519/0x1F215B56867C5D9A
created: 2016-07-07 expires: 2017-02-02 usage: E
[ultimate] (1). OpenStack Infra (Some Cycle) <infra-root@openstack.org>
gpg> addkey
Please select what kind of key you want:
(3) DSA (sign only)
(4) RSA (sign only)
(5) Elgamal (encrypt only)
(6) RSA (encrypt only)
(7) DSA (set your own capabilities)
(8) RSA (set your own capabilities)
(10) ECC (sign only)
(11) ECC (set your own capabilities)
(12) ECC (encrypt only)
(13) Existing key
Your selection? 10
Please select which elliptic curve you want:
(1) Curve 25519
(3) NIST P-256
(4) NIST P-384
(5) NIST P-521
(6) Brainpool P-256
(7) Brainpool P-384
(8) Brainpool P-512
(9) secp256k1
Your selection? 1
Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y
Really create? (y/N) y
Key is protected.
You need a passphrase to unlock the secret key for
user: "OpenStack Infra (Some Cycle) <infra-root@openstack.org>"
ID 0x120D3C23C6D5584D, created 2016-07-07
Enter passphrase: ********************************
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
+++++
........+++++
sec ed25519/0x120D3C23C6D5584D
created: 2016-07-07 expires: 2017-02-02 usage: SC
trust: ultimate validity: ultimate
ssb cv25519/0x1F215B56867C5D9A
created: 2016-07-07 expires: 2017-02-02 usage: E
ssb ed25519/0xC0224DB5F541FB68
created: 2016-07-07 expires: never usage: S
[ultimate] (1). OpenStack Infra (Some Cycle) <infra-root@openstack.org>
gpg> save
Next, sign the new master key with the key from the previous cycle
(specified with the ``--default-key`` option). This proves that the
new key was created by a party with access to its predecessor, so
provides some added assurance of its validity:
.. code-block:: shell-session
root@bridge:~# gpg --homedir signing.gnupg --default-key 0x70CA2E45DF30B1B8 --sign-key 0x120D3C23C6D5584D
This key is due to expire on 2017-02-02.
Are you sure that you want to sign this key with your
key "OpenStack Infra (Previous Cycle) <infra-root@openstack.org>" (0x70CA2E45DF30B1B8)
Really sign? (y/N) y
You need a passphrase to unlock the secret key for
user: "OpenStack Infra (Previous Cycle) <infra-root@openstack.org>"
ID 0x70CA2E45DF30B1B8, created 2016-11-03
Enter passphrase: ********************************
Now send the master key to the keyserver network. The subkeys are
all submitted along with it, so do not need to be specified
separately:
.. code-block:: shell-session
root@bridge:~# gpg --homedir signing.gnupg --send-keys 0x120D3C23C6D5584D
gpg: sending key 0x120D3C23C6D5584D to hkps://keys.openpgp.org
Check the infra-root inbox (the address associated with the key) for
a notification about the key upload to keys.openpgp.org and follow
the URL within it. Once there, click the button to send a
verification message. Now check the inbox again and follow the URL
provided in the new message which should arrive. Once that's done,
the key will be searchable on the keyserver.
The rest of this process shouldn't happen until we're ready for the
signing system to transition to our new key. In a typical,
non-emergency rotation this should not happen until release
activities for the previous cycle have concluded so that we don't
inadvertently sign their artifacts with the new key.
Create a new GnuPG keychain by exporting a copy of just the signing
subkey to a file and then importing that (and only that) in a new
GnuPG directory:
.. code-block:: shell-session
root@bridge:~# umask 077
root@bridge:~# mkdir temporary.gnupg
root@bridge:~# gpg --homedir signing.gnupg \
> --output temporary.gnupg/secret-subkeys \
> --export-secret-subkeys 0xC0224DB5F541FB68\!
root@bridge:~# gpg --homedir temporary.gnupg \
> --import temporary.gnupg/secret-subkeys
gpg: keyring `temporary.gnupg/secring.gpg' created
gpg: keyring `temporary.gnupg/pubring.gpg' created
gpg: key C6D5584D: secret key imported
gpg: temporary.gnupg/trustdb.gpg: trustdb created
gpg: key C6D5584D: public key "OpenStack Infra (Some Cycle) <infra-root@openstack.org>" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: secret keys read: 1
gpg: secret keys imported: 1
Check that the exported version does not contain a usable primary
secret key by listing all secret keys and looking for a ``sec#`` in
front of it instead of just ``sec``:
.. code-block:: shell-session
root@bridge:~# gpg --homedir temporary.gnupg --list-secret-keys
/root/temporary.gnupg/pubring.kbx
---------------------------------
sec# ed25519 2016-07-07 [SC] [expires: 2017-02-02]
120D3C23C6D5584D
uid [unknown] OpenStack Infra (Some Cycle) <infra-root@openstack.org>
ssb ed25519 2016-07-07 [S]
So that our CI jobs will be able to make use of this subkey without
interactively supplying a passphrase, the old passphrase (exported
from the master key) must be reset to an empty string in the new
temporary copy. Here we override the default pinentry mode to
loopback as a workaround for other pinentry frontends refusing to
accept an empty passphrase (unfortunately the prompting and feedback
from the loopback pinentry leaves something to be desired). Note
that the first password prompt will be for the original key's
password, but the second password prompt should be left blank thus
removing that password. This is again done using an interactive key
editor session:
.. code-block:: shell-session
root@bridge:~# gpg --homedir temporary.gnupg --pinentry-mode loopback \
> --edit-key 0xC0224DB5F541FB68
gpg (GnuPG) 2.2.4; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Secret subkeys are available.
pub ed25519/0x120D3C23C6D5584D created: 2016-07-07 expires: 2017-02-02 usage: SC
trust: unknown validity: unknown
sub ed25519/0xC0224DB5F541FB68 created: 2016-07-07 expires: never usage: S
[ unknown] (1). OpenStack Infra (Some Cycle) <infra-root@openstack.org>
gpg> passwd
gpg: key 120D3C23C6D5584D/120D3C23C6D5584D: error changing passphrase: No secret key
Enter passphrase: ********************************
Enter passphrase:
gpg> save
Key not changed so no update needed.
Test the subkey can be used without a passphrase::
root@bridge:~# echo foo | gpg --homedir temporary.gnupg --sign --armor
-----BEGIN PGP MESSAGE-----
SR43vh1iK66BbmlsONWxII74fIPEDHDeCqVnkzxdhleDf7DOd9HhYmI8WNOKtTIU
7hcy6cYqHBjEgVr5oViNiveiwGsKlOUhh8x1eYDIxEEoGQEHDJDKq9YOMMjRdsO8
fOw0TD/1r8Lmi8QLkCfGvFdrSY6EoCHqCMx3+JmGUD+iFGp2rCOucw==
=LxND
-----END PGP MESSAGE-----
This leaves us with a temporary keyring containing only an unencrypted
copy of the signing subkey. Export this keyring so that we can add it
as a secret to Zuul for use by release jobs.
.. code-block:: shell-session
root@bridge:~# gpg --homedir temporary.gnupg \
> --output temporary.gnupg/for-zuul --armor \
> --export-secret-subkeys 0xC0224DB5F541FB68\!
root@bridge:~# wget https://opendev.org/zuul/zuul/raw/branch/master/tools/encrypt_secret.py
root@bridge:~# python3 encrypt_secret.py --tenant openstack \
> --infile temporary.gnupg/for-zuul --outfile temporary.gnupg/zuul.yaml \
> https://zuul.opendev.org openstack/project-config
writing RSA key
Public key length: 4096 bits (512 bytes)
Max plaintext length per chunk: 470 bytes
Input plaintext length: 1490 bytes
Number of chunks: 4
Copy ``temporary.gnupg/zuul.yaml`` to your workstation and make a
commit to ``zuul.d/secrets.yaml`` file in the
``openstack/project-config`` repo to update the ``gpg_key`` secret
with its contents. Be sure to replace ``<name>`` and
``<fieldname>`` as appropriate.
Safely clean up, doing your best to securely remove the temporary
copy of the unencrypted signing subkey and any associated files:
.. code-block:: shell-session
root@bridge:~# find temporary.gnupg/ -type f -exec shred {} \;
root@bridge:~# rm -rf temporary.gnupg encrypt_secret.py
To document this transition, export a minimal text version of the
public master key:
.. code-block:: shell-session
root@bridge:~# ( gpg --fingerprint --list-sigs \
> 0x120d3c23c6d5584d6fc2464664dbb05acc5e7c28 ; gpg --armor \
> --export 0x120d3c23c6d5584d6fc2464664dbb05acc5e7c28 ) > \
> 0x120d3c23c6d5584d6fc2464664dbb05acc5e7c28.txt
Add the file to a change for the ``openstack/releases`` repo placing
it in the ``doc/source/static`` directory, and then link it similarly
to other exported public keys are linked in the `Cryptographic
Signatures
<https://releases.openstack.org/#cryptographic-signatures>` section
of ``doc/source/index.rst`` (noting the appropriate end date for use
of the prior key as the start date for the new one).
Attestation
-----------
Since the collapse of the SKS keyserver network, no popular
keyservers publish third-party key signatures any longer (the
pollution thereof was a big part of SKS's downfall). As such, until
something like a caff-style signature approval mechanism is
integrated into the OpenPGP keyserver infrastructure or some
alternative comes along, there's little point in generating our own
individual key signatures we can't easily distribute to users. For
now, we simply make sure the signature made by the previous key is
included in the export we publish from our own sites so that users
can have some path to confirm new keys.
Still, please retrieve a copy of the
``/root/signing.gnupg/some.revoke.asc`` fallback revocation
certificate (``some`` to be replaced with the lower-cased release
name) from the management bastion and keep it stashed somewhere
secure, for emergency use in the (hopefully very unlikely) event
that our OpenPGP master private key is completely lost to us (for
example, if we lose the file containing its decryption passphrase
and all backups thereof).
View Git Blame Copy Permalink