19 KiB
single: template single: creating specs
Pegleg Secret Generation and Encryption
Pegleg is responsible for shepherding deployment manifest documents from their resting places in Git repositories to a consumable format that is ready for ingestion into Airship. This spec expands its responsibility to account for secure generation and encryption of secrets that are required within an Airship-based deployment.
Links
The work to author and implement this spec will be tracked under this Storyboard Story.
Problem description
Airship supports the ability to identify secret information required for functioning deployments, such as passwords and keys; to ingest it into the site in a least-privilege-oriented fashion; and to encrypt it at rest within Deckhand. However, lifecycle management of the secrets outside the site should be made automatable and repeatable, to facilitate operational needs such as periodic password rotation, and to ensure that unencrypted secrets are only accessible by authorized individuals.
Impacted components
The following Airship components will be impacted by this solution:
- Pegleg: enhanced to generate, rotate, encrypt, and decrypt secrets.
- Promenade: PKICatalog will move to Pegleg.
- Treasuremap: update site manifests to use new Catalogs.
- Airship-in-a-Bottle: update site manifests to use new Catalogs.
Proposed change
PeglegManagedDocument
With this spec, the role of Pegleg grows from being a custodian of
deployment manifests to additionally being the author of certain
manifests. A new YAML schema will be created to describe these
documents: pegleg/PeglegManagedDocument/v1
. Documents of
this type will have one or both of the following data elements, although
more may be added in the future: generated
,
encrypted
. PeglegManagedDocuments serve as wrappers around
other documents, and the wrapping serves to capture additional metadata
that is necessary, but separate from the managed document proper. The
managed document data will live in the data.managedDocument
portion of a PeglegManagedDocument.
If a PeglegManagedDocument is generated
, then its
contents have been created by Pegleg, and it must include provenance
information per this example:
schema: pegleg/PeglegManagedDocument/v1
metadata:
name: matches-document-name
schema: deckhand/Document/v1
labels:
matching: wrapped-doc
layeringDefinition:
abstract: false
# Pegleg will initially support generation at site level only
layer: site
storagePolicy: cleartext
data:
generated:
at: <timestamp>
by: <author>
specifiedBy:
repo: <...>
reference: <git ref-head or similar>
path: <PKICatalog/PassphraseCatalog details>
managedDocument:
schema: <as appropriate for wrapped document>
metadata:
storagePolicy: encrypted
schema: <as appropriate for wrapped document>
<metadata from parent PeglegManagedDocument>
<any other metadata as appropriate>
data: <generated data>
If a PeglegManagedDocument is encrypted
, then its
contents have been encrypted by Pegleg, and it must include provenance
information per this example:
schema: pegleg/PeglegManagedDocument/v1
metadata:
name: matches-document-name
schema: deckhand/Document/v1
labels:
matching: wrapped-doc
layeringDefinition:
abstract: false
layer: matching-wrapped-doc
storagePolicy: cleartext
data:
encrypted:
at: <timestamp>
by: <author>
managedDocument:
schema: <as appropriate for wrapped document>
metadata:
storagePolicy: encrypted
schema: <as appropriate for wrapped document>
<metadata from parent PeglegManagedDocument>
<any other metadata as appropriate>
data: <encrypted string blob>
A PeglegManagedDocument that is both generated via a Catalog, and
encrypted (as specified by the catalog) will contain both
generated
and encrypted
stanzas.
Note that this encrypted
key has a different purpose
than the Deckhand storagePolicy: encrypted
metadata, which
indicates an intent for Deckhand to store a document encrypted
at rest in the cluster. The two can be used together to ensure security,
however: if a document is marked as
storagePolicy: encrypted
, then automation may validate that
it is only persisted (e.g. to a Git repository) if it is in fact
encrypted within a PeglegManagedDocument.
Note also that the Deckhand storagePolicy
of the
PeglegManagedDocument itself is always cleartext
, since its
data stanza is not encrypted -- it only wraps a document that
is storagePolicy: encrypted
. This should be
implemented as a Pegleg lint rule.
Document Generation
Document generation will follow the pattern established by
Promenade's PKICatalog pattern. In fact, PKICatalog management
responsibility will move to Pegleg as part of this effort. The types of
documents that are expected to be generated are certificates and keys,
which are defined via PKICatalog documents now, and passphrases, which
will be defined via a new pegleg/PassphraseCatalog/v1
document. Longer-term, these specifications may be combined, or split
further (into a CertificateCatalog and KeypairCatalog), but this is not
needed in the initial implementation in Pegleg. A collection of
manifests may define more than one of each of these secret catalog
documents if desired.
The documents generated via PKICatalog and PassphraseCatalog will
follow the PeglegManagedDocument schema above; note that this is a
change to existing PKICatalog behavior. The PKICatalog schema and
associated code should be copied to Pegleg (and renamed to
pegleg/PKICatalog/v1
), and during a transition period the
old and new PKICatalog implementations will exist side-by-side with
slightly different semantics. Promenade's PKICatalog can be removed once
all deployment manifests have been updated to use the new one.
Pegleg will place generated document files in
<site>/secrets/passphrases/
,
<site>/secrets/certificates
, or
<site>/secrets/keypairs
as appropriate:
- The generated filenames for passphrases will follow the pattern
<passphrase-doc-name>.yaml
. - The generated filenames for certificate authorities will follow the
pattern
<ca-name>_ca.yaml
. - The generated filenames for certificates will follow the pattern
<ca-name>_<certificate-doc-name>_certificate.yaml
. - The generated filenames for certificate keys will follow the pattern
<ca-name>_<certificate-doc-name>_key.yaml
. - The generated filenames for keypairs will follow the pattern
<keypair-doc-name>.yaml
. - Dashes in the document names will be converted to underscores for consistency.
A PassphraseCatalog will capture the following example structure:
schema: pegleg/PassphraseCatalog/v1
metadata:
schema: metadata/Document/v1
name: cluster-passphrases
layeringDefinition:
abstract: false
layer: site
storagePolicy: cleartext
data:
passphrases:
- document_name: osh-nova-password
description: Service password for Nova
encrypted: true
- document_name: osh-nova-oslo-db-password
description: Database password for Nova
encrypted: true
length: 12
The nonobvious bits of the document described above are:
encrypted
is optional, and denotes whether the generated PeglegManagedDocument will beencrypted
, as well as whether the wrapped document will havestoragePolicy: encrypted
orstoragePolicy: cleartext
metadata. If absent,encrypted
defaults totrue
.document_name
is required, and is used to create the filename of the generated PeglegManagedDocument manifest, and themetadata.name
of the wrappeddeckhand/Passphrase/v1
document. In both cases, Pegleg will replace dashes in thedocument_name
with underscores.length
is optional, and denotes the length in characters of the generated cleartext passphrase data. If absent,length
defaults to24
. Note that with this length and the selected character set there will be less than 8x10^48 probability of getting a new passphrase that is identical to the previous passphrase. This is sufficiently random to ensure no duplication of rotated passphrases in practice.description
is optional.
The encrypted
key will be added to the PKICatalog
schema, and adds the same semantics to PKICatalog-based generation as
are described above for PassphraseCatalog.
Pegleg CLI Changes
The Pegleg CLI interface will be extended as follows. These commands will create PeglegManagedDocument manifests in the local repository. Committing and pushing the changes will be left to the operator or to script-based automation.
For the CLI commands below which encrypt or decrypt secrets, an
environment variable (e.g. PEGLEG_PASSPHRASE
will be use to
capture the master passphrase to use.
pegleg site secrets rotate
will use a second variable (e.g.
PEGLEG_PREVIOUS_PASSPHRASE
) to hold the key/passphrase
being rotated out. The contents of these keys/passphrases are not
generated by Pegleg, but are created externally and set by a deployment
engineer or tooling. A configurable minimum length (default 24) for
master passphrases will be checked by all CLI commands which use the
passphrase. All other criteria around passphrase strength are assumed to
be enforced elsewhere, as it is an external secret that is consumed/used
by Pegleg.
pegleg site secrets generate passphrases
: Generate
passphrases according to all PassphraseCatalog documents in the site.
Note that regenerating passphrases can be accomplished simply by
re-running pegleg site secrets generate passphrases
.
pegleg generate passphrase
: A standalone version of
passphrase generation. This generates a single passphrase based on the
default length, character set, and implementation described above, and
outputs it to the console. The PassphraseCatalog is not involved in this
operation. This command is suitable for generation of a highly-secure
Pegleg master passphrase.
pegleg site secrets generate pki
: Generate certificates
and keys according to all PKICatalog documents in the site. Note that
regenerating certificates can be accomplished simply by re-running
pegleg site secrets generate pki
.
pegleg site secrets generate
: Combines the two commands
above. May be expanded in the future to include other manifest
generation activities.
pegleg site bootstrap
: For now, a synonym for
pegleg site secrets generate
, and may be expanded in the
future to include other bootstrapping activities.
pegleg site secrets encrypt
: Encrypt all site documents
which have metadata.storagePolicy: encrypted
, and which are
not already encrypted within a wrapping PeglegManagedDocument. Note that
the pegleg site secrets generate
commands encrypt generated
secrets as specified, so pegleg site secrets encrypt
is
intended mainly for external-facing secrets which a deployment engineer
brings to the site manifests. The output PeglegManagedDocument will be
written back to the filename that served as its source.
pegleg site secrets decrypt <document YAML file>
:
Decrypt a specific PeglegManagedDocument manifest, unwrapping it and
outputting the cleartext original document YAML to standard output. This
is intended to be used when an authorized deployment engineer needs to
determine a particular cleartext secret for a specific operational
purpose.
pegleg site secrets rotate passphrases
: This action
re-encrypts encrypted passphrases with a new key/passphrase, and it
takes the previously-used key and a new key as input. It accomplishes
its task via two activities:
- For encrypted passphrases that were imported from outside of Pegleg
(i.e. PeglegManagedDocuments which lack the
generated
stanza), decrypt them with the old key (in-memory), re-encrypt them with the new key, and output the results. - Perform a fresh
pegleg site secrets generate passphrases
process using the new key. This will replace allgenerated
passphrases with new secret values for added security. There is an assumption here that the only actors that need to know generated secrets are the services within the Airship-managed cluster, not external services or deployment engineers, except perhaps for point-in-time troubleshooting or operational exercises.
Similar functionality for rotating certificates (which is expected to have a different cadence than passphrase rotation, typically) will be added in the future.
Driving deployment of a site directly via Pegleg is follow-on
functionality which will collect site documents, use them to create the
genesis.sh
script, and then interact directly with Shipyard
to drive deployments. Its details are beyond the scope of this spec, but
when implemented, it should decrypt documents wrapped by applicable
PeglegManagedDocuments at the lst responsible moment, and take care not
to write, log, or stdout them to disk as cleartext.
Note that existing pegleg collect
functionality should
not be changed to decrypt encrypted secrets; this is
because it writes its output to disk. If pegleg collect
is
called, at this point in time, the PeglegManagedDocuments will be
written (encrypted) to disk. To enable special case full site secret
decryption, a --force-decrypt
flag will be added to
pegleg collect
to do this under controlled circumstances,
and to help bridge the gap with existing CICD pipelines until
Pegleg-driven site deployment is in place. It will leverage the
PEGLEG_PASSPHRASE
variable described above.
Secret Generation
The rstr
library should be invoked to generate secrets
of the appropriate length and character set. This library uses the
os.urandom()
function, which in turn leverages
/dev/urandom
on Linux, and it is suitable for cryptographic
purposes.
Characters in generated secrets will be evenly distributed across lower-and upper-case letters, digits, and punctuation in !"#$%&'()*+,-./:;<=>?@[]^_`{|}~. Note this is equivalent to the union of Python string.ascii_letters, string.digits, and string.punctuation.
Secret Encryption
The Python cryptography
library has been chosen to
implement the encryption and decryption of secrets within Pegleg.
cryptography
aims to be the standard cryptographic approach
for Python, and takes pains to make it difficult to do encryption poorly
(via its recipes
layer), while still allowing access to the
algorithmic details when truly needed (via its hazmat
layer). cryptography
is actively maintained and is the
target encryption library for OpenStack as well.
The cryptography.fernet
module will be used for
symmetric encryption. It uses AES with a 128-bit key for encryption, and
HMAC using SHA256 for encryption.
Fernet requires as input a URL-safe, base64-encoded 32-byte
encryption key, which will be derived from the master passphrase passed
into Pegleg via PEGLEG_PASSPHRASE
as described above. The
example for password-based encryption from the Fernet
documentation should be followed as a guide. The salt
to be used in key derivation will be configurable, and will be set to a
fixed value within a built Pegleg container via an environment variable
passed into the Pegleg Dockerfile. This will allow the salt to be
different on an operator-by-operator basis.
The cryptography.exceptions.InvalidSignature
exception
is thrown by cryptography
when an attempt is made to
decrypt a message with a key that is different than the one used to
encrypt a message, i.e., when the user has supplied an incorrect
phassphrase. It should be handled gracefully by Pegleg, resulting in an
informative message back to the user.
Security impact
These changes will result in a system that handles site secrets in a highly secure manner, in the face of multiple roles and day 2 operational needs.
Performance impact
Performance impact to existing flows will be minimal. Pegleg will need to additionally decrypt secrets as part of site deployment, but this will be an efficient operation performed once per deployment.
Alternatives
The Python secrets
library presents a convenient
interface for generating random strings. However, it was introduced in
Python 3.6, and it would be limiting to introduce this constraint on
Airship CICD pipelines.
The strgen
library presents an even more convenient
interface for generating pseudo-random strings; however, it leverages
the Python random
library, which is unsuitably random for
cryptographic purposes.
Deckhand already supports a storagePolicy
element which
indicates whether whether Deckhand will persist document data in an
encrypted state, and this flag could have been re-used by Pegleg to
indicate whether a secret is (or should be) encrypted. However, "should
this data be encrypted" is a fundamentally different question than "is
this data encrypted now", and additional metadata-esque parameters
(generated
, generatedLength
) were desired as
well, so this proposal adds data.encrypted
to indicate the
point-in-time encryption status. storagePolicy
is still
valuable in this context to make sure everything that should be
encrypted is, prior to performing actions with it (e.g. Git
commits).
The PyCrypto
library is a popular solution for
encryption in Python; however, it is no longer actively maintained.
Following the lead of OpenStack and others, we opted instead for the
cryptography
library.
This proposed implementation writes the output of generation/encryption events back to the same source files from which the original data came. This is a destructive operation; however, it wasn't evident that it is problematic in any anticipated workflow. In addition, it sidesteps challenges around naming of generated files, and cleanup of original files.
Implementation
Please refer to the Storyboard Story for implementation planning information.
Dependencies
This work should be based on the patchset to add Git branch and revision support to Pegleg, if it is not merged by the time implementation begins. This patchset alters the CLI interface and Git repository management code, and basing on it will avoid future refactoring.