pki: Port Promenade's PKI catalog into Pegleg
This patch set implements the PKICatalog [0] requirements as well as PeglegManagedDocument [1] generation requirements outlined in the spec [2]. Included in this patch set: * New CLI entry point called "pegleg site secrets generate-pki" * PeglegManagedDocument generation logic in engine.cache.managed_document * Refactored PKICatalog logic in engine.cache.pki_catalog derived from the Promenade PKI implementation [3], responsible for generating certificates, CAs, and keypairs * Refactored PKIGenerator logic in engine.cache.pki_generator derived from Promenade Generator implementation [4], responsible for reading in pegleg/PKICatalog/v1 documents (as well as promenade/PKICatalog/v1 documents for backwards compatibility) and generating required secrets and storing them into the paths specified under [0] * Unit tests for all of the above [5] * Example pki-catalog.yaml document under pegleg/site_yamls * Validation schema for pki-catalog.yaml (TODO: implement validation logic here: [6]) * Updates to CLI documentation and inclusion of PKICatalog and PeglegManagedDocument documentation * Documentation updates with PKI information [7] TODO (in follow-up patch sets): * Expand on overview documentation to include new Pegleg responsibilities * Allow the original repository (not the copied one) to be the destination where the secrets are written to * Finish up cert expiry/revocation logic [0] https://airship-specs.readthedocs.io/en/latest/specs/approved/pegleg-secrets.html#document-generation [1] https://airship-specs.readthedocs.io/en/latest/specs/approved/pegleg-secrets.html#peglegmanageddocument [2] https://airship-specs.readthedocs.io/en/latest/specs/approved/pegleg-secrets.html [3] https://github.com/openstack/airship-promenade/blob/master/promenade/pki.py [4] https://github.com/openstack/airship-promenade/blob/master/promenade/generator.py [5] https://review.openstack.org/#/c/611739/ [6] https://review.openstack.org/#/c/608159/ [7] https://review.openstack.org/#/c/611738/ Change-Id: I3010d04cac6d22c656d144f0dafeaa5e19a13068
This commit is contained in:
parent
40da373023
commit
2a8d2638b3
16
.zuul.yaml
16
.zuul.yaml
@ -18,11 +18,13 @@
|
||||
check:
|
||||
jobs:
|
||||
- openstack-tox-pep8
|
||||
- airship-pegleg-tox-py36
|
||||
- airship-pegleg-doc-build
|
||||
- airship-pegleg-docker-build-gate
|
||||
gate:
|
||||
jobs:
|
||||
- openstack-tox-pep8
|
||||
- airship-pegleg-tox-py36
|
||||
- airship-pegleg-doc-build
|
||||
- airship-pegleg-docker-build-gate
|
||||
post:
|
||||
@ -35,6 +37,20 @@
|
||||
- name: primary
|
||||
label: ubuntu-xenial
|
||||
|
||||
- job:
|
||||
name: airship-pegleg-tox-py36
|
||||
description: |
|
||||
Executes unit tests under Python 3.6
|
||||
parent: openstack-tox-py36
|
||||
pre-run:
|
||||
- tools/gate/playbooks/install-cfssl.yaml
|
||||
irrelevant-files:
|
||||
- ^.*\.rst$
|
||||
- ^doc/.*$
|
||||
- ^etc/.*$
|
||||
- ^releasenotes/.*$
|
||||
- ^setup.cfg$
|
||||
|
||||
- job:
|
||||
name: airship-pegleg-doc-build
|
||||
description: |
|
||||
|
@ -81,10 +81,10 @@ CLI Options
|
||||
|
||||
Enable debug logging.
|
||||
|
||||
.. _site:
|
||||
.. _repo-group:
|
||||
|
||||
Repo
|
||||
====
|
||||
Repo Group
|
||||
==========
|
||||
|
||||
Allows you to perform repository-level operations.
|
||||
|
||||
@ -127,8 +127,10 @@ a specific site, see :ref:`site-level linting <cli-site-lint>`.
|
||||
|
||||
See :ref:`linting` for more information.
|
||||
|
||||
Site
|
||||
====
|
||||
.. _site-group:
|
||||
|
||||
Site Group
|
||||
==========
|
||||
|
||||
Allows you to perform site-level operations.
|
||||
|
||||
@ -303,7 +305,7 @@ Show details for one site.
|
||||
|
||||
Name of site.
|
||||
|
||||
**-o /--output** (Optional).
|
||||
**-o/--output** (Optional).
|
||||
|
||||
Where to output.
|
||||
|
||||
@ -331,7 +333,7 @@ Render documents via `Deckhand`_ for one site.
|
||||
|
||||
Name of site.
|
||||
|
||||
**-o /--output** (Optional).
|
||||
**-o/--output** (Optional).
|
||||
|
||||
Where to output.
|
||||
|
||||
@ -418,6 +420,39 @@ Usage:
|
||||
|
||||
./pegleg.sh site <options> upload <site_name> --context-marker=<uuid>
|
||||
|
||||
Site Secrets Group
|
||||
==================
|
||||
|
||||
Subgroup of :ref:`site-group`.
|
||||
|
||||
Generate PKI
|
||||
------------
|
||||
|
||||
Generate certificates and keys according to all PKICatalog documents in the
|
||||
site using the PKI module. Regenerating certificates can be
|
||||
accomplished by re-running this command.
|
||||
|
||||
Pegleg places generated document files in ``<site>/secrets/passphrases``,
|
||||
``<site>/secrets/certificates``, or ``<site>/secrets/keypairs`` as
|
||||
appropriate:
|
||||
|
||||
* The generated filenames for passphrases will follow the pattern
|
||||
:file:`<passphrase-doc-name>.yaml`.
|
||||
* The generated filenames for certificate authorities will follow the pattern
|
||||
:file:`<ca-name>_ca.yaml`.
|
||||
* The generated filenames for certificates will follow the pattern
|
||||
:file:`<ca-name>_<certificate-doc-name>_certificate.yaml`.
|
||||
* The generated filenames for certificate keys will follow the pattern
|
||||
:file:`<ca-name>_<certificate-doc-name>_key.yaml`.
|
||||
* The generated filenames for keypairs will follow the pattern
|
||||
:file:`<keypair-doc-name>.yaml`.
|
||||
|
||||
Dashes in the document names will be converted to underscores for consistency.
|
||||
|
||||
**site_name** (Required).
|
||||
|
||||
Name of site.
|
||||
|
||||
Examples
|
||||
^^^^^^^^
|
||||
|
||||
@ -427,6 +462,14 @@ Examples
|
||||
upload <site_name> <options>
|
||||
|
||||
|
||||
::
|
||||
|
||||
./pegleg.sh site -r <site_repo> -e <extra_repo> \
|
||||
secrets generate-pki \
|
||||
<site_name> \
|
||||
-o <output> \
|
||||
-f <filename>
|
||||
|
||||
.. _command-line-repository-overrides:
|
||||
|
||||
Secrets
|
||||
@ -571,13 +614,13 @@ Example:
|
||||
|
||||
|
||||
CLI Repository Overrides
|
||||
------------------------
|
||||
========================
|
||||
|
||||
Repository overrides should only be used for entries included underneath
|
||||
the ``repositories`` field for a given :file:`site-definition.yaml`.
|
||||
|
||||
Overrides are specified via the ``-e`` flag for all :ref:`site` commands. They
|
||||
have the following format:
|
||||
Overrides are specified via the ``-e`` flag for all :ref:`site-group` commands.
|
||||
They have the following format:
|
||||
|
||||
::
|
||||
|
||||
@ -611,7 +654,7 @@ Where:
|
||||
.. _self-contained-repo:
|
||||
|
||||
Self-Contained Repository
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
-------------------------
|
||||
|
||||
For self-contained repositories, specification of extra repositories is
|
||||
unnecessary. The following command can be used to deploy the manifests in
|
||||
|
@ -100,8 +100,8 @@ directory):
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# Quick way of building a venv and installing all required dependencies into
|
||||
# it.
|
||||
# Quick way of building a virtualenv and installing all required
|
||||
# dependencies into it.
|
||||
tox -e py36 --notest
|
||||
source .tox/py36/bin/activate
|
||||
pip install -e .
|
||||
@ -128,11 +128,11 @@ Unit Tests
|
||||
|
||||
To run all unit tests, execute::
|
||||
|
||||
$ tox -epy36
|
||||
$ tox -e py36
|
||||
|
||||
To run unit tests using a regex, execute::
|
||||
|
||||
$ tox -epy36 -- <regex>
|
||||
$ tox -e py36 -- <regex>
|
||||
|
||||
.. _Airship: https://airshipit.readthedocs.io
|
||||
.. _Deckhand: https://airship-deckhand.readthedocs.io/
|
||||
|
@ -63,3 +63,11 @@ Authentication Exceptions
|
||||
.. autoexception:: pegleg.engine.util.shipyard_helper.AuthValuesError
|
||||
:members:
|
||||
:undoc-members:
|
||||
|
||||
PKI Exceptions
|
||||
--------------
|
||||
|
||||
.. autoexception:: pegleg.engine.exceptions.IncompletePKIPairError
|
||||
:members:
|
||||
:show-inheritance:
|
||||
:undoc-members:
|
||||
|
@ -21,13 +21,14 @@ Getting Started
|
||||
What is Pegleg?
|
||||
---------------
|
||||
|
||||
Pegleg is a document aggregator that will aggregate all the documents in a
|
||||
repository and pack them into a single YAML file. This allows for operators to
|
||||
Pegleg is a document aggregator that aggregates all the documents in a
|
||||
repository and packs them into a single YAML file. This allows for operators to
|
||||
structure their site definitions in a maintainable directory layout, while
|
||||
providing them with the automation and tooling needed to aggregate, lint, and
|
||||
render those documents for deployment.
|
||||
|
||||
For more information on the documents that Pegleg works on see `Document Fundamentals`_.
|
||||
For more information on the documents that Pegleg works on see
|
||||
`Document Fundamentals`_.
|
||||
|
||||
Basic Usage
|
||||
-----------
|
||||
|
Binary file not shown.
Before Width: | Height: | Size: 37 KiB After Width: | Height: | Size: 37 KiB |
@ -1,5 +1,6 @@
|
||||
ARG FROM=python:3.6
|
||||
FROM ${FROM}
|
||||
ARG CFSSLURL=https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
|
||||
|
||||
LABEL org.opencontainers.image.authors='airship-discuss@lists.airshipit.org, irc://#airshipit@freenode'
|
||||
LABEL org.opencontainers.image.url='https://airshipit.org'
|
||||
@ -14,5 +15,8 @@ WORKDIR /var/pegleg
|
||||
COPY requirements.txt /opt/pegleg/requirements.txt
|
||||
RUN pip3 install --no-cache-dir -r /opt/pegleg/requirements.txt
|
||||
|
||||
COPY tools/install-cfssl.sh /opt/pegleg/tools/install-cfssl.sh
|
||||
RUN /opt/pegleg/tools/install-cfssl.sh ${CFSSLURL}
|
||||
|
||||
COPY . /opt/pegleg
|
||||
RUN pip3 install -e /opt/pegleg
|
||||
|
@ -20,6 +20,7 @@ import click
|
||||
|
||||
from pegleg import config
|
||||
from pegleg import engine
|
||||
from pegleg.engine import catalog
|
||||
from pegleg.engine.util.shipyard_helper import ShipyardHelper
|
||||
|
||||
LOG = logging.getLogger(__name__)
|
||||
@ -130,7 +131,6 @@ def main(*, verbose):
|
||||
|
||||
* site: site-level actions
|
||||
* repo: repository-level actions
|
||||
* stub (DEPRECATED)
|
||||
|
||||
"""
|
||||
|
||||
@ -208,7 +208,7 @@ def site(*, site_repository, clone_path, extra_repositories, repo_key,
|
||||
* list: list available sites in a manifests repo
|
||||
* lint: lint a site along with all its dependencies
|
||||
* render: render a site using Deckhand
|
||||
* show: show a sites' files
|
||||
* show: show a site's files
|
||||
|
||||
"""
|
||||
|
||||
@ -375,6 +375,39 @@ def upload(ctx, *, os_project_domain_name,
|
||||
click.echo(ShipyardHelper(ctx).upload_documents())
|
||||
|
||||
|
||||
@site.group(name='secrets', help='Commands to manage site secrets documents')
|
||||
def secrets():
|
||||
pass
|
||||
|
||||
|
||||
@secrets.command(
|
||||
'generate-pki',
|
||||
help="""
|
||||
Generate certificates and keys according to all PKICatalog documents in the
|
||||
site. Regenerating certificates can be accomplished by re-running this command.
|
||||
""")
|
||||
@click.option(
|
||||
'-a',
|
||||
'--author',
|
||||
'author',
|
||||
help="""Identifying name of the author generating new certificates. Used
|
||||
for tracking provenance information in the PeglegManagedDocuments. An attempt
|
||||
is made to automatically determine this value, but should be provided.""")
|
||||
@click.argument('site_name')
|
||||
def generate_pki(site_name, author):
|
||||
"""Generate certificates, certificate authorities and keypairs for a given
|
||||
site.
|
||||
|
||||
"""
|
||||
|
||||
engine.repository.process_repositories(site_name,
|
||||
overwrite_existing=True)
|
||||
pkigenerator = catalog.pki_generator.PKIGenerator(site_name, author=author)
|
||||
output_paths = pkigenerator.generate()
|
||||
|
||||
click.echo("Generated PKI files written to:\n%s" % '\n'.join(output_paths))
|
||||
|
||||
|
||||
@main.group(help='Commands related to types')
|
||||
@MAIN_REPOSITORY_OPTION
|
||||
@REPOSITORY_CLONE_PATH_OPTION
|
||||
@ -409,11 +442,6 @@ def list_types(*, output_stream):
|
||||
engine.type.list_types(output_stream)
|
||||
|
||||
|
||||
@site.group(name='secrets', help='Commands to manage site secrets documents')
|
||||
def secrets():
|
||||
pass
|
||||
|
||||
|
||||
@secrets.command(
|
||||
'encrypt',
|
||||
help='Command to encrypt and wrap site secrets '
|
||||
@ -437,7 +465,9 @@ def secrets():
|
||||
'documents')
|
||||
@click.argument('site_name')
|
||||
def encrypt(*, save_location, author, site_name):
|
||||
engine.repository.process_repositories(site_name)
|
||||
engine.repository.process_repositories(site_name, overwrite_existing=True)
|
||||
if save_location is None:
|
||||
save_location = config.get_site_repo()
|
||||
engine.secrets.encrypt(save_location, author, site_name)
|
||||
|
||||
|
||||
@ -453,4 +483,9 @@ def encrypt(*, save_location, author, site_name):
|
||||
@click.argument('site_name')
|
||||
def decrypt(*, file_name, site_name):
|
||||
engine.repository.process_repositories(site_name)
|
||||
engine.secrets.decrypt(file_name, site_name)
|
||||
try:
|
||||
click.echo(engine.secrets.decrypt(file_name, site_name))
|
||||
except FileNotFoundError:
|
||||
raise click.exceptions.FileError("Couldn't find file %s, "
|
||||
"check your arguments and try "
|
||||
"again." % file_name)
|
||||
|
@ -25,7 +25,8 @@ except NameError:
|
||||
'extra_repos': [],
|
||||
'clone_path': None,
|
||||
'site_path': 'site',
|
||||
'type_path': 'type'
|
||||
'site_rev': None,
|
||||
'type_path': 'type',
|
||||
}
|
||||
|
||||
|
||||
@ -49,6 +50,16 @@ def set_clone_path(p):
|
||||
GLOBAL_CONTEXT['clone_path'] = p
|
||||
|
||||
|
||||
def get_site_rev():
|
||||
"""Get site revision derived from the site repo URL/path, if provided."""
|
||||
return GLOBAL_CONTEXT['site_rev']
|
||||
|
||||
|
||||
def set_site_rev(r):
|
||||
"""Set site revision derived from the site repo URL/path."""
|
||||
GLOBAL_CONTEXT['site_rev'] = r
|
||||
|
||||
|
||||
def get_extra_repo_overrides():
|
||||
"""Get extra repository overrides specified via ``-e`` CLI flag."""
|
||||
return GLOBAL_CONTEXT.get('extra_repo_overrides', [])
|
||||
|
17
pegleg/engine/catalog/__init__.py
Normal file
17
pegleg/engine/catalog/__init__.py
Normal file
@ -0,0 +1,17 @@
|
||||
# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# flake8: noqa
|
||||
from pegleg.engine.catalog import pki_utility
|
||||
from pegleg.engine.catalog import pki_generator
|
307
pegleg/engine/catalog/pki_generator.py
Normal file
307
pegleg/engine/catalog/pki_generator.py
Normal file
@ -0,0 +1,307 @@
|
||||
# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import collections
|
||||
import itertools
|
||||
import logging
|
||||
import os
|
||||
|
||||
import yaml
|
||||
|
||||
from pegleg import config
|
||||
from pegleg.engine.catalog import pki_utility
|
||||
from pegleg.engine.common import managed_document as md
|
||||
from pegleg.engine import exceptions
|
||||
from pegleg.engine import util
|
||||
from pegleg.engine.util.pegleg_managed_document import \
|
||||
PeglegManagedSecretsDocument
|
||||
|
||||
__all__ = ['PKIGenerator']
|
||||
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PKIGenerator(object):
|
||||
"""Generates certificates, certificate authorities and keypairs using
|
||||
the ``PKIUtility`` class.
|
||||
|
||||
Pegleg searches through a given "site" to derive all the documents
|
||||
of kind ``PKICatalog``, which are in turn parsed for information related
|
||||
to the above secret types and passed to ``PKIUtility`` for generation.
|
||||
|
||||
These secrets are output to various subdirectories underneath
|
||||
``<site>/secrets/<subpath>``.
|
||||
|
||||
"""
|
||||
|
||||
def __init__(self, sitename, block_strings=True, author=None):
|
||||
"""Constructor for ``PKIGenerator``.
|
||||
|
||||
:param str sitename: Site name for which to retrieve documents used for
|
||||
certificate and keypair generation.
|
||||
:param bool block_strings: Whether to dump out certificate data as
|
||||
block-style YAML string. Defaults to true.
|
||||
:param str author: Identifying name of the author generating new
|
||||
certificates.
|
||||
|
||||
"""
|
||||
|
||||
self._sitename = sitename
|
||||
self._documents = util.definition.documents_for_site(sitename)
|
||||
self._author = author
|
||||
|
||||
self.keys = pki_utility.PKIUtility(block_strings=block_strings)
|
||||
self.outputs = collections.defaultdict(dict)
|
||||
|
||||
# Maps certificates to CAs in order to derive certificate paths.
|
||||
self._cert_to_ca_map = {}
|
||||
|
||||
def generate(self):
|
||||
for catalog in util.catalog.iterate(
|
||||
documents=self._documents, kind='PKICatalog'):
|
||||
for ca_name, ca_def in catalog['data'].get(
|
||||
'certificate_authorities', {}).items():
|
||||
ca_cert, ca_key = self.get_or_gen_ca(ca_name)
|
||||
|
||||
for cert_def in ca_def.get('certificates', []):
|
||||
document_name = cert_def['document_name']
|
||||
self._cert_to_ca_map.setdefault(document_name, ca_name)
|
||||
cert, key = self.get_or_gen_cert(
|
||||
document_name,
|
||||
ca_cert=ca_cert,
|
||||
ca_key=ca_key,
|
||||
cn=cert_def['common_name'],
|
||||
hosts=_extract_hosts(cert_def),
|
||||
groups=cert_def.get('groups', []))
|
||||
|
||||
for keypair_def in catalog['data'].get('keypairs', []):
|
||||
document_name = keypair_def['name']
|
||||
self.get_or_gen_keypair(document_name)
|
||||
|
||||
return self._write(config.get_site_repo())
|
||||
|
||||
def get_or_gen_ca(self, document_name):
|
||||
kinds = [
|
||||
'CertificateAuthority',
|
||||
'CertificateAuthorityKey',
|
||||
]
|
||||
return self._get_or_gen(self.gen_ca, kinds, document_name)
|
||||
|
||||
def get_or_gen_cert(self, document_name, **kwargs):
|
||||
kinds = [
|
||||
'Certificate',
|
||||
'CertificateKey',
|
||||
]
|
||||
return self._get_or_gen(self.gen_cert, kinds, document_name, **kwargs)
|
||||
|
||||
def get_or_gen_keypair(self, document_name):
|
||||
kinds = [
|
||||
'PublicKey',
|
||||
'PrivateKey',
|
||||
]
|
||||
return self._get_or_gen(self.gen_keypair, kinds, document_name)
|
||||
|
||||
def gen_ca(self, document_name, **kwargs):
|
||||
return self.keys.generate_ca(document_name, **kwargs)
|
||||
|
||||
def gen_cert(self, document_name, *, ca_cert, ca_key, **kwargs):
|
||||
ca_cert_data = ca_cert['data']['managedDocument']['data']
|
||||
ca_key_data = ca_key['data']['managedDocument']['data']
|
||||
return self.keys.generate_certificate(
|
||||
document_name, ca_cert=ca_cert_data, ca_key=ca_key_data, **kwargs)
|
||||
|
||||
def gen_keypair(self, document_name):
|
||||
return self.keys.generate_keypair(document_name)
|
||||
|
||||
def _get_or_gen(self, generator, kinds, document_name, *args, **kwargs):
|
||||
docs = self._find_docs(kinds, document_name)
|
||||
if not docs:
|
||||
docs = generator(document_name, *args, **kwargs)
|
||||
else:
|
||||
docs = [PeglegManagedSecretsDocument(doc).pegleg_document
|
||||
for doc in docs]
|
||||
|
||||
# Adding these to output should be idempotent, so we use a dict.
|
||||
|
||||
for wrapper_doc in docs:
|
||||
wrapped_doc = wrapper_doc['data']['managedDocument']
|
||||
schema = wrapped_doc['schema']
|
||||
name = wrapped_doc['metadata']['name']
|
||||
self.outputs[schema][name] = wrapper_doc
|
||||
|
||||
return docs
|
||||
|
||||
def _find_docs(self, kinds, document_name):
|
||||
schemas = ['deckhand/%s/v1' % k for k in kinds]
|
||||
docs = self._find_among_collected(schemas, document_name)
|
||||
if docs:
|
||||
if len(docs) == len(kinds):
|
||||
LOG.debug('Found docs in input config named %s, kinds: %s',
|
||||
document_name, kinds)
|
||||
return docs
|
||||
else:
|
||||
raise exceptions.IncompletePKIPairError(
|
||||
kinds=kinds, name=document_name)
|
||||
|
||||
else:
|
||||
docs = self._find_among_outputs(schemas, document_name)
|
||||
if docs:
|
||||
LOG.debug('Found docs in current outputs named %s, kinds: %s',
|
||||
document_name, kinds)
|
||||
return docs
|
||||
# TODO(felipemonteiro): Should this be a critical error?
|
||||
LOG.debug('No docs existing docs named %s, kinds: %s', document_name,
|
||||
kinds)
|
||||
return []
|
||||
|
||||
def _find_among_collected(self, schemas, document_name):
|
||||
result = []
|
||||
for schema in schemas:
|
||||
doc = _find_document_by(
|
||||
self._documents, schema=schema, name=document_name)
|
||||
# If the document wasn't found, then means it needs to be
|
||||
# generated.
|
||||
if doc:
|
||||
result.append(doc)
|
||||
return result
|
||||
|
||||
def _find_among_outputs(self, schemas, document_name):
|
||||
result = []
|
||||
for schema in schemas:
|
||||
if document_name in self.outputs.get(schema, {}):
|
||||
result.append(self.outputs[schema][document_name])
|
||||
return result
|
||||
|
||||
def _write(self, output_dir):
|
||||
documents = self.get_documents()
|
||||
output_paths = set()
|
||||
|
||||
# First, delete each of the output paths below because we do an append
|
||||
# action in the `open` call below. This means that for regeneration
|
||||
# of certs, the original paths must be deleted.
|
||||
for document in documents:
|
||||
output_file_path = md.get_document_path(
|
||||
sitename=self._sitename,
|
||||
wrapper_document=document,
|
||||
cert_to_ca_map=self._cert_to_ca_map)
|
||||
output_path = os.path.join(output_dir, 'site', output_file_path)
|
||||
# NOTE(felipemonteiro): This is currently an entirely safe
|
||||
# operation as these files are being removed in the temporarily
|
||||
# replicated versions of the local repositories.
|
||||
if os.path.exists(output_path):
|
||||
os.remove(output_path)
|
||||
|
||||
# Next, generate (or regenerate) the certificates.
|
||||
for document in documents:
|
||||
output_file_path = md.get_document_path(
|
||||
sitename=self._sitename,
|
||||
wrapper_document=document,
|
||||
cert_to_ca_map=self._cert_to_ca_map)
|
||||
output_path = os.path.join(output_dir, 'site', output_file_path)
|
||||
dir_name = os.path.dirname(output_path)
|
||||
|
||||
if not os.path.exists(dir_name):
|
||||
LOG.debug('Creating secrets path: %s', dir_name)
|
||||
os.makedirs(dir_name)
|
||||
|
||||
with open(output_path, 'a') as f:
|
||||
# Don't use safe_dump so we can block format certificate
|
||||
# data.
|
||||
yaml.dump(
|
||||
document,
|
||||
stream=f,
|
||||
default_flow_style=False,
|
||||
explicit_start=True,
|
||||
indent=2)
|
||||
|
||||
output_paths.add(output_path)
|
||||
return output_paths
|
||||
|
||||
def get_documents(self):
|
||||
return list(
|
||||
itertools.chain.from_iterable(
|
||||
v.values() for v in self.outputs.values()))
|
||||
|
||||
|
||||
def get_host_list(service_names):
|
||||
service_list = []
|
||||
for service in service_names:
|
||||
parts = service.split('.')
|
||||
for i in range(len(parts)):
|
||||
service_list.append('.'.join(parts[:i + 1]))
|
||||
return service_list
|
||||
|
||||
|
||||
def _extract_hosts(cert_def):
|
||||
hosts = cert_def.get('hosts', [])
|
||||
hosts.extend(get_host_list(cert_def.get('kubernetes_service_names', [])))
|
||||
return hosts
|
||||
|
||||
|
||||
def _find_document_by(documents, **kwargs):
|
||||
try:
|
||||
return next(_iterate(documents, **kwargs))
|
||||
except StopIteration:
|
||||
return None
|
||||
|
||||
|
||||
def _iterate(documents, *, kind=None, schema=None, labels=None, name=None):
|
||||
if kind is not None:
|
||||
if schema is not None:
|
||||
raise AssertionError('Logic error: specified both kind and schema')
|
||||
schema = 'promenade/%s/v1' % kind
|
||||
|
||||
for document in documents:
|
||||
if _matches_filter(document, schema=schema, labels=labels, name=name):
|
||||
yield document
|
||||
|
||||
|
||||
def _matches_filter(document, *, schema, labels, name):
|
||||
matches = True
|
||||
|
||||
if md.is_managed_document(document):
|
||||
document = document['data']['managedDocument']
|
||||
else:
|
||||
document_schema = document['schema']
|
||||
if document_schema in md.SUPPORTED_SCHEMAS:
|
||||
# Can't use the filter value as they might not be an exact match.
|
||||
document_metadata = document['metadata']
|
||||
document_labels = document_metadata.get('labels', {})
|
||||
document_name = document_metadata['name']
|
||||
LOG.warning('Detected deprecated unmanaged document during PKI '
|
||||
'generation. Details: schema=%s, name=%s, labels=%s.',
|
||||
document_schema, document_labels, document_name)
|
||||
|
||||
if schema is not None and not document.get('schema',
|
||||
'').startswith(schema):
|
||||
matches = False
|
||||
|
||||
if labels is not None:
|
||||
document_labels = _mg(document, 'labels', [])
|
||||
for key, value in labels.items():
|
||||
if key not in document_labels:
|
||||
matches = False
|
||||
else:
|
||||
if document_labels[key] != value:
|
||||
matches = False
|
||||
|
||||
if name is not None:
|
||||
if _mg(document, 'name') != name:
|
||||
matches = False
|
||||
|
||||
return matches
|
||||
|
||||
|
||||
def _mg(document, field, default=None):
|
||||
return document.get('metadata', {}).get(field, default)
|
330
pegleg/engine/catalog/pki_utility.py
Normal file
330
pegleg/engine/catalog/pki_utility.py
Normal file
@ -0,0 +1,330 @@
|
||||
# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
from datetime import datetime
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
# Ignore bandit false positive: B404:blacklist
|
||||
# The purpose of this module is to safely encapsulate calls via fork.
|
||||
import subprocess # nosec
|
||||
import tempfile
|
||||
|
||||
from dateutil import parser
|
||||
import pytz
|
||||
import yaml
|
||||
|
||||
from pegleg.engine.util.pegleg_managed_document import \
|
||||
PeglegManagedSecretsDocument
|
||||
|
||||
LOG = logging.getLogger(__name__)
|
||||
_ONE_YEAR_IN_HOURS = '8760h' # 365 * 24
|
||||
|
||||
__all__ = ['PKIUtility']
|
||||
|
||||
|
||||
# TODO(felipemonteiro): Create an abstract base class for other future Catalog
|
||||
# classes.
|
||||
|
||||
|
||||
class PKIUtility(object):
|
||||
"""Public Key Infrastructure utility class.
|
||||
|
||||
Responsible for generating certificate and CA documents using ``cfssl`` and
|
||||
keypairs using ``openssl``. These secrets are all wrapped in instances
|
||||
of ``pegleg/PeglegManagedDocument/v1``.
|
||||
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def cfssl_exists():
|
||||
"""Checks whether cfssl command exists. Useful for testing."""
|
||||
try:
|
||||
subprocess.check_output( # nosec
|
||||
['which', 'cfssl'], stderr=subprocess.STDOUT)
|
||||
return True
|
||||
except subprocess.CalledProcessError:
|
||||
return False
|
||||
|
||||
def __init__(self, *, block_strings=True):
|
||||
self.block_strings = block_strings
|
||||
self._ca_config_string = None
|
||||
|
||||
@property
|
||||
def ca_config(self):
|
||||
if not self._ca_config_string:
|
||||
self._ca_config_string = json.dumps({
|
||||
'signing': {
|
||||
'default': {
|
||||
# TODO(felipemonteiro): Make this configurable.
|
||||
'expiry':
|
||||
_ONE_YEAR_IN_HOURS,
|
||||
'usages': [
|
||||
'signing', 'key encipherment', 'server auth',
|
||||
'client auth'
|
||||
],
|
||||
},
|
||||
},
|
||||
})
|
||||
return self._ca_config_string
|
||||
|
||||
def generate_ca(self, ca_name):
|
||||
"""Generate CA cert and associated key.
|
||||
|
||||
:param str ca_name: Name of Certificate Authority in wrapped document.
|
||||
:returns: Tuple of (wrapped CA cert, wrapped CA key)
|
||||
:rtype: tuple[dict, dict]
|
||||
|
||||
"""
|
||||
|
||||
result = self._cfssl(
|
||||
['gencert', '-initca', 'csr.json'],
|
||||
files={
|
||||
'csr.json': self.csr(name=ca_name),
|
||||
})
|
||||
|
||||
return (self._wrap_ca(ca_name, result['cert']),
|
||||
self._wrap_ca_key(ca_name, result['key']))
|
||||
|
||||
def generate_keypair(self, name):
|
||||
"""Generate keypair.
|
||||
|
||||
:param str name: Name of keypair in wrapped document.
|
||||
:returns: Tuple of (wrapped public key, wrapped private key)
|
||||
:rtype: tuple[dict, dict]
|
||||
|
||||
"""
|
||||
|
||||
priv_result = self._openssl(['genrsa', '-out', 'priv.pem'])
|
||||
pub_result = self._openssl(
|
||||
['rsa', '-in', 'priv.pem', '-pubout', '-out', 'pub.pem'],
|
||||
files={
|
||||
'priv.pem': priv_result['priv.pem'],
|
||||
})
|
||||
|
||||
return (self._wrap_pub_key(name, pub_result['pub.pem']),
|
||||
self._wrap_priv_key(name, priv_result['priv.pem']))
|
||||
|
||||
def generate_certificate(self,
|
||||
name,
|
||||
*,
|
||||
ca_cert,
|
||||
ca_key,
|
||||
cn,
|
||||
groups=None,
|
||||
hosts=None):
|
||||
"""Generate certificate and associated key given CA cert and key.
|
||||
|
||||
:param str name: Name of certificate in wrapped document.
|
||||
:param str ca_cert: CA certificate.
|
||||
:param str ca_key: CA certificate key.
|
||||
:param str cn: Common name associated with certificate.
|
||||
:param list groups: List of groups associated with certificate.
|
||||
:param list hosts: List of hosts associated with certificate.
|
||||
:returns: Tuple of (wrapped certificate, wrapped certificate key)
|
||||
:rtype: tuple[dict, dict]
|
||||
|
||||
"""
|
||||
|
||||
if groups is None:
|
||||
groups = []
|
||||
if hosts is None:
|
||||
hosts = []
|
||||
|
||||
result = self._cfssl(
|
||||
[
|
||||
'gencert', '-ca', 'ca.pem', '-ca-key', 'ca-key.pem', '-config',
|
||||
'ca-config.json', 'csr.json'
|
||||
],
|
||||
files={
|
||||
'ca-config.json': self.ca_config,
|
||||
'ca.pem': ca_cert,
|
||||
'ca-key.pem': ca_key,
|
||||
'csr.json': self.csr(name=cn, groups=groups, hosts=hosts),
|
||||
})
|
||||
|
||||
return (self._wrap_cert(name, result['cert']),
|
||||
self._wrap_cert_key(name, result['key']))
|
||||
|
||||
def csr(self,
|
||||
*,
|
||||
name,
|
||||
groups=None,
|
||||
hosts=None,
|
||||
key={
|
||||
'algo': 'rsa',
|
||||
'size': 2048
|
||||
}):
|
||||
if groups is None:
|
||||
groups = []
|
||||
if hosts is None:
|
||||
hosts = []
|
||||
|
||||
return json.dumps({
|
||||
'CN': name,
|
||||
'key': key,
|
||||
'hosts': hosts,
|
||||
'names': [{
|
||||
'O': g
|
||||
} for g in groups],
|
||||
})
|
||||
|
||||
def cert_info(self, cert):
|
||||
"""Retrieve certificate info via ``cfssl``.
|
||||
|
||||
:param str cert: Client certificate that contains the public key.
|
||||
:returns: Information related to certificate.
|
||||
:rtype: dict
|
||||
|
||||
"""
|
||||
|
||||
return self._cfssl(
|
||||
['certinfo', '-cert', 'cert.pem'], files={
|
||||
'cert.pem': cert,
|
||||
})
|
||||
|
||||
def check_expiry(self, cert):
|
||||
"""Chek whether a given certificate is expired.
|
||||
|
||||
:param str cert: Client certificate that contains the public key.
|
||||
:returns: True if certificate is expired, else False.
|
||||
:rtype: bool
|
||||
|
||||
"""
|
||||
|
||||
info = self.cert_info(cert)
|
||||
expiry_str = info['not_after']
|
||||
expiry = parser.parse(expiry_str)
|
||||
# expiry is timezone-aware; do the same for `now`.
|
||||
now = pytz.utc.localize(datetime.utcnow())
|
||||
return now > expiry
|
||||
|
||||
def _cfssl(self, command, *, files=None):
|
||||
"""Executes ``cfssl`` command via ``subprocess`` call."""
|
||||
if not files:
|
||||
files = {}
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
for filename, data in files.items():
|
||||
with open(os.path.join(tmp, filename), 'w') as f:
|
||||
f.write(data)
|
||||
|
||||
# Ignore bandit false positive:
|
||||
# B603:subprocess_without_shell_equals_true
|
||||
# This method wraps cfssl calls originating from this module.
|
||||
result = subprocess.check_output( # nosec
|
||||
['cfssl'] + command, cwd=tmp, stderr=subprocess.PIPE)
|
||||
if not isinstance(result, str):
|
||||
result = result.decode('utf-8')
|
||||
return json.loads(result)
|
||||
|
||||
def _openssl(self, command, *, files=None):
|
||||
"""Executes ``openssl`` command via ``subprocess`` call."""
|
||||
if not files:
|
||||
files = {}
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
for filename, data in files.items():
|
||||
with open(os.path.join(tmp, filename), 'w') as f:
|
||||
f.write(data)
|
||||
|
||||
# Ignore bandit false positive:
|
||||
# B603:subprocess_without_shell_equals_true
|
||||
# This method wraps openssl calls originating from this module.
|
||||
subprocess.check_call( # nosec
|
||||
['openssl'] + command,
|
||||
cwd=tmp,
|
||||
stderr=subprocess.PIPE)
|
||||
|
||||
result = {}
|
||||
for filename in os.listdir(tmp):
|
||||
if filename not in files:
|
||||
with open(os.path.join(tmp, filename)) as f:
|
||||
result[filename] = f.read()
|
||||
|
||||
return result
|
||||
|
||||
def _wrap_ca(self, name, data):
|
||||
return self.wrap_document(kind='CertificateAuthority', name=name,
|
||||
data=data, block_strings=self.block_strings)
|
||||
|
||||
def _wrap_ca_key(self, name, data):
|
||||
return self.wrap_document(kind='CertificateAuthorityKey', name=name,
|
||||
data=data, block_strings=self.block_strings)
|
||||
|
||||
def _wrap_cert(self, name, data):
|
||||
return self.wrap_document(kind='Certificate', name=name, data=data,
|
||||
block_strings=self.block_strings)
|
||||
|
||||
def _wrap_cert_key(self, name, data):
|
||||
return self.wrap_document(kind='CertificateKey', name=name, data=data,
|
||||
block_strings=self.block_strings)
|
||||
|
||||
def _wrap_priv_key(self, name, data):
|
||||
return self.wrap_document(kind='PrivateKey', name=name, data=data,
|
||||
block_strings=self.block_strings)
|
||||
|
||||
def _wrap_pub_key(self, name, data):
|
||||
return self.wrap_document(kind='PublicKey', name=name, data=data,
|
||||
block_strings=self.block_strings)
|
||||
|
||||
@staticmethod
|
||||
def wrap_document(kind, name, data, block_strings=True):
|
||||
"""Wrap document ``data`` with PeglegManagedDocument pattern.
|
||||
|
||||
:param str kind: The kind of document (found in ``schema``).
|
||||
:param str name: Name of the document.
|
||||
:param dict data: Document data.
|
||||
:param bool block_strings: Whether to dump out certificate data as
|
||||
block-style YAML string. Defaults to true.
|
||||
:return: the wrapped document
|
||||
:rtype: dict
|
||||
"""
|
||||
|
||||
wrapped_schema = 'deckhand/%s/v1' % kind
|
||||
wrapped_metadata = {
|
||||
'schema': 'metadata/Document/v1',
|
||||
'name': name,
|
||||
'layeringDefinition': {
|
||||
'abstract': False,
|
||||
'layer': 'site',
|
||||
}
|
||||
}
|
||||
wrapped_data = PKIUtility._block_literal(
|
||||
data, block_strings=block_strings)
|
||||
|
||||
document = {
|
||||
"schema": wrapped_schema,
|
||||
"metadata": wrapped_metadata,
|
||||
"data": wrapped_data
|
||||
}
|
||||
|
||||
return PeglegManagedSecretsDocument(document).pegleg_document
|
||||
|
||||
@staticmethod
|
||||
def _block_literal(data, block_strings=True):
|
||||
if block_strings:
|
||||
return block_literal(data)
|
||||
else:
|
||||
return data
|
||||
|
||||
|
||||
class block_literal(str):
|
||||
pass
|
||||
|
||||
|
||||
def block_literal_representer(dumper, data):
|
||||
return dumper.represent_scalar('tag:yaml.org,2002:str', data, style='|')
|
||||
|
||||
|
||||
yaml.add_representer(block_literal, block_literal_representer)
|
0
pegleg/engine/common/__init__.py
Normal file
0
pegleg/engine/common/__init__.py
Normal file
115
pegleg/engine/common/managed_document.py
Normal file
115
pegleg/engine/common/managed_document.py
Normal file
@ -0,0 +1,115 @@
|
||||
# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import os
|
||||
|
||||
from pegleg import config
|
||||
from pegleg.engine.util import git
|
||||
|
||||
MANAGED_DOCUMENT_SCHEMA = 'pegleg/PeglegManagedDocument/v1'
|
||||
SUPPORTED_SCHEMAS = (
|
||||
'deckhand/CertificateAuthority/v1',
|
||||
'deckhand/CertificateAuthorityKey/v1',
|
||||
'deckhand/Certificate/v1',
|
||||
'deckhand/CertificateKey/v1',
|
||||
'deckhand/PublicKey/v1',
|
||||
'deckhand/PrivateKey/v1',
|
||||
)
|
||||
|
||||
_KIND_TO_PATH = {
|
||||
'CertificateAuthority': 'certificates',
|
||||
'CertificateAuthorityKey': 'certificates',
|
||||
'Certificate': 'certificates',
|
||||
'CertificateKey': 'certificates',
|
||||
'PublicKey': 'keypairs',
|
||||
'PrivateKey': 'keypairs'
|
||||
}
|
||||
|
||||
|
||||
def is_managed_document(document):
|
||||
"""Utility for determining whether a document is wrapped by
|
||||
``pegleg/PeglegManagedDocument/v1`` pattern.
|
||||
|
||||
:param dict document: Document to check.
|
||||
:returns: True if document is managed, else False.
|
||||
:rtype: bool
|
||||
|
||||
"""
|
||||
|
||||
return document.get('schema') == "pegleg/PeglegManagedDocument/v1"
|
||||
|
||||
|
||||
def get_document_path(sitename, wrapper_document, cert_to_ca_map=None):
|
||||
"""Get path for outputting generated certificates or keys to.
|
||||
|
||||
Also updates the provenance path (``data.generated.specifiedBy.path``)
|
||||
for ``wrapper_document``.
|
||||
|
||||
* Certificates ar written to: ``<site>/secrets/certificates``
|
||||
* Keypairs are written to: ``<site>/secrets/keypairs``
|
||||
* Passphrases are written to: ``<site>/secrets/passphrases``
|
||||
|
||||
* The generated filenames for passphrases will follow the pattern
|
||||
``<passphrase-doc-name>.yaml``.
|
||||
* The generated filenames for certificate authorities will follow the
|
||||
pattern ``<ca-name>_ca.yaml``.
|
||||
* The generated filenames for certificates will follow the pattern
|
||||
``<ca-name>_<certificate-doc-name>_certificate.yaml``.
|
||||
* The generated filenames for certificate keys will follow the pattern
|
||||
``<ca-name>_<certificate-doc-name>_key.yaml``.
|
||||
* The generated filenames for keypairs will follow the pattern
|
||||
``<keypair-doc-name>.yaml``.
|
||||
|
||||
:param str sitename: Name of site.
|
||||
:param dict wrapper_document: Generated ``PeglegManagedDocument``.
|
||||
:param dict cert_to_ca_map: Dict that maps certificate names to
|
||||
their respective CA name.
|
||||
:returns: Path to write document out to.
|
||||
:rtype: str
|
||||
|
||||
"""
|
||||
|
||||
cert_to_ca_map = cert_to_ca_map or {}
|
||||
|
||||
managed_document = wrapper_document['data']['managedDocument']
|
||||
kind = managed_document['schema'].split("/")[1]
|
||||
name = managed_document['metadata']['name']
|
||||
|
||||
path = "%s/secrets/%s" % (sitename, _KIND_TO_PATH[kind])
|
||||
|
||||
if 'authority' in kind.lower():
|
||||
filename_structure = '%s_ca.yaml'
|
||||
elif 'certificate' in kind.lower():
|
||||
ca_name = cert_to_ca_map[name]
|
||||
filename_structure = ca_name + '_%s_certificate.yaml'
|
||||
elif 'public' in kind.lower() or 'private' in kind.lower():
|
||||
filename_structure = '%s.yaml'
|
||||
|
||||
# Dashes in the document names are converted to underscores for
|
||||
# consistency.
|
||||
filename = (filename_structure % name).replace('-', '_')
|
||||
fullpath = os.path.join(path, filename)
|
||||
|
||||
# Not all managed documents are generated. Only update path provenance
|
||||
# information for those that are.
|
||||
if wrapper_document['data'].get('generated'):
|
||||
wrapper_document['data']['generated']['specifiedBy']['path'] = fullpath
|
||||
return fullpath
|
||||
|
||||
|
||||
def _get_repo_url_and_rev():
|
||||
repo_path_or_url = config.get_site_repo()
|
||||
repo_url = git.repo_url(repo_path_or_url)
|
||||
repo_rev = config.get_site_rev()
|
||||
return repo_url, repo_rev
|
@ -65,3 +65,13 @@ class GitConfigException(PeglegBaseException):
|
||||
class GitInvalidRepoException(PeglegBaseException):
|
||||
"""Exception raised when an invalid repository is detected."""
|
||||
message = 'The repository path or URL is invalid: %(repo_path)s'
|
||||
|
||||
|
||||
#
|
||||
# PKI EXCEPTIONS
|
||||
#
|
||||
|
||||
|
||||
class IncompletePKIPairError(PeglegBaseException):
|
||||
"""Exception for incomplete private/public keypair."""
|
||||
message = ("Incomplete keypair set %(kinds)s for name: %(name)s")
|
||||
|
@ -42,18 +42,19 @@ def _clean_temp_folders():
|
||||
shutil.rmtree(r, ignore_errors=True)
|
||||
|
||||
|
||||
def process_repositories(site_name):
|
||||
def process_repositories(site_name, overwrite_existing=False):
|
||||
"""Process and setup all repositories including ensuring we are at the
|
||||
right revision based on the site's own site-definition.yaml file.
|
||||
|
||||
:param site_name: Site name for which to clone relevant repos.
|
||||
:param overwrite_existing: Whether to overwrite an existing directory
|
||||
|
||||
"""
|
||||
|
||||
# Only tracks extra repositories - not the site (primary) repository.
|
||||
extra_repos = []
|
||||
|
||||
site_repo = process_site_repository()
|
||||
site_repo = process_site_repository(overwrite_existing=overwrite_existing)
|
||||
|
||||
# Retrieve extra repo data from site-definition.yaml files.
|
||||
site_data = util.definition.load_as_params(
|
||||
@ -94,7 +95,9 @@ def process_repositories(site_name):
|
||||
"repo_username=%s, revision=%s", repo_alias, repo_url_or_path,
|
||||
repo_key, repo_user, repo_revision)
|
||||
|
||||
temp_extra_repo = _process_repository(repo_url_or_path, repo_revision)
|
||||
temp_extra_repo = _process_repository(
|
||||
repo_url_or_path, repo_revision,
|
||||
overwrite_existing=overwrite_existing)
|
||||
extra_repos.append(temp_extra_repo)
|
||||
|
||||
# Overwrite the site repo and extra repos in the config because further
|
||||
@ -105,12 +108,13 @@ def process_repositories(site_name):
|
||||
config.set_extra_repo_list(extra_repos)
|
||||
|
||||
|
||||
def process_site_repository(update_config=False):
|
||||
def process_site_repository(update_config=False, overwrite_existing=False):
|
||||
"""Process and setup site repository including ensuring we are at the right
|
||||
revision based on the site's own site-definition.yaml file.
|
||||
|
||||
:param bool update_config: Whether to update Pegleg config with computed
|
||||
site repo path.
|
||||
:param overwrite_existing: Whether to overwrite an existing directory
|
||||
|
||||
"""
|
||||
|
||||
@ -122,8 +126,10 @@ def process_site_repository(update_config=False):
|
||||
|
||||
repo_url_or_path, repo_revision = _extract_repo_url_and_revision(
|
||||
site_repo_or_path)
|
||||
config.set_site_rev(repo_revision)
|
||||
repo_url_or_path = _format_url_with_repo_username(repo_url_or_path)
|
||||
new_repo_path = _process_repository(repo_url_or_path, repo_revision)
|
||||
new_repo_path = _process_repository(repo_url_or_path, repo_revision,
|
||||
overwrite_existing=overwrite_existing)
|
||||
|
||||
if update_config:
|
||||
# Overwrite the site repo in the config because further processing will
|
||||
@ -134,17 +140,19 @@ def process_site_repository(update_config=False):
|
||||
return new_repo_path
|
||||
|
||||
|
||||
def _process_repository(repo_url_or_path, repo_revision):
|
||||
def _process_repository(repo_url_or_path, repo_revision,
|
||||
overwrite_existing=False):
|
||||
"""Process a repository located at ``repo_url_or_path``.
|
||||
|
||||
:param str repo_url_or_path: Path to local repo or URL of remote URL.
|
||||
:param str repo_revision: branch, commit or ref in the repo to checkout.
|
||||
:param overwrite_existing: Whether to overwrite an existing directory
|
||||
|
||||
"""
|
||||
|
||||
global __REPO_FOLDERS
|
||||
|
||||
if os.path.exists(repo_url_or_path):
|
||||
if os.path.exists(repo_url_or_path) and not overwrite_existing:
|
||||
repo_name = util.git.repo_name(repo_url_or_path)
|
||||
parent_temp_path = tempfile.mkdtemp()
|
||||
__REPO_FOLDERS.setdefault(repo_name, parent_temp_path)
|
||||
|
@ -75,12 +75,13 @@ def decrypt(file_path, site_name):
|
||||
:type file_path: string
|
||||
:param site_name: The name of the site to search for the file.
|
||||
:type site_name: string
|
||||
:return: The decrypted secrets
|
||||
:rtype: list
|
||||
"""
|
||||
|
||||
LOG.info('Started decrypting...')
|
||||
if (os.path.isfile(file_path) and
|
||||
[s for s in file_path.split(os.path.sep) if s == site_name]):
|
||||
PeglegSecretManagement(file_path).decrypt_secrets()
|
||||
return PeglegSecretManagement(file_path).decrypt_secrets()
|
||||
else:
|
||||
LOG.info('File: {} was not found. Check your file path and name, '
|
||||
'and try again.'.format(file_path))
|
||||
|
@ -13,7 +13,8 @@
|
||||
# limitations under the License.
|
||||
|
||||
# flake8: noqa
|
||||
from . import definition
|
||||
from . import files
|
||||
from . import deckhand
|
||||
from . import git
|
||||
from pegleg.engine.util import catalog
|
||||
from pegleg.engine.util import definition
|
||||
from pegleg.engine.util import deckhand
|
||||
from pegleg.engine.util import files
|
||||
from pegleg.engine.util import git
|
||||
|
52
pegleg/engine/util/catalog.py
Normal file
52
pegleg/engine/util/catalog.py
Normal file
@ -0,0 +1,52 @@
|
||||
# Copyright 2018 AT&T Intellectual Property. All other rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
"""Utility functions for catalog files such as pki-catalog.yaml."""
|
||||
|
||||
import logging
|
||||
|
||||
from pegleg.engine.util import definition
|
||||
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
||||
__all__ = ('iterate', )
|
||||
|
||||
|
||||
def iterate(kind, sitename=None, documents=None):
|
||||
"""Retrieve the list of catalog documents by catalog schema ``kind``.
|
||||
|
||||
:param str kind: The schema kind of the catalog. For example, for schema
|
||||
``pegleg/PKICatalog/v1`` kind should be "PKICatalog".
|
||||
:param str sitename: (optional) Site name for retrieving documents.
|
||||
Multually exclusive with ``documents``.
|
||||
:param str documents: (optional) Documents to search through. Mutually
|
||||
exclusive with ``sitename``.
|
||||
:return: All catalog documents for ``kind``.
|
||||
:rtype: generator[dict]
|
||||
|
||||
"""
|
||||
|
||||
if not any([sitename, documents]):
|
||||
raise ValueError('Either `sitename` or `documents` must be specified')
|
||||
|
||||
documents = documents or definition.documents_for_site(sitename)
|
||||
for document in documents:
|
||||
schema = document.get('schema')
|
||||
# TODO(felipemonteiro): Remove 'promenade/%s/v1' once site manifest
|
||||
# documents switch to new 'pegleg' namespace.
|
||||
if schema == 'pegleg/%s/v1' % kind:
|
||||
yield document
|
||||
elif schema == 'promenade/%s/v1' % kind:
|
||||
LOG.warning('The schema promenade/%s/v1 is deprecated. Use '
|
||||
'pegleg/%s/v1 instead.', kind, kind)
|
||||
yield document
|
@ -41,10 +41,10 @@ def load_schemas_from_docs(documents):
|
||||
return schema_set, errors
|
||||
|
||||
|
||||
def deckhand_render(documents=[],
|
||||
def deckhand_render(documents=None,
|
||||
fail_on_missing_sub_src=False,
|
||||
validate=False):
|
||||
|
||||
documents = documents or []
|
||||
errors = []
|
||||
rendered_documents = []
|
||||
|
||||
|
@ -11,6 +11,7 @@
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
"""Utility functions for site-definition.yaml files."""
|
||||
|
||||
import os
|
||||
|
||||
|
@ -26,7 +26,8 @@ from pegleg.engine import exceptions
|
||||
|
||||
LOG = logging.getLogger(__name__)
|
||||
|
||||
__all__ = ('git_handler', )
|
||||
__all__ = ('git_handler', 'is_repository', 'is_equal', 'repo_url', 'repo_name',
|
||||
'normalize_repo_path')
|
||||
|
||||
|
||||
def git_handler(repo_url,
|
||||