Browse Source

pki: Port Promenade's PKI catalog into Pegleg

This patch set implements the PKICatalog [0] requirements
as well as PeglegManagedDocument [1] generation requirements
outlined in the spec [2].

Included in this patch set:

* New CLI entry point called "pegleg site secrets generate-pki"
* PeglegManagedDocument generation logic in
  engine.cache.managed_document
* Refactored PKICatalog logic in engine.cache.pki_catalog derived
  from the Promenade PKI implementation [3], responsible for
  generating certificates, CAs, and keypairs
* Refactored PKIGenerator logic in engine.cache.pki_generator
  derived from Promenade Generator implementation [4],
  responsible for reading in pegleg/PKICatalog/v1 documents (as
  well as promenade/PKICatalog/v1 documents for backwards
  compatibility) and generating required secrets and storing
  them into the paths specified under [0]
* Unit tests for all of the above [5]
* Example pki-catalog.yaml document under pegleg/site_yamls
* Validation schema for pki-catalog.yaml (TODO: implement
  validation logic here: [6])
* Updates to CLI documentation and inclusion of PKICatalog
  and PeglegManagedDocument documentation
* Documentation updates with PKI information [7]

TODO (in follow-up patch sets):

* Expand on overview documentation to include new Pegleg
  responsibilities
* Allow the original repository (not the copied one) to
  be the destination where the secrets are written to
* Finish up cert expiry/revocation logic

[0] https://airship-specs.readthedocs.io/en/latest/specs/approved/pegleg-secrets.html#document-generation
[1] https://airship-specs.readthedocs.io/en/latest/specs/approved/pegleg-secrets.html#peglegmanageddocument
[2] https://airship-specs.readthedocs.io/en/latest/specs/approved/pegleg-secrets.html
[3] https://github.com/openstack/airship-promenade/blob/master/promenade/pki.py
[4] https://github.com/openstack/airship-promenade/blob/master/promenade/generator.py
[5] https://review.openstack.org/#/c/611739/
[6] https://review.openstack.org/#/c/608159/
[7] https://review.openstack.org/#/c/611738/

Change-Id: I3010d04cac6d22c656d144f0dafeaa5e19a13068
Felipe Monteiro 6 months ago
parent
commit
2a8d2638b3

+ 16
- 0
.zuul.yaml View File

@@ -18,11 +18,13 @@
18 18
     check:
19 19
       jobs:
20 20
         - openstack-tox-pep8
21
+        - airship-pegleg-tox-py36
21 22
         - airship-pegleg-doc-build
22 23
         - airship-pegleg-docker-build-gate
23 24
     gate:
24 25
       jobs:
25 26
         - openstack-tox-pep8
27
+        - airship-pegleg-tox-py36
26 28
         - airship-pegleg-doc-build
27 29
         - airship-pegleg-docker-build-gate
28 30
     post:
@@ -35,6 +37,20 @@
35 37
       - name: primary
36 38
         label: ubuntu-xenial
37 39
 
40
+- job:
41
+    name: airship-pegleg-tox-py36
42
+    description: |
43
+      Executes unit tests under Python 3.6
44
+    parent: openstack-tox-py36
45
+    pre-run:
46
+      - tools/gate/playbooks/install-cfssl.yaml
47
+    irrelevant-files:
48
+      - ^.*\.rst$
49
+      - ^doc/.*$
50
+      - ^etc/.*$
51
+      - ^releasenotes/.*$
52
+      - ^setup.cfg$
53
+
38 54
 - job:
39 55
     name: airship-pegleg-doc-build
40 56
     description: |

+ 54
- 11
doc/source/cli/cli.rst View File

@@ -81,10 +81,10 @@ CLI Options
81 81
 
82 82
 Enable debug logging.
83 83
 
84
-.. _site:
84
+.. _repo-group:
85 85
 
86
-Repo
87
-====
86
+Repo Group
87
+==========
88 88
 
89 89
 Allows you to perform repository-level operations.
90 90
 
@@ -127,8 +127,10 @@ a specific site, see :ref:`site-level linting <cli-site-lint>`.
127 127
 
128 128
 See :ref:`linting` for more information.
129 129
 
130
-Site
131
-====
130
+.. _site-group:
131
+
132
+Site Group
133
+==========
132 134
 
133 135
 Allows you to perform site-level operations.
134 136
 
@@ -303,7 +305,7 @@ Show details for one site.
303 305
 
304 306
 Name of site.
305 307
 
306
-**-o /--output** (Optional).
308
+**-o/--output** (Optional).
307 309
 
308 310
 Where to output.
309 311
 
@@ -331,7 +333,7 @@ Render documents via `Deckhand`_ for one site.
331 333
 
332 334
 Name of site.
333 335
 
334
-**-o /--output** (Optional).
336
+**-o/--output** (Optional).
335 337
 
336 338
 Where to output.
337 339
 
@@ -418,6 +420,39 @@ Usage:
418 420
 
419 421
     ./pegleg.sh site <options> upload <site_name> --context-marker=<uuid>
420 422
 
423
+Site Secrets Group
424
+==================
425
+
426
+Subgroup of :ref:`site-group`.
427
+
428
+Generate PKI
429
+------------
430
+
431
+Generate certificates and keys according to all PKICatalog documents in the
432
+site using the PKI module. Regenerating certificates can be
433
+accomplished by re-running this command.
434
+
435
+Pegleg places generated document files in ``<site>/secrets/passphrases``,
436
+``<site>/secrets/certificates``, or ``<site>/secrets/keypairs`` as
437
+appropriate:
438
+
439
+* The generated filenames for passphrases will follow the pattern
440
+  :file:`<passphrase-doc-name>.yaml`.
441
+* The generated filenames for certificate authorities will follow the pattern
442
+  :file:`<ca-name>_ca.yaml`.
443
+* The generated filenames for certificates will follow the pattern
444
+  :file:`<ca-name>_<certificate-doc-name>_certificate.yaml`.
445
+* The generated filenames for certificate keys will follow the pattern
446
+  :file:`<ca-name>_<certificate-doc-name>_key.yaml`.
447
+* The generated filenames for keypairs will follow the pattern
448
+  :file:`<keypair-doc-name>.yaml`.
449
+
450
+Dashes in the document names will be converted to underscores for consistency.
451
+
452
+**site_name** (Required).
453
+
454
+Name of site.
455
+
421 456
 Examples
422 457
 ^^^^^^^^
423 458
 
@@ -427,6 +462,14 @@ Examples
427 462
       upload <site_name> <options>
428 463
 
429 464
 
465
+::
466
+
467
+  ./pegleg.sh site -r <site_repo> -e <extra_repo> \
468
+    secrets generate-pki \
469
+    <site_name> \
470
+    -o <output> \
471
+    -f <filename>
472
+
430 473
 .. _command-line-repository-overrides:
431 474
 
432 475
 Secrets
@@ -571,13 +614,13 @@ Example:
571 614
 
572 615
 
573 616
 CLI Repository Overrides
574
-------------------------
617
+========================
575 618
 
576 619
 Repository overrides should only be used for entries included underneath
577 620
 the ``repositories`` field for a given :file:`site-definition.yaml`.
578 621
 
579
-Overrides are specified via the ``-e`` flag for all :ref:`site` commands. They
580
-have the following format:
622
+Overrides are specified via the ``-e`` flag for all :ref:`site-group` commands.
623
+They have the following format:
581 624
 
582 625
 ::
583 626
 
@@ -611,7 +654,7 @@ Where:
611 654
 .. _self-contained-repo:
612 655
 
613 656
 Self-Contained Repository
614
-^^^^^^^^^^^^^^^^^^^^^^^^^
657
+-------------------------
615 658
 
616 659
 For self-contained repositories, specification of extra repositories is
617 660
 unnecessary. The following command can be used to deploy the manifests in

+ 4
- 4
doc/source/developer-overview.rst View File

@@ -100,8 +100,8 @@ directory):
100 100
 
101 101
 .. code-block:: console
102 102
 
103
-  # Quick way of building a venv and installing all required dependencies into
104
-  # it.
103
+  # Quick way of building a virtualenv and installing all required
104
+  # dependencies into it.
105 105
   tox -e py36 --notest
106 106
   source .tox/py36/bin/activate
107 107
   pip install -e .
@@ -128,11 +128,11 @@ Unit Tests
128 128
 
129 129
 To run all unit tests, execute::
130 130
 
131
-  $ tox -epy36
131
+  $ tox -e py36
132 132
 
133 133
 To run unit tests using a regex, execute::
134 134
 
135
-  $ tox -epy36 -- <regex>
135
+  $ tox -e py36 -- <regex>
136 136
 
137 137
 .. _Airship: https://airshipit.readthedocs.io
138 138
 .. _Deckhand: https://airship-deckhand.readthedocs.io/

+ 8
- 0
doc/source/exceptions.rst View File

@@ -63,3 +63,11 @@ Authentication Exceptions
63 63
 .. autoexception:: pegleg.engine.util.shipyard_helper.AuthValuesError
64 64
    :members:
65 65
    :undoc-members:
66
+
67
+PKI Exceptions
68
+--------------
69
+
70
+.. autoexception:: pegleg.engine.exceptions.IncompletePKIPairError
71
+   :members:
72
+   :show-inheritance:
73
+   :undoc-members:

+ 4
- 3
doc/source/getting_started.rst View File

@@ -21,13 +21,14 @@ Getting Started
21 21
 What is Pegleg?
22 22
 ---------------
23 23
 
24
-Pegleg is a document aggregator that will aggregate all the documents in a
25
-repository and pack them into a single YAML file. This allows for operators to
24
+Pegleg is a document aggregator that aggregates all the documents in a
25
+repository and packs them into a single YAML file. This allows for operators to
26 26
 structure their site definitions in a maintainable directory layout, while
27 27
 providing them with the automation and tooling needed to aggregate, lint, and
28 28
 render those documents for deployment.
29 29
 
30
-For more information on the documents that Pegleg works on see `Document Fundamentals`_.
30
+For more information on the documents that Pegleg works on see
31
+`Document Fundamentals`_.
31 32
 
32 33
 Basic Usage
33 34
 -----------

BIN
doc/source/images/architecture-pegleg.png View File


+ 4
- 0
images/pegleg/Dockerfile View File

@@ -1,5 +1,6 @@
1 1
 ARG FROM=python:3.6
2 2
 FROM ${FROM}
3
+ARG CFSSLURL=https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
3 4
 
4 5
 LABEL org.opencontainers.image.authors='airship-discuss@lists.airshipit.org, irc://#airshipit@freenode'
5 6
 LABEL org.opencontainers.image.url='https://airshipit.org'
@@ -14,5 +15,8 @@ WORKDIR /var/pegleg
14 15
 COPY requirements.txt /opt/pegleg/requirements.txt
15 16
 RUN pip3 install --no-cache-dir -r /opt/pegleg/requirements.txt
16 17
 
18
+COPY tools/install-cfssl.sh /opt/pegleg/tools/install-cfssl.sh
19
+RUN /opt/pegleg/tools/install-cfssl.sh ${CFSSLURL}
20
+
17 21
 COPY . /opt/pegleg
18 22
 RUN pip3 install -e /opt/pegleg

+ 44
- 9
pegleg/cli.py View File

@@ -20,6 +20,7 @@ import click
20 20
 
21 21
 from pegleg import config
22 22
 from pegleg import engine
23
+from pegleg.engine import catalog
23 24
 from pegleg.engine.util.shipyard_helper import ShipyardHelper
24 25
 
25 26
 LOG = logging.getLogger(__name__)
@@ -130,7 +131,6 @@ def main(*, verbose):
130 131
 
131 132
     * site: site-level actions
132 133
     * repo: repository-level actions
133
-    * stub (DEPRECATED)
134 134
 
135 135
     """
136 136
 
@@ -208,7 +208,7 @@ def site(*, site_repository, clone_path, extra_repositories, repo_key,
208 208
     * list: list available sites in a manifests repo
209 209
     * lint: lint a site along with all its dependencies
210 210
     * render: render a site using Deckhand
211
-    * show: show a sites' files
211
+    * show: show a site's files
212 212
 
213 213
     """
214 214
 
@@ -375,6 +375,39 @@ def upload(ctx, *, os_project_domain_name,
375 375
     click.echo(ShipyardHelper(ctx).upload_documents())
376 376
 
377 377
 
378
+@site.group(name='secrets', help='Commands to manage site secrets documents')
379
+def secrets():
380
+    pass
381
+
382
+
383
+@secrets.command(
384
+    'generate-pki',
385
+    help="""
386
+Generate certificates and keys according to all PKICatalog documents in the
387
+site. Regenerating certificates can be accomplished by re-running this command.
388
+""")
389
+@click.option(
390
+    '-a',
391
+    '--author',
392
+    'author',
393
+    help="""Identifying name of the author generating new certificates. Used
394
+for tracking provenance information in the PeglegManagedDocuments. An attempt
395
+is made to automatically determine this value, but should be provided.""")
396
+@click.argument('site_name')
397
+def generate_pki(site_name, author):
398
+    """Generate certificates, certificate authorities and keypairs for a given
399
+    site.
400
+
401
+    """
402
+
403
+    engine.repository.process_repositories(site_name,
404
+                                           overwrite_existing=True)
405
+    pkigenerator = catalog.pki_generator.PKIGenerator(site_name, author=author)
406
+    output_paths = pkigenerator.generate()
407
+
408
+    click.echo("Generated PKI files written to:\n%s" % '\n'.join(output_paths))
409
+
410
+
378 411
 @main.group(help='Commands related to types')
379 412
 @MAIN_REPOSITORY_OPTION
380 413
 @REPOSITORY_CLONE_PATH_OPTION
@@ -409,11 +442,6 @@ def list_types(*, output_stream):
409 442
     engine.type.list_types(output_stream)
410 443
 
411 444
 
412
-@site.group(name='secrets', help='Commands to manage site secrets documents')
413
-def secrets():
414
-    pass
415
-
416
-
417 445
 @secrets.command(
418 446
     'encrypt',
419 447
     help='Command to encrypt and wrap site secrets '
@@ -437,7 +465,9 @@ def secrets():
437 465
     'documents')
438 466
 @click.argument('site_name')
439 467
 def encrypt(*, save_location, author, site_name):
440
-    engine.repository.process_repositories(site_name)
468
+    engine.repository.process_repositories(site_name, overwrite_existing=True)
469
+    if save_location is None:
470
+        save_location = config.get_site_repo()
441 471
     engine.secrets.encrypt(save_location, author, site_name)
442 472
 
443 473
 
@@ -453,4 +483,9 @@ def encrypt(*, save_location, author, site_name):
453 483
 @click.argument('site_name')
454 484
 def decrypt(*, file_name, site_name):
455 485
     engine.repository.process_repositories(site_name)
456
-    engine.secrets.decrypt(file_name, site_name)
486
+    try:
487
+        click.echo(engine.secrets.decrypt(file_name, site_name))
488
+    except FileNotFoundError:
489
+        raise click.exceptions.FileError("Couldn't find file %s, "
490
+                                         "check your arguments and try "
491
+                                         "again." % file_name)

+ 12
- 1
pegleg/config.py View File

@@ -25,7 +25,8 @@ except NameError:
25 25
         'extra_repos': [],
26 26
         'clone_path': None,
27 27
         'site_path': 'site',
28
-        'type_path': 'type'
28
+        'site_rev': None,
29
+        'type_path': 'type',
29 30
     }
30 31
 
31 32
 
@@ -49,6 +50,16 @@ def set_clone_path(p):
49 50
     GLOBAL_CONTEXT['clone_path'] = p
50 51
 
51 52
 
53
+def get_site_rev():
54
+    """Get site revision derived from the site repo URL/path, if provided."""
55
+    return GLOBAL_CONTEXT['site_rev']
56
+
57
+
58
+def set_site_rev(r):
59
+    """Set site revision derived from the site repo URL/path."""
60
+    GLOBAL_CONTEXT['site_rev'] = r
61
+
62
+
52 63
 def get_extra_repo_overrides():
53 64
     """Get extra repository overrides specified via ``-e`` CLI flag."""
54 65
     return GLOBAL_CONTEXT.get('extra_repo_overrides', [])

+ 17
- 0
pegleg/engine/catalog/__init__.py View File

@@ -0,0 +1,17 @@
1
+# Copyright 2018 AT&T Intellectual Property.  All other rights reserved.
2
+#
3
+# Licensed under the Apache License, Version 2.0 (the "License");
4
+# you may not use this file except in compliance with the License.
5
+# You may obtain a copy of the License at
6
+#
7
+#     http://www.apache.org/licenses/LICENSE-2.0
8
+#
9
+# Unless required by applicable law or agreed to in writing, software
10
+# distributed under the License is distributed on an "AS IS" BASIS,
11
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+# See the License for the specific language governing permissions and
13
+# limitations under the License.
14
+
15
+# flake8: noqa
16
+from pegleg.engine.catalog import pki_utility
17
+from pegleg.engine.catalog import pki_generator

+ 307
- 0
pegleg/engine/catalog/pki_generator.py View File

@@ -0,0 +1,307 @@
1
+# Copyright 2018 AT&T Intellectual Property.  All other rights reserved.
2
+#
3
+# Licensed under the Apache License, Version 2.0 (the "License");
4
+# you may not use this file except in compliance with the License.
5
+# You may obtain a copy of the License at
6
+#
7
+#     http://www.apache.org/licenses/LICENSE-2.0
8
+#
9
+# Unless required by applicable law or agreed to in writing, software
10
+# distributed under the License is distributed on an "AS IS" BASIS,
11
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+# See the License for the specific language governing permissions and
13
+# limitations under the License.
14
+
15
+import collections
16
+import itertools
17
+import logging
18
+import os
19
+
20
+import yaml
21
+
22
+from pegleg import config
23
+from pegleg.engine.catalog import pki_utility
24
+from pegleg.engine.common import managed_document as md
25
+from pegleg.engine import exceptions
26
+from pegleg.engine import util
27
+from pegleg.engine.util.pegleg_managed_document import \
28
+    PeglegManagedSecretsDocument
29
+
30
+__all__ = ['PKIGenerator']
31
+
32
+LOG = logging.getLogger(__name__)
33
+
34
+
35
+class PKIGenerator(object):
36
+    """Generates certificates, certificate authorities and keypairs using
37
+    the ``PKIUtility`` class.
38
+
39
+    Pegleg searches through a given "site" to derive all the documents
40
+    of kind ``PKICatalog``, which are in turn parsed for information related
41
+    to the above secret types and passed to ``PKIUtility`` for generation.
42
+
43
+    These secrets are output to various subdirectories underneath
44
+    ``<site>/secrets/<subpath>``.
45
+
46
+    """
47
+
48
+    def __init__(self, sitename, block_strings=True, author=None):
49
+        """Constructor for ``PKIGenerator``.
50
+
51
+        :param str sitename: Site name for which to retrieve documents used for
52
+            certificate and keypair generation.
53
+        :param bool block_strings: Whether to dump out certificate data as
54
+            block-style YAML string. Defaults to true.
55
+        :param str author: Identifying name of the author generating new
56
+            certificates.
57
+
58
+        """
59
+
60
+        self._sitename = sitename
61
+        self._documents = util.definition.documents_for_site(sitename)
62
+        self._author = author
63
+
64
+        self.keys = pki_utility.PKIUtility(block_strings=block_strings)
65
+        self.outputs = collections.defaultdict(dict)
66
+
67
+        # Maps certificates to CAs in order to derive certificate paths.
68
+        self._cert_to_ca_map = {}
69
+
70
+    def generate(self):
71
+        for catalog in util.catalog.iterate(
72
+                documents=self._documents, kind='PKICatalog'):
73
+            for ca_name, ca_def in catalog['data'].get(
74
+                    'certificate_authorities', {}).items():
75
+                ca_cert, ca_key = self.get_or_gen_ca(ca_name)
76
+
77
+                for cert_def in ca_def.get('certificates', []):
78
+                    document_name = cert_def['document_name']
79
+                    self._cert_to_ca_map.setdefault(document_name, ca_name)
80
+                    cert, key = self.get_or_gen_cert(
81
+                        document_name,
82
+                        ca_cert=ca_cert,
83
+                        ca_key=ca_key,
84
+                        cn=cert_def['common_name'],
85
+                        hosts=_extract_hosts(cert_def),
86
+                        groups=cert_def.get('groups', []))
87
+
88
+            for keypair_def in catalog['data'].get('keypairs', []):
89
+                document_name = keypair_def['name']
90
+                self.get_or_gen_keypair(document_name)
91
+
92
+        return self._write(config.get_site_repo())
93
+
94
+    def get_or_gen_ca(self, document_name):
95
+        kinds = [
96
+            'CertificateAuthority',
97
+            'CertificateAuthorityKey',
98
+        ]
99
+        return self._get_or_gen(self.gen_ca, kinds, document_name)
100
+
101
+    def get_or_gen_cert(self, document_name, **kwargs):
102
+        kinds = [
103
+            'Certificate',
104
+            'CertificateKey',
105
+        ]
106
+        return self._get_or_gen(self.gen_cert, kinds, document_name, **kwargs)
107
+
108
+    def get_or_gen_keypair(self, document_name):
109
+        kinds = [
110
+            'PublicKey',
111
+            'PrivateKey',
112
+        ]
113
+        return self._get_or_gen(self.gen_keypair, kinds, document_name)
114
+
115
+    def gen_ca(self, document_name, **kwargs):
116
+        return self.keys.generate_ca(document_name, **kwargs)
117
+
118
+    def gen_cert(self, document_name, *, ca_cert, ca_key, **kwargs):
119
+        ca_cert_data = ca_cert['data']['managedDocument']['data']
120
+        ca_key_data = ca_key['data']['managedDocument']['data']
121
+        return self.keys.generate_certificate(
122
+            document_name, ca_cert=ca_cert_data, ca_key=ca_key_data, **kwargs)
123
+
124
+    def gen_keypair(self, document_name):
125
+        return self.keys.generate_keypair(document_name)
126
+
127
+    def _get_or_gen(self, generator, kinds, document_name, *args, **kwargs):
128
+        docs = self._find_docs(kinds, document_name)
129
+        if not docs:
130
+            docs = generator(document_name, *args, **kwargs)
131
+        else:
132
+            docs = [PeglegManagedSecretsDocument(doc).pegleg_document
133
+                    for doc in docs]
134
+
135
+        # Adding these to output should be idempotent, so we use a dict.
136
+
137
+        for wrapper_doc in docs:
138
+            wrapped_doc = wrapper_doc['data']['managedDocument']
139
+            schema = wrapped_doc['schema']
140
+            name = wrapped_doc['metadata']['name']
141
+            self.outputs[schema][name] = wrapper_doc
142
+
143
+        return docs
144
+
145
+    def _find_docs(self, kinds, document_name):
146
+        schemas = ['deckhand/%s/v1' % k for k in kinds]
147
+        docs = self._find_among_collected(schemas, document_name)
148
+        if docs:
149
+            if len(docs) == len(kinds):
150
+                LOG.debug('Found docs in input config named %s, kinds: %s',
151
+                          document_name, kinds)
152
+                return docs
153
+            else:
154
+                raise exceptions.IncompletePKIPairError(
155
+                    kinds=kinds, name=document_name)
156
+
157
+        else:
158
+            docs = self._find_among_outputs(schemas, document_name)
159
+            if docs:
160
+                LOG.debug('Found docs in current outputs named %s, kinds: %s',
161
+                          document_name, kinds)
162
+                return docs
163
+        # TODO(felipemonteiro): Should this be a critical error?
164
+        LOG.debug('No docs existing docs named %s, kinds: %s', document_name,
165
+                  kinds)
166
+        return []
167
+
168
+    def _find_among_collected(self, schemas, document_name):
169
+        result = []
170
+        for schema in schemas:
171
+            doc = _find_document_by(
172
+                self._documents, schema=schema, name=document_name)
173
+            # If the document wasn't found, then means it needs to be
174
+            # generated.
175
+            if doc:
176
+                result.append(doc)
177
+        return result
178
+
179
+    def _find_among_outputs(self, schemas, document_name):
180
+        result = []
181
+        for schema in schemas:
182
+            if document_name in self.outputs.get(schema, {}):
183
+                result.append(self.outputs[schema][document_name])
184
+        return result
185
+
186
+    def _write(self, output_dir):
187
+        documents = self.get_documents()
188
+        output_paths = set()
189
+
190
+        # First, delete each of the output paths below because we do an append
191
+        # action in the `open` call below. This means that for regeneration
192
+        # of certs, the original paths must be deleted.
193
+        for document in documents:
194
+            output_file_path = md.get_document_path(
195
+                sitename=self._sitename,
196
+                wrapper_document=document,
197
+                cert_to_ca_map=self._cert_to_ca_map)
198
+            output_path = os.path.join(output_dir, 'site', output_file_path)
199
+            # NOTE(felipemonteiro): This is currently an entirely safe
200
+            # operation as these files are being removed in the temporarily
201
+            # replicated versions of the local repositories.
202
+            if os.path.exists(output_path):
203
+                os.remove(output_path)
204
+
205
+        # Next, generate (or regenerate) the certificates.
206
+        for document in documents:
207
+            output_file_path = md.get_document_path(
208
+                sitename=self._sitename,
209
+                wrapper_document=document,
210
+                cert_to_ca_map=self._cert_to_ca_map)
211
+            output_path = os.path.join(output_dir, 'site', output_file_path)
212
+            dir_name = os.path.dirname(output_path)
213
+
214
+            if not os.path.exists(dir_name):
215
+                LOG.debug('Creating secrets path: %s', dir_name)
216
+                os.makedirs(dir_name)
217
+
218
+            with open(output_path, 'a') as f:
219
+                # Don't use safe_dump so we can block format certificate
220
+                # data.
221
+                yaml.dump(
222
+                    document,
223
+                    stream=f,
224
+                    default_flow_style=False,
225
+                    explicit_start=True,
226
+                    indent=2)
227
+
228
+            output_paths.add(output_path)
229
+        return output_paths
230
+
231
+    def get_documents(self):
232
+        return list(
233
+            itertools.chain.from_iterable(
234
+                v.values() for v in self.outputs.values()))
235
+
236
+
237
+def get_host_list(service_names):
238
+    service_list = []
239
+    for service in service_names:
240
+        parts = service.split('.')
241
+        for i in range(len(parts)):
242
+            service_list.append('.'.join(parts[:i + 1]))
243
+    return service_list
244
+
245
+
246
+def _extract_hosts(cert_def):
247
+    hosts = cert_def.get('hosts', [])
248
+    hosts.extend(get_host_list(cert_def.get('kubernetes_service_names', [])))
249
+    return hosts
250
+
251
+
252
+def _find_document_by(documents, **kwargs):
253
+    try:
254
+        return next(_iterate(documents, **kwargs))
255
+    except StopIteration:
256
+        return None
257
+
258
+
259
+def _iterate(documents, *, kind=None, schema=None, labels=None, name=None):
260
+    if kind is not None:
261
+        if schema is not None:
262
+            raise AssertionError('Logic error: specified both kind and schema')
263
+        schema = 'promenade/%s/v1' % kind
264
+
265
+    for document in documents:
266
+        if _matches_filter(document, schema=schema, labels=labels, name=name):
267
+            yield document
268
+
269
+
270
+def _matches_filter(document, *, schema, labels, name):
271
+    matches = True
272
+
273
+    if md.is_managed_document(document):
274
+        document = document['data']['managedDocument']
275
+    else:
276
+        document_schema = document['schema']
277
+        if document_schema in md.SUPPORTED_SCHEMAS:
278
+            # Can't use the filter value as they might not be an exact match.
279
+            document_metadata = document['metadata']
280
+            document_labels = document_metadata.get('labels', {})
281
+            document_name = document_metadata['name']
282
+            LOG.warning('Detected deprecated unmanaged document during PKI '
283
+                        'generation. Details: schema=%s, name=%s, labels=%s.',
284
+                        document_schema, document_labels, document_name)
285
+
286
+    if schema is not None and not document.get('schema',
287
+                                               '').startswith(schema):
288
+        matches = False
289
+
290
+    if labels is not None:
291
+        document_labels = _mg(document, 'labels', [])
292
+        for key, value in labels.items():
293
+            if key not in document_labels:
294
+                matches = False
295
+            else:
296
+                if document_labels[key] != value:
297
+                    matches = False
298
+
299
+    if name is not None:
300
+        if _mg(document, 'name') != name:
301
+            matches = False
302
+
303
+    return matches
304
+
305
+
306
+def _mg(document, field, default=None):
307
+    return document.get('metadata', {}).get(field, default)

+ 330
- 0
pegleg/engine/catalog/pki_utility.py View File

@@ -0,0 +1,330 @@
1
+# Copyright 2018 AT&T Intellectual Property.  All other rights reserved.
2
+#
3
+# Licensed under the Apache License, Version 2.0 (the "License");
4
+# you may not use this file except in compliance with the License.
5
+# You may obtain a copy of the License at
6
+#
7
+#     http://www.apache.org/licenses/LICENSE-2.0
8
+#
9
+# Unless required by applicable law or agreed to in writing, software
10
+# distributed under the License is distributed on an "AS IS" BASIS,
11
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+# See the License for the specific language governing permissions and
13
+# limitations under the License.
14
+
15
+from datetime import datetime
16
+import json
17
+import logging
18
+import os
19
+# Ignore bandit false positive: B404:blacklist
20
+# The purpose of this module is to safely encapsulate calls via fork.
21
+import subprocess  # nosec
22
+import tempfile
23
+
24
+from dateutil import parser
25
+import pytz
26
+import yaml
27
+
28
+from pegleg.engine.util.pegleg_managed_document import \
29
+    PeglegManagedSecretsDocument
30
+
31
+LOG = logging.getLogger(__name__)
32
+_ONE_YEAR_IN_HOURS = '8760h'  # 365 * 24
33
+
34
+__all__ = ['PKIUtility']
35
+
36
+
37
+# TODO(felipemonteiro): Create an abstract base class for other future Catalog
38
+# classes.
39
+
40
+
41
+class PKIUtility(object):
42
+    """Public Key Infrastructure utility class.
43
+
44
+    Responsible for generating certificate and CA documents using ``cfssl`` and
45
+    keypairs using ``openssl``. These secrets are all wrapped in instances
46
+    of ``pegleg/PeglegManagedDocument/v1``.
47
+
48
+    """
49
+
50
+    @staticmethod
51
+    def cfssl_exists():
52
+        """Checks whether cfssl command exists. Useful for testing."""
53
+        try:
54
+            subprocess.check_output(  # nosec
55
+                ['which', 'cfssl'], stderr=subprocess.STDOUT)
56
+            return True
57
+        except subprocess.CalledProcessError:
58
+            return False
59
+
60
+    def __init__(self, *, block_strings=True):
61
+        self.block_strings = block_strings
62
+        self._ca_config_string = None
63
+
64
+    @property
65
+    def ca_config(self):
66
+        if not self._ca_config_string:
67
+            self._ca_config_string = json.dumps({
68
+                'signing': {
69
+                    'default': {
70
+                        # TODO(felipemonteiro): Make this configurable.
71
+                        'expiry':
72
+                            _ONE_YEAR_IN_HOURS,
73
+                        'usages': [
74
+                            'signing', 'key encipherment', 'server auth',
75
+                            'client auth'
76
+                        ],
77
+                    },
78
+                },
79
+            })
80
+        return self._ca_config_string
81
+
82
+    def generate_ca(self, ca_name):
83
+        """Generate CA cert and associated key.
84
+
85
+        :param str ca_name: Name of Certificate Authority in wrapped document.
86
+        :returns: Tuple of (wrapped CA cert, wrapped CA key)
87
+        :rtype: tuple[dict, dict]
88
+
89
+        """
90
+
91
+        result = self._cfssl(
92
+            ['gencert', '-initca', 'csr.json'],
93
+            files={
94
+                'csr.json': self.csr(name=ca_name),
95
+            })
96
+
97
+        return (self._wrap_ca(ca_name, result['cert']),
98
+                self._wrap_ca_key(ca_name, result['key']))
99
+
100
+    def generate_keypair(self, name):
101
+        """Generate keypair.
102
+
103
+        :param str name: Name of keypair in wrapped document.
104
+        :returns: Tuple of (wrapped public key, wrapped private key)
105
+        :rtype: tuple[dict, dict]
106
+
107
+        """
108
+
109
+        priv_result = self._openssl(['genrsa', '-out', 'priv.pem'])
110
+        pub_result = self._openssl(
111
+            ['rsa', '-in', 'priv.pem', '-pubout', '-out', 'pub.pem'],
112
+            files={
113
+                'priv.pem': priv_result['priv.pem'],
114
+            })
115
+
116
+        return (self._wrap_pub_key(name, pub_result['pub.pem']),
117
+                self._wrap_priv_key(name, priv_result['priv.pem']))
118
+
119
+    def generate_certificate(self,
120
+                             name,
121
+                             *,
122
+                             ca_cert,
123
+                             ca_key,
124
+                             cn,
125
+                             groups=None,
126
+                             hosts=None):
127
+        """Generate certificate and associated key given CA cert and key.
128
+
129
+        :param str name: Name of certificate in wrapped document.
130
+        :param str ca_cert: CA certificate.
131
+        :param str ca_key: CA certificate key.
132
+        :param str cn: Common name associated with certificate.
133
+        :param list groups: List of groups associated with certificate.
134
+        :param list hosts: List of hosts associated with certificate.
135
+        :returns: Tuple of (wrapped certificate, wrapped certificate key)
136
+        :rtype: tuple[dict, dict]
137
+
138
+        """
139
+
140
+        if groups is None:
141
+            groups = []
142
+        if hosts is None:
143
+            hosts = []
144
+
145
+        result = self._cfssl(
146
+            [
147
+                'gencert', '-ca', 'ca.pem', '-ca-key', 'ca-key.pem', '-config',
148
+                'ca-config.json', 'csr.json'
149
+            ],
150
+            files={
151
+                'ca-config.json': self.ca_config,
152
+                'ca.pem': ca_cert,
153
+                'ca-key.pem': ca_key,
154
+                'csr.json': self.csr(name=cn, groups=groups, hosts=hosts),
155
+            })
156
+
157
+        return (self._wrap_cert(name, result['cert']),
158
+                self._wrap_cert_key(name, result['key']))
159
+
160
+    def csr(self,
161
+            *,
162
+            name,
163
+            groups=None,
164
+            hosts=None,
165
+            key={
166
+                'algo': 'rsa',
167
+                'size': 2048
168
+            }):
169
+        if groups is None:
170
+            groups = []
171
+        if hosts is None:
172
+            hosts = []
173
+
174
+        return json.dumps({
175
+            'CN': name,
176
+            'key': key,
177
+            'hosts': hosts,
178
+            'names': [{
179
+                'O': g
180
+            } for g in groups],
181
+        })
182
+
183
+    def cert_info(self, cert):
184
+        """Retrieve certificate info via ``cfssl``.
185
+
186
+        :param str cert: Client certificate that contains the public key.
187
+        :returns: Information related to certificate.
188
+        :rtype: dict
189
+
190
+        """
191
+
192
+        return self._cfssl(
193
+            ['certinfo', '-cert', 'cert.pem'], files={
194
+                'cert.pem': cert,
195
+            })
196
+
197
+    def check_expiry(self, cert):
198
+        """Chek whether a given certificate is expired.
199
+
200
+        :param str cert: Client certificate that contains the public key.
201
+        :returns: True if certificate is expired, else False.
202
+        :rtype: bool
203
+
204
+        """
205
+
206
+        info = self.cert_info(cert)
207
+        expiry_str = info['not_after']
208
+        expiry = parser.parse(expiry_str)
209
+        # expiry is timezone-aware; do the same for `now`.
210
+        now = pytz.utc.localize(datetime.utcnow())
211
+        return now > expiry
212
+
213
+    def _cfssl(self, command, *, files=None):
214
+        """Executes ``cfssl`` command via ``subprocess`` call."""
215
+        if not files:
216
+            files = {}
217
+        with tempfile.TemporaryDirectory() as tmp:
218
+            for filename, data in files.items():
219
+                with open(os.path.join(tmp, filename), 'w') as f:
220
+                    f.write(data)
221
+
222
+            # Ignore bandit false positive:
223
+            #   B603:subprocess_without_shell_equals_true
224
+            # This method wraps cfssl calls originating from this module.
225
+            result = subprocess.check_output(  # nosec
226
+                ['cfssl'] + command, cwd=tmp, stderr=subprocess.PIPE)
227
+            if not isinstance(result, str):
228
+                result = result.decode('utf-8')
229
+            return json.loads(result)
230
+
231
+    def _openssl(self, command, *, files=None):
232
+        """Executes ``openssl`` command via ``subprocess`` call."""
233
+        if not files:
234
+            files = {}
235
+
236
+        with tempfile.TemporaryDirectory() as tmp:
237
+            for filename, data in files.items():
238
+                with open(os.path.join(tmp, filename), 'w') as f:
239
+                    f.write(data)
240
+
241
+            # Ignore bandit false positive:
242
+            #   B603:subprocess_without_shell_equals_true
243
+            # This method wraps openssl calls originating from this module.
244
+            subprocess.check_call(  # nosec
245
+                ['openssl'] + command,
246
+                cwd=tmp,
247
+                stderr=subprocess.PIPE)
248
+
249
+            result = {}
250
+            for filename in os.listdir(tmp):
251
+                if filename not in files:
252
+                    with open(os.path.join(tmp, filename)) as f:
253
+                        result[filename] = f.read()
254
+
255
+            return result
256
+
257
+    def _wrap_ca(self, name, data):
258
+        return self.wrap_document(kind='CertificateAuthority', name=name,
259
+                                  data=data, block_strings=self.block_strings)
260
+
261
+    def _wrap_ca_key(self, name, data):
262
+        return self.wrap_document(kind='CertificateAuthorityKey', name=name,
263
+                                  data=data, block_strings=self.block_strings)
264
+
265
+    def _wrap_cert(self, name, data):
266
+        return self.wrap_document(kind='Certificate', name=name, data=data,
267
+                                  block_strings=self.block_strings)
268
+
269
+    def _wrap_cert_key(self, name, data):
270
+        return self.wrap_document(kind='CertificateKey', name=name, data=data,
271
+                                  block_strings=self.block_strings)
272
+
273
+    def _wrap_priv_key(self, name, data):
274
+        return self.wrap_document(kind='PrivateKey', name=name, data=data,
275
+                                  block_strings=self.block_strings)
276
+
277
+    def _wrap_pub_key(self, name, data):
278
+        return self.wrap_document(kind='PublicKey', name=name, data=data,
279
+                                  block_strings=self.block_strings)
280
+
281
+    @staticmethod
282
+    def wrap_document(kind, name, data, block_strings=True):
283
+        """Wrap document ``data`` with PeglegManagedDocument pattern.
284
+
285
+        :param str kind: The kind of document (found in ``schema``).
286
+        :param str name: Name of the document.
287
+        :param dict data: Document data.
288
+        :param bool block_strings: Whether to dump out certificate data as
289
+            block-style YAML string. Defaults to true.
290
+        :return: the wrapped document
291
+        :rtype: dict
292
+        """
293
+
294
+        wrapped_schema = 'deckhand/%s/v1' % kind
295
+        wrapped_metadata = {
296
+            'schema': 'metadata/Document/v1',
297
+            'name': name,
298
+            'layeringDefinition': {
299
+                'abstract': False,
300
+                'layer': 'site',
301
+            }
302
+        }
303
+        wrapped_data = PKIUtility._block_literal(
304
+            data, block_strings=block_strings)
305
+
306
+        document = {
307
+            "schema": wrapped_schema,
308
+            "metadata": wrapped_metadata,
309
+            "data": wrapped_data
310
+        }
311
+
312
+        return PeglegManagedSecretsDocument(document).pegleg_document
313
+
314
+    @staticmethod
315
+    def _block_literal(data, block_strings=True):
316
+        if block_strings:
317
+            return block_literal(data)
318
+        else:
319
+            return data
320
+
321
+
322
+class block_literal(str):
323
+    pass
324
+
325
+
326
+def block_literal_representer(dumper, data):
327
+    return dumper.represent_scalar('tag:yaml.org,2002:str', data, style='|')
328
+
329
+
330
+yaml.add_representer(block_literal, block_literal_representer)

+ 0
- 0
pegleg/engine/common/__init__.py View File


+ 115
- 0
pegleg/engine/common/managed_document.py View File

@@ -0,0 +1,115 @@
1
+# Copyright 2018 AT&T Intellectual Property.  All other rights reserved.
2
+#
3
+# Licensed under the Apache License, Version 2.0 (the "License");
4
+# you may not use this file except in compliance with the License.
5
+# You may obtain a copy of the License at
6
+#
7
+#     http://www.apache.org/licenses/LICENSE-2.0
8
+#
9
+# Unless required by applicable law or agreed to in writing, software
10
+# distributed under the License is distributed on an "AS IS" BASIS,
11
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+# See the License for the specific language governing permissions and
13
+# limitations under the License.
14
+
15
+import os
16
+
17
+from pegleg import config
18
+from pegleg.engine.util import git
19
+
20
+MANAGED_DOCUMENT_SCHEMA = 'pegleg/PeglegManagedDocument/v1'
21
+SUPPORTED_SCHEMAS = (
22
+    'deckhand/CertificateAuthority/v1',
23
+    'deckhand/CertificateAuthorityKey/v1',
24
+    'deckhand/Certificate/v1',
25
+    'deckhand/CertificateKey/v1',
26
+    'deckhand/PublicKey/v1',
27
+    'deckhand/PrivateKey/v1',
28
+)
29
+
30
+_KIND_TO_PATH = {
31
+    'CertificateAuthority': 'certificates',
32
+    'CertificateAuthorityKey': 'certificates',
33
+    'Certificate': 'certificates',
34
+    'CertificateKey': 'certificates',
35
+    'PublicKey': 'keypairs',
36
+    'PrivateKey': 'keypairs'
37
+}
38
+
39
+
40
+def is_managed_document(document):
41
+    """Utility for determining whether a document is wrapped by
42
+    ``pegleg/PeglegManagedDocument/v1`` pattern.
43
+
44
+    :param dict document: Document to check.
45
+    :returns: True if document is managed, else False.
46
+    :rtype: bool
47
+
48
+    """
49
+
50
+    return document.get('schema') == "pegleg/PeglegManagedDocument/v1"
51
+
52
+
53
+def get_document_path(sitename, wrapper_document, cert_to_ca_map=None):
54
+    """Get path for outputting generated certificates or keys to.
55
+
56
+    Also updates the provenance path (``data.generated.specifiedBy.path``)
57
+    for ``wrapper_document``.
58
+
59
+    * Certificates ar written to: ``<site>/secrets/certificates``
60
+    * Keypairs are written to: ``<site>/secrets/keypairs``
61
+    * Passphrases are written to: ``<site>/secrets/passphrases``
62
+
63
+    * The generated filenames for passphrases will follow the pattern
64
+      ``<passphrase-doc-name>.yaml``.
65
+    * The generated filenames for certificate authorities will follow the
66
+      pattern ``<ca-name>_ca.yaml``.
67
+    * The generated filenames for certificates will follow the pattern
68
+      ``<ca-name>_<certificate-doc-name>_certificate.yaml``.
69
+    * The generated filenames for certificate keys will follow the pattern
70
+      ``<ca-name>_<certificate-doc-name>_key.yaml``.
71
+    * The generated filenames for keypairs will follow the pattern
72
+      ``<keypair-doc-name>.yaml``.
73
+
74
+    :param str sitename: Name of site.
75
+    :param dict wrapper_document: Generated ``PeglegManagedDocument``.
76
+    :param dict cert_to_ca_map: Dict that maps certificate names to
77
+        their respective CA name.
78
+    :returns: Path to write document out to.
79
+    :rtype: str
80
+
81
+    """
82
+
83
+    cert_to_ca_map = cert_to_ca_map or {}
84
+
85
+    managed_document = wrapper_document['data']['managedDocument']
86
+    kind = managed_document['schema'].split("/")[1]
87
+    name = managed_document['metadata']['name']
88
+
89
+    path = "%s/secrets/%s" % (sitename, _KIND_TO_PATH[kind])
90
+
91
+    if 'authority' in kind.lower():
92
+        filename_structure = '%s_ca.yaml'
93
+    elif 'certificate' in kind.lower():
94
+        ca_name = cert_to_ca_map[name]
95
+        filename_structure = ca_name + '_%s_certificate.yaml'
96
+    elif 'public' in kind.lower() or 'private' in kind.lower():
97
+        filename_structure = '%s.yaml'
98
+
99
+    # Dashes in the document names are converted to underscores for
100
+    # consistency.
101
+    filename = (filename_structure % name).replace('-', '_')
102
+    fullpath = os.path.join(path, filename)
103
+
104
+    # Not all managed documents are generated. Only update path provenance
105
+    # information for those that are.
106
+    if wrapper_document['data'].get('generated'):
107
+        wrapper_document['data']['generated']['specifiedBy']['path'] = fullpath
108
+    return fullpath
109
+
110
+
111
+def _get_repo_url_and_rev():
112
+    repo_path_or_url = config.get_site_repo()
113
+    repo_url = git.repo_url(repo_path_or_url)
114
+    repo_rev = config.get_site_rev()
115
+    return repo_url, repo_rev

+ 10
- 0
pegleg/engine/exceptions.py View File

@@ -65,3 +65,13 @@ class GitConfigException(PeglegBaseException):
65 65
 class GitInvalidRepoException(PeglegBaseException):
66 66
     """Exception raised when an invalid repository is detected."""
67 67
     message = 'The repository path or URL is invalid: %(repo_path)s'
68
+
69
+
70
+#
71
+# PKI EXCEPTIONS
72
+#
73
+
74
+
75
+class IncompletePKIPairError(PeglegBaseException):
76
+    """Exception for incomplete private/public keypair."""
77
+    message = ("Incomplete keypair set %(kinds)s for name: %(name)s")

+ 15
- 7
pegleg/engine/repository.py View File

@@ -42,18 +42,19 @@ def _clean_temp_folders():
42 42
         shutil.rmtree(r, ignore_errors=True)
43 43
 
44 44
 
45
-def process_repositories(site_name):
45
+def process_repositories(site_name, overwrite_existing=False):
46 46
     """Process and setup all repositories including ensuring we are at the
47 47
     right revision based on the site's own site-definition.yaml file.
48 48
 
49 49
     :param site_name: Site name for which to clone relevant repos.
50
+    :param overwrite_existing: Whether to overwrite an existing directory
50 51
 
51 52
     """
52 53
 
53 54
     # Only tracks extra repositories - not the site (primary) repository.
54 55
     extra_repos = []
55 56
 
56
-    site_repo = process_site_repository()
57
+    site_repo = process_site_repository(overwrite_existing=overwrite_existing)
57 58
 
58 59
     # Retrieve extra repo data from site-definition.yaml files.
59 60
     site_data = util.definition.load_as_params(
@@ -94,7 +95,9 @@ def process_repositories(site_name):
94 95
                  "repo_username=%s, revision=%s", repo_alias, repo_url_or_path,
95 96
                  repo_key, repo_user, repo_revision)
96 97
 
97
-        temp_extra_repo = _process_repository(repo_url_or_path, repo_revision)
98
+        temp_extra_repo = _process_repository(
99
+            repo_url_or_path, repo_revision,
100
+            overwrite_existing=overwrite_existing)
98 101
         extra_repos.append(temp_extra_repo)
99 102
 
100 103
     # Overwrite the site repo and extra repos in the config because further
@@ -105,12 +108,13 @@ def process_repositories(site_name):
105 108
     config.set_extra_repo_list(extra_repos)
106 109
 
107 110
 
108
-def process_site_repository(update_config=False):
111
+def process_site_repository(update_config=False, overwrite_existing=False):
109 112
     """Process and setup site repository including ensuring we are at the right
110 113
     revision based on the site's own site-definition.yaml file.
111 114
 
112 115
     :param bool update_config: Whether to update Pegleg config with computed
113 116
         site repo path.
117
+    :param overwrite_existing: Whether to overwrite an existing directory
114 118
 
115 119
     """
116 120
 
@@ -122,8 +126,10 @@ def process_site_repository(update_config=False):
122 126
 
123 127
     repo_url_or_path, repo_revision = _extract_repo_url_and_revision(
124 128
         site_repo_or_path)
129
+    config.set_site_rev(repo_revision)
125 130
     repo_url_or_path = _format_url_with_repo_username(repo_url_or_path)
126
-    new_repo_path = _process_repository(repo_url_or_path, repo_revision)
131
+    new_repo_path = _process_repository(repo_url_or_path, repo_revision,
132
+                                        overwrite_existing=overwrite_existing)
127 133
 
128 134
     if update_config:
129 135
         # Overwrite the site repo in the config because further processing will
@@ -134,17 +140,19 @@ def process_site_repository(update_config=False):
134 140
     return new_repo_path
135 141
 
136 142
 
137
-def _process_repository(repo_url_or_path, repo_revision):
143
+def _process_repository(repo_url_or_path, repo_revision,
144
+                        overwrite_existing=False):
138 145
     """Process a repository located at ``repo_url_or_path``.
139 146
 
140 147
     :param str repo_url_or_path: Path to local repo or URL of remote URL.
141 148
     :param str repo_revision: branch, commit or ref in the repo to checkout.
149
+    :param overwrite_existing: Whether to overwrite an existing directory
142 150
 
143 151
     """
144 152
 
145 153
     global __REPO_FOLDERS
146 154
 
147
-    if os.path.exists(repo_url_or_path):
155
+    if os.path.exists(repo_url_or_path) and not overwrite_existing:
148 156
         repo_name = util.git.repo_name(repo_url_or_path)
149 157
         parent_temp_path = tempfile.mkdtemp()
150 158
         __REPO_FOLDERS.setdefault(repo_name, parent_temp_path)

+ 3
- 2
pegleg/engine/secrets.py View File

@@ -75,12 +75,13 @@ def decrypt(file_path, site_name):
75 75
     :type file_path: string
76 76
     :param site_name: The name of the site to search for the file.
77 77
     :type site_name: string
78
+    :return: The decrypted secrets
79
+    :rtype: list
78 80
     """
79
-
80 81
     LOG.info('Started decrypting...')
81 82
     if (os.path.isfile(file_path) and
82 83
             [s for s in file_path.split(os.path.sep) if s == site_name]):
83
-        PeglegSecretManagement(file_path).decrypt_secrets()
84
+        return PeglegSecretManagement(file_path).decrypt_secrets()
84 85
     else:
85 86
         LOG.info('File: {} was not found. Check your file path and name, '
86 87
                  'and try again.'.format(file_path))

+ 5
- 4
pegleg/engine/util/__init__.py View File

@@ -13,7 +13,8 @@
13 13
 # limitations under the License.
14 14
 
15 15
 # flake8: noqa
16
-from . import definition
17
-from . import files
18
-from . import deckhand
19
-from . import git
16
+from pegleg.engine.util import catalog
17
+from pegleg.engine.util import definition
18
+from pegleg.engine.util import deckhand
19
+from pegleg.engine.util import files
20
+from pegleg.engine.util import git

+ 52
- 0
pegleg/engine/util/catalog.py View File

@@ -0,0 +1,52 @@
1
+# Copyright 2018 AT&T Intellectual Property.  All other rights reserved.
2
+#
3
+# Licensed under the Apache License, Version 2.0 (the "License");
4
+# you may not use this file except in compliance with the License.
5
+# You may obtain a copy of the License at
6
+#
7
+#     http://www.apache.org/licenses/LICENSE-2.0
8
+#
9
+# Unless required by applicable law or agreed to in writing, software
10
+# distributed under the License is distributed on an "AS IS" BASIS,
11
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+# See the License for the specific language governing permissions and
13
+# limitations under the License.
14
+"""Utility functions for catalog files such as pki-catalog.yaml."""
15
+
16
+import logging
17
+
18
+from pegleg.engine.util import definition
19
+
20
+LOG = logging.getLogger(__name__)
21
+
22
+__all__ = ('iterate', )
23
+
24
+
25
+def iterate(kind, sitename=None, documents=None):
26
+    """Retrieve the list of catalog documents by catalog schema ``kind``.
27
+
28
+    :param str kind: The schema kind of the catalog. For example, for schema
29
+        ``pegleg/PKICatalog/v1`` kind should be "PKICatalog".
30
+    :param str sitename: (optional) Site name for retrieving documents.
31
+        Multually exclusive with ``documents``.
32
+    :param str documents: (optional) Documents to search through. Mutually
33
+        exclusive with ``sitename``.
34
+    :return: All catalog documents for ``kind``.
35
+    :rtype: generator[dict]
36
+
37
+    """
38
+
39
+    if not any([sitename, documents]):
40
+        raise ValueError('Either `sitename` or `documents` must be specified')
41
+
42
+    documents = documents or definition.documents_for_site(sitename)
43
+    for document in documents:
44
+        schema = document.get('schema')
45
+        # TODO(felipemonteiro): Remove 'promenade/%s/v1' once site manifest
46
+        # documents switch to new 'pegleg' namespace.
47
+        if schema == 'pegleg/%s/v1' % kind:
48
+            yield document
49
+        elif schema == 'promenade/%s/v1' % kind:
50
+            LOG.warning('The schema promenade/%s/v1 is deprecated. Use '
51
+                        'pegleg/%s/v1 instead.', kind, kind)
52
+            yield document

+ 2
- 2
pegleg/engine/util/deckhand.py View File

@@ -41,10 +41,10 @@ def load_schemas_from_docs(documents):
41 41
     return schema_set, errors
42 42
 
43 43
 
44
-def deckhand_render(documents=[],
44
+def deckhand_render(documents=None,
45 45
                     fail_on_missing_sub_src=False,
46 46
                     validate=False):
47
-
47
+    documents = documents or []
48 48
     errors = []
49 49
     rendered_documents = []
50 50
 

+ 1
- 0
pegleg/engine/util/definition.py View File

@@ -11,6 +11,7 @@
11 11
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 12
 # See the License for the specific language governing permissions and
13 13
 # limitations under the License.
14
+"""Utility functions for site-definition.yaml files."""
14 15
 
15 16
 import os
16 17
 

+ 35
- 12
pegleg/engine/util/git.py View File

@@ -26,7 +26,8 @@ from pegleg.engine import exceptions
26 26
 
27 27
 LOG = logging.getLogger(__name__)
28 28
 
29
-__all__ = ('git_handler', )
29
+__all__ = ('git_handler', 'is_repository', 'is_equal', 'repo_url', 'repo_name',
30
+           'normalize_repo_path')
30 31
 
31 32
 
32 33
 def git_handler(repo_url,
@@ -377,21 +378,26 @@ def is_equal(first_repo, other_repo):
377 378
         return False
378 379
 
379 380
 
380
-def repo_name(repo_path):
381
-    """Get the repository name for local repo at ``repo_path``.
381
+def repo_url(repo_url_or_path):
382
+    """Get the repository URL for the local or remote repo at
383
+    ``repo_url_or_path``.
382 384
 
383
-    :param repo_path: Path to local Git repo.
385
+    :param repo_url_or_path: URL of remote Git repo or path to local Git repo.
384 386
     :returns: Corresponding repo name.
385 387
     :rtype: str
386 388
     :raises GitConfigException: If the path is not a valid Git repo.
387 389
 
388 390
     """
389 391
 
390
-    if not is_repository(normalize_repo_path(repo_path)[0]):
391
-        raise exceptions.GitConfigException(repo_path=repo_path)
392
+    # If ``repo_url_or_path`` is already a URL, no point in checking.
393
+    if not os.path.exists(repo_url_or_path):
394
+        return repo_url_or_path
395
+
396
+    if not is_repository(normalize_repo_path(repo_url_or_path)[0]):
397
+        raise exceptions.GitConfigException(repo_url=repo_url_or_path)
392 398
 
393 399
     # TODO(felipemonteiro): Support this for remote URLs too?
394
-    repo = Repo(repo_path, search_parent_directories=True)
400
+    repo = Repo(repo_url_or_path, search_parent_directories=True)
395 401
     config_reader = repo.config_reader()
396 402
     section = 'remote "origin"'
397 403
     option = 'url'
@@ -408,9 +414,24 @@ def repo_name(repo_path):
408 414
                 else:
409 415
                     return repo_url.split('/')[-1]
410 416
         except Exception:
411
-            raise exceptions.GitConfigException(repo_path=repo_path)
417
+            raise exceptions.GitConfigException(repo_url=repo_url_or_path)
418
+
419
+    raise exceptions.GitConfigException(repo_url=repo_url_or_path)
420
+
421
+
422
+def repo_name(repo_url_or_path):
423
+    """Get the repository name for the local or remote repo at
424
+    ``repo_url_or_path``.
425
+
426
+    :param repo_url_or_path: URL of remote Git repo or path to local Git repo.
427
+    :returns: Corresponding repo name.
428
+    :rtype: str
429
+    :raises GitConfigException: If the path is not a valid Git repo.
430
+
431
+    """
412 432
 
413
-    raise exceptions.GitConfigException(repo_path=repo_path)
433
+    _repo_url = repo_url(repo_url_or_path)
434
+    return _repo_url.split('/')[-1].split('.git')[0]
414 435
 
415 436
 
416 437
 def normalize_repo_path(repo_url_or_path):
@@ -435,7 +456,7 @@ def normalize_repo_path(repo_url_or_path):
435 456
     """
436 457
 
437 458
     repo_url_or_path = repo_url_or_path.rstrip('/')
438
-    orig_repo_path = repo_url_or_path
459
+    orig_repo_url_or_path = repo_url_or_path
439 460
     sub_path = ""
440 461
     is_local_repo = os.path.exists(repo_url_or_path)
441 462
 
@@ -455,8 +476,10 @@ def normalize_repo_path(repo_url_or_path):
455 476
             repo_url_or_path = os.path.abspath(repo_url_or_path)
456 477
 
457 478
     if not repo_url_or_path or not is_repository(repo_url_or_path):
458
-        msg = "The repo_path=%s is not a valid Git repo" % (orig_repo_path)
479
+        msg = "The repo_path=%s is not a valid Git repo" % (
480
+            orig_repo_url_or_path)
459 481
         LOG.error(msg)
460
-        raise exceptions.GitInvalidRepoException(repo_path=repo_url_or_path)
482
+        raise exceptions.GitInvalidRepoException(
483
+            repo_path=orig_repo_url_or_path)
461 484
 
462 485
     return repo_url_or_path, sub_path

+ 4
- 4
pegleg/engine/util/pegleg_secret_management.py View File

@@ -15,7 +15,6 @@
15 15
 import logging
16 16
 import os
17 17
 import re
18
-import sys
19 18
 
20 19
 import click
21 20
 import yaml
@@ -130,9 +129,10 @@ class PeglegSecretManagement(object):
130 129
         included in a site secrets file, and print the result to the standard
131 130
         out."""
132 131
 
133
-        yaml.safe_dump_all(
134
-            self.get_decrypted_secrets(),
135
-            sys.stdout,
132
+        secrets = self.get_decrypted_secrets()
133
+
134
+        return yaml.safe_dump_all(
135
+            secrets,
136 136
             explicit_start=True,
137 137
             explicit_end=True,
138 138
             default_flow_style=False)

+ 44
- 0
pegleg/schemas/PKICatalog.yaml View File

@@ -0,0 +1,44 @@
1
+# TODO(felipemonteiro): Implement validation and use this.
2
+---
3
+schema: deckhand/DataSchema/v1
4
+metadata:
5
+  schema: metadata/Control/v1
6
+  name: pegleg/PKICatalog/v1
7
+  labels:
8
+    application: pegleg
9
+data:
10
+  $schema: http://json-schema.org/schema#
11
+  certificate_authorities:
12
+    type: array
13
+    items:
14
+      type: object
15
+      properties:
16
+        description:
17
+          type: string
18
+        certificates:
19
+          type: array
20
+          items:
21
+            type: object
22
+            properties:
23
+              document_name:
24
+                type: string
25
+              description:
26
+                type: string
27
+              common_name:
28
+                type: string
29
+              hosts:
30
+                type: array
31
+                items: string
32
+              groups:
33
+                type: array
34
+                items: string
35
+  keypairs:
36
+    type: array
37
+    items:
38
+      type: object
39
+      properties:
40
+        name:
41
+          type: string
42
+        description:
43
+          type: string
44
+...

+ 3
- 0
requirements.txt View File

@@ -3,5 +3,8 @@ click==6.7
3 3
 jsonschema==2.6.0
4 4
 pyyaml==3.12
5 5
 cryptography==2.3.1
6
+python-dateutil==2.7.3
7
+
8
+# External dependencies
6 9
 git+https://github.com/openstack/airship-deckhand.git@7d697012fcbd868b14670aa9cf895acfad5a7f8d
7 10
 git+https://github.com/openstack/airship-shipyard.git@44f7022df6438de541501c2fdd5c46df198b82bf#egg=shipyard_client&subdirectory=src/bin/shipyard_client

+ 23
- 0
site_yamls/site/pki-catalog.yaml View File

@@ -0,0 +1,23 @@
1
+# Basic example of pki-catalog.yaml for k8s.
2
+---
3
+schema: promenade/PKICatalog/v1
4
+metadata:
5
+  schema: metadata/Document/v1
6
+  name: cluster-certificates-addition
7
+  layeringDefinition:
8
+    abstract: false
9
+    layer: site
10
+  storagePolicy: cleartext
11
+data:
12
+  certificate_authorities:
13
+    kubernetes:
14
+      description: CA for Kubernetes components
15
+      certificates:
16
+        - document_name: kubelet-n3
17
+          common_name: system:node:n3
18
+          hosts:
19
+            - n3
20
+            - 192.168.77.13
21
+          groups:
22
+            - system:nodes
23
+...

+ 1
- 0
site_yamls/site/site-definition.yaml View File

@@ -1,3 +1,4 @@
1
+# TODO(felipemonteiro): Update `data` section below with new values.
1 2
 ---
2 3
 data:
3 4
   revision: v1.0

tests/unit/engine/test_encryption.py → tests/unit/engine/test_secrets.py View File

@@ -12,23 +12,29 @@
12 12
 # See the License for the specific language governing permissions and
13 13
 # limitations under the License.
14 14
 
15
-import click
16 15
 import os
17
-import tempfile
16
+from os import listdir
18 17
 
18
+import click
19 19
 import mock
20 20
 import pytest
21 21
 import yaml
22
+import tempfile
22 23
 
23
-from pegleg.engine.util import encryption as crypt
24
-from tests.unit import test_utils
24
+from pegleg import config
25
+from pegleg.engine import secrets
26
+from pegleg.engine.catalog import pki_utility
27
+from pegleg.engine.catalog.pki_generator import PKIGenerator
28
+from pegleg.engine.util import encryption as crypt, catalog, git
29
+from pegleg.engine.util import files
25 30
 from pegleg.engine.util.pegleg_managed_document import \
26 31
     PeglegManagedSecretsDocument
27
-from pegleg.engine.util.pegleg_secret_management import PeglegSecretManagement
28 32
 from pegleg.engine.util.pegleg_secret_management import ENV_PASSPHRASE
29 33
 from pegleg.engine.util.pegleg_secret_management import ENV_SALT
30
-from tests.unit.fixtures import temp_path
31
-from pegleg.engine.util import files
34
+from pegleg.engine.util.pegleg_secret_management import PeglegSecretManagement
35
+from tests.unit import test_utils
36
+from tests.unit.fixtures import temp_path, create_tmp_deployment_files, _gen_document
37
+from tests.unit.test_cli import TestSiteSecretsActions, BaseCLIActionTest, TEST_PARAMS
32 38
 
33 39
 TEST_DATA = """
34 40
 ---
@@ -69,6 +75,44 @@ def test_short_passphrase():
69 75
         PeglegSecretManagement('file_path')
70 76
 
71 77
 
78
+@mock.patch.dict(os.environ, {
79
+    ENV_PASSPHRASE: 'ytrr89erARAiPE34692iwUMvWqqBvC',
80
+    ENV_SALT: 'MySecretSalt'})
81
+def test_secret_encrypt_and_decrypt(create_tmp_deployment_files, tmpdir):
82
+    site_dir = tmpdir.join("deployment_files", "site", "cicd")
83
+    passphrase_doc = """---
84
+schema: deckhand/Passphrase/v1
85
+metadata:
86
+  schema: metadata/Document/v1
87
+  name: {0}
88
+  storagePolicy: {1}
89
+  layeringDefinition:
90
+    abstract: False
91
+    layer: {2}
92
+data: {0}-password
93
+...
94
+""".format("cicd-passphrase-encrypted", "encrypted",
95
+           "site")
96
+    with open(os.path.join(str(site_dir), 'secrets',
97
+                           'passphrases',
98
+                           'cicd-passphrase-encrypted.yaml'), "w") \
99
+            as outfile:
100
+        outfile.write(passphrase_doc)
101
+
102
+    save_location = tmpdir.mkdir("encrypted_files")
103
+    save_location_str = str(save_location)
104
+
105
+    secrets.encrypt(save_location_str, "pytest", "cicd")
106
+    encrypted_files = listdir(save_location_str)
107
+    assert len(encrypted_files) > 0
108
+
109
+    # for _file in encrypted_files:
110
+    decrypted = secrets.decrypt(str(save_location.join(
111
+        "site/cicd/secrets/passphrases/"
112
+        "cicd-passphrase-encrypted.yaml")), "cicd")
113
+    assert yaml.load(decrypted) == yaml.load(passphrase_doc)
114
+
115
+
72 116
 def test_pegleg_secret_management_constructor():
73 117
     test_data = yaml.load(TEST_DATA)
74 118
     doc = PeglegManagedSecretsDocument(test_data)
@@ -141,3 +185,52 @@ def test_encrypt_decrypt_using_docs(temp_path):
141 185
         'name']
142 186
     assert test_data[0]['metadata']['storagePolicy'] == decrypted_data[0][
143 187
         'metadata']['storagePolicy']
188
+
189
+
190
+@pytest.mark.skipif(
191
+    not pki_utility.PKIUtility.cfssl_exists(),
192
+    reason='cfssl must be installed to execute these tests')
193
+def test_generate_pki_using_local_repo_path(create_tmp_deployment_files):
194
+    """Validates ``generate-pki`` action using local repo path."""
195
+    # Scenario:
196
+    #
197
+    # 1) Generate PKI using local repo path
198
+
199
+    repo_path = str(git.git_handler(TEST_PARAMS["repo_url"],
200
+                                    ref=TEST_PARAMS["repo_rev"]))
201
+    with mock.patch.dict(config.GLOBAL_CONTEXT, {"site_repo": repo_path}):
202
+        pki_generator = PKIGenerator(sitename=TEST_PARAMS["site_name"])
203
+        generated_files = pki_generator.generate()
204
+
205
+        assert len(generated_files), 'No secrets were generated'
206
+        for generated_file in generated_files:
207
+            with open(generated_file, 'r') as f:
208
+                result = yaml.safe_load_all(f)  # Validate valid YAML.
209
+                assert list(result), "%s file is empty" % generated_file.name
210
+
211
+
212
+@pytest.mark.skipif(
213
+    not pki_utility.PKIUtility.cfssl_exists(),
214
+    reason='cfssl must be installed to execute these tests')
215
+def test_check_expiry(create_tmp_deployment_files):
216
+    """ Validates check_expiry """
217
+    repo_path = str(git.git_handler(TEST_PARAMS["repo_url"],
218
+                                    ref=TEST_PARAMS["repo_rev"]))
219
+    with mock.patch.dict(config.GLOBAL_CONTEXT, {"site_repo": repo_path}):
220
+        pki_generator = PKIGenerator(sitename=TEST_PARAMS["site_name"])
221
+        generated_files = pki_generator.generate()
222
+
223
+        pki_util = pki_utility.PKIUtility()
224
+
225
+        assert len(generated_files), 'No secrets were generated'
226
+        for generated_file in generated_files:
227
+            if "certificate" not in generated_file:
228
+                continue
229
+            with open(generated_file, 'r') as f:
230
+                results = yaml.safe_load_all(f)  # Validate valid YAML.
231
+                for result in results:
232
+                    if result['data']['managedDocument']['schema'] == \
233
+                            "deckhand/Certificate/v1":
234
+                        cert = result['data']['managedDocument']['data']
235
+                        assert not pki_util.check_expiry(cert), \
236
+                            "%s is expired!" % generated_file.name

+ 4
- 2
tests/unit/fixtures.py View File

@@ -30,7 +30,7 @@ schema: deckhand/Passphrase/v1
30 30
 metadata:
31 31
   schema: metadata/Document/v1
32 32
   name: %(name)s
33
-  storagePolicy: cleartext
33
+  storagePolicy: %(storagePolicy)s
34 34
   layeringDefinition:
35 35
     abstract: False
36 36
     layer: %(layer)s
@@ -40,6 +40,8 @@ data: %(name)s-password
40 40
 
41 41
 
42 42
 def _gen_document(**kwargs):
43
+    if "storagePolicy" not in kwargs:
44
+        kwargs["storagePolicy"] = "cleartext"
43 45
     test_document = TEST_DOCUMENT % kwargs
44 46
     return yaml.load(test_document)
45 47
 
@@ -154,7 +156,7 @@ schema: pegleg/SiteDefinition/v1
154 156
         cicd_path = os.path.join(str(p), files._site_path(site))
155 157
         files._create_tree(cicd_path, tree=test_structure)
156 158
 
157
-    yield
159
+    yield tmpdir
158 160
 
159 161
 
160 162
 @pytest.fixture()

+ 105
- 6
tests/unit/test_cli.py View File

@@ -19,14 +19,25 @@ from click.testing import CliRunner
19 19
 from mock import ANY
20 20
 import mock
21 21
 import pytest
22
+import yaml
22 23
 
23 24
 from pegleg import cli
25
+from pegleg.engine.catalog import pki_utility
24 26
 from pegleg.engine import errorcodes
25 27
 from pegleg.engine.util import git
26 28
 from tests.unit import test_utils
27 29
 from tests.unit.fixtures import temp_path
28 30
 
29 31
 
32
+TEST_PARAMS = {
33
+    "site_name": "airship-seaworthy",
34
+    "site_type": "foundry",
35
+    "repo_rev": '6b183e148b9bb7ba6f75c98dd13451088255c60b',
36
+    "repo_name": "airship-treasuremap",
37
+    "repo_url": "https://github.com/openstack/airship-treasuremap.git",
38
+}
39
+
40
+
30 41
 @pytest.mark.skipif(
31 42
     not test_utils.is_connected(),
32 43
     reason='git clone requires network connectivity.')
@@ -50,13 +61,13 @@ class BaseCLIActionTest(object):
50 61
         cls.runner = CliRunner()
51 62
 
52 63
         # Pin so we know that airship-seaworthy is a valid site.
53
-        cls.site_name = "airship-seaworthy"
54
-        cls.site_type = "foundry"
64
+        cls.site_name = TEST_PARAMS["site_name"]
65
+        cls.site_type = TEST_PARAMS["site_type"]
55 66
 
56
-        cls.repo_rev = '6b183e148b9bb7ba6f75c98dd13451088255c60b'
57
-        cls.repo_name = "airship-treasuremap"
58
-        repo_url = "https://github.com/openstack/%s.git" % cls.repo_name
59
-        cls.treasuremap_path = git.git_handler(repo_url, ref=cls.repo_rev)
67
+        cls.repo_rev = TEST_PARAMS["repo_rev"]
68
+        cls.repo_name = TEST_PARAMS["repo_name"]
69
+        cls.treasuremap_path = git.git_handler(TEST_PARAMS["repo_url"],
70
+                                                  ref=TEST_PARAMS["repo_rev"])
60 71
 
61 72
 
62 73
 class TestSiteCLIOptions(BaseCLIActionTest):
@@ -428,6 +439,94 @@ class TestRepoCliActions(BaseCLIActionTest):
428 439
         assert not result.output
429 440
 
430 441
 
442
+class TestSiteSecretsActions(BaseCLIActionTest):
443
+    """Tests site secrets-related CLI actions."""
444
+
445
+    def _validate_generate_pki_action(self, result):
446
+        assert result.exit_code == 0
447
+
448
+        generated_files = []
449
+        output_lines = result.output.split("\n")
450
+        for line in output_lines:
451
+            if self.repo_name in line:
452
+                generated_files.append(line)
453
+
454
+        assert len(generated_files), 'No secrets were generated'
455
+        for generated_file in generated_files:
456
+            with open(generated_file, 'r') as f:
457
+                result = yaml.safe_load_all(f)  # Validate valid YAML.
458
+                assert list(result), "%s file is empty" % filename
459
+
460
+    @pytest.mark.skipif(
461
+        not pki_utility.PKIUtility.cfssl_exists(),
462
+        reason='cfssl must be installed to execute these tests')
463
+    def test_site_secrets_generate_pki_using_remote_repo_url(self):
464
+        """Validates ``generate-pki`` action using remote repo URL."""
465
+        # Scenario:
466
+        #
467
+        # 1) Generate PKI using remote repo URL
468
+
469
+        repo_url = 'https://github.com/openstack/%s@%s' % (self.repo_name,
470
+                                                           self.repo_rev)
471
+
472
+        secrets_opts = ['secrets', 'generate-pki', self.site_name]
473
+
474
+        result = self.runner.invoke(cli.site, ['-r', repo_url] + secrets_opts)
475
+        self._validate_generate_pki_action(result)
476
+
477
+    @pytest.mark.skipif(
478
+        not pki_utility.PKIUtility.cfssl_exists(),
479
+        reason='cfssl must be installed to execute these tests')
480
+    def test_site_secrets_generate_pki_using_local_repo_path(self):
481
+        """Validates ``generate-pki`` action using local repo path."""
482
+        # Scenario:
483
+        #
484
+        # 1) Generate PKI using local repo path
485
+
486
+        repo_path = self.treasuremap_path
487
+        secrets_opts = ['secrets', 'generate-pki', self.site_name]
488
+
489
+        result = self.runner.invoke(cli.site, ['-r', repo_path] + secrets_opts)
490
+        self._validate_generate_pki_action(result)
491
+
492
+    @pytest.mark.skipif(
493
+        not pki_utility.PKIUtility.cfssl_exists(),
494
+        reason='cfssl must be installed to execute these tests')
495
+    @mock.patch.dict(os.environ, {
496
+            "PEGLEG_PASSPHRASE": "123456789012345678901234567890",
497
+            "PEGLEG_SALT": "123456"
498
+        })
499
+    def test_site_secrets_encrypt_local_repo_path(self):
500
+        """Validates ``generate-pki`` action using local repo path."""
501
+        # Scenario:
502
+        #
503
+        # 1) Encrypt a file in a local repo
504
+
505
+        repo_path = self.treasuremap_path
506
+        with open(os.path.join(repo_path, "site", "airship-seaworthy",
507
+                               "secrets", "passphrases", "ceph_fsid.yaml"), "r") \
508
+                as ceph_fsid_fi:
509
+            ceph_fsid = yaml.load(ceph_fsid_fi)
510
+            ceph_fsid["metadata"]["storagePolicy"] = "encrypted"
511
+
512
+        with open(os.path.join(repo_path, "site", "airship-seaworthy",
513
+                               "secrets", "passphrases", "ceph_fsid.yaml"), "w") \
514
+                as ceph_fsid_fi:
515
+            yaml.dump(ceph_fsid, ceph_fsid_fi)
516
+
517
+        secrets_opts = ['secrets', 'encrypt', '-a', 'test', self.site_name]
518
+        result = self.runner.invoke(cli.site, ['-r', repo_path] + secrets_opts)
519
+
520
+        assert result.exit_code == 0
521
+
522
+        with open(os.path.join(repo_path, "site", "airship-seaworthy",
523
+                               "secrets", "passphrases", "ceph_fsid.yaml"), "r") \
524
+                as ceph_fsid_fi:
525
+            ceph_fsid = yaml.load(ceph_fsid_fi)
526
+            assert "encrypted" in ceph_fsid["data"]
527
+            assert "managedDocument" in ceph_fsid["data"]
528
+
529
+
431 530
 class TestTypeCliActions(BaseCLIActionTest):
432 531
     """Tests type-level CLI actions."""
433 532
 

+ 23
- 0
tools/gate/playbooks/install-cfssl.yaml View File

@@ -0,0 +1,23 @@
1
+# Copyright 2018 AT&T Intellectual Property.  All other rights reserved.
2
+#
3
+# Licensed under the Apache License, Version 2.0 (the "License");
4
+# you may not use this file except in compliance with the License.
5
+# You may obtain a copy of the License at
6
+#
7
+#     http://www.apache.org/licenses/LICENSE-2.0
8
+#
9
+# Unless required by applicable law or agreed to in writing, software
10
+# distributed under the License is distributed on an "AS IS" BASIS,
11
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+# See the License for the specific language governing permissions and
13
+# limitations under the License.
14
+
15
+- hosts: all
16
+  gather_facts: False
17
+  tasks:
18
+    - name: Install cfssl for Ubuntu
19
+      shell: |-
20
+        ./tools/install-cfssl.sh
21
+      become: yes
22
+      args:
23
+          chdir: "{{ zuul.project.src_dir }}"

+ 1
- 0
tools/gate/whitespace-linter.sh View File

@@ -8,6 +8,7 @@ RES=$(find . \
8 8
   -not -path "*/htmlcov/*" \
9 9
   -not -name "*.tgz" \
10 10
   -not -name "*.pyc" \
11
+  -not -name "*.html" \
11 12
   -type f -exec egrep -l " +$" {} \;)
12 13
 
13 14
 if [[ -n $RES ]]; then

+ 22
- 0
tools/install-cfssl.sh View File

@@ -0,0 +1,22 @@
1
+#!/usr/bin/env bash
2
+
3
+set -ex
4
+
5
+if [ $# -eq 1 ]; then
6
+  CFSSLURL=$1
7
+else
8
+  CFSSLURL=${CFSSLURL:="http://pkg.cfssl.org/R1.2/cfssl_linux-amd64"}
9
+fi
10
+
11
+if [ -z $(which cfssl) ]; then
12
+  if [ $(whoami) == "root" ]; then
13
+    curl -Lo /usr/local/bin/cfssl ${CFSSLURL}
14
+    chmod 555 /usr/local/bin/cfssl
15
+  else
16
+    if [ ! -d ~/.local/bin ]; then
17
+      mkdir -p ~/.local/bin
18
+    fi
19
+    curl -Lo ~/.local/bin/cfssl ${CFSSLURL}
20
+    chmod 555 ~/.local/bin/cfssl
21
+  fi
22
+fi

+ 6
- 1
tox.ini View File

@@ -57,7 +57,12 @@ deps =
57 57
   -r{toxinidir}/requirements.txt
58 58
   -r{toxinidir}/test-requirements.txt
59 59
 commands =
60
-  pytest --cov=pegleg --cov-report html:cover --cov-report xml:cover/coverage.xml --cov-report term --cov-fail-under 84 tests/
60
+  {toxinidir}/tools/install-cfssl.sh
61
+  bash -c 'PATH=$PATH:~/.local/bin; pytest --cov=pegleg --cov-report \
62
+      html:cover --cov-report xml:cover/coverage.xml --cov-report term \
63
+      --cov-fail-under 84 tests/'
64
+whitelist_externals =
65
+  bash
61 66
 
62 67
 [testenv:releasenotes]
63 68
 basepython = python3

Loading…
Cancel
Save