Merge "Adding a place for external secrets to be stored on site level"

docs/source/secrets-guidelines.md

@@ -12,7 +12,7 @@ Airshipctl uses kustomize along with different krm-functions that extend its fun
 Replacement krm-function that is needed to avoid duplication of data in documents
 Templater krm-function that is needed to produce new yaml documents based on the provided parameters.
 There is a standard catalog of [krm-functions](https://github.com/GoogleContainerTools/kpt-functions-catalog).
-It includes the standard krm-function: `gcr.io/kpt-functions/sops` that can be used to perform decryption and encryption right in kustomize. Please refer to the example configurations that can be used to encrypt and decrypt the set of [existing yamls](https://github.com/GoogleContainerTools/kpt-functions-catalog/blob/master/examples/sops/local-configs/function.yaml).
+It includes the standard krm-function: `gcr.io/kpt-fn-contrib/sops` that can be used to perform decryption and encryption right in kustomize. Please refer to the example configurations that can be used to encrypt and decrypt the set of [existing yamls](https://github.com/GoogleContainerTools/kpt-functions-catalog/blob/master/examples/contrib/sops/function.yaml).
 
 Please note that to make that krm-function work it’s necessary to provide the following ENV variables:
 
@@ -57,9 +57,11 @@ In order to implement all that functionality it was necessary to introduce a new
 
 Krm-functions accept a set of yamls and config as input and return a modified set of yamls.
 GenericContainer executor may just output it to stdout. Or it may store it as `kpt fn sink` does.
-In particular we’re using the second option to store our generated and encrypted yamls to the specific place from which other manifests will take [that file](manifests/site/test-site/target/generator/results/generated/secrets.yaml).
+In particular we’re using the second option to store our generated and encrypted yamls to the specific place from which other manifests will take [that file](manifests/site/test-site/target/encrypted/results/generated/secrets.yaml).
 
-As it’s possible to see [generator kustomization](manifests/site/test-site/target/generator/results/kustomization.yaml) performs decryption using sops krm-function.
+There is a way to provide external secrets, that shouldn't be generated. That secrets must be stored in encrypted way in [another file](manifests/site/test-site/target/encrypted/results/imported/secrets.yaml).
+
+As it’s possible to see [encrypted kustomization](manifests/site/test-site/target/encrypted/results/kustomization.yaml) performs decryption using sops krm-function.
 
 # Step-by-step Operator instructions
 
@@ -130,7 +132,7 @@ We hope that it won’t be necessary to do these actions manually, because airsh
 sops <file name>
 ```
 
-This will decrypt the file and will open it in the editor. It will be possible to perform needed modifications. Once finished just close the editor and sops will encrypt the modified document and put it back. This may be a really-really useful command for some users and very simple at the same time.
+This will decrypt the file and will open it in the editor. It will be possible to perform needed modifications. Once finished just close the editor and sops will encrypt the modified document and put it back. This may be a really-really useful command for some users and very simple at the same time. This approach may be used in order to modify [imported secrets](manifests/site/test-site/target/encrypted/results/imported/secrets.yaml).
 
 ## Generation/Regeneration and encryption of secrets in manifests
 
@@ -167,8 +169,6 @@ This config file defines the following structure of VariableCatalogue:
     labels:
       airshipit.org/deploy-k8s: "false"
     name: generated-secrets
-    annotations:
-      config.kubernetes.io/path: secrets.yaml
   ephemeralClusterCa:...
   ephemeralKubeconfig:..
   targetClusterCa:...
@@ -180,6 +180,8 @@ Please pay attention to the annotation `config.kubernetes.io/path` - it defines
 
 When this template is executed it generates keys/certs/passwords and renders them as a Variable catalog with the name `generated-secrets`.
 
+Please pay attention that the special annotation `config.kubernetes.io/path` is getting added in the fileplacement transformer - it defines the name of the file where this document will be stored by phase. It’s possible to define several VariableCatalogues with unique names of files (it even may contain directories).
+
 Now if we refer back to the Phase descritption we’ll see that it’s type is GenericContainer with the name `encrypter`.
 
 The definition of that executor is the following:
@@ -192,7 +194,7 @@ metadata:
   labels:
     airshipit.org/deploy-k8s: "false"
 spec:
-  sinkOutputDir: "target/generator/results/generated"
+  sinkOutputDir: "target/generator/results"
   image: gcr.io/kpt-fn-contrib/sops:v0.1.0
   envVars:
   - SOPS_IMPORT_PGP
@@ -205,28 +207,29 @@ config: |
     unencrypted-regex: '^(kind|apiVersion|group|metadata)$'
 ```
 
-Basically this executor accepts the bundle, runs krm-function `gcr.io/kpt-fn-contrib/sops:v0.1.0` with configuration from `config` field and stores the result to the directory `target/generator/results/generated` based on the filenames/hierarchy defined by annotation `config.kubernetes.io/path`. Sops krm-function in its turn encrypts documents and that means that `target/generator/results/generated` will contain encrypted yamls. To make that work the user will need just to specify 2 environment variables:
+Basically this executor accepts the bundle, runs krm-function `gcr.io/kpt-fn-contrib/sops:v0.1.0` with configuration from `config` field and stores the result to the directory `target/generator/results` based on the filenames/hierarchy defined by annotation `config.kubernetes.io/path`. Sops krm-function in its turn encrypts documents and that means that `target/generator/results/` will contain encrypted yamls. To make that work the user will need just to specify 2 environment variables:
 
 - `SOPS_IMPORT_PGP`
 - `SOPS_PGP_FP`
 
-Possible option how to encrypt `externally provided secrets`
+Possible option how to encrypt `externally provided secrets`:
-First of all - it’s possible to make as many phases as needed, each phase will cover its separate procedure, e.g.: change of LDAP credentials, update some external passwords.
+This feature is already in place - it's possible to update improted secrtets manually.
+Futher possible improvements are to make as many phases as needed, each phase will cover its separate procedure, e.g.: change of LDAP credentials, update some external passwords.
 The only limitation is that each procedure has to have it’s own VariableCatalogues - that just allows not to decrypte/re-encrypt values from all VariableCatalogues.
 
-For `Externally provided secrets` we should use some unencrypted VariableCatalogue as a resource instead of templater. The encryption executor configuration will look the same.
+We should use some unencrypted VariableCatalogue as a resource and be able to encrypt that and put to imported secrets.
 
 Moreover, it’s possible to combine several secret sources in 1 phase, e.g. if we need to encrypt generated and externally provided secrets, just create another directory with kustomization, and put there different resources:
 
 1. Local files with `externally provided secrets` in form of unencrypted variable catalogues
-2. Directory `target/generator`.
+2. Directory `target/encrypted`.
 
 Update phase’s documentEntryPoint with the new path to the created directory. Now when you run the phase - all these files along with newly generated secrets will be encrypted.
 
 ## Decryption of secrets and using them
 
 The current implementation of manifests doesn’t require explicit decryption of files. All secrets are decrypted on the spot. Here are the details of how it was achieved:
-All encrypted documents are listed in the [following kustomization file](https://github.com/airshipit/airshipctl/blob/master/manifests/site/test-site/target/generator/results/kustomization.yaml).
+All encrypted documents are listed in the [following kustomization file](https://github.com/airshipit/airshipctl/blob/master/manifests/site/test-site/target/encrypted/results/kustomization.yaml).
 This kustomization file performs decryption by invoking `decrypt-secrets` transformer, that is just a sops krm-function configuration that decrypts all encrypted documents.
 Note: we made a special kustomization for decrypt-secrets configuration just to be able to modify it a bit depending on the environment variable `TOLERATE_DECRYPTION_FAILURES` value. If it’s true we’re adding parameter `cmd-tolerate-failures: true` to sops configuration.
 

manifests/function/generate-secrets-example/README.md

@@ -14,7 +14,7 @@ mark it with `deploy-k8s: true` annotation.
 Make a copy of this folder to the appropraite site for which secrets has to
 be generated and then edit the [secret-generation.yaml](secret-generation.yaml)
 with the required secret generation details.
-For example refer to [generator](../../site/test-site/target/generator/) folder.
+For example refer to [generator](../../site/test-site/target/encrypted/) folder.
 
 Once the secret definitions are in place in the site manifests, we can
 add a new phase to generate secrets pointing to the folder in site manifests.
@@ -30,7 +30,7 @@ config:
     apiVersion: airshipit.org/v1alpha1
     kind: GenericContainer
     name: encrypter
-  documentEntryPoint: target/generator
+  documentEntryPoint: target/encrypted
 ```
 
 The executorRef is of `kind: GenericContainer` and should also have the
@@ -44,7 +44,7 @@ metadata:
   name: encrypter
   labels:
     airshipit.org/deploy-k8s: "false"
-kustomizeSinkOutputDir: "target/generator/results/generated"
+kustomizeSinkOutputDir: "target/encrypted/results/generated"
 spec:
   container:
     image: gcr.io/kpt-fn-contrib/sops:v0.1.0
@@ -98,7 +98,7 @@ metadata:
   name: encrypter
   labels:
     airshipit.org/deploy-k8s: "false"
-kustomizeSinkOutputDir: "target/generator/results/generated"
+kustomizeSinkOutputDir: "target/encrypted/results/generated"
 spec:
   container:
     image: quay.io/airshipit/templater:v2
@@ -109,7 +109,7 @@ config: |
 ## Decrypt to read the secrets
 
 To decrypt the secrets for readability purposes run the kustomize build
-command on the generated secrets folder with the [kustomization.yaml](../../site/test-site/target/generator/results/kustomization.yaml) and [decrypt-secrets.yaml](../../site/test-site/target/generator/results/decrypt-secrets.yaml)
+command on the generated secrets folder with the [kustomization.yaml](../../site/test-site/target/encrypted/results/kustomization.yaml) and [decrypt-secrets.yaml](../../site/test-site/target/encrypted/results/decrypt-secrets.yaml)
 files in place in the same folder.
 
 Kustomize command to decrypt:

manifests/function/hardwareprofile-example/cleanup/kustomization.yaml

@@ -1,2 +0,0 @@
-resources:
-- smp.yaml

manifests/function/hardwareprofile-example/cleanup/smp.yaml

@@ -1,11 +0,0 @@
-apiVersion: builtin
-kind: PatchStrategicMergeTransformer
-metadata:
-  name: smp-hwparofile
-patches: |-
-  ---
-  apiVersion: airshipit.org/v1alpha1
-  kind: VariableCatalogue
-  metadata:
-    name: hardwareprofile-example
-  $patch: delete

manifests/function/hostgenerator-m3/cleanup/kustomization.yaml

@@ -1,2 +0,0 @@
-resources:
-- smp.yaml

manifests/phases/executors.yaml

@@ -55,7 +55,7 @@ metadata:
     airshipit.org/deploy-k8s: "false"
 spec:
   type: krm
-  sinkOutputDir: "target/generator/results/generated"
+  sinkOutputDir: "target/encrypted/results"
   image: gcr.io/kpt-fn-contrib/sops:v0.1.0
   envVars:
     - SOPS_IMPORT_PGP

manifests/phases/phases.yaml

@@ -224,7 +224,7 @@ config:
     apiVersion: airshipit.org/v1alpha1
     kind: GenericContainer
     name: encrypter
-  documentEntryPoint: target/generator
+  documentEntryPoint: target/encrypted/generator
 ---
 apiVersion: airshipit.org/v1alpha1
 kind: Phase
@@ -235,7 +235,7 @@ config:
     apiVersion: airshipit.org/v1alpha1
     kind: GenericContainer
     name: decrypter
-  documentEntryPoint: target/generator/results
+  documentEntryPoint: target/encrypted/results
 ---
 apiVersion: airshipit.org/v1alpha1
 kind: Phase
@@ -246,7 +246,7 @@ config:
     apiVersion: airshipit.org/v1alpha1
     kind: GenericContainer
     name: encrypter
-  documentEntryPoint: target/generator/results
+  documentEntryPoint: target/encrypted/results
 ---
 apiVersion: airshipit.org/v1alpha1
 kind: Phase

manifests/site/test-site/target/catalogues/kustomization.yaml

@@ -4,7 +4,7 @@ kind: Kustomization
 resources:
   - ../../../../type/gating/shared/catalogues
   - hosts.yaml
-  - ../generator/results
+  - ../encrypted/results
 
 patchesStrategicMerge:
   - versions-airshipctl.yaml

manifests/site/test-site/target/encrypted/generator/overridegeneration/kustomization.yaml

@@ -0,0 +1,2 @@
+resources:
+- ../../../../../../type/gating/target/generator/

manifests/site/test-site/target/encrypted/generator/overrideplacement/kustomization.yaml

@@ -0,0 +1,2 @@
+resources:
+  - ../../../../../../type/gating/target/generator/fileplacement

manifests/site/test-site/target/encrypted/importer/overrideplacement/kustomization.yaml

@@ -0,0 +1,2 @@
+resources:
+  - ../../../../../../type/gating/target/importer/fileplacement

manifests/site/test-site/target/encrypted/results/imported/secrets.yaml

@@ -0,0 +1,33 @@
+apiVersion: airshipit.org/v1alpha1
+kind: VariableCatalogue
+metadata:
+    labels:
+        airshipit.org/deploy-k8s: "false"
+    name: imported-secrets
+dummySecrets: ENC[AES256_GCM,data:wksRVJ1SVPJ8wIcnVA00,iv:wt6FmbfFh+31g/pBcTTlerrwHoUoF8Hv3Cw9q//bSWs=,tag:PTidwzah8PiqAtGnYSa1+w==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age: []
+    lastmodified: "2021-04-14T16:28:51Z"
+    mac: ENC[AES256_GCM,data:sHiCLqMg7TU4eXgThM5q+0Jq67uWoDunk1AbTqXOCKUA9gBHtKflgfgxLvhz8am7pOGf/i8UikFJx5Gb/TiAyf4GGKsfFbKDXc+JwnMYbKoibRJ1cxfRKgcwXdCohcb1g4bSiX2iHmEaVKHlF5ydvfn1OMWR5hQpavSgrb8JemA=,iv:3fg3EgYQjaLCluTL9Yu1axyucAOWwH0SREQMyvMeuak=,tag:lhA5n06vB2adYiv+cGskuA==,type:str]
+    pgp:
+        - created_at: "2021-04-14T16:28:50Z"
+          enc: |-
+            -----BEGIN PGP MESSAGE-----
+
+            wcBMAyUpShfNkFB/AQgAXrMxHATnkcDVixx+LpHMRFZeEnJsnKhFMkYIC+fhtpJD
+            V73hTSSBhbFTlko81oBohS151qNrS1qtVbOhS5anlTOrgZnZ2Fmwt7YIxc68xbhO
+            1fX8XbDAU2NmtWSdmc9sFLCrnpKakHuJfq6PMii8Cga126K7Smt62XuyFWw7r1Z+
+            gyJXgIE2c1GNZmgKhhPgcVBZuyGr6x1Ml4t6qPqMTn1af64T4co+ujI0n1n3dXie
+            p/xCPD2QrI1DSMZgvJzPf8SVUD9jh70LUGJR4fY4U1kTazs/K1JwrPLSH/rWStUA
+            +MNtXkitYzQa4KH4Zr+3rEH6pC8ezgtntT99Wkj8g9LmAcyF+naWCh0Akaui+UNJ
+            21DterDBlRsuJWUgFB54Kw+rSh7T1E4XbZzGQrb86EtCx9+gtPaRJoGYBi98wWAR
+            MORhPC2ylZX46XzMj9DTfMN44rvitTcA
+            =mdwS
+            -----END PGP MESSAGE-----
+          fp: FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4
+    unencrypted_regex: ^(kind|apiVersion|group|metadata)$
+    version: 3.7.1

manifests/site/test-site/target/encrypted/results/kustomization.yaml

@@ -0,0 +1,8 @@
+resources:
+  - generated/secrets.yaml
+  - imported/secrets.yaml
+
+transformers:
+  - decrypt-secrets
+  - ../generator/overrideplacement
+  - ../importer/overrideplacement

manifests/site/test-site/target/generator/overridegeneration/kustomization.yaml

@@ -1,2 +0,0 @@
-resources:
-- ../../../../../type/gating/target/generator/

manifests/site/test-site/target/generator/overrideplacement/kustomization.yaml

@@ -1,2 +0,0 @@
-resources:
-- ../../../../../type/gating/target/generator/fileplacement

manifests/site/test-site/target/generator/results/kustomization.yaml

@@ -1,6 +0,0 @@
-resources:
-  - generated/secrets.yaml
-
-transformers:
-  - decrypt-secrets
-  - ../overrideplacement

manifests/type/gating/target/generator/fileplacement/filepaths.yaml

@@ -1,11 +1,11 @@
 apiVersion: builtin
 kind: PatchTransformer
 metadata:
-  name: filnames-patch
+  name: generated-filnames-patch
 patch: |
   apiVersion: airshipit.org/v1alpha1
   kind: VariableCatalogue
   metadata:
     name: generated-secrets
     annotations:
-      config.kubernetes.io/path: secrets.yaml
+      config.kubernetes.io/path: generated/secrets.yaml

manifests/type/gating/target/importer/cleanup/kustomization.yaml

@@ -0,0 +1,2 @@
+resources:
+- secret-cleanup.yaml

manifests/type/gating/target/importer/cleanup/secret-cleanup.yaml

@@ -1,11 +1,11 @@
 apiVersion: builtin
 kind: PatchStrategicMergeTransformer
 metadata:
-  name: smp-hgc
+  name: smp_cleanup_imported
 patches: |-
   ---
   apiVersion: airshipit.org/v1alpha1
   kind: VariableCatalogue
   metadata:
-    name: host-generation-catalogue
+    name: imported-secrets
   $patch: delete

manifests/type/gating/target/importer/fileplacement/filepaths.yaml

@@ -0,0 +1,11 @@
+apiVersion: builtin
+kind: PatchTransformer
+metadata:
+  name: imported-filnames-patch
+patch: |
+  apiVersion: airshipit.org/v1alpha1
+  kind: VariableCatalogue
+  metadata:
+    name: imported-secrets
+    annotations:
+      config.kubernetes.io/path: imported/secrets.yaml

manifests/type/gating/target/importer/fileplacement/kustomization.yaml

@@ -0,0 +1,2 @@
+resources:
+- filepaths.yaml

tools/deployment/23_generate_secrets.sh

@@ -48,13 +48,13 @@ if [[ -z "${decrypted1}" ]]; then
 fi
 
 #make sure that generated file has right FP
-grep "${SOPS_PGP_FP}" "${AIRSHIP_CONFIG_MANIFEST_DIRECTORY}/$(basename ${AIRSHIP_CONFIG_PHASE_REPO_URL})/manifests/site/$SITE/target/generator/results/generated/secrets.yaml"
+grep "${SOPS_PGP_FP}" "${AIRSHIP_CONFIG_MANIFEST_DIRECTORY}/$(basename ${AIRSHIP_CONFIG_PHASE_REPO_URL})/manifests/site/$SITE/target/encrypted/results/generated/secrets.yaml"
 
 #set new FP and reencrypt
 export SOPS_PGP_FP=${SOPS_PGP_FP_REENCRYPT}
 airshipctl phase run secret-reencrypt
 #make sure that generated file has right FP
-grep "${SOPS_PGP_FP}" "${AIRSHIP_CONFIG_MANIFEST_DIRECTORY}/$(basename ${AIRSHIP_CONFIG_PHASE_REPO_URL})/manifests/site/$SITE/target/generator/results/generated/secrets.yaml"
+grep "${SOPS_PGP_FP}" "${AIRSHIP_CONFIG_MANIFEST_DIRECTORY}/$(basename ${AIRSHIP_CONFIG_PHASE_REPO_URL})/manifests/site/$SITE/target/encrypted/results/generated/secrets.yaml"
 
 #make sure that decrypted valus stay the same
 decrypted2=$(airshipctl phase run secret-show)