SOPS improvements

Added 2 phases: 1. secret-reencrypt - This phase can be used to reecnrypt the existing secrets with new key. To do so SOPS_IMPORT_PGP must contain old public key and new private key (but may also include other data). SOPS_PGP_FP must contain fingerprint of new private key. 2. secret-show - This phase may be useful for some users that need to see what generated by secret-generate phase. Disabled SOPS debug by default. To enable it back run commands with env variable DEBUG_SOPS_GPG=true Change-Id: Id7fe13d6943d386577df25dba4aaa83e62e58980
2021-03-15 20:26:43 +00:00 · 2021-03-15 20:26:43 +00:00 · a7e332f9ec
commit a7e332f9ec
parent bfe23f4755
15 changed files with 210 additions and 9 deletions
--- a/docs/source/secrets-guidelines.md
+++ b/docs/source/secrets-guidelines.md
@ -25,7 +25,7 @@ To decrypt:

 The gating scripts set that env variables [here](https://github.com/airshipit/airshipctl/blob/master/playbooks/airshipctl-gate-runner.yaml#L17).

-## Templater krm-function use-cases overivew
+## Templater krm-function use-cases overview

 Templater krm-function allows users to call [Sprig functions](http://masterminds.github.io/sprig/). Sprig has a set of [functions that may generate random values, passwords, CAs, keys and certificates](http://masterminds.github.io/sprig/crypto.html). If it’s not possible to use the standard set of sprig functions for some important Airshipctl use-cases, it’s always possible to extend that set of functions: the latest version of templater krm-function introduces [extension library](https://github.com/airshipit/airshipctl/tree/master/pkg/document/plugin/templater/extlib) where this can be done. The set of already added functions can be found [here](https://github.com/airshipit/airshipctl/blob/master/pkg/document/plugin/templater/extlib/funcmap.go).

@ -236,8 +236,45 @@ And it’s possible to use their values as a source for replacement transformer.

 To get even more familiar with that approach and understand all details please refer to the [following commit] (https://github.com/airshipit/airshipctl/commit/a252b248bcc9be2c8aca6f544f99541dce5012a3).

+## Decryption and printing the generated secrets to the screen
+
+In some cases it may be necessary to see what was generated by the templater in unencrypted form. For example, new SSH-keys were generated and it's necessary to get
+the private in order to be able to login to the node. Since in general it maybe very useful another phase called `secret-show` has been introduced.
+It decrypts and prints out the generated secrets.
+
+## Master key rotation
+
+This procedure may be done in many different ways depending on the organizational processes.
+There are 2 different approaches that may be used:
+
+1. when we create a new key - all secrets are getting re-encrypted with that new key
+2. when we create a new key - we're using it for generation/encryption of new secrets, but the old one stays valid till the last secret encrypted with it is getting regenerated and encrypted with new one. That means that old and new keys are used for decryption in parallel during some 'overlap' period. This is be similar to the approach that [Sealed secrets project](https://github.com/bitnami-labs/sealed-secrets) selected.
+
+Both approaches are possible taking into account that fact that SOPS allows you to have several private keys to decrypt data and it selects the needed one automatically.
+
+Nevertheless for the sake of simplicity we're currently implemented the first approach in our manifests. There is a phase called `secret-reecnrypt` that allows to perform master key rotation.
+
+In order to do so please follow the following steps:
+
+1. generate new master key pair using, e.g. using gpg wizard:
+
+``` sh
+gpg --full-generate-key
+
+```
+Note: please make sure you know the fingerprint of the newly generated key.
+
+2. append the env variable `SOPS_IMPORT_PGP` with the new keypair (don't delete the prvious one at this step, because it's needed for decryption).
+3. set the env variable `SOPS_PGP_FP` to the value of the NEW private key fingerprint. That means that the new key will be used for encryption.
+4. run `airshipctl phase run secret-reecnrypt`. make sure it runs successfully.
+5. check that all encrypted files were updated and that pgp.fp field for all of them equal to the value you specified in `SOPS_PGP_FP`.
+6. now it's possible to delete the old master key from `SOPS_IMPORT_PGP`. Once done it's possible to run `airshipctl phase run secret-show` to ensure that the keys will be decrypted properly.
+8. commit the changes to the site manifests.
+
 # Troubleshooting typical cases

+Note: In order to make troubleshotting possible please set env variable `DEBUG_SOPS_GPG=true` to see all debug output.
+
 ## Validate keys fingerprints

 Sops function fails with the following typical output:
@ -322,6 +359,9 @@ It’s clear that the imported bundle with public and private key didn’t have

 ## Validate format of the encrypted message

+UPD:
+the root-cause of that behavior was identified [here](https://github.com/airshipit/airshipctl/issues/471).
+
 Here is another typical output:

 ```
--- a/manifests/phases/executors.yaml
+++ b/manifests/phases/executors.yaml
@ -55,6 +55,24 @@ config: |
    cmd: encrypt
    unencrypted-regex: '^(kind|apiVersion|group|metadata)$'
 ---
+apiVersion: airshipit.org/v1alpha1
+kind: GenericContainer
+metadata:
+  name: decrypter
+  labels:
+    airshipit.org/deploy-k8s: "false"
+spec:
+  type: krm
+  image: gcr.io/kpt-fn-contrib/sops:v0.1.0
+  envVars:
+  - SOPS_IMPORT_PGP
+  - SOPS_PGP_FP
+config: |
+  apiVersion: v1
+  kind: ConfigMap
+  data:
+    cmd: decrypt
+---
 # This executor launchs a bootstrap container, which creates
 # an Azure Kubernetes Service (AKS) cluster
 apiVersion: airshipit.org/v1alpha1
--- a/manifests/phases/phases.yaml
+++ b/manifests/phases/phases.yaml
@ -228,6 +228,28 @@ config:
 ---
 apiVersion: airshipit.org/v1alpha1
 kind: Phase
+metadata:
+  name: secret-show
+config:
+  executorRef:
+    apiVersion: airshipit.org/v1alpha1
+    kind: GenericContainer
+    name: decrypter
+  documentEntryPoint: target/generator/results
+---
+apiVersion: airshipit.org/v1alpha1
+kind: Phase
+metadata:
+  name: secret-reencrypt
+config:
+  executorRef:
+    apiVersion: airshipit.org/v1alpha1
+    kind: GenericContainer
+    name: encrypter
+  documentEntryPoint: target/generator/results
+---
+apiVersion: airshipit.org/v1alpha1
+kind: Phase
 metadata:
  name: remotedirect-ephemeral
 config:
--- a/manifests/site/test-site/target/generator/kustomization.yaml
+++ b/manifests/site/test-site/target/generator/kustomization.yaml
@ -1,2 +1,4 @@
 generators:
- override
+- overridegeneration
+transformers:
+- overrideplacement
--- a/manifests/site/test-site/target/generator/overridegeneration/kustomization.yaml
+++ b/manifests/site/test-site/target/generator/overridegeneration/kustomization.yaml
--- a/manifests/site/test-site/target/generator/overrideplacement/kustomization.yaml
+++ b/manifests/site/test-site/target/generator/overrideplacement/kustomization.yaml
@ -0,0 +1,2 @@
+resources:
+- ../../../../../type/gating/target/generator/fileplacement
--- a/manifests/site/test-site/target/generator/results/decrypt-secrets/configurable-decryption.yaml
+++ b/manifests/site/test-site/target/generator/results/decrypt-secrets/configurable-decryption.yaml
@ -8,8 +8,10 @@ metadata:
        image: quay.io/airshipit/templater:latest
        envs:
        - TOLERATE_DECRYPTION_FAILURES
+        - DEBUG_SOPS_GPG
 template: |
  {{- $tolerate := env "TOLERATE_DECRYPTION_FAILURES" }}
+  {{- $debug := env "DEBUG_SOPS_GPG" }}
  apiVersion: v1
  kind: ConfigMap
  metadata:
@ -26,3 +28,6 @@ template: |
    {{- if eq $tolerate "true" }}
    cmd-tolerate-failures: true
    {{- end }}
+    {{- if not (eq $debug "true")  }}
+    override-preexec-cmd: '[ "$SOPS_IMPORT_PGP" == "" ] || (echo "$SOPS_IMPORT_PGP" | gpg --import 2>/dev/null)'
+    {{- end }}
--- a/manifests/site/test-site/target/generator/results/kustomization.yaml
+++ b/manifests/site/test-site/target/generator/results/kustomization.yaml
@ -3,3 +3,4 @@ resources:

 transformers:
  - decrypt-secrets
+  - ../overrideplacement
--- a/manifests/type/gating/target/generator/fileplacement/filepaths.yaml
+++ b/manifests/type/gating/target/generator/fileplacement/filepaths.yaml
@ -0,0 +1,11 @@
+apiVersion: builtin
+kind: PatchTransformer
+metadata:
+  name: filnames-patch
+patch: |
+  apiVersion: airshipit.org/v1alpha1
+  kind: VariableCatalogue
+  metadata:
+    name: generated-secrets
+    annotations:
+      config.kubernetes.io/path: secrets.yaml
--- a/manifests/type/gating/target/generator/fileplacement/kustomization.yaml
+++ b/manifests/type/gating/target/generator/fileplacement/kustomization.yaml
@ -0,0 +1,2 @@
+resources:
+- filepaths.yaml
--- a/manifests/type/gating/target/generator/secret-template.yaml
+++ b/manifests/type/gating/target/generator/secret-template.yaml
@ -28,8 +28,6 @@ template: |
    labels:
      airshipit.org/deploy-k8s: "false"
    name: generated-secrets
-    annotations:
-      config.kubernetes.io/path: secrets.yaml
  {{- $ephemeralClusterCa := genCAEx .ephemeralCluster.ca.subj .ephemeralCluster.ca.validity }}
  {{- $ephemeralKubeconfigCert := genSignedCertEx .ephemeralCluster.kubeconfigCert.subj nil nil .ephemeralCluster.kubeconfigCert.validity $ephemeralClusterCa }}
  ephemeralClusterCa:
--- a/playbooks/airshipctl-gate-runner.yaml
+++ b/playbooks/airshipctl-gate-runner.yaml
@ -15,7 +15,8 @@
    - vars/test-config.yaml
  environment:
    SOPS_IMPORT_PGP: "{{ airship_config_pgp }}"
-    SOPS_PGP_FP: "{{ airship_config_pgp_fp }}"
+    SOPS_PGP_FP_ENCRYPT: "{{ airship_config_pgp_fp1 }}"
+    SOPS_PGP_FP_REENCRYPT: "{{ airship_config_pgp_fp2 }}"
    AZURE_SUBSCRIPTION_ID_B64: "UGxlYXNlLCBwcm92aWRlIHlvdXIgQXp1cmUgc3Vic2NyaXB0aW9uIGlkIGhlcmUK"
    AZURE_TENANT_ID_B64: "UGxlYXNlLCBwcm92aWRlIHlvdXIgQXp1cmUgdGVuYW50IGlkIGhlcmUK"
    AZURE_CLIENT_ID_B64: "UGxlYXNlLCBwcm92aWRlIHlvdXIgQXp1cmUgc2VydmljZSBwcmluY2lwYWwgaWQgaGVyZQo="
--- a/playbooks/vars/test-config.yaml
+++ b/playbooks/vars/test-config.yaml
@ -20,7 +20,8 @@ airship_config_site_path: manifests/site/test-site
 airship_config_ca_data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNU1USXlOakE0TWpneU5Gb1hEVEk1TVRJeU16QTRNamd5TkZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTTFSClM0d3lnajNpU0JBZjlCR0JUS1p5VTFwYmdDaGQ2WTdJektaZWRoakM2K3k1ZEJpWm81ZUx6Z2tEc2gzOC9YQ1MKenFPS2V5cE5RcDN5QVlLdmJKSHg3ODZxSFZZNjg1ZDVYVDNaOHNyVVRzVDR5WmNzZHAzV3lHdDM0eXYzNi9BSQoxK1NlUFErdU5JemN6bzNEdWhXR0ZoQjk3VjZwRitFUTBlVWN5bk05c2hkL3AwWVFzWDR1ZlhxaENENVpzZnZUCnBka3UvTWkyWnVGUldUUUtNeGpqczV3Z2RBWnBsNnN0L2ZkbmZwd1Q5cC9WTjRuaXJnMEsxOURTSFFJTHVrU2MKb013bXNBeDJrZmxITWhPazg5S3FpMEloL2cyczRFYTRvWURZemt0Y2JRZ24wd0lqZ2dmdnVzM3pRbEczN2lwYQo4cVRzS2VmVGdkUjhnZkJDNUZNQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFJek9BL00xWmRGUElzd2VoWjFuemJ0VFNURG4KRHMyVnhSV0VnclFFYzNSYmV3a1NkbTlBS3MwVGR0ZHdEbnBEL2tRYkNyS2xEeFF3RWg3NFZNSFZYYkFadDdsVwpCSm90T21xdXgxYThKYklDRTljR0FHRzFvS0g5R29jWERZY0JzOTA3ckxIdStpVzFnL0xVdG5hN1dSampqZnBLCnFGelFmOGdJUHZIM09BZ3B1RVVncUx5QU8ya0VnelZwTjZwQVJxSnZVRks2TUQ0YzFmMnlxWGxwNXhrN2dFSnIKUzQ4WmF6d0RmWUVmV3Jrdld1YWdvZ1M2SktvbjVEZ0Z1ZHhINXM2Snl6R3lPVnZ0eG1TY2FvOHNxaCs3UXkybgoyLzFVcU5ZK0hlN0x4d04rYkhwYkIxNUtIMTU5ZHNuS3BRbjRORG1jSTZrVnJ3MDVJMUg5ZGRBbGF0bz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
 airship_config_client_cert_data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQwRENDQXJnQ0ZFdFBveEZYSjVrVFNWTXQ0OVlqcHBQL3hCYnlNQTBHQ1NxR1NJYjNEUUVCQ3dVQU1CVXgKRXpBUkJnTlZCQU1UQ210MVltVnlibVYwWlhNd0hoY05NakF3TVRJME1Ua3hOVEV3V2hjTk1qa3hNakF5TVRreApOVEV3V2pBME1Sa3dGd1lEVlFRRERCQnJkV0psY201bGRHVnpMV0ZrYldsdU1SY3dGUVlEVlFRS0RBNXplWE4wClpXMDZiV0Z6ZEdWeWN6Q0NBaUl3RFFZSktvWklodmNOQVFFQkJRQURnZ0lQQURDQ0Fnb0NnZ0lCQU1iaFhUUmsKVjZiZXdsUjBhZlpBdTBGYWVsOXRtRThaSFEvaGtaSHhuTjc2bDZUUFltcGJvaDRvRjNGMFFqbzROS1o5NVRuWgo0OWNoV240eFJiZVlPU25EcDBpV0Qzd0pXUlZ5aVFvVUFyYTlNcHVPNkVFU1FpbFVGNXNxc0VXUVdVMjBETStBCkdxK1k0Z2c3eDJ1Q0hTdk1GUmkrNEw5RWlXR2xnRDIvb1hXUm5NWEswNExQajZPb3Vkb2Zid2RmT3J6dTBPVkUKUzR0eGtuS1BCY1BUU3YxMWVaWVhja0JEVjNPbExENEZ3dTB3NTcwcnczNzAraEpYdlZxd3Zjb2RjZjZEL1BXWQowamlnd2ppeUJuZ2dXYW04UVFjd1Nud3o0d05sV3hKOVMyWUJFb1ptdWxVUlFaWVk5ZXRBcEpBdFMzTjlUNlQ2ClovSlJRdEdhZDJmTldTYkxEck5qdU1OTGhBYWRMQnhJUHpBNXZWWk5aalJkdEMwU25pMlFUMTVpSFp4d1RxcjQKakRQQ0pYRXU3KytxcWpQVldUaUZLK3JqcVNhS1pqVWZVaUpHQkJWcm5RZkJENHNtRnNkTjB5cm9tYTZOYzRMNQpKS21RV1NHdmd1aG0zbW5sYjFRaVRZanVyZFJQRFNmdmwrQ0NHbnA1QkkvZ1pwMkF1SHMvNUpKVTJlc1ZvL0xsCkVPdHdSOXdXd3dXcTAvZjhXS3R4bVRrMTUyOUp2dFBGQXQweW1CVjhQbHZlYnVwYmJqeW5pL2xWbTJOYmV6dWUKeCtlMEpNbGtWWnFmYkRSS243SjZZSnJHWW1CUFV0QldoSVkzb1pJVTFEUXI4SUlIbkdmYlZoWlR5ME1IMkFCQQp1dlVQcUtSVk80UGkxRTF4OEE2eWVPeVRDcnB4L0pBazVyR2RBZ01CQUFFd0RRWUpLb1pJaHZjTkFRRUxCUUFECmdnRUJBSWNFM1BxZHZDTVBIMnJzMXJESk9ESHY3QWk4S01PVXZPRi90RjlqR2EvSFBJbkh3RlVFNEltbldQeDYKVUdBMlE1bjFsRDFGQlU0T0M4eElZc3VvS1VQVHk1T0t6SVNMNEZnL0lEcG54STlrTXlmNStMR043aG8rblJmawpCZkpJblVYb0tERW1neHZzSWFGd1h6bGtSTDJzL1lKYUZRRzE1Uis1YzFyckJmd2dJOFA5Tkd6aEM1cXhnSmovCm04K3hPMGhXUmJIYklrQ21NekRib2pCSWhaL00rb3VYR1doei9TakpodXhZTVBnek5MZkFGcy9PMTVaSjd3YXcKZ3ZoSGc3L2E5UzRvUCtEYytPa3VrMkV1MUZjL0E5WHpWMzc5aWhNWW5ub3RQMldWeFZ3b0ZZQUg0NUdQcDZsUApCQmwyNnkxc2JMbjl6aGZYUUJIMVpFN0EwZVE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
 airship_config_client_key_data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlKS1FJQkFBS0NBZ0VBeHVGZE5HUlhwdDdDVkhScDlrQzdRVnA2WDIyWVR4a2REK0dSa2ZHYzN2cVhwTTlpCmFsdWlIaWdYY1hSQ09qZzBwbjNsT2RuajF5RmFmakZGdDVnNUtjT25TSllQZkFsWkZYS0pDaFFDdHIweW00N28KUVJKQ0tWUVhteXF3UlpCWlRiUU16NEFhcjVqaUNEdkhhNElkSzh3VkdMN2d2MFNKWWFXQVBiK2hkWkdjeGNyVApncytQbzZpNTJoOXZCMTg2dk83UTVVUkxpM0dTY284Rnc5TksvWFY1bGhkeVFFTlhjNlVzUGdYQzdURG52U3ZECmZ2VDZFbGU5V3JDOXloMXgvb1A4OVpqU09LRENPTElHZUNCWnFieEJCekJLZkRQakEyVmJFbjFMWmdFU2htYTYKVlJGQmxoajE2MENra0MxTGMzMVBwUHBuOGxGQzBacDNaODFaSnNzT3MyTzR3MHVFQnAwc0hFZy9NRG05VmsxbQpORjIwTFJLZUxaQlBYbUlkbkhCT3F2aU1NOElsY1M3djc2cXFNOVZaT0lVcjZ1T3BKb3BtTlI5U0lrWUVGV3VkCkI4RVBpeVlXeDAzVEt1aVpybzF6Z3Zra3FaQlpJYStDNkdiZWFlVnZWQ0pOaU82dDFFOE5KKytYNElJYWVua0UKaitCbW5ZQzRlei9ra2xUWjZ4V2o4dVVRNjNCSDNCYkRCYXJUOS94WXEzR1pPVFhuYjBtKzA4VUMzVEtZRlh3KwpXOTV1Nmx0dVBLZUwrVldiWTF0N081N0g1N1FreVdSVm1wOXNORXFmc25wZ21zWmlZRTlTMEZhRWhqZWhraFRVCk5DdndnZ2VjWjl0V0ZsUExRd2ZZQUVDNjlRK29wRlU3ZytMVVRYSHdEcko0N0pNS3VuSDhrQ1Rtc1owQ0F3RUEKQVFLQ0FnQUJ2U1N3ZVpRZW5HSDhsUXY4SURMQzdvU1ZZd0xxNWlCUDdEdjJsN00wYStKNWlXcWwzV2s4ZEVOSQpOYWtDazAwNmkyMCtwVDROdW5mdEZJYzBoTHN6TjBlMkpjRzY1dVlGZnZ2ZHY3RUtZZnNZU3hhU3d4TWJBMlkxCmNCa2NjcGVsUzBhMVpieFYvck16T1RxVUlRNGFQTzJPU3RUeU55b3dWVjhhcXh0QlNPV2pBUlA2VjlBOHNSUDIKNlVGeVFnM2thdjRla3d0S0M5TW85MEVvcGlkSXNnYy9IYk5kQm5tMFJDUnY0bU1DNmVPTXp0NGx0UVNldG0rcwpaRkUwZkM5cjkwRjE4RUVlUjZHTEYxdGhIMzlKTWFFcjYrc3F6TlZXU1VPVGxNN2M5SE55QTJIcnJudnhVUVNOCmF3SkZWSEFOY1hJSjBqcW9icmR6MTdMbGtIRVFGczNLdjRlcDR3REJKMlF0eisxdUFvY1JoV3ZSaWJxWEQ3THgKVmpPdGRyT1h3ZFQxY2ZrKzZRc1RMWUFKR3ptdDdsY1M2QjNnYzJHWmNJWGwyNVlqTUQ1ZVhpa1dEc3hYWmt1UAorb3MzVGhxeGZIS25ITmxtYk9SSVpDMW92Q1NkSTRWZVpzalk0MUs5K0dNaXdXSk1kektpRkp3NlR2blRSUldTCkxod2EzUTlBVmMvTEg0SC9PbU9qWDc0QTNZSWwrRDFVUHd3VzAvMmw4S3BNM0VWZ21XalJMV1ZIRnBNTGJNSlcKZVZKd3dKUmF3bWZLdHZ6bU9KRHlhTXJJblhqTDMvSE1EaWtwU3JhRzFyTnc1SUozOXJZdEFIUUQ1L1VuZlRkSApLNXVjakVucTdPdDMyR1ozcHJvRTU1ZGFBY0hQbktuOGpYZ1ZKTUQyOWh5cEZvL2ZRUUtDQVFFQStBbjRoSDFFCm9GK3FlcWlvYXR3N2cwaVdQUDNCeklxOEZWbWtsRlZBYVF5U28wU2QxWFBybmErR0RFQVd0cHlsVjF5ZkZkR2oKSHc4YXU5NnpUZnRuNWZCRkQxWG1NTkNZeTcrM293V3ArK1NwYUMvMTYzN1dvb3lLRjBjVFNvcWEzZEVuRUtSSwp4TGF2a0lFUTI3OXRBNFVUK0dVK3pTb0NPUFBNNE1JS3poR0FDczZ1anRySzFNcXpwK0JhYldzRlBuN2J1bStVCkRHSFIrNCtab2tBL1Q2N2luYlRxZUwwVzJCNjRMckFURHpZL3Y4NlRGbW1aallEaHRKR1JIWVZUOU9XSXR0RVkKNnZtUDN0a1dOTWt0R2w4bTFiQ0FHQ1JlcGtycUhxWXNMWG5GQ2ZZSFFtOXNpaGgvM3JFVjZ1MUYxZCt0U3JFMgprU1ZVOHhVWDUwbHFNUUtDQVFFQXpVTjZaS0lRNldkT09FR3ZyMExRL1hVczI0bUczN3lGMjhJUDJEcWFBWWVzCnJza2xTdjdlSU9TZWV3MW1CRHVCRkl2bkZvcTVsRlA3cXhWcEIyWjNNSGlDMVNaclZSZjlQTjdCNGFzcmNyMCsKdDB2S0NXWFFIaTVQQXhucXdYb2E2N0Q1bnkwdnlvV0lVUXAyZEZMdkIwQmp0b3MvajJFaHpJZk5WMm1UOW15bgpWQXZOWEdtZnc4SVJCL1diMGkzQ3c0Wityb1l1dTJkRHo2UUwzUFVvN1hLS3ljZzR1UzU1eksvcWZPc09lYm5mCnpsd3ZqbGxNSitmVFFHNzMrQnpINE5IWGs2akZZQzU4eXBrdXd0cmJmYk1pSkZOWThyV1ptL01Nd1VDWlZDQ3kKeUlxQ3FHQVB6b2kyU05zSEtaTlJqN3ZZQ3dQQVd6TzFidjFGcC9hM0xRS0NBUUVBeG0zTGw4cFROVzF6QjgrWApkRzJkV3FpZU1FcmRXRklBcDUvZ1R4NW9lZUdxQ2QxaDJ4cHlldUtwZlhGaitsRVU0Ty9qQU9TRjk5bndqQzFjCkNsMit2Ni9ZdjZ6N2l6L0ZqUEpoNlpRbGFiT0RaeXMvTkZkelEvVGtvRHluRFRJWE5LOFc3blJRc0ZCcDRWT3YKZGUwTlBBeWhiazBvMFo3eXlqY1lSeEpVN0lnSmhCdldmOGcvRGI3ZnZNUjU4eUR6d0F4aW9pS1RNTmlzMFBBUAplMEtrbzQySUU1eGhHNWhDQjBHRUhTMlZBYzFuY0gzRkk5LzFETVAzVEtwTGltOVlQQW5JdG1CTzYrUWNtYTNYCjJ3QzZDV2ZudkhvSDc4aGd3KzRZbjg1V2QwYjhQN3pJRC9qdHZ3aGNlMzMxeDh4cjJ1Nm5ScUxBd1pzNCs0SjcKYmZkSWNRS0NBUUFDL2JlNzNheTNhZnoyenVZN2ZKTEZEcjhQbCtweU9qSU5LTC9JVzlwQXFYUjN1NUNpamlJNApnbnhZdUxKQzM0Y2JBSXJtaGpEOEcxa3dmZ2hneGpwNFoxa290LzJhYU5ZVTIvNGhScmhFWE1PY01pdUloWVpKCjJrem1jNnM3RklkdDVjOU5aWUFyeUZSYk1mYlY3UnQwbEppZllWb1V3Y3FYUzJkUG5jYzlNUW9qTEdUYXN1TlUKRy9EWmw5ZWtjV3hFSXlLWGNuY2QzZnhiK3p6OUJFbUxaRDduZjlacnhHU2IrZmhGeDdzWFJRRWc1YkQvdHdkbwpFWFcvbTU1YmJEZnhhNzFqZG5NaDJxdVEzRGlWT0ZFNGZMTERxcjlDRWlsaDMySFJNeHJJNGcwWTVRUFFaazMwCnFZTldmbktWUllOTHYrWC9DeGZ6ZkVacGpxRkVPRkVsQW9JQkFRQ0t6R2JGdmx6d1BaUmh4czd2VXYxOXlIUXAKQzFmR3gwb0tpRDFSNWZwWVBrT0VRQWVudEFKRHNyYVRsNy9rSDY5V09VbUQ1T3gxbWpyRFB0a1M4WnhXYlJXeApGYjJLK3JxYzRtcGFacGROV09OTkszK3RNZmsrb0FRcWUySU1JV253NUhmbVpjNE1QY0t0bkZQYlJTTkF0aktwCkQ2aG9oL3BXMmdjRFA0cVpNWVZvRW04MVZYZEZDUGhOYitNYnUvU3gyaFB4U0dXYTVGaTczeEtwWWp5M3BISlQKWFoyY2lHN0VNQ3NKZW9HS2FRdmNCY1kvNGlSRGFoV0hWcmlsSVhJQXJQdXdmVUIybzZCZFR0allHeU5sZ2NmeApxWEt4aXBTaEE2VlNienVnR3pkdEdNeEUyekRHVEkxOXFSQy96OUNEREM1ZTJTQUZqbEJUV0QyUHJjcU4KLS0tLS1FTkQgUlNBIFBSSVZBVEUgS0VZLS0tLS0K
-airship_config_pgp_fp: "FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4"
+airship_config_pgp_fp1: "FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4"
+airship_config_pgp_fp2: "D7229043384BCC60326C6FB9D8720D957C3D3074"
 airship_config_pgp: |-
  -----BEGIN PGP PRIVATE KEY BLOCK-----

@ -80,3 +81,37 @@ airship_config_pgp: |-
  gLk//M3qDixOxiurECkFrMvt/bDxEGpN5GVy550MmyUZQrkuqg==
  =VjGL
  -----END PGP PRIVATE KEY BLOCK-----
+  -----BEGIN PGP PRIVATE KEY BLOCK-----
+
+  lQHYBF1oQYgBBADPuVP6Jdk/J/TbNa9dXirp/zzwK18ZqNudNqQGN3H+2aSgxXwL
+  wlRfzy7rB3CU6Ewjzk9EVYeYztTIkGHL0JZ1CCTiBJArlHO0bHQQ7CPeKPkhIhkj
+  eA8yu9dcU77oYC2xbwgf43KYzfMKSGEybg+sBO+bH+Y6paJK54V2cuS3GwARAQAB
+  AAP+Jjf5BXtVP1OAr5xvCYS77JWzhpTUSIpS7dgR0br91GAC9DmhmyBEGeSqwz95
+  LUyYRbY9y1rZOfpEGCrIc5GLPOQytO9XMIzaS3dpzfGhla/spaKN4vJDvIOl+ruT
+  bInDdCRSmqXCfm2478OhOquc0H0a46eSmoaYeKdE3E8QZiECANxUL/dFk5j8NyPo
+  ZcwXw9Mv0A8UrynRcqht3Scti9k7dbsHylcObM305LFdcoNnSfNAIJhxfjbiXyGW
+  vwT2/qMCAPFatq3gvVjy6wKKylioi5cVwbLv9L+OaRXdR/Dy2bh/t3ujnsliV4+R
+  f7k3rHOQeaMLTnyfcz8AenL5IOe8RSkCANFpBgyzxCcV48Mm+FWDxjrSJ4/msRnN
+  gxqAPRrdpm7e1uebtBkPh4ch4oCW5/lLsRN23LUVIXYJRwyFfRjehCio0rRTU09Q
+  UyBGdW5jdGlvbmFsIFRlc3RzIEtleSAyIChodHRwczovL2dpdGh1Yi5jb20vbW96
+  aWxsYS9zb3BzLykgPHNlY29wc0Btb3ppbGxhLmNvbT6IzgQTAQgAOBYhBNcikEM4
+  S8xgMmxvudhyDZV8PTB0BQJdaEGIAhsDBQsJCAcCBhUKCQgLAgQWAgMBAh4BAheA
+  AAoJENhyDZV8PTB0R2cD/2YwaJ43iGueaAzByFnl+mUEBQJ4HhH4p7BIdx6B9AjE
+  3yLe8I4dqqYXxyZzaJ9d+KiqxJBT0l1GXt3H5M32yDJZqzXB9PTWP3yx8+Q1CuCs
+  7EL/bhJD1/sLdumVc77bmQtcI9NSiYyPzN/2ZqtV5RU14Loh24VFEjuHGvO0jI3+
+  nQHYBF1oQYgBBAChXi00fmpEs0Jiq0zOyYm9i749VoOsNReoB/5ix1QCimwVZKe1
+  D37IP5Qqysxy+LIQc4lJ+Q8foNOx1Aev5+TDyv+iU82D9xr9uPLLbA82k3AZ04Or
+  BjrZ/Yt1NZhuaHzciZCPpmqzF9kqVqAZc+vMiKZL1WZjS7O1FwaidY1vXwARAQAB
+  AAP+L0wUQeOfsD0+gv8khyPJTJZOD1pxQ6NYKLcXF8rG0+vQnECha098YKNKAXTp
+  kfVU8795iQYIKcQQ6Hl2O1fj1AxJE/iZYrqfm7UZz3bQ7ROSsAEPZ5GDOjKfbwsz
+  E6bWVH+PhS1azlvtTs9JezUtK0Wl9s+81FOrZtnUUskmWtECAMNNs9ujUt6GHv/J
+  NXVaSmk1z8QXitPHbAJLDMj4xVDysJWZV95eplC+RUSiLz5HeP2AQgh1D9Rv2bA5
+  c7OcJ3kCANOEkA0hVpXCI0FKrsihOf0NUOaAtS6CQNFlaIkrLwssJQY8pGYbRfRa
+  3krNJPyOlXmezV2/CsX3EqA9KXXen5cB/iSmMJO4WndGJTe7YzUEnnY/P2TKg1fN
+  s6v5Lf39j5Ll8V5rVDT7ApAw0IKS8fzpbdHP0HcizutlF6l44YaAXMGfhoi2BBgB
+  CAAgFiEE1yKQQzhLzGAybG+52HINlXw9MHQFAl1oQYgCGwwACgkQ2HINlXw9MHTD
+  HwQAv+ui718AT2hw2pK9JaNuTxjllrH+KPMlrov0P8oXHPCohC5cxM5sJ6tCQ0qH
+  XyeWoDE8V31btqFVAQyrr0wy0gntl1L/trnwMHoP8a/xa0RHNk5C7hmcuhTHbQey
+  JNbiRJZpCIZ1OyrF17+q6u9YBPjwqp8KrJ/0ryy2kyb7ZRM=
+  =+tJ6
+  -----END PGP PRIVATE KEY BLOCK-----
--- a/tools/deployment/23_generate_secrets.sh
+++ b/tools/deployment/23_generate_secrets.sh
@ -15,14 +15,43 @@
 set -xe

 echo "Generating secrets using airshipctl"
+export SOPS_PGP_FP=${SOPS_PGP_FP_ENCRYPT:-"${SOPS_PGP_FP}"}
 airshipctl phase run secret-generate

+echo "Generating ~/.airship/kubeconfig"
 export AIRSHIP_CONFIG_MANIFEST_DIRECTORY=${AIRSHIP_CONFIG_MANIFEST_DIRECTORY:-"/tmp/airship"}
 export AIRSHIP_CONFIG_PHASE_REPO_URL=${AIRSHIP_CONFIG_PHASE_REPO_URL:-"https://review.opendev.org/airship/airshipctl"}
 export EXTERNAL_KUBECONFIG=${EXTERNAL_KUBECONFIG:-""}
-
-echo "Generating ~/.airship/kubeconfig"
 if [[ -z "$EXTERNAL_KUBECONFIG" ]]; then
   # TODO: use airshipctl cluster get-kubeconfig command when it's implemented
   KUSTOMIZE_PLUGIN_HOME=./ kustomize build --enable_alpha_plugins "${AIRSHIP_CONFIG_MANIFEST_DIRECTORY}/$(basename ${AIRSHIP_CONFIG_PHASE_REPO_URL})/manifests/site/test-site/kubeconfig/" | yq '.config' --yaml-output > ~/.airship/kubeconfig
 fi
+
+#backward compatibility with previous behavior
+if [[ -z "${SOPS_PGP_FP_ENCRYPT}" ]]; then
+	#skipping sanity checks
+	exit 0
+fi
+
+echo "Sanity check for secret-reencrypt phase"
+decrypted1=$(airshipctl phase run secret-show)
+if [[ -z "${decrypted1}" ]]; then
+	echo "Got empty decrypted value"
+	exit 1
+fi
+
+#make sure that generated file has right FP
+grep "${SOPS_PGP_FP}" "${AIRSHIP_CONFIG_MANIFEST_DIRECTORY}/$(basename ${AIRSHIP_CONFIG_PHASE_REPO_URL})/manifests/site/test-site/target/generator/results/generated/secrets.yaml"
+
+#set new FP and reencrypt
+export SOPS_PGP_FP=${SOPS_PGP_FP_REENCRYPT}
+airshipctl phase run secret-reencrypt
+#make sure that generated file has right FP
+grep "${SOPS_PGP_FP}" "${AIRSHIP_CONFIG_MANIFEST_DIRECTORY}/$(basename ${AIRSHIP_CONFIG_PHASE_REPO_URL})/manifests/site/test-site/target/generator/results/generated/secrets.yaml"
+
+#make sure that decrypted valus stay the same
+decrypted2=$(airshipctl phase run secret-show)
+if [ "${decrypted1}" != "${decrypted2}" ]; then
+	echo "reencrypted decrypted value is different from the original"
+	exit 1
+fi
--- a/tools/gate/config_template.yaml
+++ b/tools/gate/config_template.yaml
@ -28,7 +28,8 @@ proxy:
 http: "$HTTP_PROXY"
 https: "$HTTPS_PROXY"
 noproxy: "$NO_PROXY"
-airship_config_pgp_fp: "FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4"
+airship_config_pgp_fp1: "FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4"
+airship_config_pgp_fp2: "D7229043384BCC60326C6FB9D8720D957C3D3074"
 airship_config_pgp: |-
  -----BEGIN PGP PRIVATE KEY BLOCK-----

@ -88,3 +89,37 @@ airship_config_pgp: |-
  gLk//M3qDixOxiurECkFrMvt/bDxEGpN5GVy550MmyUZQrkuqg==
  =VjGL
  -----END PGP PRIVATE KEY BLOCK-----
+  -----BEGIN PGP PRIVATE KEY BLOCK-----
+
+  lQHYBF1oQYgBBADPuVP6Jdk/J/TbNa9dXirp/zzwK18ZqNudNqQGN3H+2aSgxXwL
+  wlRfzy7rB3CU6Ewjzk9EVYeYztTIkGHL0JZ1CCTiBJArlHO0bHQQ7CPeKPkhIhkj
+  eA8yu9dcU77oYC2xbwgf43KYzfMKSGEybg+sBO+bH+Y6paJK54V2cuS3GwARAQAB
+  AAP+Jjf5BXtVP1OAr5xvCYS77JWzhpTUSIpS7dgR0br91GAC9DmhmyBEGeSqwz95
+  LUyYRbY9y1rZOfpEGCrIc5GLPOQytO9XMIzaS3dpzfGhla/spaKN4vJDvIOl+ruT
+  bInDdCRSmqXCfm2478OhOquc0H0a46eSmoaYeKdE3E8QZiECANxUL/dFk5j8NyPo
+  ZcwXw9Mv0A8UrynRcqht3Scti9k7dbsHylcObM305LFdcoNnSfNAIJhxfjbiXyGW
+  vwT2/qMCAPFatq3gvVjy6wKKylioi5cVwbLv9L+OaRXdR/Dy2bh/t3ujnsliV4+R
+  f7k3rHOQeaMLTnyfcz8AenL5IOe8RSkCANFpBgyzxCcV48Mm+FWDxjrSJ4/msRnN
+  gxqAPRrdpm7e1uebtBkPh4ch4oCW5/lLsRN23LUVIXYJRwyFfRjehCio0rRTU09Q
+  UyBGdW5jdGlvbmFsIFRlc3RzIEtleSAyIChodHRwczovL2dpdGh1Yi5jb20vbW96
+  aWxsYS9zb3BzLykgPHNlY29wc0Btb3ppbGxhLmNvbT6IzgQTAQgAOBYhBNcikEM4
+  S8xgMmxvudhyDZV8PTB0BQJdaEGIAhsDBQsJCAcCBhUKCQgLAgQWAgMBAh4BAheA
+  AAoJENhyDZV8PTB0R2cD/2YwaJ43iGueaAzByFnl+mUEBQJ4HhH4p7BIdx6B9AjE
+  3yLe8I4dqqYXxyZzaJ9d+KiqxJBT0l1GXt3H5M32yDJZqzXB9PTWP3yx8+Q1CuCs
+  7EL/bhJD1/sLdumVc77bmQtcI9NSiYyPzN/2ZqtV5RU14Loh24VFEjuHGvO0jI3+
+  nQHYBF1oQYgBBAChXi00fmpEs0Jiq0zOyYm9i749VoOsNReoB/5ix1QCimwVZKe1
+  D37IP5Qqysxy+LIQc4lJ+Q8foNOx1Aev5+TDyv+iU82D9xr9uPLLbA82k3AZ04Or
+  BjrZ/Yt1NZhuaHzciZCPpmqzF9kqVqAZc+vMiKZL1WZjS7O1FwaidY1vXwARAQAB
+  AAP+L0wUQeOfsD0+gv8khyPJTJZOD1pxQ6NYKLcXF8rG0+vQnECha098YKNKAXTp
+  kfVU8795iQYIKcQQ6Hl2O1fj1AxJE/iZYrqfm7UZz3bQ7ROSsAEPZ5GDOjKfbwsz
+  E6bWVH+PhS1azlvtTs9JezUtK0Wl9s+81FOrZtnUUskmWtECAMNNs9ujUt6GHv/J
+  NXVaSmk1z8QXitPHbAJLDMj4xVDysJWZV95eplC+RUSiLz5HeP2AQgh1D9Rv2bA5
+  c7OcJ3kCANOEkA0hVpXCI0FKrsihOf0NUOaAtS6CQNFlaIkrLwssJQY8pGYbRfRa
+  3krNJPyOlXmezV2/CsX3EqA9KXXen5cB/iSmMJO4WndGJTe7YzUEnnY/P2TKg1fN
+  s6v5Lf39j5Ll8V5rVDT7ApAw0IKS8fzpbdHP0HcizutlF6l44YaAXMGfhoi2BBgB
+  CAAgFiEE1yKQQzhLzGAybG+52HINlXw9MHQFAl1oQYgCGwwACgkQ2HINlXw9MHTD
+  HwQAv+ui718AT2hw2pK9JaNuTxjllrH+KPMlrov0P8oXHPCohC5cxM5sJ6tCQ0qH
+  XyeWoDE8V31btqFVAQyrr0wy0gntl1L/trnwMHoP8a/xa0RHNk5C7hmcuhTHbQey
+  JNbiRJZpCIZ1OyrF17+q6u9YBPjwqp8KrJ/0ryy2kyb7ZRM=
+  =+tJ6
+  -----END PGP PRIVATE KEY BLOCK-----