Update spec of kubernetes root CA certificate update

Some minor adjustments to kubernetes root CA certficate update
spec based on implementation for accuracy.

Story: 2008675
Signed-off-by: Andy Ning <andy.ning@windriver.com>
Change-Id: I0eecee32e2ca17c64194151d8e96076c4754f7b6
This commit is contained in:
Andy Ning 2021-09-07 09:40:31 -04:00
parent f738144690
commit bf8218355f
1 changed files with 194 additions and 102 deletions

View File

@ -14,6 +14,9 @@ This feature introduces CLI/REST APIs and execution orchestration for updating
Kubernetes root CA certficate and certificates issued by the root CA in a
rolling fashion so that the impact on the system is minimized.
This is the updated version of the approved spec security-2008675-kubernetes-rootca-update.rst.
This version reflects the adjustments from implementation.
Problem description
===================
@ -84,11 +87,12 @@ Sysinv operations for root CA certificate update
A new set of sysinv CLI commands will be introduced to simplify the update
procedure. It will be a procedure similar to software upgrade, with a start,
execute and complete cycle. There won't be support for "abort", but user can
retry the command if it fails. And user can choose to restart the update
procedure by uploading or re-generating a new root CA certficate. This also
provides a mechanism to resume to the original CA certificate if user chooses
to upload the original CA certificate.
execute and complete cycle. User can retry a step if it fails. There will also
be support for "abort", where user can choose to exit an on-going update. But
the user is supposed to restart the update procedure with either uploading or
re-generating a root CA certficate and run the update to full complete. This
also provides a mechanism to restore the original CA certificate if user
chooses to upload the original CA certificate.
The following is a summary of the CLI commands and the steps to perform
kubernetes root CA certificate update.
@ -112,28 +116,28 @@ kubernetes root CA certificate update.
certificate and private key from a file instead of generating one
* Change progress state to update-new-rootca-cert-uploaded
3. system kube-rootca-host-update <hostname> --phase=trustBothCAs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
3. system kube-rootca-host-update <hostname> --phase=trust-both-cas
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* Update apiserver's trusted CAs to include the new CA cert
* Update scheduler's trusted CAs to include the new CA cert
* Update controller-manager's trusted CAs to include the new CA cert
* Update kubelet's trusted CAs to include the new CA cert
* Update admin.conf's trusted CAs to include the new CA cert
* Change progress state to updated-host-trustBothCAs on success
* Change progress state to updating-host-trustBothCAs-failed on failure
* Change progress state to updated-host-trust-both-cas on success
* Change progress state to updating-host-trust-both-cas-failed on failure
4. system kube-rootca-pods-update --phase=trustBothCAs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
4. system kube-rootca-pods-update --phase=trust-both-cas
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* Annotate Daemonsets and Deployments to trigger pod replacement in a safer
rolling fashion, to ensure pods to pick up the new root CA cert as its trusted
CA along with the old root CA certificate
* Change progess state to updated-pods-trustBothCAs on success
* Change progess state to updating-pods-trustBothCAs-failed on failure
* Change progess state to updated-pods-trust-both-cas on success
* Change progess state to updating-pods-trust-both-cas-failed on failure
5. system kube-rootca-host-update <hostname> --phase=updateCerts
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
5. system kube-rootca-host-update <hostname> --phase=update-certs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* Update admin.conf's client cert/key data with new ones signed by the
new root CA
@ -143,27 +147,27 @@ kubernetes root CA certificate update.
* Update controller-manager's client cert/key with new one signed by the new
root CA
* Update kubelet's client cert/key with new one signed by the new root CA
* Change progress state to updated-host-updateCerts on success
* Chante progress state to updating-host-updateCerts-failed on failure
* Change progress state to updated-host-update-certs on success
* Chante progress state to updating-host-update-certs-failed on failure
6. system kube-rootca-host-update <hostname> --phase=trustNewCA
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
6. system kube-rootca-host-update <hostname> --phase=trust-new-ca
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* Update admin.conf's trusted CAs to remove the old root CA
* Update apiserver's trusted CAs to remove the old root CA
* Update controller-manager's trusted CAs to remove the old root CA
* Update scheduler's trusted CAs to remove the old root CA
* Update kubelet's trusted CAs to remove the old root CA
* Change progress state to updated-host-trustNewCA on success
* Change progress state to updating-host-trustNewCA-failed on failure
* Change progress state to updated-host-trust-new-ca on success
* Change progress state to updating-host-trust-new-ca-failed on failure
7. system kube-rootca-pods-update --phase=trustNewCA
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
7. system kube-rootca-pods-update --phase=trust-new-ca
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* Annotate Daemonsets and Deployments to trigger pod replacement in a safer
rolling fashion, to remove the old root CA from pods trusted CA list
* Change progress state to updated-pods-trustNewCA on success
* Change progress state to updating-pods-trustNewCA-failed on failure
* Change progress state to updated-pods-trust-new-ca on success
* Change progress state to updating-pods-trust-new-ca-failed on failure
8. system kube-rootca-host-update complete
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -171,17 +175,22 @@ kubernetes root CA certificate update.
* Post-check to verify the update
* Change the progress state to update-complete
system kube-rootca-update-list
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
9. system kube-rootca-host-update-list
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* Run this command anytime to show the update status of all hosts in the
cluster
system kube-rootca-update-show
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
10. system kube-rootca-update-show
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* Run this command anytime to show the overall update status
11. system kube-rootca-update-abort
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* Run this command to abort the update at any step
VIM Orchestration Operations
----------------------------
@ -272,96 +281,121 @@ each host.
The following is the list of REST resources and APIs to be added:
The new resource /kube_update_ca is added
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The new resource /kube_rootca_update is added
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* URLS:
* /v1/kube_update_ca
* /v1/kube_rootca_update
* Request Methods:
* POST /v1/kube_update_ca
* POST /v1/kube_rootca_update
* Creates (starts) a new root CA cert update
* Response body example::
{"from_rootca_cert": "kubenetes-5118144266510589551",
{"uuid": "47dff2b6-17ba-45a2-b3d3-8b2a85a5dba9",
"to_rootca_cert": null,
"created_at": "2021-08-25T14:57:13.006034+00:00",
"from_rootca_cert": "d70efa2daaee06f8-91764",
"updated_at": null,
"state": "update-started",
"uuid": "223ba65e-45d1-4383-baa7-f03bb4c46773",
"created_at": "2021-03-25T12:04:10.372399+00:00",
"updated_at": "2021-03-25T12:04:10.372399+00:00"}
"id": 1}
* GET /v1/kube_update_ca
* GET /v1/kube_rootca_update
* Return the current kube_update_ca
* Return the current root CA update
* Response body example::
{"from_rootca_cert": "kubenetes-5118144266510589551",
"to_rootca_cert": "kubenetes-6118144266510589551",
{"uuid": "47dff2b6-17ba-45a2-b3d3-8b2a85a5dba9",
"to_rootca_cert": null,
"created_at": "2021-08-25T14:57:13.006034+00:00",
"from_rootca_cert": "d70efa2daaee06f8-91764",
"updated_at": null,
"state": "update-started",
"uuid": "223ba65e-45d1-4383-baa7-f03bb4c46773",
"created_at": "2021-03-25T12:04:10.372399+00:00",
"updated_at": "2021-03-25T14:45:43.252964+00:00"}
"id": 1}
* PATCH /v1/kube_update_ca
* PATCH /v1/kube_rootca_update
* Modifies the current rootca_update. Used to update the state of the
update (e.g. to update_complete).
update (e.g. to update_complete, or update_aborted).
* Request body example::
[{"path": "/state",
"value": "update-completed",
"op": "replace"}]
[{"path": "/state",
"value": "update-aborted",
"op": "replace"}]
* Response body example::
{"from_rootca_cert": "kubenetes-5118144266510589551",
"to_rootca_cert": "kubenetes-6118144266510589551",
"state": "update-complete",
"uuid": "223ba65e-45d1-4383-baa7-f03bb4c46773",
"created_at": "2021-03-25T12:04:10.372399+00:00",
"updated_at": "2021-03-25T14:45:43.252964+00:00"}
{"uuid": "fb882423-ea26-42bf-b645-fd9de4248fd4",
"to_rootca_cert": "d70efa2daaee06f8-176046114160516196064588947858918572907",
"created_at": "2021-08-24T13:40:13.318822+00:00",
"from_rootca_cert": "d70efa2daaee06f8-199590289735612744821302170157251522966",
"updated_at": "2021-08-24T13:52:21.547899+00:00",
"state": "update-completed",
"id": 20}
* DELETE /v1/kube_update_ca
{"uuid": "7d07e384-f06d-4213-8e61-5e300aeb9d1c",
"to_rootca_cert": null,
"created_at": "2021-08-24T13:38:55.376395+00:00",
"from_rootca_cert": "d70efa2daaee06f8-199590289735612744821302170157251522966",
"updated_at": "2021-08-24T13:39:47.108582+00:00",
"state": "update-aborted",
"id": 19}
* Deletes the current rootca_update (after it is completed)
The new resource /kube_rootca_certificate/upload is added
The new resource /kube_rootca_update/upload_cert is added
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* URLS:
* /v1/kube_rootca_certificate/upload
* /v1/kube_rootca_update/upload_cert
* Request Methods:
* POST /v1/kube_rootca_certificate/upload
* POST /v1/kube_rootca_update/upload_cert
* Upload a root CA cert and key from a file
* Request body example::
* Request body example:
(The contents of the body is from a file containing both private key and certificate)::
{"ca.crt": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMyRENDQWNDZ0..."
"ca.key": "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcGdJQk..."}
{"-----BEGIN PRIVATE KEY----- ...... -----END PRIVATE KEY----- ...... -----BEGIN CERTIFICATE----- ...... -----END CERTIFICATE-----}
* Return body example::
{"cert_id": "kubenetes-5118144266510589551"}
{"success": "8503e172a63b23e6-12808492498813125379",
"error": ""}
The new resource /v1/kube_rootca_certificate/generate is added
The new resource /v1/kube_rootca_update/generate_cert is added
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* URLS:
* /v1/kube_rootca_certificate/generate
* /v1/kube_rootca_update/generate_cert
* Request Methods:
* POST /v1/kube_rootca_certificate/generate
* POST /v1/kube_rootca_update/generate_cert
* Tell sysinv to generate a new root CA cert and key pair
* Request body example::
{"expiry_date": "2022-08-25",
"subject": "C=CA O=Company CN=kubernetes"}
* Return body example::
{"cert_id": "kubenetes-5118144266510589551"}
{"success": "a8942428863f292b-253592702972967198587817983178843995169",
"error": ""}
The existing resource /ihosts is modified to add new actions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -378,51 +412,97 @@ The existing resource /ihosts is modified to add new actions
* Request body example::
{"phase", "trustBothCAs"}
{"phase", "trust-both-cas"}
* Response body example::
{"id": "4",
{"target_rootca_cert": "8503e172a63b23e6-12808492498813125379",
"created_at": "2021-08-25T17:13:22.571151+00:00",
"hostname": "controller-1",
"updated_at": "2021-08-25T17:58:59.809264+00:00",
"state": "updating-host-trust-both-cas",
"personality": "controller",
"target_rootca_cert": "kubenetes-6118144266510589551",
"effective_rootca_cert": "kubenetes-5118144266510589551",
"state": "updating-host-trustBothCAs"}
"id": 8,
"effective_rootca_cert": "d70efa2daaee06f8-91764",
"uuid": "a597c090-731f-48f8-9f3f-344997c41317"}
The new resource /kube_hosts_update_ca
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The new resource /kube_rootca_update/hosts is added
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* URLs:
* /v1/kube_hosts_update_ca
* /v1/kube_rootca_update/hosts
* Request Methods:
* GET /v1/kube_hosts_update_ca
* GET /v1/kube_rootca_update/hosts
* Returns the update details of all hosts
* Response body example::
{
"hosts": [
{"id": "2",
"hostname": "controller-1",
"personality": "controller",
"target_rootca_cert": "kubenetes-6118144266510589551",
"effective_rootca_cert": "kubenetes-5118144266510589551",
"state": "updating-host-trustBothCAs"
},
{"id": "4",
"hostname": "compute-0",
"personality": "compute",
"target_rootca_cert": "kubenetes-6118144266510589551",
"effective_rootca_cert": "kubenetes-5118144266510589551",
"state": "updating-host-updateCerts"
}
]
"kube_host_updates": [
{"target_rootca_cert": null,
"created_at": "2021-08-25T17:13:22.558411+00:00",
"hostname": "controller-0",
"updated_at": null,
"state": null,
"personality": "controller",
"id": 7,
"effective_rootca_cert": "d70efa2daaee06f8-91764",
"uuid": "7d7d05dd-900f-4004-951d-d92536faac8e"
},
{"target_rootca_cert": "8503e172a63b23e6-12808492498813125379",
"created_at": "2021-08-25T17:13:22.571151+00:00",
"hostname": "controller-1",
"updated_at": "2021-08-25T17:59:16.097029+00:00",
"state": "updated-host-trust-both-cas",
"personality": "controller",
"id": 8,
"effective_rootca_cert": "d70efa2daaee06f8-91764",
"uuid": "a597c090-731f-48f8-9f3f-344997c41317"
},
{"target_rootca_cert": null,
"created_at": "2021-08-25T17:13:22.584500+00:00",
"hostname": "worker-0",
"updated_at": null,
"state": null,
"personality": "worker",
"id": 9,
"effective_rootca_cert": "d70efa2daaee06f8-91764",
"uuid": "a4ca4eed-9b2f-4b4c-8ee7-45bbc573a55f"
}
]
}
The new resource /kube_rootca_update/pods is added
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* URLs:
* /v1/kube_rootca_update/pods
* Request Methods:
* POST /v1/kube_rootca_update/pods
* Update root CA cert for pods
* Request body example::
{"phase", "trust-both-cas"}
* Response body example::
{"uuid": "6cf4157b-75ff-4e86-bc96-8b08e4c9836d",
"to_rootca_cert": "8503e172a63b23e6-12808492498813125379",
"created_at": "2021-08-25T17:13:22.535798+00:00",
"from_rootca_cert": "d70efa2daaee06f8-91764",
"updated_at": "2021-08-25T18:37:02.574836+00:00",
"state": "updating-pods-trust-both-cas",
"id": 3}
Security impact
---------------
@ -496,8 +576,10 @@ Repos Impacted
--------------
Impacted repo from this spec:
* config
* stx-puppet
* fault
Work Items
----------
@ -526,7 +608,7 @@ Sysinv
* root CA certficate and issuer creation in cert-manager
* calculate the ID of the new root certificate
* kube-rootca-host-update <hostname> --phase=trustBothCAs CLI/API
* kube-rootca-host-update <hostname> --phase=trust-both-cas CLI/API
* basic infrastructure
* semantic checks
@ -535,14 +617,14 @@ Sysinv
* agent RPC/implementation (apply puppet manifest, report back config
status, etc...)
* kube-rootca-pods-update --phase=trustBothCAs CLI/API
* kube-rootca-pods-update --phase=trust-both-cas CLI/API
* basic infrastructure
* semantic checks
* conductor implementation (generate hieradata, trigger puppet
manifests apply, handle apply result, update progress state etc...)
* kube-rootca-host-update <hostname> --phase=updateCerts CLI/API
* kube-rootca-host-update <hostname> --phase=update-certs CLI/API
* basic infrastructure
* semantic checks
@ -552,7 +634,7 @@ Sysinv
* agent RPC/implementation (apply puppet manifest, report back config
status, etc...)
* kube-rootca-host-update <hostname> --phase=trustNewCA CLI/API
* kube-rootca-host-update <hostname> --phase=trust-new-ca CLI/API
* basic infrastructure
* semantic checks
@ -561,7 +643,7 @@ Sysinv
* agent RPC/implementation (apply puppet manifest, report back config
status, etc...)
* kube-rootca-pods-update --phase=trustNewCA CLI/API
* kube-rootca-pods-update --phase=trust-new-ca CLI/API
* basic infrastructure
* semantic checks
@ -578,19 +660,29 @@ Sysinv
* kube-rootca-update-show CLI/API
* basic infrastructure
* condutor database query
* conductor database query
* kube-rootca-update-list CLI/API
* kube-rootca-host-update-list CLI/API
* basic infrastructure
* condutor database query
* conductor database query
* kube-rootca-update-abort CLI/API
* basic infrastructure
* semantic checks
* system health checks for update abort
* clear 'kube root CA update in progress' alarm
* raise 'kube root CA update aborted' alarm
Puppet
^^^^^^
* runtime manifest for host update trustBothCAs phase
* runtime manifest for host update updateCerts phase
* runtime manifest for host update trustNewCA phase
* runtime manifest for host update trust-both-cas phase
* runtime manifest for host update update-certs phase
* runtime manifest for host update trust-new-ca phase
* runtime manifest for pods update trust-both-cas phase
* runtime manifest for pods update trust-new-ca phase
System Upgrade
^^^^^^^^^^^^^^