starlingx/docs
Code Issues Proposed changes

Updates on K8S Root CA Certificate managed by cert-manager

Browse Source 
Updated output

Editorial fixes

Merged sections

Fixed typos and indentation

Updated sections titles

Reordered sections in index

Fixed minor grammar issues

Added alarms exception

Described syntax of subject and expiry_date in example

Added references

Replaced K8s for Kubernetes

Story: 2008675
Task: 42625

Signed-off-by: Elisamara Aoki Goncalves <elisamaraaoki.goncalves@windriver.com>
Change-Id: I178fe9747c558d13c05b5cf61271fcaff59f6c26
This commit is contained in:
Elisamara Aoki Goncalves 2021-11-29 19:16:41 -03:00
parent baff361d45
commit ee2848e5fa
7 changed files with 809 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
doc/source/cli_ref/sw-manager.rst
View File

@ -137,3 +137,25 @@ cluster.
``kube-upgrade-strategy show``
Show a strategy.
***************************************
Kubernetes Root CA update orchestration
***************************************
The :command:`kube-rootca-update-strategy` commands create, apply and monitor
the orchestration of Kubernetes Root |CA| certificate updates across all hosts
of a |prod| cluster.
``kube-rootca-update-strategy create``
Create a strategy.
``kube-rootca-update-strategy delete``
Delete a strategy.
``kube-rootca-update-strategy apply``
Apply a strategy.
``kube-rootca-update-strategy abort``
Abort a strategy.
``kube-rootca-update-strategy show``
Show a strategy.

18
doc/source/dist_cloud/kubernetes/index.rst
View File

@ -15,7 +15,6 @@ Introduction
.. toctree::
:maxdepth: 1
:caption: Contents:
overview-of-distributed-cloud
distributed-cloud-architecture
@ -29,7 +28,6 @@ Installation
.. toctree::
:maxdepth: 1
:caption: Contents:
installing-and-provisioning-the-central-cloud
installing-and-provisioning-a-subcloud
@ -44,7 +42,6 @@ Operation
.. toctree::
:maxdepth: 1
:caption: Contents:
monitoring-subclouds-using-horizon
managing-subclouds-using-the-cli
@ -67,7 +64,6 @@ Manage Subcloud Groups
.. toctree::
:maxdepth: 1
:caption: Contents:
managing-subcloud-groups
creating-subcloud-groups
@ -79,7 +75,6 @@ Update (Patch) management
.. toctree::
:maxdepth: 1
:caption: Contents:
update-management-for-distributed-cloud
reviewing-update-status-for-distributed-cloud-using-horizon
@ -110,7 +105,6 @@ FPGA device image update management
.. toctree::
:maxdepth: 1
:caption: Contents:
device-image-update-orchestration
@ -120,18 +114,25 @@ Kubernetes Version Upgrade Distributed Cloud Orchestration
.. toctree::
:maxdepth: 1
:caption: Contents:
the-kubernetes-distributed-cloud-update-orchestration-process
configuring-kubernetes-update-orchestration-on-distributed-cloud
---------------------------------------------------------
Kubernetes Root CA Update Distributed Cloud Orchestration
---------------------------------------------------------
.. toctree::
:maxdepth: 1
orchestration-commands-for-dcmanager-4947f9fb9588
------------------
Upgrade management
------------------
.. toctree::
:maxdepth: 1
:caption: Contents:
upgrade-management-overview
upgrading-the-systemcontroller-using-the-cli
@ -156,7 +157,6 @@ Appendix
.. toctree::
:maxdepth: 1
:caption: Contents:
distributed-cloud-ports-reference
certificate-management-for-admin-rest-api-endpoints

217
doc/source/dist_cloud/kubernetes/orchestration-commands-for-dcmanager-4947f9fb9588.rst Normal file
View File

@ -0,0 +1,217 @@
.. _orchestration-commands-for-dcmanager-4947f9fb9588:
=========================================================================
Kubernetes Root CA Certificate Update for Distributed Cloud Orchestration
=========================================================================
You can use the :command:`dcmanager` command to orchestrate the update of the
Kubernetes Root |CA| certificate(s) for one or more subclouds in a Distributed
Cloud Environment.
The Kubernetes Root |CA| Update Distributed Cloud Orchestration commands for
DCManager use the keyword ``kube-rootca-update-strategy`` and provide the same
five subcommands as the other orchestrations: :command:`create, delete, apply,
abort, show`.
DCManager Kubernetes Root |CA| update orchestration considers a subcloud to be
'out of sync' and needing to be orchestrated based on the
``kube-rootca_sync_status`` field, which is updated based on the presence of
alarms in the subcloud related to the Kubernetes Root |CA| certificate expiring
soon (or expired).
- To see synchronization details for a subcloud.
.. code-block::
~(keystone_admin)]$ dcmanager subcloud show subcloud1
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
|id | 1 |
| name | subcloud1 |
| description | Ottawa Site |
| location | YOW |
| software_version | nn.nn |
| management | managed |
| availability | online |
| deploy_status | complete |
| management_subnet | 192.168.101.0/24 |
| management_start_ip | 192.168.101.2 |
| management_end_ip | 192.168.101.50 |
| management_gateway_ip | 192.168.101.1 |
| systemcontroller_gateway_ip | 192.168.204.101 |
| group_id | 1 |
| created_at | 2021-10-04 15:04:13.045076 |
| updated_at | 2021-10-25 21:16:23.713858 |
| dc-cert_sync_status | in-sync |
| firmware_sync_status | in-sync |
| identity_sync_status | in-sync |
| kubernetes_sync_status | in-sync |
| kube-rootca_sync_status | in-sync |
| load_sync_status | in-sync |
| patching_sync_status | in-sync |
| platform_sync_status | in-sync |
+-----------------------------+----------------------------+
- A user can pass "help" to see all the arguments for the strategy create
command.
.. code-block::
~(keystone_admin)]$ dcmanager help kube-rootca-update-strategy create
usage: dcmanager kube-rootca-update-strategy create [-h]
[-f \{json,shell,table,value,yaml}]
[-c COLUMN]
[--max-width <integer>]
[--fit-width]
[--print-empty]
[--noindent]
[--prefix PREFIX]
[--subcloud-apply-type \{parallel,serial}]
[--max-parallel-subclouds MAX_PARALLEL_SUBCLOUDS]
[--stop-on-failure]
[--force] [--group GROUP]
[--subject SUBJECT]
[--expiry-date EXPIRY_DATE]
[--cert-file CERT_FILE]
[cloud_name]
Create a kube rootca update strategy. This strategy supports: expiry-date, subject and cert-file.
positional arguments:
cloud_name Name of a single subcloud to update.
optional arguments:
-h, --help show this help message and exit
--subcloud-apply-type {parallel,serial}
Subcloud apply type (parallel or serial).
--max-parallel-subclouds MAX_PARALLEL_SUBCLOUDS
Maximum number of parallel subclouds.
--stop-on-failure
Do not update any additional subclouds after a failure.
--force
Disregard subcloud availability status, intended for some upgrade recovery scenarios. Subcloud name can be specified.
--group GROUP
Name or ID of subcloud group to update.
--subject 'C=CA ST=ON L=OTT O=WR OU=STX CN=OTHER'
Only applicable if not specifying '--cert-file', this will be the subject for the auto-generated rootca certificate.
--expiry-date YYYY-MM-DD
Only applicable if not specifying '--cert-file', this will be the expiry date for the auto-generated rootca certificate; expected format is YYYY-MM-DD.
--cert-file CERT_FILE
Path to a certificate to upload.
A subcloud can have its Kubernetes Root |CA| updated by the orchestrator even
if it is 'in-sync' by using the :command:`--force` command.
The :command:`--force` command can be used to orchestrate all subclouds, or
used with other arguments to orchestrate just one subcloud or subcloud group.
.. rubric:: |eg|
This is an example of how to orchestrate a new certificate for all subclouds,
including those that are in-sync that will expire in one year.
#. Create a Root |CA| update strategy.
.. code-block::
~(keystone_admin)]$ dcmanager kube-rootca-update-strategy create --force --expiry-date YYYY-MM-DD
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| strategy type | kube-rootca-update |
| subcloud apply type | None |
| max parallel subclouds | None |
| stop on failure | False |
| state | initial |
| created_at | 2021-10-26T14:35:50.675988 |
| updated_at | None |
+-----------------------------+----------------------------+
#. Verify that the strategy will orchestrate the subcloud(s).
.. code-block::
~(keystone_admin)]$ dcmanager strategy-step list
+-----------+-------+---------+---------+------------+-------------+
| cloud | stage | state | details | started_at | finished_at |
+-----------+-------+---------+---------+------------+-------------+
| subcloud1 | 2 | initial | | None | None |
+-----------+-------+---------+---------+------------+-------------+
#. Apply the strategy.
.. code-block::
~(keystone_admin)]$ dcmanager kube-rootca-update-strategy apply
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| strategy type | kube-rootca-update |
| subcloud apply type | None |
| max parallel subclouds | None |
| stop on failure | False |
| state | applying |
| created_at | 2021-10-26T14:36:30.327317 |
| updated_at | 2021-10-26T14:37:36.865776 |
+-----------------------------+----------------------------+
#. You can view the status of the strategy.
.. code-block::
~(keystone_admin)]$ dcmanager kube-rootca-update-strategy show
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| strategy type | kube-rootca-update |
| subcloud apply type | None |
| max parallel subclouds | None |
| stop on failure | False |
| state | applying |
| created_at | 2021-10-26 14:36:30.327317 |
| updated_at | 2021-10-26 14:37:36.865776 |
+-----------------------------+----------------------------+
It is typically more useful to monitor the progress of the strategy as it runs
in the subclouds.
In this example, the |DC| strategy is running the VIM strategy in the subcloud.
.. code-block::
~(keystone_admin)]$ dcmanager strategy-step list
+-----------+-------+------------------------------------------+----------------------------+----------------------------+-------------+
| cloud | stage | state | details | started_at | finished_at |
+-----------+-------+------------------------------------------+----------------------------+----------------------------+-------------+
| subcloud1 | 2 | applying vim kube rootca update strategy | apply phase is 0% complete | 2021-10-26 14:37:46.404736 | None |
+-----------+-------+------------------------------------------+----------------------------+----------------------------+-------------+
#. Wait for the strategy to complete. If there are failures, the
:command:`show` command in the previous step can indicate where the failure
occurred.
#. Only one type of DCManager strategy can exist at a time. Once completed,
remember to delete it.
.. code-block::
~(keystone_admin)]$ dcmanager kube-rootca-update-strategy delete
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| strategy type | kube-rootca-update |
| subcloud apply type | None |
| max parallel subclouds | None |
| stop on failure | False |
| state | deleting |
| created_at | 2021-10-26T14:27:44.856345 |
| updated_at | 2021-10-26T14:30:53.557978 |
+-----------------------------+----------------------------+

2
doc/source/security/kubernetes/kubernetes-certificates-f4196d7cae9c.rst
View File

@ -128,3 +128,5 @@ This certificate is configured to auto renew.
kubernetes-root-ca-certificate
update-renew-kubernetes-certificates-52b00bd0bdae
manual-kubernetes-root-ca-certificate-update-8e9df2cd7fb9
kubernetes-root-ca-certificate-update-cloud-orchestration-a627f9d02d6d

139
doc/source/security/kubernetes/kubernetes-root-ca-certificate-update-cloud-orchestration-a627f9d02d6d.rst Normal file
View File

@ -0,0 +1,139 @@
.. _kubernetes-root-ca-certificate-update-cloud-orchestration-a627f9d02d6d:
=========================================================
Kubernetes Root CA Certificate Update Cloud Orchestration
=========================================================
.. rubric:: |context|
You can update Kubernetes Root |CA| certificate on a running system, with
either an uploaded certificate or an auto generated certificate.
.. warning::
Do **not** let the Kubernetes Root |CA| certificate expire on your system
and ensure that certificates with valid/adequate expiry dates are used
during renewal as there is no easy way to recover a system if the
Kubernetes Root |CA| certificate expires.
Special care should be taken when updating the Root |CA| certificate.
.. rubric:: |prereq|
- The system is clear of alarms \(with the exception of alarms for locked
hosts, stopped instances, certificate expiring soon, certificate expired,
and Kubernetes root ca update in progress\).
- All hosts must be unlocked, enabled and available.
- All Kubernetes pods must be ready.
- Cert-manager app is applied.
- A file containing a self-signed certificate and corresponding private key
if choose to upload a new Root |CA| certificate.
.. rubric:: |proc|
Before starting the update, it is highly recommended to backup the existing
Kubernetes Root |CA| certficiate and key, i.e. ``/etc/kubernetes/pki/ca.crt``
and ``/etc/kubernetes/pki/ca.key``.
#. Create the strategy.
.. code-block::
~(keystone_admin)$ sw-manager kube-rootca-update-strategy create --subject "C=CA ST=ON L=OTT O=WR OU=STX CN=STX" --expiry-date YYYY-MM-DD
Strategy Kubernetes RootCA Update Strategy:
strategy-uuid: 47163c5b-44ac-432a-bd25-6e5c353046e9
controller-apply-type: serial
storage-apply-type: serial
worker-apply-type: serial
default-instance-action: stop-start
alarm-restrictions: strict
current-phase: build
current-phase-completion: 0%
state: building
inprogress: true
.. code-block::
~(keystone_admin)$ sw-manager kube-rootca-update-strategy create --cert-file some_cert.pem
strategy-uuid: 9575f1ea-4d66-4f13-8013-b04c2f420eff
controller-apply-type: serial
storage-apply-type: serial
worker-apply-type: serial
default-instance-action: stop-start
alarm-restrictions: strict
current-phase: build
current-phase-completion: 0%
state: building
inprogress: true
``--expiry-date``
Optional argument to specify the expiry date of the new certificate. It has
to be in the "YYYY-MM-DD" format. If not specified, the new certificate
will have the same valid period as the existing one (normally 10 years).
``--subject``
Optional argument to specify the distinguished name of the new certificate.
It has to be in the format ``C=<Country> ST=<State/Province> L=<Locality>
O=<Organization> OU=<OrganizationUnit> CN=<commonName>``. If not specified,
the new certificate will have "Kubernetes" as default.
``--cert-file``
Optional argument to upload a self-signed certificate as the new Root |CA|
certificate.
.. note::
Passing ``--cert-file`` uses an existing certificate, but
``--expiry-date`` and ``--subject`` generate a certificate. Using an
existing certificate will ignore any arguments to generate a
certificate.
#. Apply the strategy.
.. code-block::
sw-manager kube-rootca-update-strategy apply
#. Show the status of the update strategy.
.. code-block::
~(keystone_admin)$ sw-manager kube-rootca-update-strategy show
Strategy Kubernetes RootCA Update Strategy:
strategy-uuid: 47163c5b-44ac-432a-bd25-6e5c353046e9
controller-apply-type: serial
storage-apply-type: serial
worker-apply-type: serial
default-instance-action: stop-start
alarm-restrictions: strict
current-phase: build
current-phase-completion: 100%
state: ready-to-apply
build-result: success
build-reason:
.. note::
Passing ``--details`` will show all the internal steps and stages for
the orchestration strategy.
Passing ``--active`` will show which step is currently running for the
orchestration strategy.
#. If you want to delete the strategy.
.. code-block::
~(keystone_admin)$ sw-manager kube-rootca-update-strategy delete
Strategy deleted

415
doc/source/security/kubernetes/manual-kubernetes-root-ca-certificate-update-8e9df2cd7fb9.rst Normal file
View File

@ -0,0 +1,415 @@
.. _manual-kubernetes-root-ca-certificate-update-8e9df2cd7fb9:
============================================
Manual Kubernetes Root CA Certificate Update
============================================
.. rubric:: |context|
You can update Kubernetes Root |CA| certificate on a running system, with
either an uploaded certificate or an auto generated certificate.
A set of 'system' |CLI| subcommands, prefixed by ``kube-rootca-update-``, are
provided to accomplish this task. These |CLI| commands need to be executed in
certain order and phases, with most of the commands to be executed host by
host.
.. warning::
Do **not** let the Kubernetes Root |CA| certificate expire on your system
and ensure that certificates with valid/adequate expiry dates are used
during renewal as there is no easy way to recover a system if the
Kubernetes Root |CA| certificate expires.
Special care should be taken when updating the Root |CA| certificate.
.. rubric:: |prereq|
- The system must be clear of alarms \(with the exception of alarms for locked
hosts, stopped instances, certificate expiring soon, certificate expired,
and Kubernetes root ca update in progress\).
- All hosts must be unlocked, enabled and available.
- All Kubernetes pods must be ready.
- Cert-manager app is applied.
- A file containing a self-signed certificate and corresponding private key
if choose to upload a new Root |CA| certificate.
.. rubric:: |proc|
Before starting the update, it is highly recommended to backup the existing
kubernetes Root |CA| certificate and key, i.e. ``/etc/kubernetes/pki/ca.crt``
and ``/etc/kubernetes/pki/ca.key``.
#. Start the update.
.. code-block::
system kube-rootca-update-start
The command supports ``--force`` option to force the update to start if
there are non mgmt_affecting alarms.
.. code-block::
~(keystone_admin)]$ system kube-rootca-update-start
+-------------------+---------------------------------------+
| Property | Value |
+-------------------+---------------------------------------+
| uuid | 95afbd19-d159-407b-abe0-9ecb6ba4eb56 |
| state | update-started |
| from_rootca_cert | 8503e172a63b23e6-12808492498813125379 |
| to_rootca_cert | None |
| created_at | 2021-08-26T14:37:31.710407+00:00 |
| updated_at | None |
+-------------------+---------------------------------------+
#. Upload or generate a new Kubernetes Root |CA| certificate.
To generate a new Kubernetes Root |CA| certificate:
.. code-block::
~(keystone_admin)]$ system kube-rootca-update-generate-cert --expiry-date="2031-08-25" --subject="C=CA ST=ON L=Ottawa O=company OU=sale CN=kubernetes"
Generated new rootca certificate: 66b4daf7bb4ca6dd-248389040031382177497040389244094812254
The value returned is the ID of the new Root |CA| certificate.
``--expiry-date``
Optional argment to specify the expiry date of the new certificate. It has
to be in the "YYYY-MM-DD" format. If not specified, the new certificate
will have the same valid period as the existing one (normally 10 years).
``--subject``
Optional argment to specify the distinguished name of the new certificate.
It has to be in the format ``C=<Country> ST=<State/Province> L=<Locality>
O=<Organization> OU=<OrganizationUnit> CN=<commonName>``. If not specified,
the new certificate will have "Kubernetes" as default.
Alternatively, to upload a self-signed certificate as the new Root |CA|
certificate:
.. code-block::
~(keystone_admin)]$ system kube-rootca-update-upload-cert ca_with_key.crt
Uploaded new rootca certificate: 8503e172a63b23e6-12808492498813125379
The value returned is the ID of the uploaded Root |CA| certificate. The
``ca_with_key.crt`` file contains both the new certifcate and the
corresponding private key.
#. Show the cluster overall update status using the
:command:`kube-rootca-update-show` command.
This command can be run anytime during the update.
For example, run the command after new certificate generated.
.. code-block::
~(keystone_admin)]$ system kube-rootca-update-show
+----------------------------------------+------------------------------------+-----------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
| uuid | state | from_rootca_cert | to_rootca_cert | created_at | updated_at |
+----------------------------------------+------------------------------------+-----------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
| 95afbd19-d159-407b-abe0-9ecb6ba4eb56 | update-new-rootca-cert-generated | 8503e172a63b23e6-12808492498813125379 | 66b4daf7bb4ca6dd-248389040031382177497040389244094812254 | 2021-08-26T14:37:31.710407+00:00 | 2021-08-26T14:47:50.728284+00:00 |
+----------------------------------------+------------------------------------+-----------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
#. Update Kubernetes components of each host to trust both the old and new
Root |CA| certificates using the :command:`system kube-rootca-host-update <hostname> --phase=trust-both-cas`
command.
The command needs to be executed sequentially on each of the nodes in the
cluster (except dedicated storage nodes if there are any).
For example, update controller-1 to trust both old and new Root |CA|
certificates.
.. code-block::
~(keystone_admin)]$ system kube-rootca-host-update controller-1 --phase=trust-both-cas
+------------------------+----------------------------------------------------------+
| Property | Value |
+------------------------+----------------------------------------------------------+
| uuid | 513d626c-559e-4df7-8e15-f92481dc190f |
| state | updating-host-trust-both-cas |
| effective_rootca_cert | 8503e172a63b23e6-12808492498813125379 |
| target_rootca_cert | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
| created_at | 2021-08-26T15:48:39.903793+00:00 |
| updated_at | 2021-08-26T15:50:14.299276+00:00 |
+------------------------+----------------------------------------------------------+
#. List the update status of all the hosts in the cluster system
``kube-rootca-host-update-list``.
The :command:`kube-rootca-host-update-list` command can be run anytime
during the update.
For example, list the status of hosts after controller-0, controller-1 have
been updated to trust both old and new |CAs|.
.. code-block::
~(keystone_admin)]$ system kube-rootca-host-update-list
+--------------+-------------+------------------------------+----------------------------------------+-----------------------------------------------------------+-----------------------------------+-----------------------------------+
| hostname | personality | state | effective_rootca_cert | target_rootca_cert | created_at | updated_at |
+--------------+-------------+------------------------------+----------------------------------------+-----------------------------------------------------------+-----------------------------------+-----------------------------------+
| controller-0 | controller | updated-host-trust-both-cas | 8503e172a63b23e6-12808492498813125379 | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 | 2021-08-26T15:48:39.888935+00:00 | 2021-08-26T15:51:47.343297+00:00 |
| controller-1 | controller | updated-host-trust-both-cas | 8503e172a63b23e6-12808492498813125379 | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 | 2021-08-26T15:48:39.903793+00:00 | 2021-08-26T15:50:30.716854+00:00 |
| worker-0 | worker | None | 8503e172a63b23e6-12808492498813125379 | None | 2021-08-26T15:48:39.915956+00:0 | None |
+--------------+-------------+------------------------------+----------------------------------------+-----------------------------------------------------------+-----------------------------------+-----------------------------------+
#. Update pods deployed by Deployments and Daemonsets to trust both old and
new Root |CA| certificates.
Run this command only once on active controller. It takes a few minutes for
all the pods to restart after updated.
.. code-block::
~(keystone_admin)]$ system kube-rootca-pods-update --phase=trust-both-cas
+-------------------+----------------------------------------------------------+
| Property | Value |
+-------------------+----------------------------------------------------------+
| uuid | 6bc2ff57-e82c-4da1-af69-4d52c67917f7 |
| state | updating-pods-trust-both-cas |
| from_rootca_cert | 8503e172a63b23e6-12808492498813125379 |
| to_rootca_cert | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
| created_at | 2021-08-26T15:48:39.860160+00:00 |
| updated_at | 2021-08-26T15:59:53.851646+00:00 |
+-------------------+----------------------------------------------------------+
#. Update client and server certificates for Kubernetes components of each
host using the new Root |CA| certificate.
The command needs to be executed sequentially on each of the nodes in the
cluster (except dedicated storage nodes if there are any).
For example, update Kubernetes client and server certificates on
controller-0 using the new Root |CA| certificate.
.. code-block::
~(keystone_admin)]$ system kube-rootca-host-update controller-0 --phase=update-certs
+------------------------+----------------------------------------------------------+
| Property | Value |
+------------------------+----------------------------------------------------------+
| uuid | 18c5b474-8d7a-4b15-bee8-06d4feb704dd |
| state | updating-host-update-certs |
| effective_rootca_cert | 8503e172a63b23e6-12808492498813125379 |
| target_rootca_cert | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
| created_at | 2021-08-26T15:48:39.888935+00:00 |
| updated_at | 2021-08-26T16:13:22.064894+00:00 |
+------------------------+----------------------------------------------------------+
#. Update Kubernetes components of each host to trust only the new Root |CA|
certificate.
The command needs to be executed sequentially on each of the nodes in the
cluster (except dedicated storage nodes if there are any).
For example, update controller-0 to trust only the new Root |CA|
certificate.
.. code-block::
~(keystone_admin)]$ system kube-rootca-host-update controller-0 --phase=trust-new-ca
+------------------------+----------------------------------------------------------+
| Property | Value |
+------------------------+----------------------------------------------------------+
| uuid | 18c5b474-8d7a-4b15-bee8-06d4feb704dd |
| state | updating-host-trust-new-ca |
| effective_rootca_cert | 8503e172a63b23e6-12808492498813125379 |
| target_rootca_cert | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
| created_at | 2021-08-26T15:48:39.888935+00:00 |
| updated_at | 2021-08-26T19:19:19.366338+00:00 |
+------------------------+----------------------------------------------------------+
#. Update pods deployed by Deployments and Daemonsets to trust only the new
Root |CA| certificate.
Run this command only once on active controller. It takes a few minutes for
all the pods to restart after updated.
.. code-block::
~(keystone_admin)]$ system kube-rootca-pods-update --phase=trust-new-ca
+-------------------+----------------------------------------------------------+
| Property | Value |
+-------------------+----------------------------------------------------------+
| uuid | 6bc2ff57-e82c-4da1-af69-4d52c67917f7 |
| state | updating-pods-trust-new-ca |
| from_rootca_cert | 8503e172a63b23e6-12808492498813125379 |
| to_rootca_cert | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
| created_at | 2021-08-26T15:48:39.860160+00:00 |
| updated_at | 2021-08-26T19:26:34.347519+00:00 |
+-------------------+----------------------------------------------------------+
#. Complete the update.
.. code-block::
~(keystone_admin)]$ system kube-rootca-update-complete
+-------------------+----------------------------------------------------------+
| Property | Value |
+-------------------+----------------------------------------------------------+
| uuid | 6bc2ff57-e82c-4da1-af69-4d52c67917f7 |
| state | update-completed |
| from_rootca_cert | 8503e172a63b23e6-12808492498813125379 |
| to_rootca_cert | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
| created_at | 2021-08-26T15:48:39.860160+00:00 |
| updated_at | 2021-08-26T20:19:36.579505+00:00 |
+-------------------+----------------------------------------------------------+
#. Abort an on-going update.
This command aborts the on-going update at any step. When an update is
aborted, alarm **900.009** will be raised and the overall update status
will be in ``update-aborted``. A new update should be started, and run to
complete to fully update kubernetes certificates.
.. code-block::
system kube-rootca-update-abort
For example, the update is aborted when:
- controller-0, controller-1 and worker-0 have been updated to trust both
|CA| certificates,
- client and server certificates have been updated on controller-0 (a step
further than controller-1, worker-0),
- overall update is in ``updating-host-update-certs`` state.
.. code-block::
~(keystone_admin)]$ system kube-rootca-update-show
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
| uuid | state | from_rootca_cert | to_rootca_cert | created_at | updated_at |
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
| 04863d56-2f36-404b-ad9d-a0b1d967939e | updating-host-update-certs | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 | d70efa2daaee06f8-18974112351299353303834821971390931938 | 2021-08-26T20:28:09.383461+00:00 | 2021-08-26T20:42:40.673674+00:00 |
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
.. code-block::
~(keystone_admin)]$ system kube-rootca-host-update-list
+--------------+-------------+------------------------------+-----------------------------------------------------------+----------------------------------------------------------+-----------------------------------+-----------------------------------+
| hostname | personality | state | effective_rootca_cert | target_rootca_cert | created_at | updated_at |
+--------------+-------------+------------------------------+-----------------------------------------------------------+----------------------------------------------------------+-----------------------------------+-----------------------------------+
| controller-0 | controller | updated-host-update-certs | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 | d70efa2daaee06f8-18974112351299353303834821971390931938 | 2021-08-26T20:28:09.404809+00:00 | 2021-08-26T20:43:49.577920+00:00 |
| controller-1 | controller | updated-host-trust-both-cas | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 | d70efa2daaee06f8-18974112351299353303834821971390931938 | 2021-08-26T20:28:09.417891+00:00 | 2021-08-26T20:33:03.754760+00:00 |
| worker-0 | worker | updated-host-trust-both-cas | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 | d70efa2daaee06f8-18974112351299353303834821971390931938 | 2021-08-26T20:28:09.430753+00:00 | 2021-08-26T20:34:13.390571+00:00 |
+--------------+-------------+------------------------------+-----------------------------------------------------------+----------------------------------------------------------+-----------------------------------+-----------------------------------+
.. code-block::
~(keystone_admin)]$ system kube-rootca-update-abort
+-------------------+----------------------------------------------------------+
| Property | Value |
+-------------------+----------------------------------------------------------+
| uuid | 04863d56-2f36-404b-ad9d-a0b1d967939e |
| state | update-aborted |
| from_rootca_cert | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 |
| to_rootca_cert | d70efa2daaee06f8-18974112351299353303834821971390931938 |
| created_at | 2021-08-26T20:28:09.383461+00:00 |
| updated_at | 2021-08-26T20:53:04.493889+00:00 |
+-------------------+----------------------------------------------------------+
.. code-block::
~(keystone_admin)]$ fm alarm-list
+-----------+------------------------------------------------------------------+-----------------+---------+---------------+
| Alarm ID | Reason Text | Entity ID | Severity | Time Stamp |
+-----------+------------------------------------------------------------------+-----------------+---------+---------------+
| 900.009 | Kubernetes root CA update aborted, certificates may not be fully | host=controller | minor | 2021-08-26T20 |
+-----------+------------------------------------------------------------------+-----------------+---------+---------------+
| | updated | | | :53:04.577578 |
+-----------+------------------------------------------------------------------+-----------------+---------+---------------+
.. code-block::
~(keystone_admin)]$ system kube-rootca-update-show
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
| uuid | state | from_rootca_cert | to_rootca_cert | created_at | updated_at |
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
| 04863d56-2f36-404b-ad9d-a0b1d967939e | update-aborted | 66b4daf7bb4ca6dd-131908807141787487410130398776303207610 | d70efa2daaee06f8-18974112351299353303834821971390931938 | 2021-08-26T20:28:09.383461+00:00 | 2021-08-26T20:53:04.493889+00:00 |
+----------------------------------------+------------------------------------+------------------------------------------------------------+------------------------------------------------------------+------------------------------------+------------------------------------+
**States of the update procedure**
``update-started``: semantic checks passed, health check passed, update started.
``update-new-rootca-cert-uploaded``: the new Root |CA| certificate is uploaded.
``update-new-rootca-cert-generated``: the new Root |CA| certificate is
generated.
``updating-host-trust-both-cas``: new Root |CA| certificate is being added to
Kubernetes components' trusted |CAs|.
``updated-host-trust-both-cas``: new Root |CA| certificate has been added to
Kubernetes components' trusted |CAs|.
``updating-host-trust-both-cas-failed``: new Root |CA| certificate failed to be
added to Kubernetes components' trusted |CAs|.
``updating-pods-trust-both-cas``: new Root |CA| certificate is being added to
pods' trusted |CA| list.
``updated-pods-trust-both-cas``: new Root |CA| certificate has been added to
pods' trusted |CA| list.
``updating-pods-trust-both-cas-failed``: new Root |CA| certificate failed to be
added to pods' trusted |CA| list.
``updating-host-update-certs``: server and client certificates is being updated
for Kubernetes components.
``updated-host-update-certs``: server and client certificates have been updated
for Kubernetes components.
``updating-host-update-certs-failed``: server and client certificates failed to
be updated for Kubernetes components.
``updating-host-trust-new-ca``: old Root |CA| certificate is being removed,
only new cert will be trusted for Kubernetes components.
``updated-host-trust-new-ca``: old Root |CA| certificate has been removed, only
new cert is trusted for Kubernetes components.
``updating-host-trust-new-ca-failed``: old Root |CA| certificate failed to be
removed, both old and new certs are trusted for Kubernetes components.
``updating-pods-trust-new-ca``: old Root |CA| certificate is being removed from
pods' trusted |CA| list.
``updated-pods-trust-new-ca``: old Root |CA| certificate has been removed from
pods' trusted |CA| list.
``updating-pods-trust-new-ca-failed``: old Root |CA| certificate failed to be
removed from pods' trusted |CA| list.
``update-compete``: Kubernetes components and pods are healthy, update
completed.
``update-aborted``: a Kubernetes Root |CA| update is aborted.

6
doc/source/security/kubernetes/update-renew-kubernetes-certificates-52b00bd0bdae.rst
View File

@ -8,7 +8,11 @@ Updating Kubernetes Root |CA| certificate is a complex process, because it is
not only the Root |CA| certificate that needs to be updated, but also all the
other Kubernetes certificates signed by it need to be regenerated and updated.
The update of the Kubernetes Root |CA| certificate is currently not supported.
See :ref:`Manual Kubernetes Root CA Certificate Update
<manual-kubernetes-root-ca-certificate-update-8e9df2cd7fb9>` or
:ref:`Kubernetes Root CA Certificate Update Cloud Orchestration
<kubernetes-root-ca-certificate-update-cloud-orchestration-a627f9d02d6d>` for
how to update the Kubernetes Root |CA| certificate.
The other leaf certificates generated from the Kubernetes Root |CA| are
monitored by a cronjob, which runs every day at midnight to check if any of